The aim is to reduce the number of long-lived credentials to as close to zero as possible. Centralise what exists, replace static credentials with short-lived or dynamically issued ones wherever the backend supports it, coordinate rotation so consumers do not break, and audit every read. Deleting a committed secret from a repository does not remove it from history or invalidate it.
What is secrets management?
Secrets management is the practice of storing, scoping, distributing, rotating and auditing credentials such as passwords, API keys, tokens and certificates, so they are not embedded in code, configuration files or CI settings. The goal is that credentials are short-lived, narrowly scoped and their use is recorded.
Where secrets actually live
Before improving anything, find the copies. In most organisations they are in more places than expected: CI variables, Kubernetes Secret objects, environment files on servers, Terraform state, a shared password manager, container image layers, and Git history from a commit three years ago that someone reverted but did not rotate.
The last one deserves emphasis because it is consistently misunderstood: deleting a file containing a credential in a later commit does not remove it from history, and it certainly does not invalidate the credential.
The hierarchy, from worst to best
| Approach | Assessment |
|---|---|
| Hard-coded in source | Unacceptable. In history forever, distributed with every clone |
| CI variables | Common and weak. Long-lived, broadly scoped, rarely rotated, no read audit |
| Kubernetes Secret objects | Better with encryption at rest and tight RBAC, but still long-lived with no rotation or audit |
| Central store with scoped access | Good. One current copy, scoped, audited, rotatable |
| Short-lived tokens per run | Better. Nothing durable is handed out |
| Dynamically issued credentials | Best. Created per lease and revoked automatically, so there is nothing to rotate |
Rotation without an outage
Rotation gets deferred because it breaks things, and it breaks things because it is usually done as a single atomic swap while consumers are still holding the old value.
- Create the new credential while the old one remains valid: both work during the overlap.
- Update consumers, tracking which have picked up the new value.
- Verify no consumer is still authenticating with the old credential, using the access audit.
- Revoke the old credential.
- Confirm nothing broke, and only then close the rotation.
The step teams skip is the third, which is precisely the one that turns rotation from risky into routine. Being able to see who last used a credential is what makes revocation safe.
When a secret leaks
Assume compromise. A credential in a repository (even a private one) has been on developer laptops, in CI caches and in clones. Rewriting history is useful housekeeping but it is not remediation.
- Rotate immediately. This is the only step that actually removes the exposure.
- Check the access audit for use from unexpected sources or at unexpected times.
- Determine the exposure window: when it was committed, when it was found.
- Find every consumer, so rotation does not break something you forgot.
- Then clean history, and add a pre-commit check so the same class of mistake is caught earlier next time.
Key takeaways
- Find every copy first: CI variables, Secret objects, state files, image layers and Git history.
- Deleting a committed secret does not remove it from history or invalidate it.
- Prefer short-lived tokens, and dynamic credentials where the backend supports them.
- Rotate with an overlap window and verify no consumer is still using the old value before revoking.
- On a leak, rotate first and clean history afterwards.
Frequently asked questions
They are base64-encoded objects stored in etcd, not encrypted by default. With encryption at rest and tight RBAC they can be acceptable, but they remain long-lived, unrotated and produce no record of who read them.
Credentials created on demand for a specific consumer and lease period, then revoked automatically when the lease ends. Because nothing persists, there is no long-lived credential to leak and no rotation to schedule.
Rotate it immediately: that is the only action that removes the exposure. Then check the access audit for unexpected use, identify all consumers, and clean history afterwards as housekeeping rather than as the fix.
For static credentials, frequently enough that a leak has a bounded window, typically 30 to 90 days. The better answer is to reduce the number of static credentials so the question applies to fewer things.
Yes, where practical. Fetching at startup or on demand with a short-lived token means the credential is not baked into an image, a manifest or an environment file, and access is recorded.