No one designs secret sprawl. It accrues: a database password in a deploy script here, an API key in a CI variable there, a certificate on a host someone configured in 2019. Then an engineer leaves, or a vendor announces a breach, and the question "what do we need to rotate, and where does it live?" has no answer. In client engagements, we have watched that question take three weeks to answer. The incident did not wait.
This is the program we run - on our own infrastructure and with clients - to get from sprawl to something you can rotate on a Tuesday afternoon.
Start with inventory, not tooling
The instinct is to deploy a secrets manager first. Resist it. Migrating unknown secrets into a vault just gives you a vault plus the same unknown remainder outside it.
The inventory pass enumerates every credential with four attributes: what it grants access to, where it is stored, what consumes it, and when it last changed. Sources worth sweeping: CI/CD variables, environment files on hosts, infrastructure-as-code state, container images (yes, baked-in secrets survive in image layers), and repository history - scanners find what code review missed years ago.
Two findings recur in every estate we have swept. First, the count is always 3-5x the initial guess. Second, a meaningful fraction of credentials guard systems that no longer exist. Deleting a secret nobody uses is the cheapest security win available, and the inventory is the only way to find it.
The hierarchy of credential quality
Not all secrets are equally dangerous. We triage everything into a hierarchy and migrate upward over time:
- Static, shared, long-lived - a database password in a config file, known to several systems and several humans. The floor. Rotation is a coordinated event, which means it rarely happens.
- Static, scoped, centrally issued - per-service credentials from a secrets manager. Rotation is possible without coordination, so it actually happens.
- Dynamic and short-lived - credentials issued on demand with a TTL of minutes or hours. Rotation is continuous and implicit; a leaked credential is stale before most attackers can use it.
The goal is not level 3 everywhere - plenty of systems only speak static passwords. The goal is knowing which level each secret sits at and having a reason when it stays low.
Rotation is the capability that matters
A credential you cannot rotate quickly is a liability with a delay timer, because rotation always arrives as an emergency: a departure, a vendor advisory, a suspicious log line. Our standing rules:
- Every secret has a documented rotation procedure, tested at least once outside an emergency
- Applications reload credentials without restarting, or restart gracefully - discovered before the emergency, not during
- Overlap windows are the default: issue the new credential, deploy it everywhere, then revoke the old one. Simultaneous cutover is how rotations cause outages
- Rotation age is a dashboard metric per secret; the emergency drill is simply doing early what the dashboard would eventually demand
Humans are the hardest tier
Machine-to-machine secrets yield to automation. The stubborn residue is human access: the shared admin password for the legacy appliance, the break-glass account, the credentials in a departing engineer's password manager. Process, not tooling: named owners for every shared credential, break-glass access that is sealed, alarmed, and audited when used, and offboarding that triggers rotation of everything the person could have known - which is answerable in an afternoon only because the inventory exists.
Secrets management is not a product you install. It is an inventory you maintain, a hierarchy you climb, and a rotation muscle you exercise before the day it is tested for real.