On one platform I work on, secrets used to get to applications by hand. Someone had the value, someone else needed it, and it moved through whatever channel was handy. I built secrets management with HashiCorp Vault using init containers, and that ended manual secret distribution. Teams can now rotate their own secrets without waiting on anyone.

Before that, when I was training new engineers, I saw the other version of this problem. An account in the training environment got hacked twice because learners committed secrets to Git. Both problems have the same fix: the secret lives in one place, and the app gets it at runtime.

If a secret is already in Git

Rotate it first. Cleaning the history comes second, because anyone who cloned the repository already has it. Then remove it from the history with a tool like git filter-repo, following GitHub's guide to removing sensitive data. Then add a secret-leak check to the pipeline so the next one gets caught before it merges.

# only after the secret is rotated
# read the old value without echoing it or saving it in shell history
read -rs OLD_SECRET
# keep the replacements file outside the clone, so it never gets committed
printf '%s==>REMOVED\n' "$OLD_SECRET" > ../replacements.txt
git filter-repo --replace-text ../replacements.txt
rm ../replacements.txt
unset OLD_SECRET

How the init container pattern works

The pod starts an init container before the app container.
The init container logs in to Vault with the pod's Kubernetes service account, using Vault's Kubernetes auth method. Vault checks the token with the cluster and maps the service account and namespace to a Vault role and its policy.
It reads the secrets that policy allows and writes them to a shared in-memory volume.
It exits. The app container starts and reads the secrets from that volume as files.

No secret is in the image, the Helm chart or Git. The manifest only says which role to use and which paths to read. The files live in memory, so they aren't written to the node's disk, and they go away with the pod.

Rotation gets simpler too. The value changes in Vault, in one place, and the next pod that starts picks it up. Nobody has to find every copy of it.

With the Vault Agent Injector, you get this by annotating the pod. vault.hashicorp.com/agent-inject: "true" turns injection on, and vault.hashicorp.com/agent-pre-populate-only: "true" keeps it to an init container with no sidecar.

On the Vault side, a policy that reads one app's paths and a role that ties it to one service account in one namespace:

vault policy write my-app-read - <<EOF
path "secret/data/my-app/*" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/my-app \
  bound_service_account_names=my-app \
  bound_service_account_namespaces=my-namespace \
  policies=my-app-read \
  ttl=1h

On the Kubernetes side, the annotations go on the pod template. This one writes the secret to /vault/secrets/db inside the pod.

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-pre-populate-only: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-db: "secret/data/my-app/db"
    spec:
      serviceAccountName: my-app

Init container or sidecar

An init container fetches secrets once, when the pod starts. It's simple, and it uses nothing while the app runs. The trade-off is that a changed secret only reaches the app when the pod restarts.

A sidecar keeps running next to the app and can renew and update the secrets while the app runs, at the cost of an extra container in every pod. For secrets that change on a schedule, an init container plus a rolling restart after rotation is usually enough. The restart is one command:

kubectl rollout restart deployment/my-app -n my-namespace

For short-lived credentials, such as dynamic database passwords, use the sidecar.

There are other ways to get Vault secrets into Kubernetes, like the Vault Secrets Operator or the Secrets Store CSI driver. The same questions apply to all of them: where the secret ends up, who can read it, and what happens when it changes.

What goes wrong

Permission denied from Vault. The Vault role is bound to a different service account or namespace than the one the pod runs as. Check the binding before anything else.
The pod hangs in init. The init container can't reach Vault. Usually it's a network policy, or a TLS certificate the container doesn't trust.
Secrets end up in Kubernetes Secrets anyway. Some setups copy Vault values into Kubernetes Secrets, which are only base64-encoded by default. That can be fine, but make the choice deliberately and lock down who can read Secrets in the namespace.
One policy that reads everything. It's easy, and it defeats the point. Give each app a role that reads its own paths and nothing else.
Rotation nobody has tested. Rotate a secret in a test environment and watch the app pick it up before you rely on it in production.

When it makes sense to get help

If you already run Vault and have one app to wire up, the HashiCorp docs will get you there. Bring in help when you're setting Vault up for the first time, when many teams need a self-service way to manage their own secrets, or when you've just found a secret in Git and need it cleaned up and the process fixed so it doesn't happen again.

Vault init containers on Kubernetes: getting secrets out of Git

If a secret is already in Git

How the init container pattern works

Init container or sidecar

What goes wrong

When it makes sense to get help