Rotating Secrets Without Breaking Anything

If you keep secrets in Oracle Cloud Infrastructure (OCI) Vault, you have probably noticed that rotating one is never a single change. Vault holds a copy, but so does everything that uses it: the database account the password belongs to, the env file a container reads at startup, the password file a nightly backup opens. For however long it takes you to update all of them, they disagree. And if you have started letting an AI coding agent run cloud commands for you, the next question arrives quickly: how much of that should it be allowed to touch?

In this post I will go over a skill I built for AI coding agents that inventories, tags, verifies and rotates OCI Vault secrets, then carries each new value through to everything downstream that holds a copy: Autonomous Database (ADB) users and ADMIN, PostgreSQL and MongoDB accounts running in Docker, Docker Compose env files, and the credential files applications read. More usefully, I will cover the ordering that keeps Vault and those copies from disagreeing, why the agent is given a confirmation phrase rather than a yes, and what broke during testing, including two failures I caused on purpose.

One secret, many copies

Anything that uses a secret is only ever as correct as the copy it reads. Change the database password first and every application fails until its config catches up. Change Vault or the config first and the same thing happens in the other direction. Do it all in a script and the question becomes what state you are left in when the script dies halfway.

OCI Vault gives you the tool to solve this, it just does not force you to use it. A secret can hold a new value as a PENDING version while CURRENT stays exactly where it was, and the pending version can be promoted later. So quite naturally, the whole design falls out of one rule: Vault CURRENT stays authoritative until every downstream copy has accepted and verified the new value.

This is not an OCI idea, and it is worth saying so. Amazon Web Services (AWS) documents the same shape for Secrets Manager rotation functions, with AWSPENDING and AWSCURRENT labels and separate create, set, test and finish steps. When two clouds arrive at the same four steps, it is a reasonable sign the steps are right.

One thing worth mentioning before going further. The purpose of this piece was not to build around a specific feature, but it would be remiss of me not to point out that OCI Vault's Application Programming Interface (API) now supports native scheduled rotation for Autonomous Database secrets, anywhere from daily to once every 360 days, plus on-demand rotation, and it can hand other targets to an OCI Function you write. If that fits what you need, use it; it is less code to own. What I wanted in addition was control over which secrets may rotate at all, a plan I approve before anything changes, independent proof afterwards that the credential still works, and a defined way out when a rotation stops halfway.

What the skill does

A skill here means a folder an AI coding agent loads on demand: a SKILL.md telling the agent when and how to act, reference documents it reads before risky steps, and scripts that do the actual work. The same folder is written for both Claude Code and OpenAI's Codex.

Everything defaults to read-only. In plain language, a user can ask for:

• "Vault status" — an inventory of every secret, its version, age and rotation tag, without retrieving a single value
• "Can I reach the database?" — a connectivity preflight covering the wallet, the Transparent Network Substrate (TNS) alias and the driver
• "Tag this secret" — set how it rotates, preserving everything else on it
• "Rotate this password" — a dry-run plan first, and an execution only after approval
• "Verify it" — proof that Vault's current value actually logs in
• "Find incomplete rotations" — any secret left holding a PENDING version

Underneath that are eleven command-line tools sharing one library, and a test suite of 120 unit tests that never touch OCI, a database or a server.

Rotation tags. Every secret carries a rotation tag in its Vault metadata, and the tag says how it rotates, not just whether. P1 means Vault only: nothing downstream holds a copy, so the new value simply has to land in Vault. P2 means something downstream has to change with it, and the part after the colon names what. P2:ADB is an Autonomous Database password; P2:PG:COMPOSE is a Postgres password that also lives in an app's Docker Compose env file. EXEMPT means it does not need rotating at all, a wallet password being the obvious case. MANUAL is kept for the secrets the skill genuinely cannot update, such as credentials a provider issues. Anything without a tag cannot rotate.

Each name after the colon is a handler: code that knows how to apply the new value to one downstream system and prove it took. There are five today:

A secret can be tagged with a handler that does not exist yet, which records the intent, but it will not rotate until every handler it names exists. Rotating the database and skipping the application that also reads the password would be worse than not rotating at all.

A tag alone is deliberately not enough. A P2:ADB secret must also be mapped to a specific database user in a configuration file, and a P1 secret must be listed there as Vault-only. The tag lives in Vault, where anyone with the right Identity and Access Management (IAM) permission can change it. The mapping lives in a file you review. Both have to agree, which is the same principle as the two OCI firewall layers I wrote about in the Wix migration post, applied on purpose this time.

Concretely, the mapping for an Autonomous Database says which secret holds ADMIN and which secrets belong to which database users:

"databases": {
  "appdb": {
    "ocid": "ocid1.autonomousdatabase.oc1.iad.…",
    "service_alias": "appdb_low",
    "wallet_dir": "~/wallets/Wallet_appdb",
    "wallet_password_secret": "appdb-wallet-pwd",
    "admin_secret": "appdb-admin-pwd",
    "user_secrets": { "appdb-app_user-pwd": "APP_USER" }
  }
}

And a secret with copies outside the database names each handler, in order, and where its copy lives:

"secret_bindings": {
  "app-db-pwd": {
    "value": { "format": "password" },
    "handlers": {
      "PG": { "host": "app-host", "container": "app-db",
              "role": "app_user", "database": "app" },
      "COMPOSE": { "host": "app-host", "project_dir": "/srv/app",
                   "env_var": "POSTGRES_PASSWORD", "services": ["app-db", "api"],
                   "url_embedded": true,
                   "container_env": [{ "service": "api", "var": "DATABASE_URL",
                                       "match": "contains" }] }
    }
  }
}

Rotating app-db-pwd changes app_user's password in the app-db container, proves it with a login, then rewrites POSTGRES_PASSWORD in /srv/app/.env, recreates only app-db and api, and checks that the running api container's DATABASE_URL now carries the new value. url_embedded tells the generator to avoid characters such as @ and / that would break the connection URL the password sits inside.

To avoid assembling that configuration by hand, a discovery script reads the tenancy's compartments, secrets and databases and drafts it. On my own tenancy it mapped all five database secrets to the right users on the first run. It also left every application credential in that vault unmapped, which is exactly where you want a naming heuristic to fail.

The transaction

A rotation runs in this order:

1. Verify the ADMIN credential actually logs in
2. Temporarily disable Vault's auto-generation, if the secret has it on
3. Confirm the version has not moved and no PENDING version exists
4. Stage a freshly generated password as the only PENDING version
5. ALTER USER in the database to the staged value
6. Log in as that user with the new value
7. Promote PENDING to CURRENT
8. Restore auto-generation

Each step is there because of what happens if it is missing. Staging comes before ALTER USER because the reverse order, with staging then failing, leaves the only copy of the new password in the memory of a process that is about to exit. Verification comes before promotion because a successful ALTER USER proves the statement ran, not that the account can log in; locked accounts and password profiles only show themselves at login. A pre-existing PENDING version blocks everything, because it is evidence of an interrupted rotation and staging on top of it would destroy that evidence.

A P1 secret runs the same transaction with Vault as the only target. With no database login to prove the new value, the proof is reading the staged version back and comparing it in memory with what was generated, before anything is promoted.

This is what a real run against an Always Free Autonomous Data Warehouse (ADW) looked like, trimmed and with names genericized:

04:28:52Z  admin_login_verified
04:29:09Z  vault_version_staged       pending_version=2
04:29:09Z  database_password_changed  database_user=APP_USER
04:29:10Z  database_login_verified
04:29:27Z  vault_version_promoted     previous_version=1  current_version=2
04:29:27Z  rotation_complete          result=SUCCESS

Note the two 17-second gaps. I will come back to those, because they are the most instructive thing in the log.

Beyond the database

Autonomous Database was the starting point, but the secrets I most wanted rotated belong to the second site on the same server: a FastAPI app with Postgres, plus a MongoDB store for a side project. Moving the copies outside the database changed three things.

Vault has to be the source of truth first. When I checked how that app read its secrets, it did not read Vault at all. The values reached it through an env file on the server, and the Vault copies were duplicates nothing checked. They happened to match, but nothing guaranteed it. So every P2 rotation now starts by proving that each downstream copy holds Vault's CURRENT, and changes nothing if one does not. An import tool fixes the drift by making Vault match what is actually in use.

Handlers run in order, and each proves itself before the next one starts. P2:PG:COMPOSE stages the new value, changes the Postgres password and proves it with a login, then rewrites the env file, recreates only the containers that read it, and proves the API connects. Only then is Vault promoted. The same recover and resume modes apply if it stops in between. This was the real rotation of that app's Postgres password, trimmed:

consumer_sync           PG=true  COMPOSE=true
vault_version_staged    pending_version=3
handler_applied         PG
handler_verified        PG
handler_applied         COMPOSE
handler_verified        COMPOSE
vault_version_promoted  previous_version=2  current_version=3
p2_rotation_complete    result=SUCCESS

A check that cannot fail proves nothing. The app's health endpoint deliberately ignores the database, so it would have passed with a broken password; the Postgres handler uses a real connection from inside the API container instead. The check commands for the MongoDB files were each run with a wrong password first, to prove they fail, before I trusted them with the right one. And the Postgres handler refuses to run unless Postgres rejects a random wrong password, because a database that trusts every connection would make the login proof meaningless.

MongoDB added one more wrinkle: the application's user is not allowed to change its own password. The MongoDB handler signs in as the administrator, whose password it reads from Vault at that moment, and changes it from there. After that rotation I checked the one thing the handlers do not: the old password was refused and the new one accepted.

Why the agent gets a phrase, not a yes

An AI agent that asks "shall I rotate the password?" and receives "yes" has received permission for whatever it currently believes the plan is. That is a weak guarantee. Plans drift, and a version number that was 4 when the plan was made may be 5 by the time anybody acts on it.

So every mutation here is two separate runs. The first is a dry run that changes nothing and ends with a confirmation phrase built from the exact state it inspected:

rotate:stage:app-user-pwd:v4

The second run refuses to do anything unless it is given that exact phrase. If the secret has moved to version 5 in the meantime, the phrase no longer matches and nothing happens. Each operation has its own form, so an approval to tag a secret cannot be replayed as an approval to rotate it:

tag:stage:app-user-pwd:v1:P2:ADB
rotate-p1:stage:webhook-token:v3
rotate:stage:app-db-pwd:v2:PG+COMPOSE
verify-login:stage:app-user-pwd:v2
recover:stage:app-user-pwd:current-v3:pending-v4
rotate-admin:stage:admin-pwd:v1

The skill's instructions tell the agent to show the plan and wait, never to construct a phrase itself, and never to batch approvals across secrets. The scripts enforce the part that matters regardless of what the agent does.

The other half of trusting an agent with this is that the agent never sees a secret value. Only the executors read values, only after approval, only into process memory, and they print metadata alone: names, version numbers, states. The agent's transcript is exactly that output, so there is nothing sensitive in it to leak, log or summarize. Error messages get the same treatment. Exception text is one of the few places a value could escape, so only the error type and OCI status code are ever reported.

ADMIN has its own runner, and the ordinary one refuses it. ADMIN is the credential the ordinary runner needs to change every other password, and a stranded ADMIN password locks you out until you reset it in the Console.

What broke, including what I broke on purpose

A crash in the one place you most need output. The unit tests caught this one before anything reached the tenancy. The helper that prints each event took the event's name as a parameter called name, and the preflight passed a field also called name for each check. Every blocked preflight would therefore have crashed with a Python error instead of listing which check failed, which is the precise moment you most need to know. A one-character fix, and a good argument for testing the failure paths as seriously as the happy one.

Recovery advice OCI would not let you follow. My first draft of the recovery documentation said that if a rotation stopped after staging, you could cancel the stranded PENDING version and try again. Before relying on that, I checked the API. OCI only allows deleting a secret version whose rotation state is DEPRECATED, so a PENDING version cannot simply be discarded. Somebody following my advice mid-incident would have hit a wall.

The fix was better than the original design. Both runners gained two recovery modes. Resume is for a stop before ALTER USER: it applies the value already staged, verifies it and promotes it, without ever generating another password. Recover is for a stop after ALTER USER: it tests both versions as the user and promotes PENDING only when PENDING logs in and CURRENT does not. If both log in, or neither does, it stops and says so rather than guessing.

Seventeen seconds of UPDATING. Back to those gaps in the log. After any change, an OCI Vault secret sits in an UPDATING state for a few seconds, around 17 in my testing, and a second change during that window is refused with a 409 IncorrectState. The first live attempt surfaced this harmlessly: tagging a secret and immediately planning its rotation was blocked with "secret is UPDATING, not ACTIVE", and nothing changed.

Looking into why, I found the uncomfortable part. The Python Software Development Kit (SDK) only classifies that 409 as a transient, retryable error when its circuit breaker is enabled. My promotion step retried transient errors, which meant that under some configurations it would treat a busy secret as a hard failure. That step runs straight after ALTER USER, so the failure would have landed at the worst possible point: database changed, Vault not yet promoted, and every consumer of the password broken. The library now waits explicitly for the secret to become ACTIVE before each change, and treats the 409 as retryable regardless of SDK settings. The two 17-second gaps in the successful run are that wait doing its job.

Breaking it on purpose. A recovery path you have only unit-tested is a theory, so I used a throwaway database user created for the purpose and left it in both interrupted states a real crash would leave.

Stopping before ALTER USER produced exactly what it should. The recovery scan found the stranded version, a fresh rotation refused to start, recover tested both logins, saw the database was still on CURRENT and declined, and resume then finished the job. Stopping after ALTER USER was the more telling test, because the first thing the verify tool reported was:

adb_login_verification_result  result=FAIL

That is what an outage looks like from the inside: Vault confidently handing out a password the database no longer accepts. Resume refused to touch it, since the database had already moved. Recover saw that only PENDING logged in, promoted it, and the same check went back to PASS. With both recovery paths proven, I rotated the real ADMIN password on the same database, then rotated the throwaway user once more to confirm everything downstream picked up the new ADMIN from Vault.

A smaller footnote. Before any of this goes near a public repository, it gets scanned for secrets. That check turned up two of the skill's own test fixtures: a fake private-key header, and a function call the scanner read as a password assignment. Neither was a secret, but both are exactly the shape secret scanners look for, and either would have blocked a commit. Test fixtures deserve the same scrutiny as code.

Trying it yourself

You will need a Python environment with the oci and oracledb packages, an OCI command-line interface (CLI) profile, and a downloaded wallet for each database. If the wallet's private key is encrypted, which it is by default, the driver needs the wallet password. I generated a fresh wallet through the API with a random password stored straight into Vault, so that password exists nowhere else. Downloading a new wallet does not invalidate existing ones; only rotating the wallet does. The other handlers reach their systems over SSH, so they need an SSH alias for each host and python3 on it.

Then draft the configuration from your tenancy, and read every mapping it produces before trusting it:

python scripts/discover_config.py --profile YOUR_PROFILE \
  --wallet-root ~/wallets --output ~/.config/oci-vault-ops/config.json
python scripts/vault_inventory.py

My strong recommendation is to do what I did and start with a throwaway database user that nothing depends on. Tag it, rotate it, then break it on purpose and recover it. Only once you have watched each path work against your own tenancy should anything real be tagged P2. A secret an application also reads belongs under a tag naming every system that holds a copy, such as P2:PG:COMPOSE. If a handler for one of them does not exist yet, the tag records the intent and the secret waits, safely unrotated.

The skill is on GitHub at BASoapbox/oci-vault-rotation-skill, MIT-licensed, with the reference documents covering the transaction, ADMIN recovery, the tag scheme, each handler and every configuration key.

Where it landed

My database passwords, and the app secrets that live in env files and password files, now rotate through a process that plans before it acts, proves the result afterwards, and has a tested answer for every point at which it could stop. The agent does the typing, but it cannot approve its own plan, and it never sees a value it could repeat.

If you take one thing from this, make it the testing rather than the code. All three real problems above were invisible on the happy path, and two of them would have surfaced for the first time in the middle of a real incident, with a database already changed. Breaking it on purpose, on an account nobody cared about, is how I found out first.

Enjoy!