The Secret, the Key and the Code You Did Not Write

Draw your architecture the way you would for a review. Users, a gateway, services, a database, a queue, a cloud account around all of it. It looks complete, and a security reviewer will still ask the same three questions that are not on the page.

Where are the secrets? Who can use the keys? And what is running in here that nobody on the team wrote?

Those are three different things, and many incidents start at one of them. A credential that was never rotated. A key that anyone could use. A dependency that changed under the build. None of them is a box on the diagram, so none of them gets reviewed.

The Best Secret Is No Secret

Start with the move most teams skip. Before you choose a place to store a credential, ask whether it needs to exist.

Take the pipeline. A build system has to deploy to your cloud, and the old way is to create a long-lived key and paste it into the CI settings. That key now lives forever, in a place many people can reach. GitHub's documentation describes the alternative, OpenID Connect: workflows "no longer need long-lived cloud credentials", and instead each job gets a signed token that the cloud provider checks against conditions you set, then exchanges for an access token that lasts only for that job. There is no static credential to leak because there is no static credential.

The detail that makes it safe is the condition. The token carries claims for the repository, the branch and the environment, and the trust you configure in the cloud can require all of them. GitHub's own example scopes access to a single environment of a single repository. A trust that says "any repository in my organisation" is a long-lived key wearing a disguise.

The same idea runs through the platform: a service that gets its permissions from the identity it runs under rather than from a password in a file. When that is possible, you remove a whole class of incident instead of managing it.

When You Cannot Remove It, Rotate It

Some secrets are unavoidable: a third party's API key, a database password. They go in a managed store, never in the repository, never baked into an image. And then comes the part that separates a vault from a plan.

Rotation, in AWS's words, "is the process of periodically updating a secret", and when you rotate you "update the credentials in both the secret and the database or service." Read that second half again. Rotation changes two things at once, so there is a moment when the application holds the old credential and the service wants the new one. If the application cannot cope, a rotation is an outage with a calendar invite.

So the test is not "do we store it in a vault". It is: if this secret changes right now, underneath the running application, what happens? A refresh on failure, or two valid credentials during a window, are the usual answers. If you do not know the answer, the secret is effectively never rotated, whatever the policy says.

Detection belongs here too. Scan the repository and the commit before it lands, because prevention fails eventually and finding it quickly is the fallback.

Encryption at Rest Is Not the Control

Every cloud encrypts data at rest by default, so "it is encrypted" is no longer an answer to anything. The better question is: who can decrypt?

The model behind it is envelope encryption. AWS describes it as "encrypting plaintext data with a data key, and then encrypting the data key under another key". The data key does the bulk work and is stored beside the data, protected by the key above it. The top key never leaves the hardware security module. This is why it is cheap, and it is also why rotating the key above does not mean re-encrypting all your data: you "re-encrypt only the data keys".

Now the control. AWS says key policies "are the primary way to control access to KMS keys", and that no principal, "including the account root user or key creator, has any permissions to a KMS key unless they are explicitly allowed". Encryption with a key that every role in the account can use protects you against a stolen disk and nothing else. The policy on the key is where the decision is made. In a regulated or multi-tenant system, separate keys per tenant give you separation, an audit trail and an honest answer to "can you delete this customer's data".

The Code You Did Not Write

The third question is the one with a recent story.

In March 2025 the US cybersecurity agency CISA published an alert about a compromise of tj-actions/changed-files, a popular GitHub Action that detects changed files in a pull request. The compromised action could disclose secrets from the workflows that used it, including access keys, tokens and private keys. CISA's advice was blunt: identified secrets "should be considered compromised" and rotated immediately.

Look at what happened there. A dependency nobody on those teams wrote, running inside the pipeline, read the secrets the pipeline held. All three of today's questions met in one incident: a secret that could be read, in a place where one change to a third party reached it, with nothing on the diagram to show it.

GitHub's own guidance points at the controls. Pin actions to a commit SHA, because "a tag can be moved or deleted if a bad actor gains access to the repository storing the action." Where you can adopt one, a workflow lockfile "is the strongest protection available". Avoid the privileged pull request triggers, which may have write access and secrets, and never check out untrusted code in them. And remember that any user with write access to the repository can read every secret configured in it.

The same logic applies below the pipeline. Base images, libraries and their dependencies are execution inside your perimeter. The layers are familiar: minimal images, scanning at build and again in the registry, a bill of materials so that when the next widespread vulnerability lands you can answer "are we affected" in minutes, and signing checked at deploy so only what your pipeline built can run.

The part that decides whether any of it survives is policy, not tooling. A scanner that blocks every build on every medium finding gets switched off within a month. Decide what blocks, what is tracked, and who owns an exception and for how long.

Four Questions for the Diagram

If I could add four lines to every architecture review, they would be:

1. Which secrets exist here, and could any of them be an identity instead?
2. What happens to each one the day it changes?
3. Who can decrypt, and where is that written?
4. What do we run that we did not write, and could we list it within the hour?

None of them needs a new tool to ask. They need someone to ask them before the diagram is called finished.