Top 100 Site Reliability Engineer Interview Questions and Answers
The questions most likely to come up in your Site Reliability Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-105Walk me through how you'd define the SLIs, SLOs and error budget for a user-facing checkout service, availability and latency, and what the error budget actually changes about how the team works?
First thing I'd do is pin down the user journey, because the SLI has to be measured where the user feels it. Checkout is: cart → payment authorisation → order confirmation. If I measure availability at the pod, I miss the load balancer, DNS, and the payment vendor. So I'd take two signals: server-side telemetry at the edge (ingress or service mesh) for the bulk, plus a blackbox synthetic that runs a real checkout against production every minute from outside the VPC.
Availability SLI: confirmed orders / valid checkout attempts. Valid means I exclude 4xx that are the user's fault: declined card, expired session, bad address. Those are not reliability failures. 5xx, timeouts, and connection resets count. Latency SLI: proportion of checkout requests completing under 1.5 s, measured at the edge. I prefer a threshold SLI over p99 because it composes with the availability SLI and is easier to explain to product: '99% of checkouts finish in under 1.5 s.' p99 is a good secondary signal for debugging.
Then the SLO. 99.9% availability over a rolling 30-day window, and 99% latency under 1.5 s over the same window. Rolling 30d, not calendar month, because a calendar reset lets a team burn the budget in the last week and start fresh on the 1st: that's gaming, not reliability. 99.9% is the target because it's the point where you can still deploy daily; each extra nine roughly doubles the cost of the next one, and 99.99% on checkout usually means multi-region active-active and a much slower release cadence. If the business genuinely loses revenue per minute, I'd argue for 99.95% and pay for it.
Error budget: 0.1% of 30 days = 43.2 minutes of full outage, or 0.1% of requests. That's the number the team gets to spend on deploys, experiments, and vendor blips.
The consequence is the part most teams skip. I'd write a policy: when the budget is exhausted, feature work stops and the team does reliability work until the budget is back above zero. But waiting for exhaustion is too late, so I'd alert on burn rate. Multi-window: a fast burn. 14.4x over 1 hour, which is 2% of the 30-day budget in an hour. Pages immediately. A slow burn (6x over 6 hours, 5% of budget) opens a ticket. That catches a slow leak before it becomes a month-long outage.
| SLI | Definition | SLO | Window | | Availability | confirmed orders / valid checkout attempts | 99.9% | rolling 30d | | Latency | checkout requests < 1.5 s / total | 99% | rolling 30d |
Failure modes: if the synthetic is the only signal, you miss partial failures that only affect logged-in users; if you only use server-side, you miss DNS. If the payment vendor is down, the SLI drops and the budget burns: that's correct, because the user can't check out. But you need a separate dependency SLO for the vendor so you can hold them accountable, not just absorb it.
Curated: · Written: · Reviewed: —
Q&A-1A product team wants to SLO on CPU idle and HTTP 200 rate. Which measurements belong in the SLO, and which do not?
Short answer: CPU idle does not belong. A 200 rate over real user-facing requests can belong, but only when the endpoint is the last hop the user waits on and 200 is a faithful proxy for the user getting what they came for. A 200 rate over /healthz is not an SLI at all: it measures whether a probe can reach the process, which is an availability input, not a user outcome.
The test that decides every candidate: write the sentence the measurement is true or false about. "The user is fine because ___." If the measurement can be green while that sentence is false, it is a diagnostic signal, not an SLO.
| Candidate | Sentence it is true/false about | In the SLO? |
|---|---|---|
| CPU idle > 40% | none: no user is in this sentence | No, diagnostic |
| HTTP 200 rate on /healthz | a probe reached the process | No |
| HTTP 200 rate on the checkout POST | the buyer's request was accepted | Only if 200 means the order was placed |
| checkout_ok / checkout_started | the buyer saw an order id | Yes |
| p99 render of search results | the shopper saw a result list | Yes |
HTTP 200 rate, in detail. It earns a place when the endpoint is the last hop the user waits on and the status code is a faithful proxy for their success. It fails when it is not: a 200 with an error body, a 200 on a cached error page, a 200 returned before the downstream write commits. It also fails when the denominator is polluted: health checks, bots and internal callers folded into the same rate make it green while real buyers fail. Detection is cheap: track the semantic success event (order id issued) next to the 200 rate. When they diverge, the 200 rate is measuring the transport, not the user, and the semantic event is the SLI.
CPU idle, in detail. It is a resource gauge, and a resource gauge belongs in an SLO only when there is no user-facing request stream to measure: a batch or queue worker, a nightly job, a training run. There the "user" is the job, and what they feel is completion time. Even then, SLO the job latency or queue drain time and keep CPU idle as the diagnostic that explains it. The deciding criteria: can a user feel it, does it change a decision (page, rollback, add capacity), and is it a proxy for something a user feels, if it is a proxy, SLO the thing, not the proxy.
A worked example, hypothetical: a team pages on CPU idle below 40% while 18% of payment confirmations stall at the bank callback. Dashboards stay green for a week; completed orders drop 12%. The CPU was idle because the workers were blocked on a network call, not because the system was healthy. The SLO that would have caught it is checkout_ok / checkout_started, not CPU idle.
So: keep the 200 rate only over the user's last hop and only when 200 means the user succeeded; keep CPU idle only where no request stream exists, and prefer the job's latency even then.
Curated: · Written: · Reviewed:
Q&A-10Mean search latency is 42 ms and users still complain that search feels slow. What are you measuring wrong?
The 60-second answer
A mean of 42 ms is total time divided by request count. Users don't experience that number; they experience their own request, and search latency is right-skewed and usually multi-modal. Report the distribution instead: a histogram with buckets around your product threshold, the fraction of requests above that threshold, and that fraction cut by segment. Device, connection type, shard, cache state. Then alert on SLO burn rate rather than on a percentile line.
Two other things are probably wrong. You're likely measuring server-side handler time, not what the user waits for; network RTT, TLS setup, render and any search-as-you-type debounce sit outside it. And you're likely aggregating across a heterogeneous population, where a healthy fleet-wide number hides one bad shard or one bad device class.
Why the mean hides the problem
A hypothetical day of 1,000,000 searches:
| Bucket | Requests | Share | Mean in bucket |
|---|---|---|---|
| 0–50 ms | 950,000 | 95.0% | 22 ms |
| 50–100 ms | 29,000 | 2.9% | 70 ms |
| 300–1000 ms | 15,000 | 1.5% | 550 ms |
| 1000–3000 ms | 6,000 | 0.6% | 1800 ms |
Mean = 42.0 ms. p50 = 22 ms. p95 = 50 ms. p99 ≈ 810 ms. Requests over 300 ms = 21,000, or 2.1%.
The mean is 42 ms and 21,000 people a day wait most of a second. Both are true. The mean is dominated by the 95% who are fine; the complaint comes from the 2.1% who aren't, and they aren't randomly distributed: they're on a slow radio, hitting a cold shard, or landing right after a deploy.
What to measure instead
Histogram first. Bucket counts add across instances, shards and time windows, so you can compute any percentile and any threshold share after the fact, and re-cut by segment without a deploy. Pick boundaries around the threshold that matters (100, 200, 300, 500, 1000 ms) not powers of ten.
Then the SLO. "99% of searches under 300 ms" gives an error budget of 10,000 requests a day. The table above spends 21,000, a burn rate of 2.1x, which exhausts a 30-day budget in about 14 days. Alert on burn rate with two windows: a fast one (2% of budget in 1 hour) that pages, and a slow one (5% in 6 hours) that files a ticket.
Trade-offs between the three
Percentiles are cheap and compact, t-digest or an HDR histogram gives you p50/p95/p99 in kilobytes per series, and they're the right thing for a dashboard. They break in three ways. You cannot average them: the fleet p99 is not the mean of the per-instance p99s, so any "average p99" panel is wrong. They're noisy at low traffic: at 100 requests per minute, p99 is the single slowest request in the window. And a p99 line can sit comfortably under threshold while 5% of users sit at 250 ms and are unhappy: the percentile value and the fraction over threshold answer different questions.
Histograms cost more memory and cardinality, and you must choose boundaries before you know what you'll ask. Burn-rate alerting needs a defined SLO and a window policy, and reacts more slowly to a sudden cliff than a plain threshold. Use all three: histogram as the source of truth, percentiles for dashboards, burn rate for paging.
Failure modes and how to catch them
- Cold cache or cold JIT after a deploy: tail spike in the first minutes. Detect with p99 by instance age.
- One hot or slow shard: fleet p99 fine, one shard terrible. Detect with p99 and threshold share per shard.
- Slow radio on mobile: desktop looks clean. Detect by segmenting on connection type.
- GC pauses: periodic tail spikes. Correlate with GC logs.
- Retry storms: the tail feeds itself. Watch requests per user alongside latency.
Close
Nobody experiences the average request. Publish the histogram, the threshold share and the burn rate; keep the mean only as a capacity and cost signal, and only after you've confirmed you're measuring the user's wait and not the server's.
Curated: · Written: · Reviewed:
Q&A-9A search index is "available" at 99.99% while new products take 47 minutes to appear. What SLI are you missing?
The place where teams go wrong on freshness SLIs for async work is usually the step before the one they are debating.
For asynchronous pipelines, SLO on the age of the newest successfully processed record, not on whether the consumer process is up.
Concretely, for a time-based freshness SLI, measure current time minus the newest completely processed source watermark and count observation intervals over the threshold as bad; alternatively measure per-record publication delay, but do not mix the two denominators.
The reason for that specificity is a failure I have seen: Paging only on consumer CPU left a stuck Kafka partition undetected for 47 minutes, and launch-day inventory for 8,400 SKUs was invisible to search.
Freshness SLO: 99% of documents searchable within 120 seconds.
# Bad minutes: watermark lag exceeds 120s.
avg_over_time((
time() - search_index_watermark_unixtime
)[30d:1m] > 120)
# 47 min stall on one partition: 47 / 43,200 min = 0.109% bad, which already
# burns a 99.9% freshness SLO (budget 43.2 min).
I would not consider it settled without evidence: Inject a tagged event at the source and assert it becomes queryable within the freshness SLO, then leave that probe running.
A worker that is alive but behind is indistinguishable from an outage for anyone waiting on the output.
Curated: · Written: · Reviewed:
Q&A-8The API returns 200 while 3.1% of invoices contain the wrong tax. Is the availability SLO enough?
No, and the gap is the point. An availability SLO counts requests that returned successfully. 3.1% of invoices carrying the wrong tax is a correctness failure that never touches that number: the API answered 200, so the SLO stays green while the business result is wrong. Availability and correctness are different SLIs over different events, and a 99.95% availability SLO says nothing about whether the tax is right.
First thing I'd clarify: what is the SLO scoped to? If the service's job is "accept and persist an invoice," availability may be the right SLI and correctness belongs to a downstream owner. If the service computes the tax, correctness is in scope and needs its own SLI and its own budget.
Then it's a design decision, how to measure correctness, and it's a trade-off, not a default:
| Approach | Coverage | Cost per check | Catches | Misses |
|---|---|---|---|---|
| Full invariant scan | 100% | compute only | internal inconsistency | a uniformly wrong rate |
| Sampled oracle | n/day | oracle call | wrong rate, if n is large enough | rare or subset-specific errors |
| Full oracle scan | 100% | oracle × volume | everything the oracle knows | the oracle's own errors |
| Downstream reconciliation | 100%, late | finance time | what finance notices | anything finance doesn't check |
Deciding criteria:
- Oracle cost per check. If the oracle is an arithmetic invariant (line items × rate == tax), it's free and you scan 100%. If it's a third-party tax engine or human review, you sample.
- Blast radius and reversibility. Money out the door, regulatory filings, or anything you can't un-send justifies the expensive oracle.
- Change rate of the logic. Rounding rules and tax tables change; a stable path needs less checking.
- Detection latency you can tolerate. Nightly is fine for invoices, not for a payment authorization.
Sampling arithmetic, because it decides the sample size: the rule of three. Zero defects in n samples bounds the rate at roughly 3/n with 95% confidence. To catch a 3.1% rate you need about 100 samples; to bound the rate below 0.1% you need ~3,000; below 0.01%, ~30,000. Sampling is cheap for gross breakage and expensive for the tail, which is exactly where a rounding bug lives.
Failure modes:
- Invariant-only checks pass a uniformly wrong rate. If every invoice gets 8% instead of 7.5%, every internal sum is consistent and nothing fires. That needs an external oracle.
- Oracle drift. If the oracle is wrong you get false alarms, and people mute the alert.
- Sampling blind spots. A uniform sample under-samples a subset: EU-only invoices, one jurisdiction. Stratify by jurisdiction and rate.
When not to build it: display-only estimates, reversible and cheap-to-correct outputs, a low change rate, or when the oracle costs more than the error it catches. Then a downstream reconciliation is the honest answer.
Concrete: a rounding change shipped behind a 99.95% availability SLO billed 14,200 invoices 1–3 cents high for 9 days before finance noticed. Availability never moved.
bad_http = events.filter(status="5xx").count() # 0.04%
bad_tax = invoices.filter(tax != oracle.tax).count() # 3.10%
correctness_sli = 1 - (bad_tax / invoices.count()) # 96.90%
# Availability SLO 99.95%: green. Correctness SLO 99.99%: exhausted on day 1.
I wouldn't call it settled without evidence: run a nightly invariant over 100% of invoices and publish the mismatch rate next to the HTTP success rate. A 200 that posts the wrong number is an outage monitoring won't name unless you ask it to.
Curated: · Written: · Reviewed:
Q&A-2A 99.9% availability SLO is green while shoppers abandon a 2.4-second search page. How do you split the objectives?
Two objectives on the same journey, answering different questions. Availability asks whether the search request returned a usable response. Latency asks whether the shopper got it before they gave up. A 99.9% availability SLO can sit green while every response arrives at 2.4 seconds, because a 200 OK at 2.4s is still a success under the availability definition.
So I'd define both on the search journey:
- Availability SLI: successful search responses / valid search attempts. Target 99.9%.
- Latency SLI: search responses served within the threshold / valid search attempts. Target 99.9%.
The threshold is the decision that matters, and it comes from product behaviour, not from the server. If abandonment climbs sharply past 2 seconds, set the threshold at 1 second, the point where conversion is still intact, not at the gateway's own timeout, which is a server-side limit, not a shopper-side one. Setting it at the timeout is the classic mistake: it turns the latency SLO into a restatement of availability.
For the budget arithmetic, 28 days is 40,320 minutes, so 99.9% leaves 40.3 minutes of allowed failure. That's the availability budget. The latency budget has the same shape but counts slow-but-successful requests: 0.1% of search attempts may exceed 1 second.
| Objective | SLI | Target | 28-day result |
|---|---|---|---|
| Availability | 2xx search responses / valid attempts | 99.9% | 99.94% (green) |
| Latency | search responses <= 1s / valid attempts | 99.9% | 91.2% (burning) |
Trace one request. A shopper searches, the gateway returns 200 at 2,410 ms. Availability counts it good. Latency counts it bad. The availability budget is untouched; the latency budget is consumed. That is exactly the state in the question: the tile is green because it is measuring the wrong thing.
For alerting, use multi-window burn rates on the latency SLO: a 5-minute window at 14.4x burn rate pages, a 1-hour window at 6x pages, a 6-hour window at 1x tickets. Availability keeps its own burn-rate alerts. Two budgets, two alert sets, one journey.
When would a single combined SLO be the right call instead? When the product genuinely treats slow as failed and you want one number to gate releases: a checkout where a 3-second response is a lost sale and nobody cares whether it eventually returned. Then define one good-event SLI, success AND under threshold, with one budget. The cost is diagnosability: when the combined number drops you cannot tell from the SLO alone whether to look at errors or latency, so you keep the two component SLIs on dashboards even if the budget is combined. For search, where a slow page still returns results and a fast error still fails, I'd keep them separate.
Watch for these failure modes. A threshold copied from the server timeout makes latency redundant with availability. A mean latency instead of a proportion-under-threshold hides the tail: a mean of 400 ms can sit on top of a p99 of 2.4s, so use the proportion. Measuring server-side only under-counts what the shopper feels, since the 2.4s includes network and render; measure at the client or from RUM where you can. And one shared budget lets a latency regression silently eat availability budget, so the on-call chases errors that aren't there.
Finally, validate the split with data: plot completion rate and p99 on the same search journey over 28 days and confirm each series can burn its own budget while the other stays inside target. If they always move together, one SLO is enough.
Curated: · Written: · Reviewed:
Q&A-14A nightly billing job has no request rate. What does an SLO look like for it?
The 60-second version. A batch job has no request rate, so the SLO cannot be "successful requests / total requests". Two SLIs carry the meaning: freshness (did the output land by the business deadline) and completeness (does the output contain every source record). Uptime and exit code are not SLIs: a job can exit 0 and still be wrong. Define both at the consumer boundary: the invoice file the downstream system reads, not the job's own "finished" log line.
SLI definitions.
- Freshness SLI = fraction of scheduled runs whose output watermark is available by the due time. The due time is a business deadline (06:00 UTC), not the cron time.
- Completeness SLI = fraction of source records present in the output, from a per-run reconciliation count.
Budget arithmetic (worked example; figures illustrative). Daily run, 30-day window, 99.5% freshness target:
- Per-run view: 0.5% × 30 runs = 0.15 runs. One late run every ~200 days. Too coarse to act on for a daily job.
- Time view: 0.5% × 30 days = 3.6 hours = 216 minutes of lateness budget per 30 days. This is the number to alert on, because lateness accumulates across runs.
Completeness for billing gets a deliberately tiny budget: target 100% row reconciliation per run, SLO 99.9% of runs reconcile exactly → 0.03 runs per 30 days, effectively zero tolerance. Missing rows are money.
What the SLI looks like in practice.
| Run date | Due | Finished | Rows in / out | Freshness | Completeness |
|---|---|---|---|---|---|
| 2026-08-02 | 06:00 | 05:12 | 1,204,331 / 1,204,331 | pass | pass |
| 2026-08-11 | 06:00 | 05:08 | 1,211,004 / 1,133,492 | pass | fail, 6.4% dropped |
| 2026-08-19 | 06:00 | 07:41 | 1,209,880 / 1,209,880 | fail, 101 min late | pass |
-- after the output is committed, once per scheduled run
SELECT
(SELECT count(*) FROM source.accounts WHERE billing_period = :period) AS src,
(SELECT count(*) FROM billing.invoices WHERE billing_period = :period) AS out;
-- completeness SLI for the run = out / src; alert when src <> out
SELECT max(committed_at) FROM billing.invoice_runs WHERE billing_period = :period;
-- freshness: late when committed_at > due_at
Failure modes and how each is detected.
- Silent filter change. The job exits 0 and drops rows: I have seen 6.4% of accounts skipped this way, with wrong invoices going out for 11 days. Detected only by count reconciliation, never by exit code.
- Upstream source late. The job starts on time but reads a partial partition. Detect by checking the source watermark before processing and failing fast rather than billing a partial day.
- Partial commit plus retry. Output count exceeds source count. Detect with a uniqueness check on the idempotency key, not just a count.
- Backfills polluting the SLI. Exclude ad-hoc runs from the scheduled-run denominator, or track them as a separate SLI.
- Timezone drift. A 06:00 UTC deadline is unambiguous across DST; a local-time deadline is not.
Alternatives, and when not to use them. An exit-code or uptime SLO is cheap and useless for correctness. A per-record SLO is precise but needs reconciliation infrastructure: worth it for billing, overkill for a nightly analytics rollup. End-to-end freshness measured at the consumer is the strongest signal, but it requires the consumer to emit a watermark.
Error budget policy. When the 216 minutes are burned, freeze schema and feature changes to the job and add reconciliation before anything else ships. Exit status 0 is not an SLO; the output either arrived on time and whole, or it did not.
Curated: · Written: · Reviewed:
Q&A-11Should a 99.9% SLO reset on the first of the month or slide every minute?
I would answer this by separating what rolling versus calendar SLO windows guarantees from what it merely usually does.
Prefer a rolling window so a disaster on the 2nd cannot hide behind a fresh calendar month, unless a contract forces calendar reporting.
Concretely, compute the SLI over the trailing 28 or 30 days every minute, keep a calendar view only for external reports, and drive freezes from the rolling number.
The reason for that specificity is a failure I have seen: A 14-hour outage on the 1st was "reset" on the 1st of the next month, so feature launches resumed while users were still inside the pain of the previous window.
14-hour outage starting 00:10 on the 1st, 99.9% / 30-day budget = 43.2 min.
| Window | Budget remaining on the 2nd | Freeze? |
|---|---|---|
| Calendar month (resets 00:00 on 1st) | 0 min until the 1st next month | freeze all month |
| Calendar month if you "start over" next month | 43.2 min on the 1st | false green |
| Rolling 30d | 0 min for the next 29 days | freeze until burn ages out |
I would not consider it settled without evidence: Replay a 14-hour outage on day 1 of a calendar month against both windows and show remaining budget on day 2.
Users do not forgive you because a spreadsheet rolled over at midnight.
Curated: · Written: · Reviewed:
Q&A-7Availability rose to 99.98% after the team started retrying inside the SLI window. What went wrong?
Nothing about the service got more available. The SLI changed what it counts. Once a retry inside the window counts as a fresh attempt, a request that failed and then succeeded is scored good, so the number rises while the user experience is unchanged. Availability went from measuring the system to measuring the retry loop.
Assumptions that matter here: the SLI is defined per logical user request and credits eventual success; the retry happens client-side or in the gateway, inside the same measurement window; and the underlying failure is transient enough that a retry usually succeeds. Change any of those and the diagnosis changes.
Why the number moved
Two counting rules produce two different numbers from identical traffic:
| Counting rule | Success | What the user experienced |
|---|---|---|
| First attempt only | 99.50% | 500 of 100,000 users hit a failure and waited for a retry |
| Any attempt succeeds (eventual) | 99.98% | the same 500 users, the same wait |
| First attempt within 300 ms | 98.10% | the same 500 users, plus everyone whose first attempt was slow |
Figures are illustrative, but the shape is the point: the 0.48-point gap between the first two rows is the retry loop, not reliability. Note the direction, because it tells you which rule you're on. If the SLI counted per attempt instead of per logical request, retries would add to both numerator and denominator and the number would fall, not rise. A rise to 99.98% is the signature of a per-logical-request SLI that now credits eventual success.
What the user actually got
Retries don't remove the failure; they hide it and add latency. The user who would have seen an error now sees a spinner for the retry's duration. p99 latency goes up while availability goes up: the two moving in opposite directions is the tell.
What to measure instead
- Success means the first attempt succeeded within the user-visible latency budget. One logical request, one outcome.
- Expose retry rate and retry success rate as separate diagnostic metrics, never folded into the SLI.
- Alert on first-attempt failure rate; it's the leading indicator the SLI just stopped showing.
Failure modes this creates
- Retry storms: retries multiply load exactly when the backend is already degraded. Three retries per client across 100k clients is 300k extra requests arriving during the incident. Use exponential backoff with jitter and a retry budget: cap retries at a fraction of total requests so the loop can't outrun capacity. AWS's Builders' Library covers the mechanics.
- Metastable failure: the retry load holds the system in a degraded state that persists after the original trigger clears (Bronson et al., "Metastable Failures in Distributed Systems", HotOS 2021).
- Non-idempotent retries: a retried payment can double-charge. Retry only idempotent operations, or carry an idempotency key.
- Goodhart: once the SLI is the target, the cheapest way to move it is to stop counting the failures users already felt.
Detection
Compare first-attempt success against eventual success over the same window. A gap above roughly 0.5 points means the SLI is contaminated by retries. Then check the latency: if p99 rises while availability rises, you're measuring recovery, not health.
Curated: · Written: · Reviewed:
Q&A-12Legal signed 99.9% monthly availability with a customer, while the team tracks 99.95% internally. How should the two relate?
The judgement in customer SLO vs internal SLO lies in what you refuse to do as much as in what you build.
Keep the internal SLO strictly tighter than the customer contract so internal burn triggers action before a credit is owed.
Concretely, choose an internal objective with enough quantified buffer to detect and mitigate burn before breaching the SLA, based on incident and response-time modeling rather than an arbitrary extra nine.
The reason for that specificity is a failure I have seen: Matching the contract at 99.9% meant the first internal page was also the first billable credit, and finance learned of the miss from the customer.
Internal 99.95% / 30d versus customer 99.9% / calendar month.
| SLO | Budget | Used as |
|---|---|---|
| Internal 99.95% rolling 30d | 21.6 min | pages, freezes |
| Customer 99.9% calendar month | 43.2 min | credits |
| Incident of 28 min | internal exhausted, contract intact | working as designed |
I would not consider it settled without evidence: For the last four incidents, show internal page time, remaining internal budget, and whether the contractual SLO was still intact.
The SLO you operate to must trip before the SLO you sold, or you have no warning channel.
Curated: · Written: · Reviewed:
Q&A-4A 30-day SLO pages only after the month is already lost. How do you alert on burn rate instead?
The useful framing here is to ask what evidence would change my mind about multi-window burn-rate alerts.
Page on a fast burn window and ticket on a slow burn window so a short outage and a quiet leak both get a human before the budget is gone.
Concretely, for a 30-day SLO, page when both 1h and 5m windows exceed 14.4× burn, or both 6h and 30m windows exceed 6×; open a ticket when both 3d and 6h windows exceed 1×, using the same bad-event ratio as the SLO.
The reason for that specificity is a failure I have seen: A 2% error leak for 11 days never crossed a 5-minute error-rate page, then the monthly report showed 0 minutes of budget left.
Multi-window burn rates for a 30-day 99.9% SLO (43.2 min budget).
# 14.4x for 1h consumes 2% of the budget; sustained 14.4x burn exhausts the full budget in about 50 hours.
(
sum(rate(sli_bad[1h])) / sum(rate(sli_total[1h]))
) / (1 - 0.999) >= 14.4
and
(
sum(rate(sli_bad[5m])) / sum(rate(sli_total[5m]))
) / (1 - 0.999) >= 14.4
I would not consider it settled without evidence: Replay the last three budget-exhausting incidents through the burn-rate rules and record page time versus the minute the budget actually hit zero.
If the page fires after the monthly slide is already red, the alerting window is the wrong length.
Curated: · Written: · Reviewed:
Q&A-3Your 30-day 99.9% SLO has 43.2 minutes of remaining budget and a risky migration is scheduled. What policy do you apply?
Answer
43.2 minutes is the entire 30-day budget for 99.9%: 43,200 minutes × 0.001. So "43.2 minutes remaining" means nothing has burned yet: the service is at 100% of budget, green band. The budget does not block the migration. What decides it is the migration's blast radius and whether it can be rolled back.
The policy I apply:
| Remaining budget | Band | Change allowed |
|---|---|---|
| > 21.6 min (>50%) | green | normal deploys, standard canary |
| 8.64–21.6 min (20–50%) | amber | risky changes need canary + 2h soak + SRE sign-off |
| < 8.64 min (<20%) | red | reliability work only; risky migrations blocked |
At 43.2 min the migration is green by budget, so it ships: with three conditions that are about risk, not budget:
- Canary at 1% of traffic for 30 minutes before ramp.
- Automated rollback, tested in staging, not a runbook step someone executes at 3am.
- An abort rule tied to burn rate, not to a human noticing.
The abort rule is where the arithmetic matters. Burn rate 1x consumes the budget exactly over 30 days. The multi-window alerts in the Google SRE Workbook for a 30-day window are: 14.4x over 1 hour (2% of budget, page), 6x over 6 hours (5%, page), 1x over 3 days (10%, ticket). At 43.2 minutes of budget, 14.4x for an hour costs 0.86 minutes. The canary trips a page long before it costs anything you'd miss, so wire the canary to that alert and roll back automatically.
Why bands instead of a single freeze line: a hard "freeze below 50%" rule means one bad Tuesday stops all work for a month, including the reliability fixes that would restore the budget. Bands let you trade change velocity against remaining budget explicitly, and they make the trade reviewable.
The failure mode this prevents: a schema rewrite shipped with 4 minutes of budget left. It ran 6 hours at full error rate: 360 minutes against a 43.2-minute budget, 8.33x the entire 30-day allowance. The SLO stayed violated until the outage aged out of the rolling window 30 days later. Detection is the burn-rate alert firing within minutes; the reason it wasn't caught is that the change had no canary and no automatic rollback, so the only signal was customer reports.
Two things I'd check before approving, because budget is a lagging indicator:
- Is the migration reversible? Dual-write + backfill + cutover can be rolled back; an in-place column type change cannot. Irreversible changes need a rollback path designed first, regardless of band.
- What is the worst-case cost in budget minutes? If the worst case exceeds the remaining budget, the change is red-band by risk even when it is green-band by budget.
A budget that never blocks a launch is a slide, not a control. But the control is the abort rule and the rollback path, not the number on the dashboard.
Curated: · Written: · Reviewed:
Q&A-13Your service SLO is 99.9% and you call four dependencies each at 99.9%. How do you budget for that?
The 60-second answer
99.9% over 30 days is 43.2 minutes of bad events. Four hard dependencies in series, each at 99.9%, multiply out to 0.999⁴ = 99.6%: 172.8 minutes, four times the budget. So the budget is not split four ways. It is allocated by contribution, and the sum of every dependency's contribution plus your own code has to fit inside 0.10%.
Contribution = (share of requests that need it) × (1 − its SLO) × (1 − fallback coverage). A dependency on 8% of requests behind a cache costs a fraction of one that is on every request with no fallback. That is the whole method: weight by exposure, then subtract what the fallback absorbs.
The arithmetic
| Dependency | Share of requests | Vendor SLO | Raw contribution | Fallback | Effective contribution |
|---|---|---|---|---|---|
| Auth | 100% | 99.9% | 0.100% | 5-min token cache; only new sessions call it (~15% of requests) | 0.015% |
| Catalog | 100% | 99.95% | 0.050% | serve stale up to 60s | 0.020% |
| Payments | 25% | 99.9% | 0.025% | none: money path | 0.025% |
| Tax API | 8% | 99.9% | 0.008% | cached rates, 24h TTL | 0.002% |
| Your own code | — | — | — | — | 0.020% |
| Total | 0.082% of 0.10% |
The fallback-coverage fractions are estimates; the point of the table is that they get replaced by measured numbers. If you cannot add fallbacks, the requirement becomes each dependency at 99.975% (0.999^0.25), which is a vendor conversation, not an engineering one.
Failure modes
Independence is usually false. The multiplication assumes the four fail separately. If all four run in one region, one AZ event takes the product to 0%, not 99.6%. The serial figure is an upper bound on availability, not a promise. Check for shared blast radius (same region, same DNS, same identity provider) before you trust the number.
Fallback coverage is measured, not assumed. A cache that is cold after a deploy absorbs nothing during the window that matters. Track the fraction of requests actually served by the fallback over 30 days; if it is 40% and you budgeted 80%, the slice is already blown.
Retries cut both ways. A retry with idempotency keys raises effective availability against transient errors and makes a saturated dependency worse: the retry storm turns a 30-second blip into a 10-minute outage. Cap retries at one, with jitter and a circuit breaker.
The slice constrains you, not the vendor. You cannot make a vendor hit their published SLO. The slice is a design constraint: it tells you which paths need a fallback before launch, and which dependency is the first candidate for replacement.
What you would actually do
Publish the table in the service spec so the constraint is visible before code is written. Alert on each dependency's measured contribution over a rolling 30 days against its slice. Write the error-budget policy down: when a dependency burns its slice, feature launches on that path freeze until a fallback ships or the vendor is replaced. Without that trigger, the table is documentation, not a budget.
Curated: · Written: · Reviewed:
Q&A-100The budget is half gone at day 12. Do you spend the rest on features or on deleting toil?
Spend it on toil. Half the budget gone on day 12 is not on pace, it is 25% over it, and features add failure surface while toil is where the burn comes from.
State the assumptions, because the numbers mean nothing without them: a 99.9% availability SLO over a 30-day window gives 43.2 minutes of error budget. Day 12 is 40% of the window, so 50% consumed means you are burning at 1.25x the sustainable rate. 21.6 minutes spent in 12 days is 1.8 min/day; the remaining 21.6 minutes runs out on day 24, six days before the window closes. That projection, not the raw 50%, is the argument.
Why toil and not features. Error budget is burned by user-visible failures, and toil is where those failures concentrate: manual failovers, hand-run backfills, alert triage a human does by hand at 3am. Each toil item is a recurring source of human error and a recurring consumer of the engineering time you would otherwise spend preventing the next incident. Deleting one removes both. A runbook that takes 30 minutes and fires three times a month is 90 minutes of on-call a month plus whatever incident it eventually causes; automating it is a one-time cost that stops the recurrence. A feature does the opposite (new code path, new dependency, new alert) and it starts burning budget the day it ships.
The exception: a feature can ship if it comes with a budget grant and a tighter canary. The canary does not lower your burn rate, it bounds the cost of being wrong. A bad release at full traffic might burn 10 minutes before you notice; at 1% it burns roughly a tenth of that before the canary catches it. That is the only reason to allow a launch in amber.
The case where I would keep shipping: if the 21.6 minutes came from a single incident that is already root-caused and fixed, the forward burn rate is near zero, not 1.8 min/day, and the day-24 projection is wrong. Check whether the burn is one spike or a slope before you reallocate the sprint.
Failure mode to watch: you spend the sprint on toil that does not touch burn (rewriting a dashboard nobody pages on, clearing a backlog) and the freeze arrives anyway with the real toil untouched. Detect it by tracking toil hours against budget burn week over week; if toil hours drop and the burn rate does not, you deleted the wrong toil.
| Signal | Value | Action |
|---|---|---|
| Budget remaining | 21.6 of 43.2 min | — |
| Elapsed | day 12 of 30 (40%) | — |
| Burn rate | 1.8 min/day | projected exhaustion day 24 |
| Decision | — | next sprint: top toil item + top incident class |
| Feature launches | — | budget grant + 1% canary only |
So: toil first, features only behind a grant and a canary, and re-check the burn rate at day 18.
Curated: · Written: · Reviewed:
Q&A-97Payments is a vendor with no public SLO. How do you still run a reliability program on that hop?
You don't need the vendor's SLO. You need a caller-observed SLI for that hop, a slice of your own error budget assigned to it, and a rule for what happens when the slice runs out. The vendor's number, if it existed, would only be an input to that; what your callers experience is what you actually promise.
Measure the hop the way your users feel it: success rate and latency of the outbound call as seen from your service, not the vendor's status page. Then decide how much of your budget it may consume.
Take a checkout service with a 99.95% success SLO over 30 days. At 1,000 rpm that's 43.2M requests, so 0.05% is 21,600 bad events. Give the payments hop 40% of that budget: 8,640 bad events, or 0.02% of requests. The vendor now has to succeed at least 99.98% as you observe it, or the hop is overrunning.
| Signal, 30d, caller-observed | Measured | Slice | Verdict |
|---|---|---|---|
| payments success | 97.20% | ≥99.98% | overrun |
| payments p99 | 1,840 ms | 250 ms | overrun |
| checkout budget burned by this hop | 80% | ≤40% | overrun |
Once it overruns you have four moves, and the choice is decided by two things: whether the hop is on the critical path, and whether you have leverage.
Negotiate a contract with credits. Works when the vendor is big enough to care and you're a meaningful share of their revenue. Doesn't work when they can't fix the underlying capacity problem, and it never fixes anything this quarter.
Fall back. Cache, queue-and-retry, or route to a secondary path. Use it when the failure is transient and the fallback is cheaper than waiting on the vendor. Don't use it when the fallback changes correctness: serving cached authorization for a payment is a different decision from serving cached catalog data.
Add a second vendor. Use it when the hop is on the critical path and the vendor's reliability is structurally below what you need. It costs integration work and reconciliation, so it's rarely worth it for a hop you can degrade.
Absorb it. Raise your own headroom and accept the risk. Reasonable when the hop isn't user-facing.
The budget-slice approach is the wrong tool when you can't measure the hop from callers (fire-and-forget async, batch jobs) or when the vendor is your only option and you have no leverage. Then the slice is bookkeeping and the real work is fallback design.
Failure modes worth naming. An unowned timeout: a 2.8% timeout rate sat for six weeks and burned 80% of the checkout budget, because nobody alerted on the hop's own SLI. Conflation: a caller-observed SLI includes your timeouts, bad requests and retry storms, so break it out by error class before blaming the vendor. Silent degradation: the vendor raises p99 without failing anything, success stays green, so watch latency as its own signal. Retry amplification: your retries inflate their load and your latency, which shows up as a vendor problem you caused.
Keep the evidence: a 30-day caller-observed SLI next to the allocated slice, and a dated decision. Contract signed, fallback shipped, or risk accepted. A dependency without an SLO is still inside your SLO; you're just not looking at it.
Curated: · Written: · Reviewed:
Q&A-5On-call spent 19 hours last week restarting a batch worker. Is that toil, and what do you do with it?
Yes, it's toil: manual, repetitive, automatable, tactical, no enduring value, and it scales with the service. But classifying it is the easy half. The 19 hours is a symptom, and the reflex to write a restart script is the trap.
Start by asking what the restart is actually doing. If the worker is being OOM-killed, restarting clears the symptom and leaves the cause: a leak, an unbounded queue, or a batch that outgrew its container limit. Automate that restart and you convert a visible 19-hour problem into an invisible one that fires at 03:00 and pages nobody. So split the response in two:
- Remove the cause. Bound the input (cgroup memory limit, queue-depth cap) fix the leak if there is one, and give the worker a retry budget so a poison message fails the batch instead of looping forever.
- Bound the blast radius. Automatic recovery with alerting: restart on OOM, but emit an event and count it. More than a handful of restarts in an hour pages someone. That's the line between self-healing and a silent crash loop.
Then classify the week honestly. Not everything on-call does is toil, and the number that matters is the toil fraction, not the raw hours:
| Activity | Hours | Toil? | Next action |
|---|---|---|---|
| Restart batch worker on OOM | 19 | yes | cgroup limit + retry budget |
| Design review for new SLO | 4 | no | keep |
| Copy-paste config to 12 shards | 7 | yes | generate from one source |
| Customer incident command | 6 | no | keep |
Those two toil rows get different owners. The 19 hours of restarts is a defect ticket: the work disappears when the cause does. The 7 hours of copy-pasting config is an automation ticket: there's no defect, just work a generator should do. Treating them the same is how teams end up automating a crash loop.
Set a ceiling and track against it. A common convention is to cap ops work at 50% of on-call time; a breach is a signal to fund work, not to ask for headcount. Track toil hours per rotation so the trend is visible before it becomes a resignation.
Verify with a before/after. Take the four weeks before the fix and the four after, and compare restart count, pages, and toil hours for the same task. If restarts drop from 19 hours a week to under 1 and the page count goes to zero, the fix worked. If hours stay flat, you automated the symptom.
Failure modes worth naming: a restart loop that hides a leak for months; an alert that fires on every restart so nobody reads it; a toil budget that becomes a target to game rather than a trigger to fix; and a retry budget set so high that a poison message burns the whole batch window before failing.
When automating is right: the restart is genuinely unavoidable (upstream dependency flaps, no fix available), it's idempotent, and you can alert on the rate. Then automate it and keep the counter: the counter is what tells you the underlying problem is still there.
Curated: · Written: · Reviewed:
Q&A-24The primary pager received 61 pages last week and acknowledged 58 of them in under 30 seconds. What is broken?
What's broken
The ack rate. 58 of 61 pages acknowledged in under 30 seconds is not a healthy rotation: it's a rotation that has learned to clear the pager without investigating. A sub-30-second ack on a page that needs no action is a dismissal, not a response. Once 95% of pages are cleared that fast, ack time stops measuring responsiveness and starts measuring how quickly someone can make the noise stop.
The defect underneath is the alert set. 61 pages a week is about 4.4 per 12-hour shift, more than double the two-events-per-shift ceiling Google's SRE guidance sets for on-call. Volume at that level trains reflexive acknowledgement, and reflexive acknowledgement is what turns the 62nd page, a real incident, into a 41-minute silence.
What to measure instead
Ack time rewards fast dismissal, so it is the wrong SLI. Track three things: pages per shift, ack-without-action rate (pages acked with no ticket opened, dashboard queried, or runbook started), and the fraction of pages that produce an incident doc. A healthy set has a high incident-doc fraction, not a low one. If nearly every page becomes an incident, the pages are real. If almost none do, you are paging on noise.
| Metric | Last week | Target |
|---|---|---|
| Pages per week | 61 | ≤ 28 |
| Pages per 12-hour shift | 4.4 | ≤ 2 |
| Acked < 30 s with no ticket or dashboard opened | 58 (95%) | < 10% |
| Pages that produced an incident doc | 2 (3%) | > 50% |
Failure mode and detection
The concrete failure: 61 pages trained the rotation to auto-ack, and the 62nd, a real 5xx event, sat acknowledged for 41 minutes before anyone looked. Ack time cannot detect this, because the ack was fast. Detection requires correlating the ack with a follow-up action: a ticket, a dashboard query, a runbook step. If a page is acked and nothing else happens within a few minutes, treat it as unacknowledged and re-page.
What to do
Set a budget of two pages per 12-hour shift and delete or downgrade alerts until traffic fits. Anything that does not need a human now goes to a ticket or a dashboard. Re-check the numbers weekly; if the incident-doc fraction stays near zero, the alert set is still noise.
Curated: · Written: · Reviewed:
Q&A-25A 7-day primary rotation ends with 19 hours of night work and a 4-line Slack handoff. How do you run load and handoff?
19 hours of night work across 7 nights is 2.7 h/night on average. That average is the wrong number to act on. Pull the per-night distribution: if two nights were 6 hours and five were 1.4, the rotation is mis-sized for one primary; if every night was 2.7, the problem is interrupt rate. Run load by capping consecutive night work at two nights, then a 24-hour break, and by tracking sleep-disrupting pages separately from total pages. A page at 03:00 that wakes you costs more than a 14:00 page, even if both are Sev-3.
Handoff: a 4-line Slack message is a pointer, not a state file. Replace it with a written state file plus a 10-minute live call. The written file must let the next primary act without a conversation. Minimum fields:
Primary: 2026-09-06 09:00 -> 2026-09-07 09:00
Open incidents: INC-4412 (Sev-2, vendor, next update 10:30)
Silences: checkout_burn_1h until 11:00 (owner: sre-lee, reason: load test)
In-flight: canary 12% on payments-v2, rollback: RB-88
Pages: 3 (2 real, 1 mis-aimed). Interrupt hours: 2.1
Follow-ups: raise HPA max 30 -> 45 after traffic report
The live call is for the incoming primary to ask "what would you do if X fires at 02:00?" and for the outgoing to flag anything not in the file. Written-only fails when a silence's reason is ambiguous; verbal-only fails when the outgoing primary is asleep or in another timezone.
Trade-offs: 7-day primary with a written handoff is fine when page volume is under one page/night and no Sev-1s. It fails when night work exceeds 4 hours on any single night or when the same person is primary for two consecutive weeks. 24-hour primary rotations cut night hours per person but add context switches; use them when the team has at least five people. Follow-the-sun removes night work entirely but needs three or more regions and a handoff window that overlaps; it is not available to a three-person team.
Failure mode: a 4-line handoff omitted a silenced burn-rate alert. The weekend primary found out when the monthly error budget hit zero on Sunday night. Detection: audit 8 consecutive handoffs for the required fields; if any field is missing, the handoff is not done. Plot interrupt hours per primary; act when the max single night exceeds 4 hours or the 7-night total exceeds 19.
When not to use a written state file: if the shift had zero open incidents, zero silences, zero in-flight changes, and zero pages, a one-line "all clear" is enough. The state file is for state that can bite the next person.
Curated: · Written: · Reviewed:
Q&A-23The on-call is paged for disk at 70% while users are fine, and not paged when checkout 5xx hits 12%. What do you page on?
Page on user-visible symptoms measured against an SLO, and page on a resource cause only when its time-to-exhaustion is shorter than the time it takes to fix. Everything else (disk at 70%, one replica at 80% CPU) is a ticket.
The mechanism is burn rate. An SLO of 99.9% over a 30-day month gives 43.2 minutes of full-failure budget. Burn rate is the fraction of budget consumed divided by the fraction of the window elapsed, so it converts any error rate into "how fast am I spending the month." Google's SRE Workbook pairs a long window (to confirm it's real) with a short window (to fire fast and clear fast):
| Long window | Short window | Burn | Budget consumed | Action |
|---|---|---|---|---|
| 1h | 5m | 14.4x | 2% in 1h | page |
| 6h | 30m | 6x | 5% in 6h | page |
| 3d | 6h | 1x | 10% in 3d | ticket |
Worked example, assumptions labelled: checkout at 100 rps, 99.9% availability SLO, 30-day month. Monthly requests = 100 × 2,592,000 = 259.2M; budget = 259,200 errors. A 12% 5xx rate for 5 minutes is 100 × 300 × 0.12 = 3,600 errors, which is 3,600 / 259,200 = 1.39% of the budget in 5 / 43,200 = 0.0116% of the month. Burn rate = 0.0139 / 0.000116 ≈ 120x. That is eight times the 14.4x page threshold: it should have paged in under a minute, not waited for someone to notice on social media.
Now the disk. A level is not a symptom; a deadline is. A 500 GB log volume at 70% has 150 GB free. Growing 3 GB/h, that is 50 hours to full: a ticket. Growing 30 GB/h, it is 5 hours, and if the fix (expand the volume, ship a log-rotation change) takes longer than 5 hours, that is a legitimate page. So the rule is time-to-exhaustion versus time-to-remediate, not a percentage.
| Alert | Fires on | Page? |
|---|---|---|
| checkout 5xx burn 14.4x / 1h | user symptom | yes |
| latency bad-event burn, requests > 300 ms | user symptom vs latency SLO | yes |
| disk 70% on a log node, 50h to full | cause, no deadline | ticket |
| disk 85%, 5h to full, fix takes 8h | cause with a deadline | yes |
| CPU > 80% on one replica | cause, HPA absorbs it | no |
Two failure modes to name. First, symptom-only paging with a single short window misses slow burns: a 1% error rate that never trips the 1h window still drains the month, which is why the 3d/6h ticket row exists. Second, cause-based paging trains the rotation to ignore the pager: the 70% disk page at 03:11 that fires 14 nights running is exactly what makes a 12% checkout failure look like noise. Detect both by auditing pages against SLO burn over 14 days: count pages that coincided with budget consumption versus pages that did not, and drive the uncorrelated ones off the pager. Track the ratio of pages that had a runbook action to pages that were closed as "no action needed"; if that ratio drops, the pager is losing authority.
The trade-off is real: pure symptom paging leaves you blind to capacity that will cause an outage in six hours but has not yet. That is what the time-to-exhaustion rule covers: it is still a cause alert, but it is admitted to the pager only when the deadline is inside the response window.
Curated: · Written: · Reviewed:
Q&A-15A CI/CD pipeline is down 9% of weekday hours. Why is that an SRE problem rather than a developer inconvenience?
The 90-second answer
It's an SRE problem because the pipeline sits on the critical path of two things SRE owns: shipping change and mitigating incidents. A pipeline that's dark 9% of weekday hours is a change freeze nobody declared, and worse, it usually takes the rollback path down with it, so the system you'd use to recover is the system that's broken.
Put a number on it. Weekday hours = 5 × 24 = 120 h/week, so 9% is 10.8 h/week, roughly 2.2 h every weekday. Count only working hours (5 × 8 = 40 h) and it's 3.6 h/week. Either way it isn't one developer's inconvenience: it's a shared dependency failing for every team at once, and the failures are correlated, not independent.
The test for "SRE problem" is blast radius plus recovery path. If the pipeline gates production deploys and the rollback job runs through it, pipeline availability is a term in your MTTR, and MTTR is an SRE number.
Worked example (hypothetical figures)
Runner outage 09:00–13:00. Sev-1 starts 09:20. Rollback job queued 09:25, can't start until runners return at 13:00, runs 22 min, done 13:22. MTTR = 4 h 2 min against a 22-minute baseline. The pipeline didn't cause the incident; it set the floor on how fast you could end it.
Where the pipeline sits
| Journey | SLI | Target | Last 30d |
|---|---|---|---|
| checkout API | success rate; p99 ≤ 300 ms | 99.90% | 99.93% |
| default-branch pipeline | start succeeds and finishes < 18 min | 99.00% | 91.00% |
| rollback job | triggered and completes < 10 min | 99.90% | not measured |
91% is the 9% in the question. The rollback row is the one that matters, and nobody was measuring it.
Three postures, and when each is right
- SLO plus page on burn (multi-window, multi-burn-rate). Right when the pipeline gates production changes and rollback, and more than a handful of teams depend on it.
- Ticket-only. Right when there's a manual fallback that's exercised regularly and the pipeline is not on the rollback path.
- Dashboard-only, reviewed weekly with a named owner. Middle ground: no pager, but the number is visible and someone is accountable.
Deciding criteria: is the pipeline on the rollback/hotfix path? How many teams block simultaneously? Is there a manual fallback, and when was it last exercised? What does paging cost: how many pages per week at the current burn rate?
When it isn't worth the overhead: docs-only or low-traffic repos; a team deploying monthly with a working manual path; a small org where the pipeline owner is the same on-call who'd get paged. You'd be paging yourself.
Failure modes
- Paging on raw pipeline failure conflates broken tests with broken infra. Split the SLI into infra-caused and test-caused failures, or the on-call learns to ignore the alert.
- Rollback sharing the runner pool or credentials with CI. Detect with a scheduled synthetic rollback drill; alert when the drill fails.
- A target set from aspiration rather than measurement. 99.9% on a pipeline that has never been above 95% produces permanent burn and alert fatigue.
Evidence to settle it
Report pipeline availability and mean time-to-green weekly next to product SLOs, and show at least one incident where pipeline downtime delayed mitigation. If the number is 91% and no incident has ever been delayed by it, you have a ticket, not a pager.
Curated: · Written: · Reviewed:
Q&A-17Three engineers are all debugging during a Sev-1 and nobody is talking to customers. How do you structure the response?
Before choosing an implementation I would establish what success looks like for incident command.
Name a single incident commander who does not debug, plus a dedicated comms lead, so diagnosis has a coordinator and customers have a voice.
Concretely, on Sev-1/2, the first responder either becomes commander or hands command within 5 minutes, assigns an ops lead and a comms lead, and records those names in the incident doc.
The reason for that specificity is a failure I have seen: Four people patched four different services for 38 minutes because nobody owned the hypothesis list, and the actual stuck deploy ran the whole time.
Roles on a Sev-1 at T+5 minutes.
| Role | Person | Allowed to SSH/debug? |
|---|---|---|
| Incident commander | SRE-2 | no |
| Ops lead | SWE-4 | yes, one change at a time |
| Comms lead | support-1 | no, owns customer + status page |
| Scribe | SRE-2 (dual-hat if short) | timeline only |
I would not consider it settled without evidence: Audit the last 8 Sev-1s for a named commander within 5 minutes and a comms update on the agreed cadence.
An incident with many helpers and no commander is just a group of people interrupting each other.
Curated: · Written: · Reviewed:
Q&A-98The status page says "degraded performance". What does on-call actually do in the first 15 minutes?
The status page is the vendor's claim about their own system. It says nothing about whether your checkout is failing. So the first 15 minutes are: confirm it's real and ours, scope it, pick a mitigation, and set a deadline to escalate, not read the vendor's feed.
Assumption I'd state out loud: we have client-side SLIs per dependency (success rate, p99, error class) and a deploy timeline. Without those, minute 2 is already lost.
| Minute | Action | Signal that decides it |
|---|---|---|
| 0–2 | Acknowledge, open incident channel, name an IC | Our dependency success rate below SLO → real |
| 2–5 | Scope: all traffic or one region/endpoint? Correlate with deploys in the last 30 min | Deploy in window + error spike → rollback |
| 5–8 | Classify: our fault vs vendor fault | Our fault → rollback/flag off; vendor → fallback or fail closed |
| 8–12 | Mitigate: flip fallback flag, shed non-critical calls, circuit-break the dependency | Error rate falling → hold; flat → escalate |
| 12–15 | Comms, write the revert condition, set a decision deadline | No revert condition written → escalate now |
A cache fallback is right for read paths where stale is tolerable: recommendations, catalog, a tax estimate. It is wrong for money-moving writes. Serving stale auth or pricing on a payment can charge the wrong amount, which is worse than a clean 5xx. Fail closed there and let the request error.
# python 3.12 — fail closed on money paths, fall back on read paths
def authorize(order):
if breaker.open("payments"):
raise FailClosed("payments degraded") # never serve stale auth
return payments.authorize(order, timeout=0.8, retries=1, backoff="jitter")
Retries multiply load on a dependency that is already struggling. Jittered backoff behind a circuit breaker, not "retry harder".
Wait vs mitigate is a real call. If the vendor's ETA is three minutes and impact is 0.2% of checkouts, waiting is cheaper than a fallback that risks inconsistent data. If checkout is failing across the board, mitigate now and reconcile later.
Failure modes, and how you catch them:
- Chasing the status page. Time-to-first-mitigation past 5 minutes is the tell: that is Slack archaeology while a rehearsed fallback would have restored checkout in 90 seconds.
- No decision deadline. Nobody escalates and the incident drifts. Detect: no revert condition written by minute 15.
- Stale cache on a money path. Wrong charges surface later. Detect: reconciliation mismatch against the vendor's ledger.
The 15 minutes end with either a mitigation in flight or an escalation, plus a written revert condition. Still reading the vendor's status page at minute 15 means you spent the window on a news feed.
Curated: · Written: · Reviewed:
Q&A-18A dashboard is wrong but checkout works. Is that Sev-1? How do you classify?
No. A wrong dashboard with checkout working is Sev-4, or Sev-3 if customers see it. It is Sev-1 only when the dashboard is the revenue path: a merchant settlement report paying out the wrong number, or a status page telling customers the site is down while it is up.
State the assumptions before you classify, because they decide the answer:
- "Dashboard" means an internal ops or metrics view. If it is customer-facing, the row changes.
- "Checkout works" means the revenue path is serving. If checkout is degraded, the dashboard is irrelevant to the severity.
- Nothing automated acts on the dashboard. If a controller reads it and scales or fails over, its wrongness is an input to a real incident, not a cosmetic bug.
The matrix
Written severity matrix, applied at T+2 minutes.
| Impact | < 5% of users | 5–50% | > 50% or revenue path |
|---|---|---|---|
| No user-visible effect | Sev-4 | Sev-4 | Sev-3 |
| Degraded, workaround exists | Sev-3 | Sev-2 | Sev-2 |
| Hard down or wrong money | Sev-2 | Sev-1 | Sev-1 |
Applying it to the scenario
- Internal ops dashboard, wrong numbers, no customer sees it: row 1, under 5% of users → Sev-4. Fix it in business hours.
- Same dashboard, but it is the only signal on-call has and it shows green during a live 5xx spike: the incident is the 5xx spike. Classify that by its own row. The dashboard bug is a contributing cause, logged as a Sev-4 follow-up.
- Customer-facing status page or merchant settlement report showing wrong state or wrong money: row 2 or 3 → Sev-3 to Sev-1 depending on the fraction of users and whether money is wrong.
Failure modes
Inflation. Calling a broken internal admin theme Sev-1 paged 22 people at 02:14. The cost is not the page itself; it is that the next Sev-1 gets triaged slower because the last three were noise.
Deflation. A 14% checkout 5xx the previous afternoon sat as Sev-3 because it "looked like a client bug". Under the matrix that is the revenue path, 5–50% band, hard down → Sev-1. It ran for hours at the wrong severity.
Both come from classifying by who noticed and how bad it felt, not by impact. Forbid upgrading severity to "get more people" without a new impact fact, if you need more responders, page them; do not relabel the incident.
Evidence it is working
Re-score the last 15 incidents against the matrix and report the mismatch rate. Target under 10%. If the mismatch is concentrated in one direction, the matrix is not the problem: the on-call's incentives are.
Severity describes harm. It is not a volume knob for attention.
Curated: · Written: · Reviewed:
Q&A-16Leadership wants MTTR under 15 minutes. What do you split out first?
The useful framing here is to ask what evidence would change my mind about MTTD versus MTTR.
Split detect time from repair time, because a 15-minute MTTR target is a lie if 40 minutes of that incident were spent noticing it.
Concretely, stamp detect, diagnose, mitigate, and recover on every Sev-1/2, report MTTD and MTTR separately, and fund detection work when MTTD dominates.
The reason for that specificity is a failure I have seen: A disk-full outage took 14.2 minutes to page on a 5-minute scrape interval with a 3-for-3 alert, 8.4 minutes to diagnose, and 6.1 minutes to mitigate; the postmortem still quoted "MTTR 28.7 minutes" as if repair were the whole problem.
Ten Sev-1s: detection, not repair, owns the clock.
| Phase | Median minutes | What moves it |
|---|---|---|
| MTTD (start to page) | 14.2 | SLI paging, not host paging |
| Diagnose | 8.4 | dashboards, traces |
| Mitigate | 6.1 | rollback, feature flag |
| "MTTR" as commonly quoted | 28.7 | hides that 14.2 min was noticing |
I would not consider it settled without evidence: For the last 10 Sev-1s, show MTTD and MTTR as stacked bars and fund the larger slice.
You cannot repair a failure you have not yet admitted is happening.
Curated: · Written: · Reviewed:
Q&A-19During an outage, Slack fills with hypotheses and the status page stays on "investigating" for 70 minutes. What is the comms contract?
My approach to incident communication starts from the constraint rather than from the technique.
Send time-boxed updates on a stated cadence with impact, current action, and next update time, even when the cause is still unknown.
Concretely, comms lead posts internally every 10 minutes and externally every 20 minutes for Sev-1, using a template that forbids silence and forbids guessing a root cause.
The reason for that specificity is a failure I have seen: A 70-minute "investigating" status page next to a 40% error rate produced a 1,200-ticket flood and a journalist story before engineering had a mitigation.
Sev-1 external update template, minute 20.
Status: identified
Impact: 38% of checkouts returning HTTP 503 since 14:12 UTC
What we are doing: rolling back deploy checkout@2026.09.07.1311
Next update: 14:52 UTC (20 min)
Cause: not confirmed; we will not speculate here
I would not consider it settled without evidence: For each Sev-1, check the incident doc for update timestamps at the cadence, and count customer tickets that arrived after the first accurate impact statement.
People will invent a narrative if you do not provide one on a clock.
Curated: · Written: · Reviewed:
Q&A-6Engineering wants a freeze after two Sev-2s, while product wants the launch date. How do you decide?
The 90-second answer
Ask two questions before anyone picks a side: what did the two Sev-2s actually cost in user-facing minutes, and what does the launch date cost if it slips? Decide on the larger number, and make whoever owns the risk sign for it.
Error budget is the currency, but it is not the whole decision. Two Sev-2s that burned 6 minutes against a 43.2-minute monthly budget (99.9% availability over 30 days) leave 37.2 minutes. That alone is not a freeze signal. Two Sev-2s that share a root cause (same dependency, same deploy path, same saturation) is a freeze signal regardless of how few minutes they burned, because the next one is not independent of the last two.
The mechanism
Budget answers "can we absorb another incident?" It does not answer "is the system stable?" or "what does delay cost?" Those are three separate inputs, and the freeze call is a function of all three:
| Signal | Freeze | Ship |
|---|---|---|
| Two Sev-2s, unrelated causes, 6 min burned, 37 min left | ✓ | |
| Two Sev-2s, same unresolved root cause | ✓ | |
| Expected launch blast radius > remaining budget | ✓ | |
| Cost of slip > expected incident cost, risk owner signs | ✓ |
Worked example (hypothetical figures)
Launch touches the same service that just failed. Estimate blast radius: 25% chance of a 60-minute outage → 15 expected minutes. Remaining budget is 37.2, so it fits. Now price the delay: launch worth $30k/day, a 5-day slip costs $150k. Price the risk: an outage minute costs roughly $2k in lost transactions, so 15 expected minutes ≈ $30k. Ship, with a rollback plan and the on-call lead's sign-off.
Change one input: 50% chance of a 90-minute outage → 45 expected minutes, which exceeds the 37.2 remaining. Now the launch spends budget the org does not have. Freeze, or gate the launch behind a canary that cannot touch the failing path.
When a freeze is warranted regardless of budget
- The two Sev-2s share a root cause and the fix is not merged.
- The launch changes the same code path that just failed.
- You cannot roll back the launch in under one deploy cycle.
- The Sev-2s were detected by users, not monitoring: you do not yet know what you do not know.
Failure modes
A freeze triggered by incident count rather than cost teaches the org that freezes are political. The opposite failure is quieter: product ships through a nearly empty budget, the next incident blows the SLO, and the miss gets relabelled a "known risk." Both are detected the same way. Freeze and unfreeze timestamps plotted against remaining budget, plus a deploy log showing whether any feature ship crossed the red band without a signed exception. An exception does not restore budget or erase the SLO miss; it only records who accepted it.
What I would say in the room
"Six minutes burned, 37 left, unrelated causes: I will not block the date. But if these two share a cause, we freeze until the fix lands, and I will trade the date for it. If you want to ship anyway, name the person who signs for the outage."
Curated: · Written: · Reviewed:
Q&A-20A postmortem draft says "the intern ignored the runbook". How do you rewrite it?
I would treat blameless postmortems as a design decision with a stated trade-off rather than a best practice.
Describe the conditions, incentives, and missing guards that made the action reasonable, and never name a person as the cause.
Concretely, require contributing factors in systems language, a timeline of facts, and a review that rejects personal attributions before the doc is marked done.
The reason for that specificity is a failure I have seen: Naming an intern ended the investigation at "training", and the same missing canary check shipped a second bad config six weeks later with a different person at the keyboard.
Rewrite the contributing factor so it survives a staffing change.
| Draft line | Rewrite |
|---|---|
| The intern ignored the runbook. | Production apply was possible without a plan file, and the UI defaulted to the last target. |
| Jane should have paged sooner. | The only page was on CPU, which stayed green while 5xx rose to 18%. |
| Fat-fingered the cluster name. | Two clusters accepted the same context name; kubectl had no confirmation prompt. |
I would not consider it settled without evidence: Scan the last 10 postmortems for personal attributions and for at least one system-level action item per doc.
If the same process with a different human would fail the same way, the human was not the defect.
Curated: · Written: · Reviewed:
Q&A-21A postmortem has 23 action items and three of them are "be more careful". What makes an action item real?
What makes an action item real: it names a contributing factor, an owner, a due date, and a check that would have changed this incident's outcome. "Be more careful" has none of the four: no factor, no owner, no date, no observable state you could test. It is a wish with a verb in it. If someone insists on it, ask what they would have needed to see at the time; the answer is usually a missing gate or a missing signal, and that is the action item.
The list is capped by the number of contributing factors you can actually close, not by how many ideas the room generated. Three items that each remove a factor beat twenty-three that describe the incident again.
The trade-off that decides the shape of each item: there are three credible kinds, and they are not interchangeable.
- Automation: the strongest. The factor is removed structurally and the check is the pipeline itself. Right when the factor is a missing gate, a bad default, or a manual step that can be encoded. Wrong when the fix costs more than the incident's blast radius justifies, or when it would block unrelated work for weeks.
- Detection and alerting: does not prevent the next incident, it shortens it. Right when the factor is "we found out from a customer" and the underlying cause is not yet understood. Be honest that this is mitigation, not a fix, and pair it with a follow-up to find the cause.
- Process or checklist change: the weakest, and "be more careful" is its degenerate case. It is right when the decision is genuinely a judgment call in a novel situation and no automation exists: a schema migration review, an incident commander's call to fail over. Then verification is not a CI gate but a sampled audit or a drill: take two of the last ten such decisions and check the checklist was followed.
When a longer list is right: when the incident has several independent contributing factors, or when the causal analysis is still open and you are buying information. Cap the items that claim to close a factor; a separate "investigate" bucket can be longer, but it is not action items and should not be counted as such.
When a non-verification approach is right: when the fix is a one-off change already applied, the verification is a regression test or a config assertion rather than a repeatable process. When the factor is a person's judgment under time pressure, the honest verification is a drill, not a promise.
Illustrative example: an apply-without-plan outage:
| Item | Owner | Due | Verification |
|---|---|---|---|
| Require terraform plan artifact in CI | platforms | 14d | apply job fails without plan |
| Page on 5xx burn, not CPU | sre | 7d | replay of incident pages at T+3 min |
| "Be more careful in prod" | — | — | rejected |
The failure mode this prevents: 23 items aged into a graveyard, including the one that would have added the plan file, and the next apply-without-plan outage was a copy of the first. Long lists do not fail loudly; they fail by dilution.
Track them where other reliability work lives, default due date 30 days, and report completion at 30 days: re-opening any item whose verification step cannot be demonstrated. An action item without a check is a wish that will lose to the next feature deadline.
Curated: · Written: · Reviewed:
Q&A-22After a year of outages, leadership asks "what keeps breaking". How do you answer without a slide of anecdotes?
First I'd pin down what leadership is actually asking. "What keeps breaking" is a question about where next quarter's engineering time goes, so the unit that matters is customer-facing minutes of impact, not incident count. A four-minute blip and a four-hour outage are not the same event, and counting them equally is how you end up funding whichever team files the most tickets.
Assumptions I'd state out loud: we already record start, end and customer impact per incident; we have at least a year of closed incidents; and someone owns the tracker. If any of those is false, the answer is "fix the tracker first", not a taxonomy.
The answer I'd give: at incident close, require exactly one primary cause class from a fixed list of five to seven. Then report customer-facing minutes by class over the trailing 90 days and fund the top class. On an illustrative quarter it looks like this:
| Class | Incidents | Minutes | Share |
|---|---|---|---|
| Change / deploy | 11 | 412 | 41% |
| Capacity / overload | 4 | 188 | 19% |
| Dependency | 6 | 171 | 17% |
| Config | 5 | 140 | 14% |
| Hardware / zone | 2 | 91 | 9% |
Change and deploy is 41% of the minutes on 11 of 28 incidents. That is the number that gets a project funded, because it points at deploy safety: canaries, progressive rollout, automated rollback. Capacity at 19% points somewhere else entirely, at headroom and load shedding.
Now the alternatives, because a fixed taxonomy is not obviously right.
The main competing scheme is symptom-first: classify by what the customer saw, elevated latency or errors or data loss, rather than by cause. That is better when your consumers are SLOs and you want to know which objective is being violated. It is worse for funding, because "elevated latency" does not tell you which team or which project fixes it.
The third option is contributing-factor tagging: multiple tags per incident, no primary. That is better for postmortem depth, since a change that shipped a bad config into an under-provisioned zone has three true tags. It is worse for prioritization, because minutes get double-counted and every class looks important.
So the deciding criteria: if you need to fund one project, use one primary class. If you need to understand one incident, use contributing factors. They do not conflict. Keep both fields, and report on the primary class.
When a fixed taxonomy is the wrong tool: if you close fewer than roughly 20 incidents a year, the distribution is noise and you should just read the postmortems. If one class already accounts for most of your minutes, you do not need a taxonomy, you need that one fix. And if your incidents are dominated by a single dependency you do not control, the taxonomy will only tell you what you already know.
Failure modes I would watch for. Label inflation, where free text creeps back in and you get 90 unique labels again; detect it by counting distinct labels per quarter and capping the list. Primary-class arguments at close, where the incident commander and the service owner disagree about whether it was change or config; detect it by tracking how often the class is reassigned after close. And survivorship, where the class that burned the most minutes last year is the one you already fixed, so always report the trailing window rather than the all-time total.
I would not call it settled without one more thing: a quarterly histogram of customer-minutes by class, with the top class mapped to a funded project and a named owner. If the histogram has not moved by the next quarter, the taxonomy was decoration.
Curated: · Written: · Reviewed:
Q&A-103An incident is mitigated and the pager is quiet. Walk me through your postmortem loop: when one is required, what the written record contains, and how action items get closed.
A strong candidate separates three things the interviewer is really asking about: the trigger policy, the written record, and the closure loop. Most teams get the first two roughly right and fail the third.
When a postmortem is required. Write the policy down, because "postmortem everything" burns out the on-call and "postmortem nothing" repeats incidents. A workable default:
| Trigger | Required? | Draft due | Review |
|---|---|---|---|
| Sev1 or Sev2 | Yes | 3 business days | 5 business days |
| Any incident that burned >5% of the monthly error budget | Yes | 3 business days | 5 business days |
| Data loss, security, or privacy event | Yes | 3 business days | 5 business days |
| Sev3 with a novel failure mode or a lucky mitigation | Optional | 10 business days | — |
| Single alert, self-resolved, no user impact | No | — | — |
The near-miss row matters. If the mitigation worked because someone remembered a runbook from 2023, that is a postmortem, not a celebration.
What the written record contains. Not a novel. One page plus a timeline. The parts that earn their space:
- Impact, quantified: 12,400 users, 18 minutes of checkout 5xx, 4.1% of the monthly error budget. If you cannot quantify it, say what you cannot measure and why.
- Timeline in UTC with detection, mitigation, and resolution timestamps. The gap between "first bad deploy" and "alert fired" is usually the most actionable number in the document.
- Contributing factors, plural. "Root cause" invites a single culprit and stops the analysis early. An incident is a chain: a config map edit bypassed validation, the canary only checked HTTP 200, the alert threshold was 5% for 5 minutes, and the rollback runbook was stale. Fixing only the edit leaves three of those live.
- What went well. This is not morale padding; it tells you which controls to keep.
- Action items with owner, due date, and a verifiable completion criterion.
Making action items land. The failure mode is a postmortem with 23 items, 19 of which are "add a test" with no owner, and a Confluence page nobody opens again. Three rules fix most of it:
- Action items live in the same backlog as feature work, not in the postmortem doc. If it is not in Jira/Linear, it does not exist.
- Every item has one owner and a due date. "The team" is not an owner.
- Aging is visible. Any item open past 30 days gets raised at the weekly ops review; past 90 days it is either done or explicitly dropped with a written reason. Dropping is allowed; silent rot is not.
A real item: "Add a validating admission webhook that rejects unknown keys in the checkout config map. Owner @priya, due 2026-03-14, verified by a CI test that applies a bad key and expects rejection." A fake item: "Be more careful with config changes."
Failure modes to name. The postmortem becomes a blame document (rewrite the sentence, not the person). The action items pile up and the same contributing factor recurs six months later: detect that by tagging each item with the contributing factor and querying for repeats. The postmortem is written but never reviewed; fix that by putting the review on the calendar with the service owner, not just the on-call.
The loop closes when the next incident's timeline shows the detection gap shrinking and the same factor not recurring. That is the metric to report.
Curated: · Written: · Reviewed: —
Q&A-28p99 latency is 1.8 s and nobody can find a slow request to inspect. What do you attach to the histogram?
What I'd say first
Attach exemplars to the tail buckets of the request latency histogram. An exemplar is a (trace_id, value) pair stored next to the bucket a sample landed in, so the p99 tile links to a real trace instead of a random one. The quantile stays the same number; it just stops being a dead end.
How it works
The client library records the trace id of the request that incremented the bucket and emits it in OpenMetrics exposition. Prometheus stores exemplars separately from the sample stream (2.26+, exemplar storage: confirm your build has it enabled) and serves them from the exemplars API, not from the PromQL result. Grafana renders them as dots on the histogram panel; clicking one opens the trace.
// client_golang, Go 1.21
hist.ObserveWithExemplar(ctx, dur.Seconds(), prometheus.Labels{"trace_id": traceID})
What that buys you, traced end to end:
histogram_quantile(0.99, sum by (le) (rate(checkout_latency_seconds_bucket[5m])))→ 1.86 s- exemplar on the
le="2.5"bucket:trace_id=9f2c41ab77de0031c0ffee4411aa22bb,duration=1.81s - click → the trace shows
payments.waitconsuming 1.79 s of that 1.81 s
Alternatives, and when exemplars are not enough
- Tail-based sampling in the OTel Collector: buffer spans, keep every trace over a latency threshold. Better when you need the whole trace shape rather than one exemplar per bucket. Costs memory proportional to in-flight traces, and the collector becomes a stateful hop.
- Trace id in logs: cheap, and it works once you already have an id from an alert. It does not help you find the slow request in the first place, which is the actual problem here.
- Continuous profiling: when the tail is CPU- or allocation-bound and no single span is slow. The request is slow because of aggregate contention, and a trace will show a flat, unremarkable span tree.
Failure modes
- Head-based sampling at 1%: the exemplar's
trace_idpoints at a trace the backend never kept. Exemplars are only as good as your trace retention; check that the sampling decision for the slow path is not made before the latency is known. - Exemplar storage is a fixed-size circular buffer, so exemplars are evicted under load. If the p99 tile has no dot, that is the first thing to check, not the last.
- One exemplar per bucket boundary: if the slow request lands in
le="2.5"but your exemplar sampling is 1-in-N, you can still miss it. Alert on the bucket, then read the exemplar.
The decision rule: exemplars when you have a percentile and no example; tail sampling when one example is not enough to see the shape; profiling when the trace is flat.
Curated: · Written: · Reviewed:
Q&A-27Someone added user_id as a Prometheus label to "debug faster". What happens, and what do you allow instead?
The 90-second answer
A Prometheus series is one unique combination of metric name and label values. Adding user_id doesn't add one dimension: it multiplies the existing ones. http_requests_total{method, route, code, region} is a few thousand series; the same metric with user_id is that count times the number of distinct users, and every one of those series costs head-block memory, an index entry, and a slot in every query that touches the metric.
So the failure isn't "the dashboard is slow". It's that Prometheus runs out of memory or falls behind its scrape interval, and the alerts you actually depend on (the burn-rate page, the SLO error budget) stop evaluating. You traded a debugging convenience for the monitoring system itself.
What I allow instead: bounded labels on metrics (route, method, status class, region, service, version), user_id as a span attribute in traces and a structured field in logs, and exemplars on histograms so a spike on the metric links straight to the trace that caused it. That last one is usually what "debug faster" actually meant.
Why it blows up
Series count is the product of the cardinalities of the labels you keep. Take a checkout service:
- method: 5
- route: 24
- code: 8
- region: 3
That's 5 × 24 × 8 × 3 = 2,880 series. Healthy.
Now add user_id with 2.1M distinct users in a day. The theoretical ceiling is 2,880 × 2.1M ≈ 6 billion. You never hit the ceiling, because a given user only touches a handful of route/code/region combinations. If the average user touches 6 of them, you land at 2.1M × 6 ≈ 12.6M series.
At roughly 1–2 KB per active series in the head block (rule of thumb, scales with sample rate), 12.6M series is ~19 GB of memory before you count the index and WAL. That's a Prometheus that OOMs on restart, or one whose scrape duration exceeds its 15s interval, so scrapes queue, targets flap to down, and you get gaps exactly when you need the data.
What to allow
| Label | Distinct values / 24h | On metrics? | Where it lives |
|---|---|---|---|
| route (templated) | 24 | yes | Prometheus |
| method | 5 | yes | Prometheus |
| code | 8 | yes | Prometheus |
| region | 3 | yes | Prometheus |
| user_id | 2.1M | no | trace attribute + log field |
Two guardrails make this stick:
- Drop it at scrape time, so a bad client can't poison the TSDB:
scrape_configs:
- job_name: checkout
metric_relabel_configs:
- action: labeldrop
regex: user_id
- Budget series in CI. Estimate the cardinality of any new label and fail the build if the metric would cross its budget (say 100k series per service). Alert on
prometheus_tsdb_head_seriesso you find out before the OOM, not after.
Failure modes and how you see them
- Head-block memory grows until the process is OOMKilled; you lose the in-memory window and restart into a replay.
- Scrape duration exceeds the interval:
up == 0for the target, gaps in graphs,TargetDownfiring. - Query timeouts on any dashboard that aggregates the metric; alert evaluation falls behind, so burn-rate alerts fire late or not at all.
- Remote-write backlog if you're shipping to a long-term store.
The tell is always the same: the metric you added for debugging is the one that takes down the metrics you needed.
Curated: · Written: · Reviewed:
Q&A-30On-call is regexing "timeout" across 14 log formats. What logging contract do you enforce?
The 90-second answer
Enforce one contract: JSON lines with five required fields, plus a registry of event names.
ts: RFC 3339, millisecond precision, UTC.sev: one of a fixed set (debug|info|warn|error).svc: the service name, matching the deploy unit.event: a stable, registered identifier such aspayments_upstream_timeout. Not a sentence.trace_id: required for request-scoped events.
Everything else (http_code, upstream_ms, retry) is optional context. The human-readable message is optional commentary, and nothing parses it.
Enforce it in two places: a shared library with no free-text API at warn and above, and a CI job that validates a sample of lines from each service against that service's schema. Index event and trace_id as first-class fields; leave the rest as a blob.
{"ts":"2026-09-07T14:12:03.441Z","sev":"error","svc":"checkout",
"event":"payments_upstream_timeout","trace_id":"9f2c41ab77de0031c0ffee4411aa22bb",
"span_id":"7a11e4c0ffee1234","http_code":503,"upstream_ms":2104,"retry":2}
On-call then queries event:payments_upstream_timeout and gets every payment hang across all 14 services, whichever one phrased it as "deadline exceeded (2000ms)".
Why JSON and not logfmt
logfmt is cheaper to write and readable in a terminal, but it is flat, and its quoting rules for values containing spaces or = are a recurring source of parser bugs. JSON costs bytes and escaping CPU; you buy nesting and a schema you can validate mechanically. At 5k lines/s, an illustrative 320-byte JSON line is ~1.6 MB/s against ~0.9 MB/s for the equivalent logfmt: real money at ingest pricing, but small next to a missed payment-hang alert.
| Option | Migration cost | Query power | Where it breaks |
|---|---|---|---|
| JSON + library | High: 14 services adopt, version, redeploy | Nested, typed, schema-validated | Library bugs become logging outages if it blocks |
| logfmt + library | Medium | Flat key=value only | Values with spaces or =; no nesting |
| Normalize at the collector | Low: no app change | Whatever the regexes capture | Regexes break on the next format change. The failure you already have |
The cost you are actually paying
A library across 14 services is not free: language coverage (Go, Java, Python, Node), a version bump and redeploy per service, and the review friction of "why can't I just log a string". Budget a quarter-long migration with a per-service checklist, not a one-week flag flip. The collector-side option is the cheap path, and it is the one that keeps failing.
When a lighter contract is enough
If a service is single-tenant, needs no cross-service correlation, and nobody pages on it, logfmt with ts, sev, svc, event is enough: skip trace_id and skip the schema CI job. The contract earns its cost where a query has to span services.
Failure modes
- The library blocks or throws on a schema violation. Logging must never be a failure path: drop the line, increment a counter, keep serving.
- Requiring
trace_ideverywhere forces fake values out of cron jobs and Kafka consumers. Require it only for request-scoped events; generate a root id otherwise. - Free-form
eventstrings explode index cardinality. Keep a registry of event names and fail CI on an unregistered one. - CI validates a sample, so it only catches what the sample covers. Validate at runtime in dev and staging, sample in prod.
Evidence it is working
Query the last 24 hours for lines missing event or trace_id; hold that rate under 0.1%. If it climbs, the library is being bypassed.
Curated: · Written: · Reviewed:
Q&A-31When do you instrument a service with RED, and when do you reach for USE?
RED on the request path, USE on the resources behind it. If a service answers requests, instrument it with RED (rate, errors, duration) because those three are the signals that map to user harm. If a component has a finite capacity that requests queue against (CPU, disk, a connection pool, a thread pool, a queue) instrument it with USE (utilization, saturation, errors) because those are the signals that show a bottleneck forming before it surfaces as latency.
The split is not stylistic. RED is measured at a request boundary and answers "are users being served?" USE is measured on a resource and answers "how much headroom is left?" A service can be clean by USE and failing by RED, and the reverse.
What to export, concretely:
- RED, per user-facing endpoint: request rate, error rate split by class (4xx vs 5xx), and duration as a histogram so you can read p50/p99 rather than a mean.
- USE, per resource: utilization as a fraction of capacity, saturation as queue length or wait time, and errors as the resource's own failure counter. Allocation failures, timeouts, refused connections.
Worked example, illustrative figures. A checkout service and the database pool it owns, same incident:
| Method | Signal | 14:12 | Reading |
|---|---|---|---|
| RED | rate | 1,840 rps | normal for this hour |
| RED | errors | 9.1% 5xx | users are failing |
| RED | duration p99 | 1,920 ms | SLO is 500 ms: burning |
| USE | pool utilization | 0.98 | 98 of 100 connections held |
| USE | pool saturation | 240 waiters | the bottleneck |
RED says users are hurting. USE names the resource. Neither alone gets you to the fix: RED without USE tells you to roll back, USE without RED tells you to add connections to a pool that may be blocked on a slow query.
The failure that argues for both: a fleet of CPUs at 30% utilization, USE-clean, while 9% of requests returned 400 because a client sent a malformed header. No resource was saturated; the users were. A USE-only dashboard is blind to that. The mirror failure is a pool at 0.99 utilization with a healthy error rate: a RED-only dashboard is blind to that until the queue overflows.
PromQL for the RED half, checked against Prometheus 2.x:
sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="checkout"}[5m]))
Alert on that ratio against a burn-rate budget, not on a raw threshold.
When not to use each: RED is meaningless for a batch job or cron worker with no request boundary. Use USE plus a job-completion metric. USE is meaningless for a resource you do not own; put RED on the edge of the dependency instead and let its owner run USE. For a service that is both, do both: RED at the edge, USE on the resources it owns.
RED tells you users are hurting; USE tells you which bottle is about to empty.
Curated: · Written: · Reviewed:
Q&A-26A 5xx spike has a red graph, 40 GB of logs, and no trace ids. How do you wire the three signals?
The short version
Assume the spike is live and the three signals were never joined: metrics have dimensions, logs have text, traces either don't exist on the failing path or aren't sampled. The answer has two halves: what you do in the next ten minutes with the data you already have, and what you wire so the next spike is a lookup instead of a grep.
Right now, without trace ids
You can't join on an id that was never emitted. Join on the dimensions you do have, in this order:
- Bisect the metric. The red graph is an aggregate; the dimensions under it are the hypothesis space. Slice by status code, route, pod/replica, upstream host, client version. The spike is usually one slice, not all of them.
- Pull logs by that dimension, not by timestamp.
replica=pay-7b AND status=503over the incident window is a few thousand lines, not 40 GB. - Force-sample traces if they exist. Turn on 100% head sampling for that route for five minutes, or tail-sample on error status.
Hypothetical arithmetic: 40 GB over 22 minutes is ~30 MB/s. A timestamp grep reads all of it. A dimension-filtered query reads the 3% of traffic that hit the bad replica: ~1.2 GB, and far less on a columnar store.
What you wire so it doesn't recur
Three joins, cheapest first:
- Logs → traces. Propagate W3C trace context on every hop and put
trace_idandspan_idin every structured log line. This is the join that turns "find the request" into a lookup. - Metrics → traces. Exemplars: attach a sampled trace id to the histogram bucket or counter increment, so the burn-rate panel links to a real request from the bad bucket.
- Traces → logs. Reverse index by trace id, so opening a span shows its log lines.
One request, three signals, one id:
metric: checkout_sli_bad{code="503",route="/checkout",replica="pay-7b"} += 1 # 14:12:03.441
trace: 00-9f2c41ab77de0031c0ffee4411aa22bb-7a11e4c0ffee1234-01 # W3C traceparent
log: {"trace_id":"9f2c41ab77de0031c0ffee4411aa22bb","span_id":"7a11e4c0ffee1234",
"msg":"payments_upstream_timeout","ms":2104,"replica":"pay-7b"}
From the burn-rate panel: click the exemplar, open the trace, jump to the log line. Seconds, not a 22-minute grep.
Trade-offs
- Head sampling is cheap but blind: a 1% sample misses a 3% error slice often enough to matter. Tail sampling keeps every error but needs a collector buffering whole traces: memory cost and an extra hop.
- Exemplars are cheap and bounded, one trace id per bucket, but they're a pointer, not a sample; you can't compute error rates from them.
- Trace ids in logs are nearly free. The cost is cardinality if you also index every id as a metric label. Index the field; don't make it a label.
Failure modes
- Context lost at async boundaries. A queue consumer or thread pool that doesn't carry context starts a new trace, orphaning the failing span. Detect: traces with a single span and no parent.
- Sampling drops the failure. If the error path is 0.5% and you head-sample at 1%, you'll see it eventually, not during the incident. Detect: compare sampled error rate against the metric's error rate.
- Log pipeline strips the field. A parser or schema change drops
trace_id. Detect: assert the field exists on a canary log line in CI. - Cardinality explosion. Trace id as a metric label. Detect: watch series count after deploy.
How I'd know it's fixed
From a burn-rate alert, open one exemplar and see the failing span and its log line without a search. If that takes more than a few clicks, the wiring isn't done.
Curated: · Written: · Reviewed:
Q&A-101Checkout p99 jumped from 180 ms to 2.4 s at 14:05 and CPU, memory and error rate are flat. Walk me through how you use traces to find the dependency that is actually slow.
Assumptions first: checkout and its dependencies are instrumented with OpenTelemetry, traces land in a backend you can query, and the p99 jump is real. The same histogram that alerted confirms it. Flat CPU, memory and error rate rule out local compute and crashes. A 13x latency jump with flat CPU is a wait, not work.
Check the sampling before you trust the traces. If you're head-sampling at 1%, the slow requests are almost certainly not in the backend: you're looking at a biased sample of fast ones. Tail sampling keeps the traces you need:
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: slow
type: latency
latency: {threshold_ms: 1000}
- name: errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 1}
Open a slow trace. The waterfall tells you where the time is. A real checkout trace from this kind of incident:
| span | service | duration | self time |
|---|---|---|---|
| POST /checkout | checkout | 2.41 s | 12 ms |
| charge | payment-client | 2.35 s | 2.35 s |
| POST /charge | payment-service | 41 ms | 8 ms |
| INSERT payment | payment-service → postgres | 33 ms | 33 ms |
The checkout span is 2.41 s but its self time is 12 ms: it is waiting. The payment client span is 2.35 s. The payment server span is 41 ms. The gap between client span and server span is 2.31 s. That gap is not the dependency's code. It is the network path, connection pool, DNS, TLS, or a proxy in between. That is the finding.
If instead the payment server span were 2.3 s and the DB span inside it 2.2 s, the dependency's own database is the problem. The client/server span comparison is the discriminator, and it only works if context propagates. OTel carries it in the W3C traceparent header. A new trace ID per hop means propagation is broken and you have two unrelated traces.
Then narrow the gap. 2.31 s on a call that normally takes 5 ms. Check connection pool wait, DNS resolution, TLS handshake. Pool exhaustion often shows as time before the request is sent, invisible unless you instrument pool acquisition: add a span around it.
Failure modes. Clock skew: span durations are computed from timestamps on different hosts. If payment-service's clock is 2 s behind, the server span appears to start before the client span and the gap is nonsense. NTP/chrony discipline is a prerequisite. Sampling bias: if you only tail-sample slow traces, your trace-derived p99 is not the real p99. Use traces for diagnosis, metrics for measurement. High cardinality: user_id and request_id belong in traces, not in a metrics pipeline. Backend drop: at 8k spans/sec the collector drops spans; check its dropped-span metric before concluding the dependency is fine.
Read the waterfall, compare client and server span durations, and the gap tells you whether to fix the network path or the dependency's internals.
Curated: · Written: · Reviewed: —
Q&A-29You cannot keep 8,000 traces per second. Which traces do you keep?
What I'd say first
Assume 8,000 traces/s is the trace rate, not the span rate: at ~20 spans/trace and ~1 KB/span that's ~160 MB/s, ~14 TB/day raw. I'd spend 2-4% of that. Keep every trace carrying a failure signal, keep a uniform baseline so the retained set still has an unbiased denominator, and bound the failure branch so a total outage doesn't become total retention.
Why tail, not head
Head sampling decides at the root span, before the request has an outcome: it cannot know the request will 500 or take 900 ms. Tail sampling buffers the whole trace for decision_wait and decides after the last span, so the decision can depend on status code, duration, or an attribute. The cost is state: the collector holds every in-flight trace, and every span of a trace must reach the same collector.
Config
processors:
tail_sampling:
decision_wait: 30s # default; must exceed p99 trace duration
num_traces: 300000 # default is 50000 — too small here
expected_new_traces_per_sec: 8000
policies:
- name: errors
type: status_code
status_code: {status_codes: [ERROR]}
- name: slo-miss
type: latency
latency: {threshold_ms: 300}
- name: new-version
type: string_attribute
string_attribute: {key: service.version, values: [v2.31.0]}
- name: baseline
type: probabilistic
probabilistic: {sampling_percentage: 1.0}
Policies are OR'd: any match retains the trace. So a rate_limiting policy is a floor for extra traffic, not a cap on the error branch: to bound errors you combine status_code with a probabilistic sub-policy under an and policy.
The arithmetic
- baseline 1% → 80 traces/s
- errors at 0.8% → 64 traces/s
- SLO misses at 2% → 160 traces/s
- total ≈ 300 traces/s ≈ 3.8% of traffic, ~2.4 MB/s
Memory is the constraint people miss: 30 s of decision wait at 8,000 traces/s is 240,000 traces in flight. At ~20 KB/trace that's ~4.8 GB of heap per collector, and the default num_traces: 50000 would evict traces before they're decided: silently. So you shard: N collectors behind a load-balancing exporter with routing_key: traceID (the default) so all spans of a trace land on one instance, each sized for its share.
Failure modes
num_tracestoo small → traces dropped before decision. Watch the processor's dropped-too-early counter underotelcol_processor_tail_sampling_*.- Collector restart → up to
decision_waitof traces lost (240k here). Drain on deploy. - Late spans after the decision → traces retained but incomplete.
decision_waitmust exceed p99 trace duration, not average. - Head sampling alone: at 0.1% you retain 8 traces/s overall, but a failure confined to one route at 0.02% of traffic (1.6 traces/s) yields one retained trace every ~10 minutes. You find out late and have nothing to compare against.
- Metrics computed from the sampled set are biased. Either weight by the sampling probability carried in
tracestate, or compute error rate from an unsampled counter.
How I'd know it works
Inject synthetic errors and compare the count of retained error traces against an unsampled error counter for the same window; they should agree within sampling noise. Then check the dropped-too-early counter is flat and that spans-per-retained-trace matches the unsampled baseline.
Curated: · Written: · Reviewed:
Q&A-35Latency doubled after you raised the worker count. How does Little's law explain that, and what do you change?
Little's law is L = λW: requests in flight equals arrival rate times latency. Worker count is what sets L: with synchronous workers, each one holds exactly one request in flight. If a shared downstream resource pins throughput λ, then W = L/λ, so doubling the workers doubles in-flight requests and doubles latency. Throughput doesn't move; every request just waits longer.
So the fix is not more workers. Find the resource pinning λ (a DB connection pool, a lock, a single-threaded service, a saturated CPU) and either cap concurrency to what that resource can actually serve, or add capacity there.
Why λ stays pinned
The bottleneck is a shared resource with a fixed service rate. Workers queue at it. Adding workers adds queue depth, not service rate. It's worse than linear: as utilization ρ = λ/μ approaches 1, queue wait grows like ρ/(1−ρ) for a single server (M/M/1), so the last few workers cost far more latency than the first.
Worked trace
Bottleneck: a single-threaded downstream, service time S = 10 ms, so μ = 100 rps. There's a backlog, so workers never idle.
| Workers | In-flight L | λ (pinned) | W = L/λ |
|---|---|---|---|
| 10 | 10 | 100 rps | 100 ms |
| 20 | 20 | 100 rps | 200 ms |
Latency doubled, throughput flat. That is the signature of a concurrency-bound system. (If workers are async and each can hold many requests, the concurrency limit plays the role of L, not the worker count: same arithmetic.)
What to change
- Cap concurrency at the bottleneck. If it's serial (one lock, one writer) in-flight at it should be about 1. Size the worker pool and connection pool to the bottleneck's real concurrency, and put a bounded queue in front with backpressure: return 429/503 above the cap instead of letting in-flight grow.
- Add capacity at the bottleneck if it's parallel: more connections, a read replica, sharding, a cache in front. That raises μ, and only then do extra workers raise λ.
- Cut work per request: batch, cache, move work off the hot path. Lower S lowers W directly.
Trade-offs
More workers is right only when the bottleneck has headroom and is parallel; then it raises λ. When the bottleneck is serial, extra workers are pure latency. Backpressure trades some throughput (shed requests) for bounded latency: right when failing fast beats queueing. Adding capacity costs money and may be impossible (single-writer DB), which is why capping concurrency is usually the first move.
Detection
Measure λ, L, and W together. If λ is flat while L climbs, you're concurrency-bound. Watch pool wait time, lock wait time, and run-queue length. p99 rising with flat throughput is the tell.
Curated: · Written: · Reviewed:
Q&A-32Error rate is still 0.02% and you want to know you will run out of capacity next Thursday. What do you alert on?
I would treat saturation signals as a design decision with a stated trade-off rather than a best practice.
Alert on saturation and time-to-exhaustion of finite pools, not only on the errors that appear after the pool is empty.
Concretely, track utilization and queue wait for connections, threads, disk, and quotas; page or ticket when time-to-limit at the current slope is inside the lead time to add capacity.
The reason for that specificity is a failure I have seen: A connection pool hit 100% at 18:42 Friday; the first 5xx page arrived 11 minutes later, after 8,400 checkout attempts queued and timed out.
Time-to-exhaustion on a 500-connection database limit.
limit, used, slope = 500, 400, 4.2 # connections per minute
minutes_left = (limit - used) / slope # 23.8 minutes
lead_time_to_scale = 15 # HPA + pool bounce
# Alert when minutes_left < lead_time_to_scale * 2 -> 23.8 < 30, page.
I would not consider it settled without evidence: In the last capacity incident, show that the saturation signal crossed its threshold before the SLI did, with enough lead time to scale.
Errors are a lagging confirmation that you ignored a full tank gauge.
Curated: · Written: · Reviewed:
Q&A-34Traffic grows 6% a week. How much unused capacity do you keep, and how do you know it is real?
The 90-second answer
"Capacity" has to mean the bottleneck resource, not the fleet. Headroom is a time budget, not a percentage: at 6%/week compounding, 30% headroom is 4.5 weeks of runway, and that is the number you compare against procurement lead time. Keep enough headroom that runway exceeds lead time plus a margin: on the DB primary that usually lands near 30%, on stateless pods near zero because autoscaling covers it. You know it is real by soaking production at the projected peak and watching the bottleneck's saturation metric, not by counting replicas.
Assumptions: 6%/week compounds (1.06^n), the trend holds for a quarter, and the bottleneck is the database primary.
Convert headroom to weeks
runway_weeks = ln(1 + h) / ln(1.06)
| Headroom | Runway |
|---|---|
| 12% | 1.9 wk |
| 30% | 4.5 wk |
| 50% | 7.0 wk |
Doubling time is ln2/ln1.06 = 11.9 weeks; a quarter of growth is 1.06^13 = 2.13x. The rule: runway_weeks ≥ lead_time_weeks + 2. The +2 covers forecast error and the week spent getting budget approved.
Worked example (figures illustrative)
| Resource | Last peak | Saturation point | Headroom | Lead time | Runway |
|---|---|---|---|---|---|
| app pods | 42 of 80 | 80 (HPA max) | 47% | 10 min | 6.6 wk |
| db primary CPU | 88% | 85% (p99 write > 50 ms) | −3 pts | 14 days | 0 |
| kafka partitions | 24 of 48 | 48 | 50% | 1 day | 7.0 wk |
The DB row is the answer to "how do you know it is real." A soak found p99 write latency crossing 50 ms at 85% CPU, and the last peak was already 88%: the headroom is negative. The 47% of idle pods sitting in front of it are decoration.
How you verify
Soak at last peak × 1.3 for a full business cycle: 7 days including the Monday 09:00 spike and the month-end batch, cache-cold, with the real connection pool size. Watch the bottleneck's saturation metric (p99 latency, queue depth, replication lag) not CPU alone. If p99 holds and queue depth stays flat, the headroom is real. If it degrades, your saturation point is lower than the table claims and the headroom is smaller than you thought.
Then alert on runway, not utilization. "db primary runway < 3 weeks" fires while you can still act; "CPU > 80%" fires after you are already late.
Trade-offs
Stateless tiers: autoscale, keep little headroom, HPA adds a pod in 10–30 s. Stateful resources (DB SKU, Kafka partitions, shard count, IP space) have lead times of days to weeks, so pre-provision. Buying headroom costs money, and the alternative to headroom is a re-architecture date. At 6%/week a year is 1.06^52 ≈ 20x, so the plan must name the date you shard or move off the single primary, not just the date you buy a bigger SKU.
Failure modes
- Headroom measured on replica count while the DB is the bottleneck.
- Soak run at average load, missing the daily peak.
- Soak with a warm cache and no batch job, so it passes and production does not.
- Growth concentrated in one tenant; track per-tenant headroom, not aggregate.
- A new feature or a dropped index eats the headroom silently, which is why the alert is on runway.
Curated: · Written: · Reviewed:
Q&A-33A load test at 10k rps passed on empty tables, then production fell over at 2.1k rps. What was the test missing?
The test measured a different system. On 200-row fixtures the whole dataset sits in the buffer pool, so every query is a memory read and the bottleneck is CPU. On 90 million rows the working set exceeds shared_buffers (default 128 MB in PostgreSQL), so the same index lookup becomes a random read from disk. The bottleneck moves from CPU to I/O and connection-pool wait, and 10k rps of memory reads says nothing about 2.1k rps of disk reads.
Four things were missing.
1. Production data volume and skew. Cardinality changes the plan. With statistics gathered on 200 rows the planner picks a nested loop: correct there, catastrophic at 90M rows, where the outer side drives millions of index probes. Fixtures also miss hot-key skew: when traffic concentrates on a small set of rows, uniform random fixtures understate contention on exactly those rows.
2. Cache state. A warm cache hides the miss path entirely. The number that matters is the cold-cache p99 after a deploy, when the cache is empty and every request falls through to the origin.
3. Dependency limits. Mocks don't enforce rate limits, connection pool caps, or quotas. A pool of 20 connections (pgbouncer's default_pool_size) caps DB throughput at 20 / query_latency; at 48 ms per query that is ~417 queries/s, well under 2,100 rps.
4. Saturation signals, not status codes. A test that asserts on HTTP 200 counts passes while the pool queue grows. You need pool wait time, DB CPU, and queue depth.
Illustrative figures: hypothetical, but this is the shape:
| Condition | rps | p99 | First saturation |
|---|---|---|---|
| 200-row fixture, warm cache | 10,000 | 18 ms | none |
| 90M-row catalog, 61% cache hit | 2,100 | 640 ms | DB CPU 96%, pool wait |
| 90M-row, cold cache after deploy | 740 | 2,400 ms | origin overload |
The arithmetic behind row 2: at 2,100 rps with a 61% hit rate, 819 rps reach the database. At 48 ms per query that needs 819 × 0.048 ≈ 39 concurrent connections. The pool has 20. Requests queue, latency compounds, and throughput plateaus at whatever the pool can drain, not at what the generator can send.
Failure modes to name and detect:
- Plan regression at scale: detect with
EXPLAIN (ANALYZE, BUFFERS)on production-sized data; watch for nested loops with high row estimates. - Pool exhaustion: detect with pool wait time and active-vs-idle connection counts; the symptom is p99 rising while CPU stays flat.
- Cache stampede after deploy: detect with an origin QPS spike in the first minutes after a release.
- Stale statistics: detect by comparing
pg_stat_user_tables.last_analyzeagainst write volume.
What a strong candidate adds: the acceptance criterion. Before a peak, the load test's dataset cardinality, cache hit rate, and dependency limits should sit within a stated margin of last week's production, and the pass condition should be a saturation signal (pool wait, DB CPU) rather than a status-code count. A green test against a toy database is not evidence about the database you run.
Curated: · Written: · Reviewed:
Q&A-44Autoscaling is still spinning up pods and latency is already 2 s. What should have rejected work earlier?
I would treat admission control as a design decision with a stated trade-off rather than a best practice.
Admit only the concurrency the service can finish inside the latency SLO, and reject the rest at the edge before queues grow.
Concretely, set a global and per-instance concurrency limit from Little's law using SLO latency and measured service time, return 429 at the load balancer, and scale on rejected QPS plus utilization.
The reason for that specificity is a failure I have seen: Waiting for HPA to add pods at 2 s p99 admitted 14k waiting requests, then the new pods spent 8 minutes draining a queue that admission control would have refused.
Concurrency cap from SLO: 300 ms, 25 ms service time, 40 pods.
slo_s, service_s, pods = 0.300, 0.025, 40
concurrency_per_pod = slo_s / service_s # 12 in-flight
cluster_cap = concurrency_per_pod * pods # 480
max_admit_rps = cluster_cap / slo_s # 1,600 rps
# Offered 3,200 rps -> admit 1,600, 429 the rest, p99 of admitted 240 ms.
I would not consider it settled without evidence: At 2x offered load, show admitted QPS flat at the cap, p99 of admitted work inside SLO, and a 429 rate that matches the overflow.
Scaling is how you raise the cap; admission control is how you survive until it rises.
Curated: · Written: · Reviewed:
Q&A-45HPA on CPU sits at 40% while p99 is 1.4 s. Which signal should scale the service?
Scale on concurrency, in-flight requests per pod, with CPU as a secondary cap. CPU is not just unhelpful here, it is actively harmful: with a 70% CPU target and utilization at 40%, the HPA is not idle, it is scaling down toward minReplicas, which lengthens the queue and pushes p99 further out. (If the target is 40%, it sits still: either way it never adds capacity.)
Why CPU reads 40% while p99 is 1.4 s. CPU utilization is one resource's occupancy, not the service's queue depth. HPA computes it as sum(pod usage) / sum(pod requests). If the bottleneck is a downstream call, a connection pool, lock contention, or GC, cores sit idle in futex and I/O waits while requests pile up. The CPU ratio cannot see the queue.
Little's Law: concurrency = throughput × latency. At constant RPS, when p99 goes 180 ms → 1.4 s, average in-flight requests rise roughly proportionally. Concurrency is the leading indicator; latency is the lagging one. Scale on the leading one.
metrics:
- type: Pods
pods:
metric: {name: checkout_inflight}
target: {type: AverageValue, averageValue: "8"}
- type: Resource
resource:
name: cpu
target: {type: Utilization, averageUtilization: 70}
behavior:
scaleDown:
stabilizationWindowSeconds: 300
Arithmetic (worked example, not a benchmark): HPA computes desired = ceil(current × currentMetric / targetMetric). Three pods at 19 in-flight each: ceil(3 × 19/8) = ceil(7.125) = 8 pods. The default 10% tolerance band means ratios between 0.9 and 1.1 do nothing; 2.375 is far outside it.
Choosing the target. 8 is not a guess: it is the per-pod concurrency at which p99 stays under SLO, measured by ramping load against a single pod. If p99 crosses 300 ms at 12 in-flight, a target of 8 leaves 33% headroom.
| Signal | Use when | Fails when |
|---|---|---|
| in-flight per pod | bottleneck is downstream or queueing | per-request cost varies wildly |
| queue wait / oldest request age | worker pools with a backlog | no queue (sync HTTP) |
| RPS per pod | uniform request cost | heavy-tailed request mix |
| p95 latency (mesh) | nothing better available | lagging → oscillation |
| CPU | compute-bound (serialization, compression) | any wait-bound bottleneck |
Failure modes.
- Scaling on latency directly: feedback loop. Pods added, latency drops, pods removed, latency rises. Fix with the scale-down stabilization window (300 s default) and by scaling on concurrency instead.
- Shared bottleneck: if the real limit is a DB connection pool or a downstream service, adding pods multiplies load on it and makes p99 worse. Check the downstream's saturation before scaling the caller.
- Metrics pipeline outage: the HPA cannot compute a ratio, sets
ScalingActive=False, and stops scaling. Alert on that condition: a silent autoscaler is worse than a wrong one. - Target too high: at target concurrency the pod is already saturated, so p99 blows the SLO before the HPA reacts.
Evidence I would want before calling it settled: a load ramp where in-flight crosses 8 before p99 crosses 300 ms, and replica count tracks it with bounded lag.
Curated: · Written: · Reviewed:
Q&A-57Pods OOM at 600 mi while CPU HPA is happy. Do you raise replicas or raise limits?
Raise the limit: replicas don't shrink a working set. Adding pods divides load, not memory. If each replica's resident set is 780 Mi against a 600 Mi limit, thirty replicas give you thirty OOMKills instead of four.
Why the CPU HPA stays quiet: HPA compares observed CPU against the container's CPU request, not its limit, and only acts when utilization crosses the target plus the tolerance (default 10%). A pod that dies on memory before CPU saturates never moves that needle. Memory is also incompressible: the kernel throttles CPU, but the only lever for memory is the OOM killer, which SIGKILLs the container (exit code 137, Reason: OOMKilled).
The one case where replicas are the right answer: when memory grows per in-flight request (large response buffers, per-connection caches) and each pod only OOMs above some concurrency. Then more replicas lower per-pod concurrency. Tell the two apart by plotting working set against QPS per pod: a flat line that crosses the limit at any QPS is a shape problem; a line that tracks concurrency is a capacity problem.
Worked example (figures illustrative):
| Working set | Limit | QPS/pod | Fix | |
|---|---|---|---|---|
| A | 780 Mi flat | 600 Mi | 4 | raise limit and request |
| B | 300 Mi at 5 QPS → 800 Mi at 25 QPS | 1 Gi | 25 | HPA |
For A, a JVM needing 512 Mi heap + ~96 Mi metaspace + 200 threads × 1 Mi stacks + 64 Mi direct buffers ≈ 870 Mi gets killed at 600 Mi no matter how many copies run. Set the limit to 1 Gi and the request to 900 Mi (or request == limit for Guaranteed QoS), and set -XX:MaxRAMPercentage=75 so the heap tracks the limit instead of a hardcoded -Xmx.
Failure modes to avoid:
- Raising the limit but leaving the request low. The pod becomes Burstable with a wide gap; under node memory pressure the kubelet evicts pods whose usage exceeds their request first. Raise both.
- Memory-based HPA.
autoscaling/v2accepts a memory metric, but memory doesn't fall after GC the way CPU does, so the deployment ratchets up and never scales down. It also targets requests, so wrong requests make the target meaningless. - VPA in Auto mode evicts pods to apply new bounds: that's a restart, and it fights HPA if both target CPU. Run VPA in
Off/recommend mode first, read the recommendation, apply it in a release. - A leak. If working set climbs monotonically across restarts, a bigger limit buys time, not a fix.
Diagnostics: kubectl describe pod for Last State: Terminated / Reason: OOMKilled / Exit Code: 137; container_memory_working_set_bytes from cAdvisor for the peak, since kubectl top shows only the current sample. If working set sits well under the limit but you still get OOMKilled, check page cache: the cgroup limit counts it even though working set excludes inactive file pages.
Curated: · Written: · Reviewed:
Q&A-88App replicas autoscale from 10 to 40 and the database starts refusing connections. What was not scaled?
What was not scaled: the connection budget. Replicas went 10 → 40, but each replica kept its per-pod pool size, so total client demand scaled linearly while the database's max_connections stayed fixed. The autoscaler scaled the thing that was elastic and left the one resource that isn't.
Assume Postgres with max_connections = 500, an HPA going 10 → 40 pods, and a pool of 50 per pod. At 10 pods that is 10 × 50 = 500: already at the ceiling, with nothing left for migrations, pg_dump, or on-call. At 40 pods it is 40 × 50 = 2,000 attempted connections against 500 slots. The first 500 win; the rest get FATAL: sorry, too many clients already and their checkouts fail. The outage moved from CPU to connections.
The fix is a budget, not a bigger pool. Reserve headroom for admin work, then divide what's left by the replica count:
db_max, reserve, pods = 500, 20, 40
pool_per_pod = (db_max - reserve) // pods # 480 // 40 = 12
# Old: 40 * 50 = 2000 attempted, 500 accepted, migrations locked out
# Alert on pool_wait_ms p99 > 20 or db_used > 400
12 per pod × 40 = 480, leaving the 20-connection reserve. Emit pool-wait metrics and fail fast when the pool is empty rather than spawning threads that queue behind a socket you'll never get.
Alternatives, and when each wins.
- PgBouncer in transaction mode. Many client connections multiplex onto few server connections, so 40 pods can hold hundreds of client-side connections against, say, 100 server-side. This is the right answer when per-pod pools are large but most connections are idle between queries. The catch: transaction mode breaks session state.
SET, advisory locks,LISTEN, and session-level prepared statements don't survive across transactions. Use session mode if you need those, and accept far less multiplexing. - A proxy sidecar per pod. Same multiplexing idea, scoped to the pod, no shared hop. Costs a container per pod and a new failure domain.
- Raise
max_connections. Each Postgres backend is a process with its own memory (order of a few MB, plus work_mem per sort/hash), so 2,000 backends is not a knob you turn: it's context-switching and memory pressure. This buys headroom, not a fix. - Read replicas or sharding. Splits the connection demand across instances. Correct when the load is genuinely read-heavy or shardable; overkill for a pool-sizing bug.
When pooling down hurts. A 12-connection pool per pod means at most 12 concurrent queries per replica. If a pod serves bursty traffic with long transactions, requests queue on the pool and p99 latency climbs even though the database is fine. Watch pool_wait_ms; if it rises while db_used sits below the budget, the pool is too small and you need PgBouncer rather than a bigger per-pod number.
How you'd know it's fixed. At max HPA replicas, db_used stays ≤ 80% of db_max and the admin reserve is intact. If it isn't, the budget is still wrong.
Curated: · Written: · Reviewed:
Q&A-36At 2.4x expected QPS the service thrashes and nobody gets an answer. What do you drop first?
First 90 seconds
Ask two things before dropping anything: what is the SLO on the revenue path, and is the 2.4x real user demand or retry amplification? Then drop in this order: retries and duplicate requests, then deferrable background work, then non-critical interactive reads. Checkout, auth and payment stay.
Why 2.4x kills everything
Thrashing is a queueing problem, not a capacity one. Take 200 worker threads at 200 ms of unloaded service time: capacity is 200 / 0.2 = 1,000 rps. Expected load is 400 rps. At 2.4x you offer 960 rps, so utilisation is 0.96: under capacity, but queue wait at that utilisation is already seconds, not milliseconds, and p99 crosses a 1 s client timeout. Clients retry. If 30% retry once, offered load becomes ~1,250 rps against 1,000 rps of capacity: utilisation above 1, the queue grows without bound, and every request times out. Including checkout. Little's Law says in-flight demand at 960 rps with a 1 s timeout is 960 concurrent requests against 200 threads.
So the first thing to drop is the amplification, not the users.
Shedding order
| Order | What | Why it goes first | Signal |
|---|---|---|---|
| 1 | Retries / duplicate requests | Pure amplification; dedupe by idempotency key costs nothing | retry rate, duplicate keys |
| 2 | Background: analytics ingest, batch, webhooks, cache warmers | Deferrable, no user waiting | queue depth, consumer lag |
| 3 | Non-critical reads: search, autocomplete, recommendations | Degrades UX, not revenue | per-class 429 rate |
| 4 | Expensive fan-out on the critical path: personalisation, oversized pages | Keep the endpoint, cut the work | p99 by endpoint |
| — | Checkout, auth, payment | Never shed | success rate vs SLO |
The figures above are illustrative for a 200-thread, 200 ms service; substitute your own.
How to shed
Shed at the edge, before the request enters the process: once you have accepted the connection and parsed the body you have paid most of the cost. Give each class its own concurrency limit and its own worker pool (bulkhead), sized by Little's Law: limit = target throughput × p99 latency. Return 429 with Retry-After for quota-style shedding and 503 for "we are overloaded"; a 429 without Retry-After just makes clients retry immediately.
Failure modes
- Shared pools defeat the classification. Shedding search at the route layer while checkout and search share one 20-connection DB pool means checkout still starves. Separate pools per class, or the table above is fiction.
- Shedding after admission. Rejecting inside the handler still consumes a thread.
- Health-check flapping. If the load balancer's health check hits a saturated endpoint, every instance is marked unhealthy and you get a full outage. Keep health checks on a separate cheap path.
- Autoscaling as the answer. If the bottleneck is a shared DB or downstream, more app instances make it worse.
Retry-Afterignored. Server-side dedupe or a token bucket is the backstop.
Verify
Run an overload test at 3x offered load and show checkout success ≥ SLO while a non-critical class absorbs the shed, with in-flight concurrency per class pinned at its limit rather than growing.
Curated: · Written: · Reviewed:
Q&A-37A 4-second payments blip produced 19 minutes of 5xx. Where did the extra load come from?
The extra load is the system's own recovery machinery, not the blip. A 4-second stall that produces 19 minutes of 5xx is a metastable failure: the failure generates the load that prevents recovery. Four sources, roughly in order of how often they dominate.
1. Retry amplification. Layered retries multiply. Three layers each retrying up to 3 times is 4^3 = 64× worst case. Under independent 20% failures the expected value is only ~1.9× (each layer: 1 + 0.2 + 0.04 + 0.008 = 1.248; cubed = 1.94). But a blip is a correlated failure, every client fails at once, so you sit near the correlated case, 1.6^3 ≈ 4.1×, and jitter-free retries arrive in synchronized waves.
2. Queue drain. Requests already accepted must still be served. Hypothetically, a service at 10k QPS that stalls 4s has 40k requests queued. It drains at 10k QPS only if arrivals stop: they don't. Add 1.5× retry load and arrivals are 15k against 10k capacity: the queue grows 5k/s. Over 19 minutes that is ~5.7M queued requests. The queue never drains because the retries are what is filling it.
3. Thundering herd on reconnect. Connection pools re-establish, TLS handshakes and DNS re-resolve, health checks flip and load balancers remove and re-add instances, and every client that backed off retries at the same instant. Each reconnect costs more than a warm request.
4. Autoscaling lag. Scale-up takes minutes and new instances start cold (empty caches, no JIT warmup), so they send more load downstream per request than the instances they replace. If the bottleneck is a downstream dependency rather than CPU, scaling the caller does not help at all.
Telling them apart. Offered QPS ≫ served QPS with a retry-rate spike points at retries. Monotonically climbing queue depth points at drain. A connection/handshake spike points at herd. Instance count flat then stepping, with cold-start latency, points at autoscale lag. Pool wait time rising to timeout then retries points at pool collapse.
| Source | Signature | Metric |
|---|---|---|
| Retry amplification | offered ≫ served | retry rate per layer |
| Queue drain | depth grows monotonically | queue depth, wait time |
| Thundering herd | connection/handshake spike | pool, TLS, DNS QPS |
| Autoscale lag | flat then step, cold | ASG events, cold-start p99 |
| Pool collapse | wait → timeout → retry | pool wait time |
What to do, and when not to. A retry budget (retries ≤ 10% of successful requests, or a token bucket) caps amplification and is the right default. Jittered backoff alone desynchronizes the herd but does not cap total offered load: it fixes source 3, not source 1. Shed-and-fail-fast is right when the work is stale by the time it would succeed (a payment authorization that has already timed out); wrong when the client can wait and the work is valuable. Breaker-only stops new calls but the first wave still lands, and the half-open probe can re-trigger the storm. Budgets are the wrong fix when the retry is the only path to success, a single flaky dependency with no alternative, because then you want to spend the budget, not cap it. Watch per-instance budgets too: 100 instances each with a 10% budget is 10× fleet-wide.
Evidence. In staging, fail 20% of a dependency for 30s and assert offered load stays under 2× and recovery under 60s. Alert on retry rate as a first-class metric, on queue-depth growth rate, and on the offered-to-served ratio.
Curated: · Written: · Reviewed:
Q&A-38After a 10-second outage, all clients retry at t=10, t=20, t=40 and knock the service over again. What is missing?
What's missing is decorrelation, and a server-side ceiling on retries.
Assumption I'd state first: the clients already do exponential backoff. 10 s, 20 s, 40 s is a doubling schedule. The growth isn't the problem. The delay is a pure function of the attempt number, so 2.1 million clients compute the same number and fire in the same millisecond. Exponential backoff without jitter is a synchronized schedule, not a spread one.
Jitter is the fix, and which jitter matters.
| Strategy | Delay for attempt n | Window width | Peak rate, 2.1M clients, base 10 s |
|---|---|---|---|
| None | 10·2ⁿ | 0 | 2.1M/s |
| Equal jitter | U(10·2ⁿ/2, 10·2ⁿ) | 5·2ⁿ | ~420k/s |
| Full jitter | U(0, 10·2ⁿ) | 10·2ⁿ | ~210k/s |
| Decorrelated | min(cap, U(base, prev·3)) | grows with prev | ~210k/s, no power-of-two clustering |
Full jitter halves the peak against equal jitter at the same base, because it spreads over twice the window. That is the whole argument for it.
import random
def sleep_seconds(attempt, base=10.0, cap=32.0, prev=None, strategy="full"):
b = min(cap, base * 2 ** attempt)
if strategy == "full":
return random.uniform(0, b)
if strategy == "equal":
return random.uniform(b / 2, b)
if strategy == "decorrelated":
return min(cap, random.uniform(base, prev * 3))
Trace: attempt 0 → b = min(32, 10·1) = 10, so full jitter returns a value in [0, 10]. attempt 1 → min(32, 20) = 20 → [0, 20]. attempt 2 → min(32, 40) = 32 → [0, 32]. attempt 3 → min(32, 80) = 32 → [0, 32]. The cap binds at attempt 2 and every later attempt draws from the same [0, 32]. Decorrelated needs prev; on the first call pass prev=base, giving [10, 30].
When jitter alone is not enough.
- Once the cap binds, every client draws from the same [0, cap] window again. If the outage outlasts the cap, the herd re-forms at cap intervals: jitter spreads each wave but does not stop the waves.
- If the fleet is large enough that even a spread window exceeds capacity, spreading only makes the overload last longer. 2.1M clients over 32 s is still 65k/s.
- If the retry trigger is a shared external event (a DNS TTL expiring, a config push, a deploy) every client starts its backoff at the same instant, and jitter decorrelates only within that wave.
- If the server cannot shed load, retries queue and amplify.
Server-side controls, the other half.
Retry-After(RFC 9110 §10.2.3): return 503 withRetry-Afterand clients that honor it stop guessing. It is the only mechanism that lets the server set the pace.- Retry budget: allow retries only while they stay under a fraction of recent successful requests, 10% is a common starting point, so a storm cannot exceed the budget however many clients misbehave.
- Load shedding: fail fast with 503 rather than queueing, so the healthy path stays healthy.
- Client-side circuit breaker: stop retrying entirely after a failure threshold.
Detection. Plot retry arrival timestamps after an induced 30 s outage. A spike at the powers of two means no jitter; a flat distribution that still exceeds capacity means jitter works but the budget is missing. Alert on retry rate as a fraction of request rate, not on absolute error count.
Curated: · Written: · Reviewed:
Q&A-39When should a client stop calling a sick dependency, and when is that the wrong tool?
Short answer. Stop calling when the failures are the dependency's, they're sustained, and your own retries and timeouts are part of the problem. It's the wrong tool when the errors are yours (bad payload, expired credential), when the dependency is slow but still succeeding, when you have no fallback so failing fast converts a degraded dependency into a hard outage, or when the failure is one host or shard that routing should already handle.
What a breaker actually decides. A local admission decision: given what I've seen from this dependency over the last N calls, is it worth spending a request on it? It is not a health check, not a cluster-wide verdict, and not a way to protect the dependency.
Scope it per caller, per dependency, per endpoint. A breaker shared across clients is the classic own-goal: one client sending 0.01% of traffic trips it and black-holes checkout for everyone.
Trip on a rolling error ratio with a minimum volume, not a raw error count. Count only 5xx and timeouts: a 4xx is your bug and shouldn't open the breaker for the whole client.
# Python 3.12
from collections import deque
class Breaker:
def __init__(self, window=20, min_calls=20, trip=0.5, open_s=10):
self.window, self.min_calls, self.trip, self.open_s = window, min_calls, trip, open_s
self.results = deque(maxlen=window) # True = failure
self.opened_at = None
def record(self, failed, now):
self.results.append(failed)
if self.opened_at is None and self._trips():
self.opened_at = now
def _trips(self):
if len(self.results) < self.min_calls:
return False
return sum(self.results) / len(self.results) >= self.trip
Trace it. 20 calls, 10 failures: 10/20 = 0.5, which is >= 0.5, so it trips. 20 calls, 5 failures: 0.25, stays closed. 3 calls, 3 failures: len 3 < min_calls 20, stays closed, which is the point, a caller doing 0.1 rps can't trip on three unlucky calls.
Size the window against the open period. At 1 rps a 20-call window is 20 seconds of history. With a 10 s open period the half-open probe fires while the window still holds pre-open failures, so one failed probe re-trips instantly and the breaker never recovers. Either clear the window when you open, or size the window in time and make it longer than the open period.
| State | Condition | Caller behaviour |
|---|---|---|
| closed | < 20 samples, or failures < 50% | call normally |
| open | >= 50% failures over >= 20 samples | fail fast, no socket |
| half-open | open_s elapsed | one probe; success closes, failure re-opens |
When it's the wrong tool.
- Slow but succeeding. A breaker keyed on errors never trips on a 3 s p99. That's a timeout plus concurrency-limit problem (bulkhead), not a breaker problem.
- No fallback. If the dependency is the only path, failing fast turns partial degradation into a full outage. You need a stale read or a degraded response first.
- Non-idempotent writes. Dropping a payment capture because the breaker is open loses money. Use an outbox and retry later.
- Your own errors. Bad credentials or a malformed request trip the breaker for every caller on that client. Classify 4xx separately.
- One bad host. If one of ten instances is dead, the load balancer's health check should route around it; a per-service breaker takes out the nine healthy ones too.
- Protecting the dependency. A breaker protects the caller. To stop a retry storm hammering a struggling service, that's server-side load shedding or adaptive concurrency limits.
Failure modes to watch. Breaker opens and your own endpoint's error rate spikes in the same second: that's the no-fallback case. Breaker never closes: window longer than the open period, or the probe path shares the same saturated connection pool. Breaker flaps: threshold sits too close to the dependency's normal error rate; set it above the steady-state baseline.
How I'd validate. Take one caller, fail its dependency path at 80% for 60 s, and confirm that caller sheds (fast 503s, no sockets held) while other callers' success rate is unchanged. Restore, and confirm the half-open probe closes the breaker within one open period.
Curated: · Written: · Reviewed:
Q&A-40Recommendations are down. Should checkout go down with them? How do you degrade?
No. Recs is presentation; checkout is money. Take the recs hit, keep checkout up, but only for dependencies you've actually classified.
The first 60 seconds: the decision isn't "is recs important?" It's "does this dependency's response change the correctness of the order or the money moved?" If it only changes what the user sees, it's optional and must fail open. If it changes the amount charged, whether stock exists, or whether payment is authorized, it's required and checkout should fail rather than lie. Recs is the first kind. So: empty module, checkout continues.
Mechanism. Three things make fail-open real, and skipping any one turns the optional dependency back into a single point of failure:
- A per-call deadline carved out of the page budget, not a fixed number copied from the recs team's SLA. If the checkout page budget is 400 ms, recs gets ~80 ms.
- A bulkhead, a separate thread/connection pool for recs, so a hung recs call can't consume the pool checkout needs.
- A fallback that is a tested code path, plus a metric when it fires.
# Python 3.11
import concurrent.futures
RECS_BUDGET_S = 0.08 # 80 ms slice of the 400 ms checkout page budget
def recs_or_empty(user_id, recs_pool):
fut = recs_pool.submit(recs_client.fetch, user_id)
try:
return fut.result(timeout=RECS_BUDGET_S)
except concurrent.futures.TimeoutError:
fut.cancel() # best effort: only cancels if not yet running
metrics.incr("recs.fallback.empty")
return [] # fail open: render the module empty
Trace: recs p99 is 2.2 s. The future raises TimeoutError at 80 ms, we return [], the template renders an empty module, checkout proceeds. fut.cancel() does nothing once the call is running, so recs_client must set its own connect/read timeout (say 200 ms) or the worker thread stays occupied: the bulkhead pool is what keeps that from mattering.
Classification table.
| Dependency | Changes order/money correctness? | On timeout | Checkout |
|---|---|---|---|
| payment auth | yes | fail the request | required |
| tax rate | yes (amount charged) | last cached rate, flag order for re-rate | required, stale ≤ 10 min |
| inventory reserve | yes (oversell) | fail the request | required |
| recs | no | empty module | optional |
| reviews | no | empty module | optional |
Failure modes, and how you'd see them.
- Pool starvation. Recs hangs, checkout threads block on the same pool. Detect: pool queue depth and saturation alerts. Fix: bulkhead.
- Untested fallback. The empty path throws because the template assumes a non-empty list: checkout 500s anyway. Detect: a canary that forces recs failure on every deploy.
- Unbounded staleness. Tax cached with no expiry charges yesterday's rate. Detect: age-of-cached-value metric with an alert at the bound.
- Retry storm. The client retries recs 3× against a dying service. Optional deps get zero retries.
- Silent permanent degradation. Checkout is green, recs has been empty for six hours. Alert on fallback rate, not just on the checkout SLO.
When not to degrade. If recs is the product, a discovery feed, then recs down is the outage, and that page should fail loudly. The rule is per-surface, not per-service.
Evidence. Kill recs in staging under load: checkout success and p99 inside SLO, recs.degraded=1. Then a prod canary. An optional dependency you've never failed on purpose is a required one you haven't met yet.
Curated: · Written: · Reviewed:
Q&A-78The payments call has no deadline and checkout threads pile up. How do you set timeouts end to end?
The engineering question underneath API timeouts is which failure is acceptable.
Give every outbound call a deadline smaller than the caller's remaining budget, and fail when it expires rather than waiting forever.
Concretely, start with the user SLO, subtract a margin, budget each hop, set client timeouts and server-side deadlines to that, and propagate remaining time.
The reason for that specificity is a failure I have seen: A 30-second default HTTP client timeout on a 300 ms SLO held 900 threads during a payments hang, then the load balancer 504'd everyone else.
300 ms checkout SLO, hop budgets, 50 ms margin.
user SLO p99: 300 ms
margin: 50 ms
auth: 40 ms deadline
catalog: 60 ms
payments: 120 ms # was 30,000 ms default
tax (cached): 30 ms
# Hang payments: caller 120 ms -> 503, threads +4% not +900.
I would not consider it settled without evidence: Hang the dependency in staging and show the caller returns inside the budget with a 503/504, and that thread count stays bounded.
An unset timeout is a promise to wait longer than the user will.
Curated: · Written: · Reviewed:
Q&A-77A partner retries failed webhooks every second with no cap. How do you protect the service and still accept the event once?
Before choosing an implementation I would establish what success looks like for webhook retry storms.
Authenticate, accept quickly with an idempotency key, and rate-limit per partner, or their retry policy becomes your outage.
Concretely, return 2xx only after the event is durably queued, dedupe on partner event id, apply a per-partner QPS cap, and publish a Retry-After when shedding.
The reason for that specificity is a failure I have seen: A 4-second 500 from our side caused a partner to retry at 1 Hz from 60 workers; 60 QPS of the same order id saturated the payments pool for 17 minutes.
Accept in < 50 ms, dedupe, cap 20 rps per partner.
| Step | Cost | Duplicate 10,000 | Over cap |
|---|---|---|---|
| auth + enqueue | 18 ms | 1 job | 429 + Retry-After: 5 |
| charge inline then 200 (old) | 210 ms | 60 QPS storms | pool 100% 17 min |
| partner workers | 60 at 1 Hz | same event id | our incident |
I would not consider it settled without evidence: Replay 10,000 duplicate webhooks of one id and show one queued job; then exceed the partner cap and show 429 with Retry-After.
A webhook endpoint that does expensive work before 200 is volunteering to run the partner's retry loop on your CPU.
Curated: · Written: · Reviewed:
Q&A-79A single API key is 40% of QPS. How do you rate-limit without becoming a Sev-1 for everyone else?
My approach to rate limiting starts from the constraint rather than from the technique.
Enforce per-principal limits at the edge, isolate noisy neighbors, and keep a reserved slice for anonymous or new keys so one key cannot consume the admission cap.
Concretely, token-bucket per API key and per IP, a global reserved band for the rest, 429 with Retry-After, and dashboards that show shed per principal.
The reason for that specificity is a failure I have seen: A global 1,600 rps cap with no per-key split let one integration eat 1,520 rps of retries and 429 everyone else, including checkout.
Edge buckets: per-key 200 rps, reserved 400 rps for others.
| Principal | Offered | Admitted | 429 |
|---|---|---|---|
| key_9f2c (broken retries) | 1,520 rps | 200 | 1,320 |
| other keys combined | 900 rps | 900 | 0 |
| global cap only (old) | 2,420 rps | 1,600 mixed | checkout in the 429s |
I would not consider it settled without evidence: Run one key at 10x its contract and show that key 429'd while other keys' success rates stay inside SLO.
A single shared bucket is how one broken client becomes a site-wide outage.
Curated: · Written: · Reviewed:
Q&A-74One bad payload retries forever and blocks the partition. How do you isolate it?
What I'd say first
Two questions decide the design: is the payload permanently bad, or is the dependency down? And does this stream need per-key ordering? If the payload is permanently bad, no retry budget saves it: you need a bounded budget and somewhere for it to go. If the dependency is down, every record fails and a dead-letter queue just fills with good data; that's a circuit-breaker problem, not a poison-message problem.
Assuming a permanently bad payload on an ordered partition: retry N times with backoff, then publish to a dead-letter topic and commit the offset so the partition advances. N is small, 3 to 5, because the failure is deterministic; extra attempts only lengthen the stall.
Why one record blocks everything
A Kafka partition is a single ordered log and a consumer commits offsets in order. A record that fails and is retried in place never advances the offset, so every record behind it waits. Same shape in SQS FIFO within a message group, or any single-threaded consumer that retries before acking. The blast radius is the partition, not the record.
The options and what each costs
| Option | Unblocks partition | Ordering for that key | At-least-once for that record | When to use |
|---|---|---|---|---|
| Bounded retry + DLQ | yes | broken (record skipped) | broken until replayed | payload is droppable or replayable |
| Quarantine topic, manual replay | yes | broken until replay | preserved if replayed | you need the payload for forensics |
| Per-key retry (partition by key) | only that key | preserved | preserved | poison is isolated to one key, and that key isn't hot |
| Block with timeout, then DLQ | after timeout | preserved until timeout | broken after timeout | ordering is contractual, e.g. a ledger |
| Circuit breaker on the consumer | yes (pauses) | preserved | preserved | the failure is the dependency, not the payload |
The honest cost of the DLQ path: the skipped record is no longer in order for its key, and it is no longer at-least-once until someone replays it. If the record is a payment or a state transition, that's a correctness break, not an availability win: you either block with a timeout or replay it back into the same partition and accept the stall. If it's telemetry, dropping is fine.
Failure modes
- DLQ as graveyard: depth grows, nobody drains it. Page on DLQ growth rate and require a named owner per topic, not just a depth threshold.
- Replay storm: replaying the whole DLQ re-poisons the partition. Replay one record at a time, and only after the fix ships.
- Misdiagnosed outage: if the DLQ fills with 40,000 records in a minute, the payload isn't bad. The dependency is. Alert on fill rate to tell the two apart.
- Hot key: per-key retry doesn't help when the poison key is the one carrying most of the traffic.
Worked example
A null-price order retried 14,000 times on one partition, stalling 40,000 good orders for 3.1 hours. With a budget of 5:
| Step | Partition lag | DLQ depth | Page |
|---|---|---|---|
| poison arrives | 0 | 0 | no |
| attempts 1–5 | +1 | 0 | no |
| attempt 6 → DLQ, offset committed | 0 | 1 | yes |
| 40,000 good orders drain | 0 | 1 | — |
| infinite retry (old behaviour) | 40,000 | 0 | 3.1 h stall |
How I'd verify
Inject a permanently failing payload and confirm partition lag stays flat, DLQ depth goes to 1, and the page fires. Then inject a dependency outage and confirm the circuit breaker trips instead of the DLQ filling: that's the case the DLQ path gets wrong.
Curated: · Written: · Reviewed:
Q&A-73Producers keep publishing while consumers lag by 2.4 million messages. What backpressure do you implement?
Clarify first: what does the consumer do with a message, and can the producer be throttled? Those two answers pick the mechanism. If the consumer is last-value state (a cache, a dashboard), the 2.4M backlog is worthless and you skip to the head. If it is an audit ledger, you cannot drop it and you archive it. If producers are internal services that can retry, you throttle them.
The arithmetic decides the shape of the answer. At 80 msg/s aggregate drain, 2.4M messages is 30,000 s ≈ 8.3 h. Producers at 200 msg/s add 120 msg/s net, so lag grows ~432k/hour. Clearing the backlog inside a 30 s freshness SLO would need 2.4e6/30 = 80,000 msg/s: 1000× the current drain. So the SLO cannot be met for the existing backlog; it can only be met for new messages. That splits the problem: stop the growth, then decide the backlog's fate.
LAG_MSGS, CONSUMERS, PER_CONSUMER_RPS, PRODUCE_RPS, SLO_LAG_S = 2_400_000, 8, 10, 200, 30
drain_rps = CONSUMERS * PER_CONSUMER_RPS # 80
lag_s = LAG_MSGS / drain_rps # 30000.0
net_rps = PRODUCE_RPS - drain_rps # 120
needed_rps_for_slo = LAG_MSGS / SLO_LAG_S # 80000.0
Trace: drain is 8 × 10 = 80 msg/s; 2.4e6 / 80 = 30,000.0 s; 200 − 80 = 120 msg/s net growth; 2.4e6 / 30 = 80,000.0 msg/s to clear the backlog inside the SLO.
What I implement, in order:
- Throttle producers on lag-in-seconds, not depth.
lag_seconds = lag_msgs / drain_rps; above 30 s return 429 (or pause the partition) withRetry-After. Depth alone is meaningless: 2.4M is 8 h at 80 rps and 40 min at 1000 rps. - Scale consumers only as far as partitions allow. 12 partitions × 10 rps = 120 rps, still under 200, so scaling alone never closes it; you would need higher per-consumer throughput (bigger batches, in-consumer parallelism) or more partitions, which forces a rebalance.
- Decide the backlog: skip to head for last-value topics, archive to object storage and replay offline for ledgers, shed by priority when one topic mixes debug telemetry with billing events.
When pushback is the wrong answer: the producer is a third party you cannot throttle (throttling just moves the failure into their SLO); the backlog is the system of record and must be processed in order; the lag is a one-off burst that drains inside the SLO on its own; or the consumer is simply broken. Throttling producers hides the bug and burns the freshness SLO anyway.
Failure modes and how I would see them:
- 429 → immediate producer retry → amplification. Watch producer retry rate and 429 rate together; if retries track 429s 1:1, add jittered backoff on the producer side.
- Scale-up triggers a consumer-group rebalance that stalls consumption for 30–60 s, spiking lag. Watch rebalances per hour.
- Autoscaler oscillation: scale up, drain, scale down, lag sawtooths. Watch scale events against lag.
- Silent priority shedding breaks downstream reconciliation. Watch the reconciliation mismatch count.
Evidence before calling it done: a consumer-pause test where producers receive 429 before lag_seconds crosses 30, and a drain trace showing lag falling once producers are throttled.
Curated: · Written: · Reviewed:
Q&A-72A popular key expires and 4,000 requests miss together. How do you stop the stampede?
Coalesce the misses so one request refills the key, hand the other 3,999 either the shared in-flight result or the stale value, and make sure the same expiry does not repeat every TTL. The target is easy to state and easy to measure: at expiry the origin sees 1 request, not 4,000.
Pick the defence from the shape of the problem
| Technique | Mechanism | Do callers wait? | Origin calls at expiry | Fails when |
|---|---|---|---|---|
Request coalescing (singleflight, per-key mutex) | first request fetches, the rest share its result | Yes, for one origin round trip | 1 per process | Origin is slow or flaky - you have put 4,000 callers on its critical path |
| Stale-while-revalidate | serve the expired value, refresh in the background | No | 1 per process | The value must not be stale (auth decisions, a price you quote) |
| Probabilistic early refresh (XFetch) | each request refetches with rising probability; trigger when now - beta * delta * ln(rand()) >= expiry, delta = expected fetch time | Only the request that triggers | A few per TTL | Traffic is low, nothing crosses the threshold, and you are back to a plain expiry |
| TTL jitter | ttl = 60 * uniform(0.9, 1.1) | No | Unchanged | One hot key - 4,000 requests still arrive in the same millisecond |
The deciding question is which failure shape you have. A warm-up planting 50,000 keys with the same TTL is fixed by jitter alone and not helped by coalescing, because those are 50,000 distinct keys. One popular key taking thousands of requests per second is the reverse: jitter does nothing, and you need coalescing, stale serving, or early refresh.
Between those three: if stale is acceptable and the origin is slow, serve stale - it is the only option where the caller path never touches the origin. If the value must be fresh and the origin is fast and healthy, coalesce and block. If you can neither block nor serve stale, refresh early so the fetch lands before the deadline, and pair it with coalescing since it is probabilistic and a few requests can trigger at once.
The arithmetic
Hot key, TTL 60 s, 4,000 req/s, origin p99 300 ms, origin pool 20 connections. Hypothetical, but the arithmetic is the point:
- No defence: at t=60.000 the key expires and 4,000 concurrent requests miss. The pool takes 20 at a time, so 4,000 / 20 = 200 waves * 300 ms = 60 s to drain. Every client with a 1 s budget 5xx's, and the key expires again while the queue is still draining - a self-sustaining outage.
- Coalescing: one caller fetches, 3,999 attach to it. Origin calls: 1. All 4,000 pay 300 ms instead of 1 ms, once per TTL.
- Stale-while-revalidate: 4,000 callers get the stale value in ~1 ms, one background fetch runs. Staleness is bounded by the fetch, so give the value a soft TTL of 60 s and a hard TTL of 300 s; past the hard TTL you block rather than serve arbitrary age.
- Early refresh, beta=1, delta=300 ms: the first request crosses the threshold a couple of seconds before expiry, writes a fresh value, and the probability for everyone behind it collapses. Expiry stops being an event.
Where this bites
singleflight is per process. Twenty pods turn 4,000 requests into 20 origin calls, not 1. If the origin cannot absorb 20 concurrent fetches, add a lease (SET lock NX PX 5000) so one pod refills while the others serve stale or give up quickly; the PX expires the lease if the holder dies.
Do not run the shared fetch under the first caller's context. singleflight.Do takes no context: a caller that gives up still waits, and if the fetch inherits that caller's cancellation, one client disconnecting kills the refresh for everyone. Use DoChan with a select on ctx.Done() for the waiting side, and a separate background context with its own timeout for the fetch.
Never cache the failure. A refill that errors and writes an error or a zero TTL turns a blip into a sustained outage; either serve stale through it or cache the error for a couple of seconds at most.
Verify by expiring the hot key under production-shaped load and counting origin requests: 1 per process, p99 unmoved, no 5xx. Repeat with the origin artificially slowed to 2 s - the right behaviour is stale served, not callers queued - and kill the lease holder mid-refresh to confirm recovery.
Curated: · Written: · Reviewed:
Q&A-71Hit rate is 97% and the origin is sized for 3%. Why is that a reliability problem?
I would answer this by separating what cache as a reliability hazard guarantees from what it merely usually does.
Treat cache hit rate as a dependency: size the origin for a miss storm, and make a cache failure degrade rather than multiply load by 1/(1-hit rate).
Concretely, provision origin for a stated miss-rate floor, add request coalescing, and pre-warm a new key generation before a staggered cutover so a deploy does not create a fleet-wide cold miss; page on origin QPS, not only cache process health.
The reason for that specificity is a failure I have seen: A redis restart dropped a 97% hit rate to 0%; origin received 33x traffic and melted in 40 seconds, taking search with it.
Origin load at 97% hit versus a flush.
| State | Hit rate | Origin QPS | User p99 |
|---|---|---|---|
| steady | 97% | 300 | 40 ms |
| sized for | 70% floor | 3,000 | 90 ms |
| redis restart | 0% | 10,000 (33x) | timeout |
| coalesced + shed | 0% | 3,000 cap | 429 on overflow |
I would not consider it settled without evidence: Fail the cache in staging and show origin QPS, shed behaviour, and user SLI; the origin must survive the floor miss rate.
A 97% hit rate is a 33x landmine unless the origin was bought for the other 3% becoming 100%.
Curated: · Written: · Reviewed:
Q&A-86After a fleet restart, every pod stamps the origin at once. How do you stagger the return to traffic?
Don't let the fleet return at once. Admit pods to the load balancer on a jittered ramp, prewarm the hot keys before a pod reports ready, and coalesce concurrent misses so a cold cache costs the origin one fetch per key rather than one per pod.
Restart is worse than failure because it is synchronized. A pod that dies takes its local cache with it, and every pod that comes back has an empty one. If all 80 pods become ready within the same few seconds, the first request for each hot key misses in every pod at once. With 20 hot keys that is 80 × 20 = 1600 origin fetches inside the readiness window, all for keys the origin was serving from cache a minute earlier. The origin sees a step function, not a ramp.
The plan, in the order I'd apply it:
- Batch the rollout.
maxUnavailable: 10%on an 80-pod fleet is 8 pods at a time, not the 50% that caused the incident below. Slower rollout, bounded burst. - Jitter readiness. Each pod waits a random 5–25 s after its containers are up before its readiness probe passes. That spreads the first-request wave across a window instead of a point.
- Prewarm before ready. The pod fetches the 20 hot keys itself, with origin concurrency capped at 4, so it enters the pool warm. Cap the concurrency or the prewarm becomes the stampede.
- Coalesce fills. Single-flight on the cache layer: concurrent misses for the same key collapse into one origin fetch, and the waiters share the result. This is what turns 8 pods × 20 keys into at most 20 fetches per batch.
# Deployment
strategy:
rollingUpdate:
maxUnavailable: 10% # 8 of 80 pods
maxSurge: 0
# Rollout controller knobs (sketch, not a k8s field)
prewarm: {keys: 20, originConcurrency: 4}
readyJitterSec: [5, 25]
A readiness gate (the readinessGates field, where a separate controller sets a Pod condition) only helps if you actually run that controller. If nothing sets the condition, the pod never becomes ready and the rollout stalls. Use it only when the controller exists.
The failure I'm designing against: a rolling restart with maxUnavailable: 50% aligned 40 pods' first requests on the same 20 hot keys. Origin QPS went 18x steady for 70 seconds. That is 40 × 20 = 800 misses, no coalescing, no jitter. The fix above keeps origin under 2x steady and p99 at 210 ms in the same staging test.
Trade-offs: jitter and prewarm lengthen the rollout. 80 pods at 8 per batch with up to 25 s of jitter each is minutes, not seconds, so this is the wrong approach for a service that must recover in under a minute. Coalescing adds latency to the waiters queued behind the first miss. Prewarming costs origin load up front, which is fine when the origin is idle during the restart and wrong when it is already saturated.
Detect it with origin QPS, cache hit ratio, and p99 during the rollout window, not after. Validate in staging: restart 50% of the fleet and require origin QPS under 2x steady with p99 inside SLO before doing it in production.
Curated: · Written: · Reviewed:
Q&A-76A payment event is delivered twice. How does the consumer avoid charging twice?
Assume at-least-once delivery, an event ID that repeats across redeliveries, and a payment provider that supports idempotency keys. Then do two things, not one: dedupe locally on that event ID in a durable inbox, and pass a provider idempotency key derived from the same ID. The inbox stops you re-entering the charge path; the provider key stops a crash between charge and record from becoming a second charge. Either alone leaves a window.
Why both. The inbox row and the charge live in two systems you cannot commit atomically. Dedupe only locally and a crash after the charge returns but before the row is marked done means the retry calls the provider again: without a key, that is a new charge. Use only the provider key and you still cannot dedupe the non-idempotent local effects (ledger row, receipt email, downstream webhook), and you cannot return the original result to the caller.
Alternatives, and when to pick them.
| Approach | Covers | Fails when |
|---|---|---|
| Local inbox keyed on event ID | Any effect, including non-idempotent local writes | The effect is an external call and you crash between call and record |
| Provider idempotency key only | The charge itself | You have other non-idempotent effects, or the provider has no key support |
| Exactly-once via Kafka transactions (read-process-write) | Kafka-to-Kafka, or Kafka-to-transactional-store pipelines | The sink is an HTTP API: the transaction cannot span it |
| Dedupe window (TTL set of seen IDs) | Cheap, bounded memory | A redelivery after the window expires; acceptable only for short-horizon, low-value effects |
Natural idempotency (INSERT ... ON CONFLICT DO NOTHING on event ID) | Row writes | You need to return the original result, or the effect is not a row |
Criteria for choosing: does the effect have a provider-supported key? Is it a local row write or an external call? How long must dedupe hold: forever, or a window? Is the event ID stable across redeliveries? Kafka gives you topic-partition-offset, SQS gives a message ID, but a re-published event with a fresh ID gives you nothing and you need a business key (payment intent ID) instead.
Failure modes.
- Crash between charge and
finish. Detected by a reconciliation job that lists provider charges in the window and left-joins them to ledger rows by idempotency key; a charge with no row is the bug. - Same event ID, different payload. Stripe returns a 400 with error type
idempotency_error; your inbox hash check should raise before you ever call. Alert on it: it means an upstream bug, not a retry. - Provider without key support. Write a
pendingcharge row with a unique constraint on the business key before calling, then on startup query the provider for charges in the last N minutes and reconcile. Slower, and that window is a real limit. - Dedupe window expiry. A redelivery after TTL charges twice; size the window against your broker's maximum redelivery age, or do not use a window for money.
The mechanism, in code.
def consume(event):
with db.tx():
row = inbox.insert_if_absent(event.id, payload_hash=sha256(event.cents))
if row is None: # unique constraint on event.id hit
existing = inbox.get(event.id)
if existing.payload_hash != sha256(event.cents):
raise KeyReuseConflict(event.id)
if existing.status == "done":
return existing.result
result = payments.charge(idempotency_key=event.id, cents=event.cents)
with db.tx():
inbox.finish(event.id, result)
return result
Trace, event evt_7 for 1900 cents, delivered twice:
- Delivery 1:
insert_if_absentinserts apendingrow.chargewith keyevt_7createsch_1.finishwritesch_1. - Delivery 2:
insert_if_absentreturnsNone. Hash matches. Status isdone, so it returnsch_1without calling the provider. One charge.
Crash variant: delivery 1 dies after the charge returns, before finish. The row is still pending. Delivery 2 falls through the done check, calls charge with the same key evt_7, and the provider returns the original ch_1 instead of creating a second charge. One charge.
The test that settles it: kill the process between the charge and the record, replay the event, assert one provider charge and one ledger row. Anything less is a hope.
Curated: · Written: · Reviewed:
Q&A-41A team wants to "run chaos" by killing random pods in production on Friday. What makes an experiment real?
The 90-second answer
A real experiment is falsifiable before it runs. Four things have to exist in writing: a steady-state hypothesis stated as an SLI with a threshold, a blast radius you can say in one sentence, an abort condition that fires without a human, and a control to compare against. "Kill random pods on Friday" has none of them: it is an outage with a name.
For a 12-pod checkout deployment:
| Field | Value | Why it matters |
|---|---|---|
| Hypothesis | Kill 1 of 12 pods in az-a; p99 stays < 300 ms and burn rate stays < 2x | Falsifiable; names the SLI and the threshold |
| Steady state | p99 190 ms, burn 1.0x, measured for 30 min before injection | Without a baseline you cannot attribute the delta |
| Blast radius | 1 pod, az-a, 15 min, max 1 concurrent | Smallest unit that still exercises the failure |
| Abort | 1h burn rate > 6 (0.83% of a 99.95% monthly budget) → stop, no human in the loop | 14.4x is the page threshold; abort well below it |
| Rollback | kubectl rollout restart plus flag revert, exercised before the experiment | An untested rollback is not a rollback |
What a pod kill actually does
kubectl delete podsends SIGTERM and removes the pod from its EndpointSlice concurrently. kube-proxy takes a few hundred ms to reprogram iptables/IPVS, so in-flight requests can still land on a pod that is shutting down. ApreStop: sleep 5closes that window; without it you get 5xx that look like the experiment found a bug.- The remaining 11 pods absorb the traffic. If each holds a local cache, that pod's share becomes cache misses against the database: the p99 spike usually comes from the DB, not from the missing pod.
- The Deployment creates a replacement, and a JVM or a connection pool takes 10–60 s to warm. During that window you run at 11/12 capacity with one cold instance.
Watch the PDB trap: kubectl delete pod, and the pod-kill action in most chaos tools, delete the pod object directly and bypass PodDisruptionBudgets. Only the eviction API honours them. If your PDB says minAvailable: 11 and you delete a pod anyway, you have proven nothing about your disruption budget.
Abort has to be automatic. A human watching a dashboard on a Friday afternoon has 3–5 minutes of detection latency; a burn-rate alert on a 5m/1h window fires in under a minute. Wire the abort to the same rule that pages you.
Alternatives, and when not to
If the service has no SLO and no error budget, there is nothing to abort on: fix that first. If it is stateful (single-writer database, quorum system), pod-kill is the wrong fault: inject latency or a network partition instead, because killing a quorum member tests your recovery procedure, not your steady state. Staging chaos is cheaper but lies: it will not reproduce the config drift, the noisy neighbour, or the real traffic mix. Run the first experiment in staging to validate the abort path, then run it in production at one pod.
Failure modes, and how you'd see them
Retry amplification: client retries across 11 survivors can triple QPS on the database. Watch retry counters, not just error rate. Leader-election storms: etcd or a Raft group re-electing repeatedly. Watch election counters. Connection-pool exhaustion on the survivors: watch pool wait time, not just p99.
The experiment is done only when the report carries the hypothesis, the abort threshold, the observed SLI, and a go/no-go for the next blast radius. If you cannot say what should stay true while you inject pain, you are not experimenting.
Curated: · Written: · Reviewed:
Q&A-42How is a game day different from a chaos experiment, and what do you score?
The 90-second answer. A chaos experiment is an automated test of a system hypothesis: you define steady state, inject a bounded fault, and score whether the system held and whether the blast radius stayed contained. A game day is a rehearsal of people: you inject a failure your runbook claims to cover and score detection, command, comms, and time-to-mitigate. Same fault, different scorecard. A game day can use a chaos experiment as its injector: what makes it a game day is that the humans are the system under test.
| Chaos experiment | Game day | |
|---|---|---|
| Question | Does the system hold? | Do the people and runbooks hold? |
| Hypothesis | Steady state defined, falsifiable | Runbook claims to cover this failure |
| Injection | Automated, scheduled, repeatable | Facilitated, one-off, ~90 min |
| Blast radius | Bounded by design, auto-abort | Bounded by abort criteria and a kill switch |
| Score | Steady state before/after, hypothesis falsified?, abort fired? | MTTD, time-to-commander, update cadence, time-to-mitigate, wrong turns |
| Output | Code or config change | Runbook edits, action items, trained responders |
Worked example. Hypothetical figures, one AZ-loss game day, 90 minutes. Injection: the AZ's egress is blackholed and its nodes are marked unhealthy.
- T+0:00 injection.
- T+4:12 first page fires → MTTD 4.2 min (target < 5).
- T+11:00 incident commander named (target < 5). 6 minutes late.
- T+13:06 traffic drained from the AZ, failover complete → 13.1 min (target < 15).
- T+34:00 first external status update (target 20 min).
- Wrong turns: 2. The team scaled the dead AZ's group before draining it, and re-ran a deploy unrelated to the fault.
| Skill | Target | This game day |
|---|---|---|
| MTTD | < 5 min | 4.2 min |
| Named commander | < 5 min | 11.0 min |
| External update cadence | 20 min | first update at 34 min |
| Mitigate (fail over) | < 15 min | 13.1 min |
| Wrong-turn count | note | 2 (scaled the dead AZ) |
The platform passed, failover took 13 minutes, while the org failed: no commander for 11 minutes and the first customer update at 34. That gap is exactly what a game day exists to expose, and a chaos experiment would have scored it as a clean pass.
Failure modes, and how you catch them.
- The game day becomes a demo. Everyone knows the time and the fault, so MTTD is meaningless. Detect it by comparing game-day MTTD against real-incident MTTD; if the rehearsal is 3× faster, the rehearsal is theatre. Run it unannounced, or inject a decoy fault alongside the announced one.
- No abort criteria. The injected fault cascades past the intended blast radius. Define abort conditions up front (error-budget burn, customer-visible errors, a second AZ degrading) and give the facilitator a kill switch they can pull without asking the incident commander.
- Scoring individuals. Wrong-turn count drops to zero and the runbook never changes. Score the system and the runbook, never the person; the health metric is action items closed per game day, not the score itself.
- A chaos experiment with no hypothesis. That is just an outage you caused. If there is no steady-state definition and no pass/fail, you are not running an experiment.
- No stakeholder pre-brief. Support and the status page get blindsided. Tell them the window and the expected customer impact before T+0:00.
When not to. Do not run a game day for a failure you have never exercised in staging: you will be rehearsing discovery, not response. Do not run chaos experiments in production before you have SLOs and automated abort; the experiment's value is the falsifiable hypothesis, and without SLOs you cannot state one.
Publish the scorecard per game day. The number that matters is whether time-to-commander and MTTD fall across the year, not whether any single run passed.
Curated: · Written: · Reviewed:
Q&A-43How do you prove a fallback works without waiting for the vendor to actually die?
What a strong answer sounds like
You can't prove a fallback works by waiting for the vendor to fail: you'd wait forever and still not know which failure mode you'd get. You prove it by injecting the failure on the call path, in a controlled environment, and asserting the fallback plus the core SLO, not just that an error was logged.
Clarify two things first: which failure modes the vendor contract actually permits (error, timeout, hang, slow success, partial response), and what the fallback's own SLO is (staleness bound, latency budget, success rate). Then inject each one.
Why the call path, not a mock
A mock replaces the client library, so it can only produce the failures you thought to write. A fault proxy sits where the real call goes and can produce the ones you didn't: a TCP connection that opens and then never responds, a 200 with an empty body, a response that arrives after your deadline. That distinction is the whole point.
The failure I've seen: a mock that always returned an empty 200 trained the fallback to look green. In production the vendor hung TCP for 30 seconds, checkout threads pinned on the connection, and the pool exhausted: the fallback never ran because the call never returned.
What to inject
Illustrative canary run, 2% of production traffic, tax API:
| Injected fault | Expected fallback | Observed |
|---|---|---|
| HTTP 500 | cached rate, ≤ 10 min old | 99.94% checkout success, stale tax 2.1 min |
| 5,000 ms hang | 250 ms deadline, then cache | p99 271 ms, no pool exhaustion |
| TCP reset | same as 500 | 99.93% checkout success |
| Empty 200 (anti-pattern) | looks fine | hides the hang, not evidence |
The assertion
def test_fallback_under_hang(proxy):
proxy.inject_hang("tax-api", ms=5000)
r = checkout(client, cart) # tax call deadline: 250 ms
assert r.status == 200
assert r.tax_source == "cache"
assert r.latency_ms < 300
Trace: the proxy holds the tax response for 5000 ms. The checkout's tax call carries a 250 ms deadline, so it aborts at ~250 ms, the fallback reads the cache, and checkout returns 200 with tax_source "cache" at ~271 ms: under the 300 ms assertion and consistent with the p99 in the table. Remove the deadline and the same test blocks for 5 s and fails the assertion, which is the point of running it.
Failure modes to watch
- Fallback never engaged: the call hung instead of erroring, so no fallback code ran. Detect with a fallback-engaged counter, not with error logs.
- Fallback engaged but stale: cache age exceeds the bound. Emit cache age as a metric and alert on it.
- Fallback engaged but slow: the fallback itself makes a network call. Budget it inside the same deadline.
- Pool exhaustion: threads pinned on the hung connection. Watch in-flight connections and pool saturation, not just latency.
When not to do this
Don't inject in production without a kill switch and a blast-radius cap; a 2% canary with automatic abort on core-SLO burn is the ceiling. Don't inject faults the vendor contract rules out: you'll build fallbacks for failures that can't happen and skip the ones that do.
A fallback that has never been exercised is a comment in the code, not a reliability control.
Curated: · Written: · Reviewed:
Q&A-48When is rollback the correct mitigation, and when is it the wrong reflex?
What a strong candidate says first. Roll back when three conditions hold together: the change is still the leading hypothesis for the regression, the previous state is still valid, and you can restore it inside the RTO. Break any one and rollback is the wrong reflex: it resets the incident clock, hides the real cause, and can leave you worse off than the forward fix.
The condition people skip is the second. Rollback restores code and config. It does not restore state. Anything the change wrote durably: rows a backfill rewrote, a column a migration dropped, events already published to a topic, emails sent, payments captured: survives the rollback and is now being read by the old code. That is the line between rollback and forward-fix, and it is decided by the change type, not by how the incident feels.
Decide by change type, before the incident.
| Change | Reversible by rollback? | What survives | Action |
|---|---|---|---|
| App binary / container image | Yes: previous ReplicaSet is retained (revisionHistoryLimit defaults to 10) | Nothing durable | kubectl rollout undo, ~1–2 min |
| Feature flag | Yes: flip is seconds | Nothing, unless the flag gated a write path | Turn it off |
| Additive migration (new nullable column) | Yes: old code ignores the column | The column | Roll back code; leave the column |
| Destructive migration (drop / rename / type change) | No: old code reads a column that is gone | The schema change | Forward-fix, or restore from backup |
| Data backfill | No: rows are already rewritten | Every rewritten row | Pause writers, forward-fix |
| Config or secret pushed to all replicas | No: the bad value is in the config store, not the artifact | The config | Fix the config; a deploy rollback does nothing |
The cost of treating rollback as a last resort. A reversible feature flag was left on for 26 minutes while the room debated a forward-fix, holding 18% 5xx the whole time. Flipping the flag took 40 seconds once someone ran it. Same error rate, 1560 s instead of 40 s: 39× the failed-request volume, for a fix that was already available.
Where rollback is the wrong reflex.
- The change isn't the cause. If the error rate doesn't recover within the rollback window, stop rolling back: you're resetting the clock and hiding a dependency failure. Detect it by watching the SLI during the rollback, not after.
- The rollback is itself a breaking change. Rolling producers back to v1 events while consumers already parse v2 breaks the consumers. Detect: consumer deserialization errors spike right after the rollback.
- The artifact is gone. If the previous image was garbage-collected or the migration is one-way, "roll back" is a plan you don't have. Detect: run the command in a game day and time it.
- Rollback loses in-flight work: draining queues, cancelling jobs, discarding a batch mid-write.
What makes it a real option. Measure time from abort-line crossed to the old version serving 99% of traffic, and hold it under the incident RTO. An abort line you can act on looks like "error rate +0.2 pts over baseline for 5 minutes": a number, not a discussion. If that measurement doesn't exist, rollback is a hope, not a mitigation.
Rollback is a designed property of the release system, not a confession that the team was wrong.
Curated: · Written: · Reviewed:
Q&A-49Should a risky behaviour change ship as a binary deploy or as a flag? How do you operate flags in production?
Answer
Flag when the behaviour is runtime-toggleable and you need to stop it without a deploy; binary deploy when it isn't toggleable or the flag would outlive the change. Operate flags as production releases: dark ship, cohort ramp behind the same SLI gates as a canary, a named owner, an expiry, and every mutation in the change log.
Mechanism: the SDK evaluates in-process against a cached rule set. The request path calls flags.bool("payments_v2", ctx, default=False); the rules are already local (streamed or polled), so evaluation is a map lookup, not a network hop. A flip changes the rule set on the flag service; the SDK picks it up on the next update: seconds to minutes, not instant. That's why a flip avoids rollback cost: the artifact stays put, connections aren't drained, and a migration already applied stays applied. A binary rollback means redeploying the old artifact and reversing whatever the new one did, which you can't always do.
# Python 3.12 — in-process evaluation, rules cached locally
ctx = {"user_id": "u-42", "plan": "pro"}
if flags.bool("payments_v2", ctx, default=False):
return new_payment_path(order)
return legacy_payment_path(order)
The SLI gate on a flag ramp checks the same things a canary does: error-rate delta against the control cohort, p99 latency delta, and the business metric the change is supposed to move (checkout success, say), each with a minimum sample size and soak window. Numbers here are illustrative: error rate +0.20 percentage points over 20 minutes at ≥5k requests in the cohort.
When the binary deploy is the better choice: schema migrations and dependency upgrades aren't toggleable. The old artifact can't run against the new schema, so there's nothing to flip. A new endpoint or a new service doesn't exist in the old binary. And a change that should be permanent doesn't need a flag; wrapping it just creates dead code you'll never remove. That's the cost side: each flag doubles the paths to test, N flags give up to 2^N combinations, and stale flags become dead code and "why is this off in prod" incidents. A flag you can't name an owner and an expiry for is a permanent branch.
Failure modes worth naming: a kill switch that defaults to on doesn't kill anything when the flag service is unreachable. Decide fail-open vs fail-closed per flag. A "flip and watch" that reads metrics inside the propagation window is reading the old rules. And the one I've seen: a long-lived flag flipped at 02:00 with no ticket, no soak, and the binary 19 versions stale, producing a Sev-1 that "was not a deploy."
| Event | Artifact | User share | Gate |
|---|---|---|---|
| Tue 11:00 | binary v441 (flag off) | 100% | standard canary |
| Tue 14:00 | flag payments_v2 | 5% | error +0.20 pts, 20 min |
| Tue 14:40 | flag payments_v2 | 25% | same |
| Wed 02:00 (anti-pattern) | flag, no ticket | 100% | none. Sev-1 |
Evidence that it's operated, not just shipped: flag changes appear in the same audit trail as deploys, each ramp has its soak metrics attached, and a weekly report lists flags older than 30 days for removal. A flag that bypasses the release process is an untracked production change with extra steps.
Curated: · Written: · Reviewed:
Q&A-46A canary looks "about the same" on a dashboard. How do you decide to proceed or roll back?
The decision rule
Write the rule before the canary starts: the SLIs, the soak window, the abort threshold, the test. Then the decision is mechanical. At the end of the soak, compare canary and control on the same SLIs. Delta crosses the threshold with confidence: roll back. It doesn't: ship. "Looks about the same" is not an input.
For a service at 0.12% error and 188 ms p99, a registered rule might be: abort on +0.20 points error or +50 ms p99 over a 15-minute soak, tested as a two-proportion comparison at alpha 0.05.
| Slice | n requests | Error % | p99 ms | Decision |
|---|---|---|---|---|
| control | 84,200 | 0.12 | 188 | — |
| canary 5% | 4,410 | 0.94 | 210 | abort, +0.82 pts |
| eyeball at 5 min | — | "looks close" | "looks close" | would have shipped |
Worked calculation for whether that soak can even see the threshold: to detect a +0.20-point error delta from a 0.12% baseline at 80% power and alpha 0.05 two-sided, equal arms need about 8,600 requests each. 7.84 × (0.0012×0.9988 + 0.0032×0.9968) / 0.002² ≈ 8,600. The canary arm here is 4,410, roughly half that, so the registered +0.20 threshold sits at the edge of what this soak resolves. The observed +0.82 points is far outside it, which is why acting on the abort is safe. A +0.25-point reading would not be: that outcome is "soak longer or widen the canary," not "ship."
Where the rule has to bend
- Small deltas are sometimes shippable. A checkout rewrite that lifts conversion 1.5% while p99 rises 20 ms is worth shipping. Register that trade before the run. A trade-off you name after seeing the numbers is a rationalization.
- Cost asymmetry sets the threshold. A config-only rollback is seconds; a schema-migration rollback is a multi-hour incident. Tighter thresholds for the irreversible change.
- Soak length changes the answer. Rare events need volume, not time: a 0.1% error rate needs tens of thousands of requests. Slow-burn failures need time, not volume: a connection-pool leak or cache-warming regression won't surface in 15 minutes. And a 15-minute soak at 02:00 samples a different traffic mix than a 24-hour soak covering the diurnal cycle.
- Peeking. Checking every minute and stopping at the first significant result inflates false positives. Use sequential bounds or a fixed horizon.
- Manual override is allowed, but only against a new registered hypothesis: a different SLI, a different threshold, a stated reason. Otherwise the override is just the eyeball with extra steps.
Failure modes
- Overlapping graphs hide small deltas. The 5-minute eyeball above read "looks close" and would have shipped a 0.8-point error regression that burned 18 minutes of monthly budget after full rollout.
- The canary shares state with control: same database, same cache, same downstream. The canary's damage lands on control and the comparison means nothing.
- Low-traffic services. 5% of 200 rps is 10 rps; 15 minutes gives 9,000 requests, enough. 5% of 5 rps gives 225. Nothing is detectable, and "no abort" is noise.
What I'd ask for
The analysis report: sample sizes per arm, the test and its bounds, and the recorded go/no-go the pipeline actually obeyed. If a human can wave a canary through, you have a waiting period, not canary analysis.
Curated: · Written: · Reviewed:
Q&A-47After a canary passes, how do you raise traffic without turning the rest of the rollout into a big-bang?
The short version. The canary passed on a slice that probably isn't the whole population: one region, one cluster, internal or low-traffic accounts. The ramp's job is to widen that population in steps small enough that a bad step costs a bounded fraction of users, with gates on the same SLIs the canary used, automatic rollback to the last good share, and a hold rather than an advance when a gate is inconclusive.
A schedule that does this (figures illustrative):
| Step | Share | Soak | Gate | Outcome |
|---|---|---|---|---|
| 1 | 1% | 20 min | error rate +0.20 pts, p99 +50 ms | pass |
| 2 | 5% | 20 min | same | pass |
| 3 | 25% | 30 min | same, evaluated per region | hold: eu-west error +0.41 pts |
| 4 | 50% | — | not started | rolled back to 0% |
A hold freezes the ramp and pages; it does not advance and does not auto-rollback, because a single region drifting is worth a look before you throw away a working deploy.
Alternatives, and when each is the right one.
- Weighted split at the load balancer or mesh. Per-request, so a user can bounce between versions mid-session unless you add stickiness. Right when the service is stateless and the risk lives in the artifact. Wrong when sessions or caches need per-user consistency.
- Ringed deployment. Deploy the artifact to ring 0, then ring 1, and so on. Right when traffic can't be split: batch jobs, stateful services, infra changes like a sidecar or kernel bump. Slower feedback: each ring is a full deploy, so you get one data point per ring instead of a continuous share.
- Feature flags. Decouple release from deploy, target cohorts, kill instantly. Right when the risky part is behavior. Wrong as the only control when the risk is in the deploy itself (config, dependencies, resource limits) because the flag never exercises those.
- Blue/green swap. Fast, but the swap is the big-bang. Only safe with warm-up and a revert you have actually run.
Deciding criteria: can traffic be split per request; does the change carry state or schema; is the risk proportional to traffic share or concentrated on a rare code path; how fast is revert. A flag flip with instant revert and no state change does not need a two-hour five-step ramp: that just delays the release. A fast ramp is wrong when the canary population is unrepresentative, when the gate's detection window is longer than the soak, or when rollback isn't a pure traffic shift.
Failure modes, and how you see them.
- Population skew. Canary ran in one region; the ramp hits a region with different config. Detect by gating per region, not on the aggregate.
- Gate lag. Gating on a 1-hour burn-rate window while soaking 10 minutes means the gate cannot fire before you advance. Soak must be at least the shortest alert window.
- Rollback that doesn't roll back. If the new version wrote schema, cache entries, or queue messages, dropping traffic to 0% leaves that data behind. Writes have to stay backward-compatible for the duration of the ramp.
- Long-lived connections. The balancer keeps existing connections on the new version after the weight hits 0. Check connection drain, not just the weight.
- Rare-path blindness. A bug on a path taken by 1 in 10,000 requests barely registers at 1% share.
That last one is why you widen share rather than only lengthening soak. On a 20,000 rps service, 1% is 200 rps, and a 20-minute soak is 240,000 requests: a bug with a 1% failure rate produces 2,400 errors and is obvious, while a bug on a 1-in-10,000 path produces 24 hits. Inside normal noise. You don't see it until 25% share, where the same window carries 6M requests and 600 hits.
Curated: · Written: · Reviewed:
Q&A-54When do you cut all traffic to a new environment, and when do you drip it?
The engineering question underneath blue-green versus canary is which failure is acceptable.
Use canary when you need production traffic to falsify a hypothesis cheaply; use blue-green when you need instant cutback and the two environments can both carry 100% load.
Concretely, run a canary as a traffic share with SLI gates, or run blue-green as two full stacks with a health-checked green, a balancer flip, and a warm blue sized for the RTO rather than a 100% flip onto an untested green.
The reason for that specificity is a failure I have seen: A blue-green flip of an unsoaked green environment took 100% of checkout to a missing IAM role, and the "instant cutback" was 11 minutes of DNS because blue had been scaled to zero to save money.
Same release, two topologies, two failure costs.
| Topology | First users exposed | Cutback | Last miss |
|---|---|---|---|
| Canary 1% | 1% for 20 min | instant to 0% | would have caught IAM miss |
| Blue-green, blue warm | 100% at flip | 40 s balancer | IAM miss = 100% Sev-1 |
| Blue-green, blue scaled 0 | 100% at flip | 11 min reprovision | freeze-time outage |
I would not consider it settled without evidence: State the RTO of the cutback and prove it in a drill; for canary, show the smallest share that would have caught the last three bugs.
Blue-green without a live blue is just a big-bang deploy with extra infrastructure.
Curated: · Written: · Reviewed:
Q&A-56A pod is killed in a loop while it is only slow to warm. Which probe is wrong?
I would treat liveness versus readiness probes as a design decision with a stated trade-off rather than a best practice.
Readiness removes a pod from traffic until it can serve; liveness restarts a pod that is deadlocked, and using liveness as readiness turns slowness into a crash loop.
Concretely, use a startup probe to protect slow initialization, readiness to report whether this pod can currently accept traffic, and liveness only for local unrecoverable process failure. Include downstream state in readiness only when losing that dependency truly makes the pod unable to serve and cannot cause a fleet-wide readiness collapse.
The reason for that specificity is a failure I have seen: A liveness probe that hit payments timed out during a 4-second vendor blip, Kubernetes restarted 80 pods, and the herd of cold starts extended a 4-second blip to 11 minutes.
Probe split for checkout, warmup 40 s, vendor blip 4 s.
readinessProbe:
httpGet: {path: /ready, port: 8080} # local ability to serve, not shared payments
initialDelaySeconds: 40
timeoutSeconds: 1
livenessProbe:
httpGet: {path: /live, port: 8080} # in-process only, no payments
periodSeconds: 10
failureThreshold: 18 # 3 min before restart
# Wrong: liveness=/ready -> 80 restarts, 11 min herd.
I would not consider it settled without evidence: Fail a shared downstream and verify the service follows its intended fallback without restart or total endpoint removal; separately deadlock the process and verify liveness restarts it.
Restarting a slow pod is how you convert a brownout into a thundering herd of cold starts.
Curated: · Written: · Reviewed:
Q&A-58A node upgrade drained 6 of 6 checkout pods at once. What should have blocked that?
The short answer
A PodDisruptionBudget on the checkout workload with minAvailable: 5 (equivalently maxUnavailable: 1) for 6 replicas, enforced through the Eviction API. kubectl drain and every conforming node-replacement controller evict via POST .../pods/<name>/eviction; the API server checks every PDB whose selector matches the pod and returns 429 TooManyRequests if the eviction would drop healthy replicas below the floor. The drain retries with backoff, so the upgrade stalls instead of the service dying.
Three things must all hold for that to work:
- A PDB exists and its selector actually matches the pods.
- The upgrade path uses the eviction subresource, not
kubectl delete pod, not a script that deletes pods directly. - Replicas are spread across nodes, so evicting one leaves a schedulable home for the replacement.
Mechanism
The PDB controller computes desiredHealthy = minAvailable (or expectedPods - maxUnavailable) and disruptionsAllowed = currentHealthy - desiredHealthy, clamped at 0. Only Ready pods count as healthy. The eviction API allows the eviction only if every matching PDB has disruptionsAllowed > 0. Percentages round up for minAvailable and down for maxUnavailable, so minAvailable: 80% of 6 is 5, not 4.
The budget for this service
Peak 1,200 rps, 250 rps/pod (illustrative figures).
| replicas | minAvailable | capacity while draining | vs 1,200 peak |
|---|---|---|---|
| 6 | 5 | 1,250 rps | +50 rps (4%) |
| 6 | 4 | 1,000 rps | −200 rps |
| 8 | 6 | 1,500 rps | +300 rps (25%) |
minAvailable: 5 is the floor that keeps checkout serving; the 4% headroom is thin, so 8 replicas with minAvailable: 6 is the safer shape.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: checkout
namespace: shop
spec:
minAvailable: 5
selector:
matchLabels:
app: checkout
What the drain looks like when the budget holds:
$ kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data
evicting pod shop/checkout-7d9f-abcde
error when evicting pods/"checkout-7d9f-abcde" -n shop (will retry after 5s):
Cannot evict pod as it would violate the pod's disruption budget.
The API returns 429 with reason: TooManyRequests; the drain waits until a replacement pod is Ready and disruptionsAllowed goes back to 1.
Why it still happens
- No PDB. The drain evicts all 6 in one pass.
- Selector mismatch. PDB selects
app: checkout, pods are labelledapp.kubernetes.io/name: checkout. The PDB matches zero pods and protects nothing. Checkkubectl get pdb -n shop: theALLOWED DISRUPTIONScolumn andstatus.expectedPodsmust match the real replica count. - Bypass.
kubectl delete pod, or node tooling that deletes pods directly, never touches the eviction API. PDBs only gate voluntary evictions. - All replicas on one node. The PDB blocks the drain entirely and the upgrade hangs: correct for the service, bad for the upgrade. Fix with
topologySpreadConstraints(maxSkew: 1,topologyKey: kubernetes.io/hostname) so 6 replicas land on ≥2 nodes, plus spare capacity for the replacement. - Replacement can't schedule. Insufficient CPU/memory, taints, or a zone-pinned PVC leaves the new pod Pending and the drain stuck after the first eviction.
- Unhealthy pods. With the default
unhealthyPodEvictionPolicy: IfHealthyBudget, if the app is already below the floor, even unready pods can't be evicted and the drain hangs.AlwaysAllow(available since 1.27) lets them go.
What a PDB does not cover
Involuntary disruption: node crash, spot reclaim, kernel panic, OOM kill. Those bypass the eviction API entirely. Cover them with replica spread across nodes and AZs plus capacity headroom, not with the budget.
Curated: · Written: · Reviewed:
Q&A-59Pods with no CPU request schedule densely, then p99 explodes under load. What do you set, and why not unlimited?
Set the CPU request to measured sustained usage (p95 of actual CPU over a representative week, not peak) and leave the CPU limit off for latency-sensitive services, or set it at least 2× the measured burst. That lands the pod in Burstable QoS, which is the right default here.
Why the request, and why not unlimited
A CPU request does two jobs. The scheduler uses it to filter and score nodes, so the sum of requests on a node is the packing contract. The kernel uses it as cpu.shares, a relative weight under contention. It is not a hard reservation: if the node is overcommitted, every pod's share shrinks proportionally. A request protects you only when the node's total requests stay under allocatable.
Unlimited isn't on the menu. The node has a fixed number of cores; the request is the only mechanism that divides them. Giving one pod no ceiling doesn't create capacity, it just lets that pod take cycles from its neighbours, which is exactly the failure in the question.
A CPU limit is a different mechanism from a request: it writes cpu.cfs_quota_us, and a cgroup that exhausts its quota is stalled until the next 100 ms period. That stall is invisible in average CPU and lands directly in p99. For a service whose tail matters, a limit converts a busy period into latency. Guaranteed QoS (requests == limits on every container, CPU and memory) buys a fixed share and eviction priority, but it forces a CPU limit, so it reintroduces throttling: I only take it when the workload genuinely needs it.
The HPA is the third reason to set a request. CPU utilization is usage ÷ request. With no request there is no denominator, the metric is unavailable, and the HPA does not scale: it sits there looking healthy while the service burns.
Worked example (illustrative numbers)
18 latency-critical pods, no requests, on a 4-core node. All BestEffort, so they contend for equal shares. Under load p99 goes 160 ms → 2.1 s, and the HPA never fires because it has no request to divide by. Adding request: 300m to each pod makes the sum 5.4 cores, so the scheduler refuses to place all 18 on that node, and the HPA has a denominator.
| Field | Value | If omitted |
|---|---|---|
request.cpu | 300m (p95 measured) | 18 pods pack onto 4 cores; HPA has no denominator |
limit.cpu | omit, or ≥ 2× measured burst | a 100m limit throttles a 280m burst |
| HPA CPU target | 70% of request | no request → no utilization → no scaling |
What I'd check before calling it done
kubectl describe node: sum of requests ≤ allocatable, with headroom for kubelet and system daemons.container_cpu_cfs_throttled_seconds_totalper pod, near zero at peak. If it climbs, the limit is too tight or the request is too low.- HPA events showing a real utilization number, not
missing request for cpu.
Curated: · Written: · Reviewed:
Q&A-50A config-map edit took checkout down. Why was that not "just config", and how do you ship it?
I would start by making the assumption behind configuration as a release explicit.
Treat production configuration as a versioned release with the same canary, audit, and rollback as binaries, because the process reads it on the hot path.
Concretely, store config in git, render it immutably, roll it to a subset of instances first, and keep a one-command rollback to the previous config hash.
The reason for that specificity is a failure I have seen: An in-cluster edit of a 2-line timeout from 250 ms to 25000 ms bypassed CI, skipped the canary, and pinned every checkout thread on a hung vendor for 14 minutes.
Config change 250 ms -> 25,000 ms, two shipping paths.
| Path | Canary | Rollback | Outcome |
|---|---|---|---|
| kubectl edit | none | unknown previous | 14 min hang, 22% 5xx |
| git hash a1c3, 5% then 100% | 20 min SLI gate | kubectl apply -f config-a1c3.yaml | abort at 5%, 0.3% extra errors |
I would not consider it settled without evidence: Diff prod config against git HEAD every 5 minutes and page on drift; show that the last 10 config changes had canaries.
The runtime does not care that the bytes came from a ConfigMap instead of a container; neither should your release bar.
Curated: · Written: · Reviewed:
Q&A-102Checkout pods enter CrashLoopBackOff only during the evening peak, and the platform team says the manifest is compliant. Walk me through how you find the cause and who owns the fix.
CrashLoopBackOff is a symptom, not a cause. Three different problems produce it, and they have three different owners, so the first move is to read the termination reason rather than guess.
kubectl get pod checkout-7f9c -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
# {"exitCode":137,"reason":"OOMKilled","startedAt":"19:02:11Z","finishedAt":"19:04:38Z"}
kubectl describe node ip-10-0-3-14 | grep -A6 'Allocated resources'
| exitCode / reason | what actually happened | first owner |
|---|---|---|
137, OOMKilled | cgroup memory limit hit | app team (working set) |
143, Error (SIGTERM) | liveness probe killed it | app team (probe semantics) |
1, Error | process exited on its own | app team |
137, no reason, node MemoryPressure | kubelet evicted it | platform (node sizing, requests) |
If it is OOMKilled, compare three numbers, not one. Say the limit is 512 Mi, requests 256 Mi, and container_memory_working_set_bytes peaks at 780 Mi at 19:00. The container dies at 512 Mi, but the node is a second, larger problem: 8 GiB allocatable with 256 Mi requests lets the scheduler pack ~30 pods onto it, and at 780 Mi each that is 23 GiB of real demand. The node crosses the kubelet default evictionHard: memory.available<100Mi, starts evicting, and pods die for reasons unrelated to their own limit. Raising the limit alone makes the node problem worse, which is why "the manifest is compliant" and "the pods are dying" are both true.
Probes are the other half. A liveness probe at initialDelaySeconds: 10, periodSeconds: 5, failureThreshold: 3 kills the container at 25 s. If warm-up takes 40 s under peak database contention, every restart loops: the pod is doing exactly what it should and the probe is wrong. Readiness gates traffic; liveness should only restart a genuinely wedged process. For slow warm-up the right tool is a startup probe (periodSeconds: 2, failureThreshold: 30 = a 60 s budget), which suppresses liveness and readiness until it passes.
On ownership, the manifest is a shared artefact and the split is worth stating explicitly:
| decision | owner | why |
|---|---|---|
| limit and request values | app team | only they know the working set |
| LimitRange defaults, quota, admission | platform | enforces a floor across every namespace |
| node size, eviction thresholds, taints | platform | cluster-wide blast radius |
| probe endpoints and what "ready" means | app team | it is application semantics |
| who gets paged for which failure | SRE | the boundary is the SRE's artefact |
Failure modes to name out loud. Raising the limit until restarts stop hides a leak: check the 24-hour working-set trend, not the peak. Deleting liveness leaves a wedged process serving 500s indefinitely. Setting requests equal to limits (Guaranteed QoS) buys predictable eviction order (BestEffort goes first, then Burstable over its requests, then Guaranteed) at the cost of burst headroom and real capacity. And if the app team owns the numbers while the platform team owns the LimitRange, the fix is a default that makes the wrong manifest fail admission, not a Slack thread at 19:15.
What I would change: a LimitRange with sane defaults, a startup probe replacing the aggressive liveness, requests set from the p95 working set rather than the idle one, and routing that pages OOMKilled to the app team and node eviction to the platform team.
Curated: · Written: · Reviewed: —
Q&A-52The default-branch pipeline is red 18% of the time and people just retry. Why is that a reliability defect?
Assumptions. Default branch, CI required on every merge, "red" means a required check failed, retries are re-runs of the same commit with no code change, and there is no quarantine or flake budget in place today. Figures below are illustrative, not measured from your repo.
The answer. It is a reliability defect because the pipeline has stopped carrying information. A red build is supposed to mean "this commit is bad." At an 18% red rate with a ~1% genuine failure rate, a red build means that about 6% of the time:
P(genuine failure | red) = real failures / all reds
= 1.1% / 18.0% ≈ 6%
Fifteen of sixteen red builds are noise. Retrying is then the individually rational move, nobody can afford to investigate a 6% hypothesis on every merge, and the collectively fatal one, because the real regression gets retried along with the noise. Detection latency for a genuine break becomes unbounded: it is however long it takes someone to get unlucky enough to actually read the log. The second-order damage is that green stops meaning anything too. A build that passes on the third attempt is not evidence the commit is good; it is evidence the test is nondeterministic, and you can no longer bisect, revert confidently, or say "main is green" during an incident.
| Metric (30 days, default branch) | Value | Budget |
|---|---|---|
| Runs | 1,140 | — |
| Failed first try | 18.0% | — |
| Failed after 3 retries (genuine) | 1.1% | — |
| Flake rate | 16.9% | 2.0% |
| Auth regression time-to-detect | 2 days / 9 retries | 1 run |
The retry tax is real but secondary: ~205 red builds per 1,140 merges, each re-run up to three times, is up to ~600 extra pipeline executions, roughly 50% more CI load, plus a re-run's wall-clock on the critical path of every affected merge.
What to do about it, and the trade-offs. Three credible approaches, and they are not interchangeable:
- Retry budget + flake SLO. Count a retry as pipeline unavailability, publish flake rate, and freeze merges when it exceeds a budget (2% is a common starting point). Cheap, measurable, and it makes the cost visible to the teams causing it. Weakness: it caps the damage without fixing anything, and it is gameable. Splitting one flaky suite into ten files changes the denominator, not the behaviour.
- Quarantine. Move tests that fail without a product change out of the required set. Restores signal immediately. Weakness: quarantine becomes a dumping ground the moment it has no owner and no expiry. Require a named owner, a TTL (say 14 days), and a per-team cap; track quarantined-test count as a first-class metric, because a quarantine that only grows is the same defect with better PR.
- Test-level isolation. Fix the actual nondeterminism: shared mutable fixtures, real clocks, unseeded randomness, network calls, fixed ports, order dependence. Highest cost, only permanent fix.
Deciding criteria. Where is the flake? If it clusters in a handful of suites, isolate: a budget just hides it. If it is spread thin across many teams and infra, a budget plus quarantine buys signal back this week while isolation work proceeds. Cost of a false red versus cost of a missed regression sets the budget: a payments service should run tighter than a docs site.
Failure modes to watch. Auto-retry with no counter (the signal is destroyed silently); quarantine with no TTL (rot); flake measured only on default branch (misses PR-level flake, which is where engineers actually feel it); and a budget that is reported but never enforced, which is just the current state with a dashboard.
What I'd want before calling it fixed. Flake rate and median retries-to-green per suite, the quarantine list with ages, and the last regression that reached production because a red build was retried rather than read. Retry is not a test strategy; it is how you launder a red signal into a green one.
Curated: · Written: · Reviewed:
Q&A-94An engineer runs terraform apply because "the plan looked fine yesterday". What rule do you enforce?
The rule: production applies only a saved plan file that CI generated from the commit being deployed. Nobody runs terraform apply against live state in prod.
A bare terraform apply refreshes state and computes a new plan at apply time. That plan is not the one anyone reviewed: it is whatever the world looks like right now. terraform apply plan.bin applies the exact actions recorded in the file, with no refresh and no recompute. That is the whole point: the reviewed change is the applied change.
Be precise about what the saved plan guarantees, because the common version of this rule overstates it. Terraform embeds the state snapshot in the plan and applies against that snapshot. It does check the state serial and lineage: if another Terraform operation wrote state after the plan was created, apply fails with Saved plan is stale. It does not detect drift made outside Terraform. Someone deleting a queue in the console, an autoscaler resizing a target group, a manual security-group edit: none of those bump the serial, so the plan applies anyway, against a world that no longer matches the snapshot. The rule buys you "the reviewed change is the applied change", not "the state is correct".
# CI, on the merge commit
terraform plan -out=plan.bin
# upload plan.bin as an artifact keyed by the commit SHA
# prod
terraform apply plan.bin # applies recorded actions; no refresh
Enforce it with three things: the prod apply job takes the artifact as its only input and fails if it is missing; state locking is on so two applies cannot interleave; and local applies are break-glass. Logged, and the operator pages the on-call before running one.
The failure this prevents is a local apply against drifted state. A plan from yesterday described a world that had since moved: 40 merges landed, someone had already removed a queue by hand, and the apply recreated and then destroyed sqs.checkout_events, dropping 18 minutes of events. The reviewed plan would have caught it; the live apply could not.
Detect drift separately, because the saved plan will not: run a scheduled terraform plan -detailed-exitcode against prod and alert on exit code 2. That is the check that catches out-of-band changes.
Curated: · Written: · Reviewed:
Q&A-68A "latest" tag on the registry was overwritten. How do you know what actually runs in production?
The 90-second answer
You can't tell from the tag. A tag is a pointer the registry can rewrite, and by the time anyone asks you, it already has. Split it into two questions and answer both from digests.
What is running now: read the digest off the running container, not off the Deployment manifest. The manifest says what was requested; the container status says what the runtime actually pulled.
What should be running: the signed provenance record for the release. Git SHA, builder identity, digest. If the running digest isn't in that record, you have an unaccounted-for binary in production, and the incident is "who pushed it", not "which version is it".
Assumption I'd state out loud: the registry allows tag overwrite. If it doesn't, the question mostly dissolves.
Getting the digest off a running pod
kubectl get pods -n checkout -l app=checkout \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].imageID}{"\n"}{end}'
Hypothetical output:
checkout-7d9f4b6c8-2xk9p sha256:3b1e77c0...
checkout-7d9f4b6c8-9mq4t sha256:3b1e77c0...
checkout-7d9f4b6c8-zz8vl sha256:9f2c41ab...
Gotcha: on containerd, imageID is the image config digest, not the manifest digest you pinned with @sha256:. They are different hashes. To get the manifest digest, ask the node's image store, crictl inspecti <imageID> and read repoDigests from the JSON, then verify that digest (cosign v2 flags):
cosign verify \
--certificate-identity-regexp '^https://github.com/acme/checkout/\.github/workflows/.*$' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
registry.example/checkout@sha256:9f2c41ab...
Two different digests inside one ReplicaSet means the tag moved between pod starts. That's the signal.
Why it's partial, not total
imagePullPolicy decides how fast a retag spreads. With IfNotPresent, a node that already has the old digest keeps serving it; only pods that restart, get rescheduled, or land on a fresh node re-resolve the tag. With Always, every container start re-resolves. So a retag arrives gradually (on deploys, OOM kills, node drains, HPA scale-ups) which is why you find out hours later and why the blast radius is a percentage rather than everything.
Worked example (hypothetical figures): :latest retagged at 11:04 to an unsigned laptop build; by 14:34, 30% of pods had restarted onto it and were serving a debug binary. Detection was a digest that didn't match the provenance log, not a failing health check.
The trade-off: pin digests vs. signed mutable tags
Pin @sha256: | Signed mutable tag | Immutable tag + signature | |
|---|---|---|---|
| Manifest churn | every deploy rewrites the manifest | stable | stable |
| Rollback | revert to a recorded digest | re-point the tag | re-point the tag |
| Human-readable version | lost; needs a digest→version map | kept | kept |
| Tooling | kubectl set image, helm --set image.tag stop working | works | works |
| Tag-move risk | none | TOCTOU between admission and pull | registry rejects the move |
Cost of digest pinning, concretely: every deploy is a manifest diff, so GitOps history is noisy; rollback needs the previous digest recorded in your CD system because the manifest no longer names a version; and kubectl set image / helm upgrade --set image.tag silently do nothing useful. You pay that for the guarantee that what you reviewed is what runs.
When not to use it: dev and staging clusters where kubectl rollout restart picking up a fresh build is the point; or when the registry enforces tag immutability (ECR imageTagMutability: IMMUTABLE, Artifact Registry, Harbor immutable-tag rules) and admission resolves tag→digest and verifies the signature on the resolved digest. Then the tag is a friendly alias for a digest and pinning buys little.
The failure mode pinning doesn't fix
TOCTOU: admission sees :latest → digest A, verifies the signature, admits the pod; the tag moves; kubelet pulls digest B. The fix is a mutating webhook that rewrites the pod spec to the resolved digest before the kubelet ever sees the tag, or pinning. Detection: alert on any running digest with no matching signature or provenance entry.
Curated: · Written: · Reviewed:
Q&A-51Live cluster state no longer matches the last merged manifest. How do you detect that, and what is the allowed repair?
The 90-second answer. Drift is any live field that differs from the manifest at the merged commit. Detect it with the GitOps controller's own comparison: Argo CD re-diffs each Application on timeout.reconciliation (default 180s) and exposes argocd_app_info{sync_status="OutOfSync"}; Flux re-applies each Kustomization on its spec.interval and exposes gotk_reconcile_condition. Worst-case detection latency equals that interval, so set it to what your SLO tolerates and alert on the metric, not on a dashboard.
The allowed repair is always "write git, then let the controller apply git." Two shapes. If the live change was unauthorized, sync HEAD over it and the live edit is discarded. If it was a legitimate emergency change, land the same change in a PR first, then sync; the live object is overwritten with identical content, so provenance is git. What is never allowed is kubectl edit → kubectl get -o yaml → commit: that captures API-server defaults, other controllers' writes and last-applied annotations, and makes the cluster the source of truth.
Detection has a blind spot. The controller only compares objects it manages. A kubectl apply of a Deployment that isn't in git produces no OutOfSync at all. Cover that with the API audit log: alert on write verbs (create, update, patch, delete) against namespaced workload kinds by human identities.
| Drift class | Signal | Blind to |
|---|---|---|
| Field changed on a managed object | sync_status=OutOfSync | — |
| Managed object deleted | OutOfSync, resource Missing | — |
Field git doesn't own (HPA replicas, injected sidecar) | permanent OutOfSync noise | real drift hides in it |
| Object not in git at all | nothing | controller |
Trace. Argo CD, selfHeal: true, timeout.reconciliation: 180s.
02:14:07 kubectl scale deploy/checkout --replicas=4 (break-glass, no PR)
02:14:07 live spec.replicas=4, git HEAD=12, app still reports Synced
02:17:07 reconciliation: live != HEAD -> OutOfSync, argocd_app_info=1
02:17:07 selfHeal sync applies HEAD -> spec.replicas=12
02:17:09 Synced
Time-to-heal is 3 minutes with no human. With selfHeal off, the same OutOfSync pages on-call, who runs argocd app diff checkout to see the exact field, then argocd app sync checkout. If git itself is wrong, the repair is git revert plus sync, or argocd app rollback <history-id>: still a write from git history, so still allowed.
Failure modes.
- Auto-heal reverts an in-flight mitigation. If someone scaled to 4 at 02:14 to shed load, selfHeal puts it back to 12 three minutes later and the incident gets worse. Pausing auto-sync (
argocd app set checkout --sync-policy none) is the escape hatch, but it opens a drift window: require an incident ticket and a time bound, and alert if the app is still unmanaged after 30 minutes. - HPA fights git over
spec.replicas. Both write the field, so the app flaps OutOfSync forever. Dropreplicasfrom the manifest, or addignoreDifferenceson/spec/replicas. - Mutating webhooks inject fields (sidecars,
securityContext) that git never wrote, producing permanent diff noise. UseignoreDifferencesor server-side apply so field ownership is explicit. - Drift in a field no one owns is invisible. If a controller writes
statusor an operator patches a CR, the diff won't show it; that's a job for audit-log alerting, not the reconciler.
Prevention. RBAC that denies update/patch on workload kinds to human roles, with break-glass as a short-lived binding that requires a PR within the hour. Then measure: drift events per week, time-to-heal p50/p99, and the count of production writes that succeeded outside break-glass. That last number should be zero.
Curated: · Written: · Reviewed:
Q&A-55How do you change a production table without a long blocking lock on checkout or making old binaries crash?
My approach to production schema migrations starts from the constraint rather than from the technique.
Expand-contract in separate releases so every running binary can read both shapes, and keep the expand step's unavoidable lock acquisition short and bounded.
Concretely, use expand-contract across separate releases; verify the database engine’s exact DDL behavior, set short lock timeouts, use online migration tooling where needed, backfill in throttled chunks, test a mixed-version fleet, and contract only after old readers and writers are gone.
The reason for that specificity is a failure I have seen: An in-place ALTER with a volatile default rewrote the checkout table while holding an ACCESS EXCLUSIVE lock, blocking 2.1 million writes over 14 minutes.
Expand-contract on orders.tax_cents, 2.1e6 rows, bounded lock and no table rewrite.
PR1: ALTER ADD COLUMN tax_cents INT NULL; # short ACCESS EXCLUSIVE lock, no rewrite
PR2: backfill 50k rows / 2s, 84 min, lag SLO 120s
PR3: binaries write tax_cents; mixed fleet 24 h
PR4: ALTER DROP COLUMN tax_legacy; # after 0 old binaries
# PostgreSQL 11+: constant DEFAULT is metadata-only; a volatile DEFAULT still rewrites rows.
# Any ALTER can wait on or block behind its ACCESS EXCLUSIVE lock, so set lock_timeout.
I would not consider it settled without evidence: Show the four PRs, the backfill rate, and that old and new binaries passed a mixed-fleet test for 24 hours.
A migration that requires a maintenance window is an admission that the change was not sequenced.
Curated: · Written: · Reviewed:
Q&A-53Retail wants a freeze from Black Friday through Cyber Monday. How do you freeze without blocking incident response?
Freeze the merge queue, not the deploy pipeline. Routine change stops for the four days; the rollback path and the security path stay open, pre-approved, and rehearsed before the freeze starts.
The choke point matters. Closing the merge queue catches every service at once and leaves the deploy pipeline itself untouched, so a rollback is still an ordinary, boring deploy. Branch protection alone does not do this: a service with a direct-push exception or a hotfix branch walks around it. And the merge queue is not the only road into production. Config-service flips, feature-flag toggles, migration jobs, IaC applies, secret rotations and vendor consoles (CDN, WAF, payment provider) all change production without a PR. Each one needs its own gate or it is the hole in the freeze.
Black Friday freeze policy, 4 days, 99.9% SLO still in force.
| Change type | Allowed? | Gate |
|---|---|---|
| Feature merge | no | merge queue closed |
| Config tweak / flag flip | no | config service locked to read-only |
| Migration job | no | job disabled in scheduler |
| Rollback to last green | yes | automated, no extra approval |
| Sev-1 security patch | yes | freeze-exception label + 2 approvers |
| Bad canary at 5% | rollback | 6 min, not a VP ticket |
The failure this guards against is a hard freeze that also blocked rollbacks. A bad canary at 5% of traffic turned a 6-minute revert into a 4-hour wait for a VP exception, during the highest-revenue window of the year. The freeze did not prevent the incident; it extended it. So the rollback job is exempt from the freeze, and it is exercised on day one rather than trusted on paper.
Detection, not just policy: alert on any deploy event outside the allowlist, and review the deploy audit log each morning of the freeze. A freeze nobody can see being violated is a suggestion.
Before the window opens, three things get demonstrated, not asserted. A rollback runs in staging end to end and the timing is recorded. The exception approver list is published with names and a paging path, so nobody is hunting for a VP at 2 a.m. A normal feature PR is pushed at the closed queue and rejected, proving the gate is live rather than configured-but-bypassed.
Two things I would flag as assumptions. First, that the rollback target is genuinely the last green build, if the previous release is also bad, the exception path is the only exit, so keep the approver list short and reachable. Second, that the freeze covers the payment and CDN consoles, which are the changes most likely to be made by someone who never sees the merge queue.
A freeze that cannot roll back is a vow to sit in any production fire you already started.
Curated: · Written: · Reviewed:
Q&A-64Is a 02:00-04:00 change window safer than shipping at 14:00 with a full staff?
The useful framing here is to ask what evidence would change my mind about change windows.
Prefer changes when detection and rollback staffing are highest, and use windows only to avoid known peak traffic, not to hide risk in the night.
Concretely, schedule risky changes in staffed hours below the traffic peak, require on-call plus a second engineer, and reserve night windows for the rare change that cannot overlap peak.
The reason for that specificity is a failure I have seen: A 03:10 schema apply with one sleepy primary took 47 minutes to notice a lock; the same change at 14:00 with two people rolled back in 6 minutes in rehearsal.
Same schema apply, two clocks.
| When | Staff | Traffic | MTTD | Rollback |
|---|---|---|---|---|
| 03:10 | 1 primary | 12% of peak | 47 min | 19 min |
| 14:00 | primary + SWE | 70% of peak | 3 min | 6 min |
| 19:00 peak | full | 100% | — | forbidden by policy |
I would not consider it settled without evidence: Compare MTTD and rollback time for staffed-hour versus night changes over a quarter.
A quiet graph at 03:00 is not safety; it is a missing audience for the alarm.
Curated: · Written: · Reviewed:
Q&A-60How do you drain a node in production without a user-visible blip?
The 90-second answer
Cordon, confirm the pods have somewhere to go, then evict through the Eviction API one node at a time, and abort the drain if the PDB returns 429 or the latency SLO starts burning. Never --force, never --grace-period=0, never --disable-eviction.
The blip is almost never the pod dying. It is the endpoint still pointing at a pod that has stopped accepting connections, or the capacity dip while replacements warm up. Draining faster fixes neither.
What actually happens, in order
kubectl cordon node-7setsspec.unschedulable=true. Nothing moves yet.kubectl draincalls the eviction API (POST /api/v1/namespaces/{ns}/pods/{name}/eviction). The API server checks the PDB:disruptionsAllowed = currentHealthy - desiredHealthy. At zero you get429 TooManyRequestsand kubectl retries until--timeoutexpires.- On success the pod gets a
deletionTimestampand the endpoints controller removes it from the EndpointSlice. preStopruns, then SIGTERM. The app stops accepting, finishes in-flight work, exits. AtterminationGracePeriodSecondsthe kubelet sends SIGKILL.- The ReplicaSet or HPA creates a replacement.
Steps 3 and 4 race. kube-proxy programs rules on endpoint change events, with a full resync every 30s as a backstop, but a cloud LB deregisters on its own schedule: an ALB target group's deregistration_delay.timeout_seconds defaults to 300s. That gap is where connection resets come from, and it is why preStop exists.
Trace: 12 checkout pods, PDB minAvailable: 10, node-7 holds 2 (worked example, not a benchmark)
t=0 kubectl cordon node-7
t=0 headroom check: HPA min 12 / max 24, CPU 45% vs 70% target -> can scale
t=~60s HPA adds 2 pods; image cached, readiness probe 5s period -> Ready
t=90s 14 healthy (12 + 2 replacements)
t=95s kubectl drain node-7 --ignore-daemonsets --timeout=8m
PDB: currentHealthy=14, desired=10, disruptionsAllowed=4
evict pod A -> 200; evict pod B -> 200
t=95s preStop sleep 10s (covers endpoint and LB propagation)
t=105s SIGTERM; app drains in-flight (p99 400ms, longest 3s)
t=108s both pods gone; 12 serving on 5 nodes
forced_evictions: 0 checkout 5xx: 0 p99: 41ms -> 44ms
terminationGracePeriodSeconds: 45 covers preStop (10s) + shutdown (30s) + margin. With grace at 30s and preStop at 10s the app gets 20s, so a 25s batch job is SIGKILLed mid-write.
Failure modes
| Failure | Symptom | Detection |
|---|---|---|
minAvailable == replicas | disruptionsAllowed: 0, drain hangs, someone force-deletes | kubectl get pdb DISRUPTIONS column |
--force --grace-period=0 | SIGKILL; in-flight requests cut mid-TCP | forced-eviction counter, 5xx at drain time |
| preStop < LB deregistration | TCP RST or 502 at the LB, pod logs clean | LB 5xx, not pod logs |
| No headroom | remaining pods saturate, p99 climbs | CPU throttling, SLO burn rate |
| Local PV or emptyDir | data loss | drain refuses without --delete-emptydir-data |
| Draining during a rollout | rollout and drain both spend the PDB budget | overlapping ReplicaSets |
When this doesn't work
A single-replica or leader-elected service has no PDB budget to spend: minAvailable: 1 on one replica blocks every drain. You either accept a blip or drain via failover: promote a replica, then drain the old leader. Say that out loud rather than pretending the sequence is universal. Also note the PDB only covers voluntary disruptions; node failure, kubelet eviction and a direct kubectl delete pod bypass it entirely.
Evidence I'd keep
Per drain: duration, forced_evictions (target 0), eviction-API 429 count, and SLI burn rate over the drain window. A drain that took 4 minutes with zero forced evictions and flat p99 is a pass; one that finished in 20 seconds is a red flag.
Curated: · Written: · Reviewed:
Q&A-69A feature flag and a JSON config both change behaviour. How do you roll config to 5% of users rather than 5% of pods?
Assume the config is read in the request path (timeouts, retry budgets, feature toggles) and that sessions are sticky to pods. Under those assumptions, "5% of users" means hashing a stable user identifier into a bucket and evaluating the config per request from a central store. It does not mean shipping the config to 5% of pods. Pod-level rollout is the right tool only when the config is binary-incompatible, such as a new wire format, a schema version, or a sidecar image. Then a pod genuinely cannot serve both variants, and you drain and replace.
Pick a key that is stable for the user across sessions and devices: an account id, not a session id or a pod name. Hash salt + user_id into a bucket, compare the bucket against the rollout percentage, and look up the config version assigned to that bucket. The salt is per config version, so re-rolling the same flag does not hit the same unlucky users. Evaluate per request against a local cache of the central store with a short TTL. Per-session caching is cheaper, but a user who logs in mid-rollout keeps the old config until the session ends.
Here is a worked example with illustrative numbers. 200 pods, 10,000 daily active users, sticky sessions, roughly 50 users pinned per pod. The change is a request timeout from 2s to 800ms, and the backend's p99 is 1.2s, so every request on the new timeout fails.
With a pod-level 5% rollout, 10 pods get the new timeout. Those 10 pods carry about 500 sticky users, all of whom now fail 100% of the affected requests. The fleet-wide error rate is 500/10,000 = 5%. A gate set at "fleet error rate < 6%" passes while 500 users are fully down.
With a cohort-level 5% rollout, the hash selects 500 users. The same 500 users fail, but the cohort SLI, error rate measured only over users on the new config, reads about 100%, and a gate at 2% aborts on the first evaluation window. Rollback is flipping the percentage or the salt; the next request re-evaluates and the user is back on the old config with no pod restart.
| Rollout unit | What 5% selects | What affected users see | What the gate measures |
|---|---|---|---|
| 5% of pods | 10 of 200 pods | about 500 sticky users fully on the new config, 100% error on the affected path | fleet-wide error rate, diluted to about 5% |
| 5% of user ids | 500 of 10,000 users | those 500 users fully on the new config, 100% error on the affected path | error rate within the cohort, about 100% |
| rollback by cohort | flip the percentage or the salt | affected users return to the old config on their next request | no pod restart, no drain |
Four failure modes matter here. Sticky sessions plus pod-level rollout produce the dilution above; detect it by comparing the error rate of users on the new config against the fleet rate, because a healthy fleet alongside a fully-down subset means the gate is measuring the wrong population. An unstable hash key, such as a session id or a rotating device id, flips users in and out of the cohort and turns the cohort SLI into noise; detect it by tracking cohort membership churn between windows. Salt reuse puts the same users in every experiment; detect it by measuring cohort overlap across flags. Cache staleness holds the old config for the SDK's TTL, so rollback is not instant; detect it by measuring the time from flipping the percentage to the cohort SLI moving.
Before calling it done I would want cohort-level SLIs broken out by config version, and a demonstrated rollback where a user hashed into treatment returns to the old config without a pod restart.
Curated: · Written: · Reviewed:
Q&A-62Developers want to mount all secrets as environment variables. What do you inject, and how?
What I'd inject
Not all of them, and not as env. I'd inject the minimum set the process needs, as files on a tmpfs volume, fetched at pod start by the workload's own identity, with a short lease.
Env is the wrong transport for three reasons that show up in real incidents. It's inherited by every child process, so a curl in a debug script or a spawned shell carries the credential with it. It's readable at /proc/<pid>/environ by anything running as the same UID. And it lands verbatim in core dumps and in the config dumps many frameworks print at startup. A file on tmpfs has none of those properties, and it can be rewritten in place when the lease renews.
So the injected set is: a projected, audience-scoped service account token (that's identity, not a secret), the secret files themselves under a read-only mount, and nothing else.
How
Two mechanisms, same shape.
Secrets Store CSI Driver (secrets-store.csi.k8s.io, v1.x):
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: payments-db
spec:
provider: vault
rotationPollInterval: 2m
parameters:
vaultAddress: "https://vault.internal:8200"
roleName: "payments-api"
objects: |
- objectName: "db-password"
secretPath: "database/creds/payments-api"
secretKey: "password"
---
spec:
containers:
- name: api
volumeMounts:
- name: secrets
mountPath: /var/run/secrets/payments
readOnly: true
volumes:
- name: secrets
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: payments-db
The driver mounts the volume as tmpfs and re-fetches on rotationPollInterval (default 2m). Do not define secretObjects: that syncs the value into a Kubernetes Secret, which is exactly the object someone then mounts as env.
Vault Agent Injector is the other option: a mutating webhook adds an init container and a sidecar that render templates into /vault/secrets/ (also tmpfs), authenticated with the pod's service account token against a Vault role.
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "payments-api"
vault.hashicorp.com/agent-inject-secret-db: "database/creds/payments-api"
Rotation is the point
A Vault database role configured with default_ttl=1h and max_ttl=24h means a leaked credential is dead within the hour. A static password in a Kubernetes Secret lives until someone rotates it, which in practice is never. That difference is the whole argument for files over env: env can't be rotated without a restart, so teams don't.
The catch: the app has to re-read the file. If it reads once at boot and caches, rotation is cosmetic. Detect that by comparing the file mtime to the process start time, or by rotating a canary and asserting a health endpoint reports the new value.
Enforcement
Block env-based injection at admission. ValidatingAdmissionPolicy (GA in Kubernetes 1.30) with CEL:
validations:
- expression: |
!object.spec.containers.exists(c,
has(c.env) && c.env.exists(e,
has(e.valueFrom) && has(e.valueFrom.secretKeyRef)))
message: "Secrets must be mounted as files, not injected as env."
Bind it with a ValidatingAdmissionPolicyBinding, or use Kyverno or Gatekeeper if you already run one. Extend the expression to initContainers and ephemeralContainers. Keep a documented exception list, some third-party charts only accept env, and make the exceptions expire.
Failure modes
- App caches the file at startup, so rotation does nothing. Detect via mtime versus process start time.
secretObjectspresent in the SecretProviderClass, so a Kubernetes Secret exists and gets mounted as env. Audit for it.- Over-scoped Vault role, so the sidecar can read paths the app never needs. Scope the role to exact paths.
kubectl debugephemeral container withshareProcessNamespace: truecan read the tmpfs mount. Keep it false and gate ephemeral containers with RBAC.
Evidence
Scan running pods for env[].valueFrom.secretKeyRef: outside the exception list the count should be zero. Scan the repo with gitleaks. And show the Vault audit log line where the pod's identity fetched a lease, with the lease TTL on it.
Curated: · Written: · Reviewed:
Q&A-63A Terraform apply in the "shared" stack took down billing and search. How do you bound blast radius?
Assume the shared stack holds the VPC, its route tables, and the IAM that billing and search both consume. That is the blast radius: a single terraform apply walks one dependency graph, takes one state lock, and can mutate every resource address in that state file. Bounding it means bounding the state, then bounding what a plan is allowed to contain.
Split state along failure domains, not team boundaries. The VPC and its core route tables stay in a small, rarely-applied state. Per-service subnets, route table associations, security groups, and IAM move into per-service states that read the VPC id from a data source rather than owning it.
| State | Owns | Apply role | Worst case from a bad CIDR |
|---|---|---|---|
| net-core | VPC, core route tables | net-admin | whole VPC |
| search-prod | search subnets, SGs, IAM | search-deploy | search only |
| billing-prod | billing subnets, SGs, IAM | billing-deploy | billing only |
Two things make the split real rather than cosmetic. Each state gets its own apply credential, an IAM role whose policy is scoped to that stack's resource ARNs, so a bad plan cannot reach billing even if someone writes the address. And cross-stack reads go through terraform_remote_state or SSM parameters, which are read-only; there is no write path from search's state into billing's.
Then make the complete plan the normal change. A targeted apply (-target) refreshes only part of the graph and leaves the rest of the state stale, so it is reserved for documented break-glass recovery. Every normal change applies the full plan for its state.
Enforce the boundary in CI. terraform show -json plan.tfplan emits resource_changes with each address; a check asserts every address matches the allowed module prefix for that state and fails the pipeline otherwise. That is the evidence that a plan for search cannot include billing resources.
A worked example, figures hypothetical: a one-character CIDR typo, 10.20.0.0/16 written as 10.2.0.0/16, in a shared route table replaced 14 route entries in one apply. Nine services lost their database routes for 23 minutes until rollback. With the split above, the same typo lands in search-prod, the plan diff shows one route table, and billing never enters the graph.
Failure modes worth naming: over-splitting into one state per resource makes ordering unmanageable, so split at failure-domain boundaries and no finer. Cross-state reads create an apply-order dependency, so net-core must apply before its dependents. A scheduled terraform plan -refresh-only catches drift when someone edits by hand. And a shared state is a blast radius you reaffirm on every apply, which is why the split is the fix and not the review process around it.
Curated: · Written: · Reviewed:
Q&A-91A whole availability zone disappears. What must already be true for checkout to stay inside SLO?
My approach to AZ outage design starts from the constraint rather than from the technique.
Run N-1 capacity across AZs, keep stateful dependencies multi-AZ, and automatically drain the dead AZ rather than waiting for DNS to guess.
Concretely, provision enough per-AZ capacity that the surviving zones carry peak traffic at the chosen headroom target; at 250 rps per pod and a 1,200-rps peak with a 70% utilization ceiling, at least seven pods must survive, so an even three-AZ layout needs four pods per AZ.
The reason for that specificity is a failure I have seen: Two-AZ deploy at 50/50 left 100% of remaining capacity at 100% utilization when AZ-b died; p99 went to 3.4 s and the SLO burned in 9 minutes.
Peak 1,200 rps, 250 rps/pod, three AZs.
| Layout | Pods per AZ | After one AZ loss | p99 |
|---|---|---|---|
| 2 AZ, 3+3 | 3 | 3 pods = 750 rps cap | 3.4 s |
| 3 AZ, 4+4+4 | 4 | 8 pods = 2,000 rps cap, 60% utilized | inside target |
| 3 AZ, 2+2+2 | 2 | 4 pods = 1,000 rps cap | SLO miss, undersized |
I would not consider it settled without evidence: In an AZ-loss game day, show remaining utilization ≤ 70%, SLI inside SLO, and no manual DNS edit.
Multi-AZ that is not N-1 is a hope that the surviving zone was bored.
Curated: · Written: · Reviewed:
Q&A-90All private-subnet egress shares one NAT. What fails when that NAT or its AZ fails?
Assume AWS, private subnets whose route tables send 0.0.0.0/0 to a single NAT gateway in AZ-a, and no VPC endpoints covering the critical paths. Under those assumptions, here is what breaks.
The route table entry does not disappear when the AZ fails. It still points at the NAT gateway's elastic network interface in AZ-a, so packets are blackholed rather than rejected with an ICMP unreachable. That distinction decides the symptom. Established TCP connections do not fail fast; they stall and retransmit until the kernel gives up, which is minutes, not seconds. New connections fail with connect timeouts. From the application's side it looks like a hung dependency, not a clean outage, which is why the first instinct is usually to blame the downstream service.
What stops working is every outbound call from every private subnet that routes through that NAT, in all AZs, not just AZ-a. That covers package and OS updates, container image pulls from public registries, calls to third-party APIs such as payments, auth and SaaS, external health checks, and the SSM and CloudWatch Logs agents if you have not given them interface endpoints. DNS is the subtle one: the VPC resolver at the .2 address does not need the NAT, but if you run a custom forwarder in a private subnet that egresses through it, name resolution breaks too, and that failure looks different from the rest.
What keeps working is anything inside the VPC, inbound traffic, and any path covered by a gateway or interface endpoint: S3 and DynamoDB gateway endpoints, interface endpoints for ECR, SSM, Secrets Manager and CloudWatch Logs. That is the lever. If your critical egress is to AWS services, endpoints take the NAT out of the path entirely.
One thing worth separating from the AZ failure: a NAT gateway is a managed service that is already redundant inside its AZ, so the failure you plan for is the AZ, not the instance. The capacity failure is different again. A NAT gateway supports roughly 55,000 simultaneous connections to a single destination, where a destination is a unique IP address, port and protocol. Past that you get port allocation errors, surfaced as the ErrorPortAllocation metric. One NAT shared by every subnet concentrates all of that on one gateway; per-AZ NATs split it.
| Design | AZ-a NAT fails | Blast radius | Approx. list cost |
|---|---|---|---|
| One NAT in AZ-a | all egress blackholed | every AZ | ~$32/mo + $0.045/GB |
| NAT per AZ, local routes | AZ-a egress dead, b/c live | one AZ | 3x NAT hourly |
| Endpoints for AWS services | unaffected | none for those paths | endpoint hourly |
A single NAT is genuinely fine for dev and test environments, non-production, batch workloads that retry, or anywhere losing egress does not take down the product. The deciding criteria are blast radius and recovery time, not best practice. With one NAT, recovery means creating a NAT gateway in another AZ and repointing route tables, a manual change measured in minutes, and it is the same change you would be making under pressure at 3am.
Detect it with a synthetic canary from each AZ's private subnet to an external endpoint, plus alarms on ErrorPortAllocation, ActiveConnectionCount and PacketsDropCount. Verify by failing the NAT in a drill and confirming the other AZs still egress.
Curated: · Written: · Reviewed:
Q&A-85A pod dies and its sticky users error until their cookie expires. How should affinity behave on failure?
The first thing to settle about sticky sessions after failure is what decision it actually changes.
Stickiness must die with the backend; failing pods must drop affinity so users are hashed onto a live replica immediately.
Concretely, use balancer affinity that is ignored when the target is unhealthy, set a short cookie TTL as a backstop, and prefer stateless tokens over pod-sticky sessions.
The reason for that specificity is a failure I have seen: A 12-hour cookie pinned 8% of users to a terminating pod; they 502'd for 11 minutes while the other 92% were fine.
Cookie TTL 12 h versus health-aware affinity.
| Policy | Pod killed | Stuck users | Recover |
|---|---|---|---|
| cookie 12 h, ignore health | 8% of traffic | 11 min 502 | cookie expiry |
| cookie 12 h, drop if unhealthy | 8% | 0 | next request, 80 ms |
| stateless JWT, no stickiness | 0% stuck | 0 | any replica |
I would not consider it settled without evidence: Kill a pod under sticky load and show those users succeed on another replica within one retry, not at cookie expiry.
Affinity to a corpse is not session continuity, it is a private outage.
Curated: · Written: · Reviewed:
Q&A-89p99 climbed and CPU is idle. How do you tell a full disk path from a full NIC, and why does it matter?
The discriminator is which queue is full, and two numbers tell you: disk await and queue depth on one side, NIC drops and TCP retransmits on the other. CPU is idle in both cases because the application threads are blocked waiting on I/O, not runnable: the scheduler has nothing to run, so CPU utilization says nothing about where the stall is.
Disk path. A read enters the block layer and queues behind whatever is already in flight. await is queue wait plus service time, so as queue depth grows, await grows with it: Little's law, in-flight = throughput × latency. Throughput plateaus at the device's IOPS or bandwidth ceiling while await keeps climbing, and p99 is just the requests that sat at the back of that queue. On a gp3 volume the baseline is 3,000 IOPS and 125 MiB/s. If your requests are 16 KiB random reads, 3,000 IOPS is only about 47 MiB/s, so you hit the IOPS ceiling long before the bandwidth one. At 3,000 IOPS and 210 ms await, in-flight = 3,000 × 0.210 ≈ 630 requests queued. That is a full disk path.
NIC path. The link is a fixed-rate serial resource. Once offered load exceeds line rate, the NIC or the switch drops, and TCP answers loss with retransmission and congestion-window collapse. The tail latency comes from the retransmit timer: Linux's minimum RTO is 200 ms, so one lost segment adds 200 ms to that request, and it lands squarely in p99. You see it as drops on the interface, a rising retransmit rate, and a link pinned near line rate, not as high await.
| Signal | Disk-bound | NIC-bound |
|---|---|---|
| CPU | idle | idle |
| disk await | hundreds of ms, rising with load | single-digit ms |
| disk throughput | flat at IOPS/bandwidth cap | normal |
| NIC utilization | normal | near line rate |
| NIC drops / TCP retransmits | ~0 | climbing |
| app threads | blocked in D state | blocked in socket read |
Why it matters: the fixes are opposite, and one of them makes things worse. If the volume is IOPS-bound, adding replicas multiplies the random read rate against a ceiling that does not move: more queue depth, worse p99. That is the failure I have seen: an HPA scaled 20 pods on idle CPU during a disk storm, and await went 18 ms → 210 ms. Replicas cannot manufacture IOPS the volume does not have, and they cannot manufacture NIC credits either. Disk-bound means more IOPS: larger volume, striping, caching, coalescing chatty queries. NIC-bound means more bandwidth: bigger instance or an extra ENI, compression, batching, fewer round trips.
Confirm before acting: check whether await scales with concurrency while throughput stays flat (disk), or whether drops and retransmits track the p99 spike (NIC). If both are clean, the stall is elsewhere (a lock, a connection pool, a downstream dependency) and neither fix applies.
Curated: · Written: · Reviewed:
Q&A-84The load balancer thinks a pod is healthy while /ready is false. What is the check wrong about?
The check is answering the wrong question. It reports "this instance is alive and accepting connections" when the routing decision needs "this instance can serve a new request right now." /ready is the app's own statement that it cannot. When the two disagree, the check is not observing that predicate at all, and there are three different things it might be observing instead. Name which one before you touch anything, because the fix differs each time.
1. Wrong predicate: the balancer never reads /ready. A TCP connect on the serving port only proves the kernel's accept queue is open. A GET /live only proves a handler ran. Neither knows about an exhausted DB pool, a full worker queue, or a pod still warming a cache. This is the most common cause, and it is often not even a check-design problem: it is a wiring problem. In Kubernetes, /ready only matters if something runs it:
readinessProbe:
httpGet: { path: /ready, port: 8000 }
periodSeconds: 10 # default
failureThreshold: 3 # default -> up to 30 s before PodReady flips
Omit that block, point it at /healthz, or aim it at the admin port, and kubelet never reads /ready. The pod stays Ready forever and everything downstream keeps forwarding.
2. Right predicate, lagging. The check is correct but the disagreement is in propagation and draining:
- kubelet probes every 10 s and needs 3 consecutive failures (defaults) to set PodReady=False: worst case ~30 s, plus up to timeoutSeconds=1 per probe.
- The EndpointSlice controller drops the pod from conditions.ready; kube-proxy re-syncs rules off the watch, normally well under a second.
- A cloud target group is an independent source of truth. AWS ALB health checks do not consult Kubernetes readiness, and its default deregistration delay is 300 s.
So "LB healthy, /ready false" is the expected state for up to a minute even when everything is wired correctly.
3. Right predicate, wrong target or wrong response contract. The probe passes through a sidecar, an ingress with response caching, or a node-local hop that answers on the pod's behalf. Or /ready returns 200 with {"ready": false} in the body while the check treats any 2xx/3xx as healthy: the contract lives in the status code and the check is reading the wrong field. /ready must return 503 when unready; anything else is a silent divergence.
| Probe | What it actually observes | Worker deadlocked, DB pool empty |
|---|---|---|
| TCP connect :8000 | accept queue open | healthy. Divergence |
| GET /live | a handler got scheduled | healthy: divergence |
| GET /ready (503 contract) | the app's own serving predicate | unhealthy, removed |
Concrete failure this produces: a stuck worker pool holds connections open and never answers. A TCP check keeps all such pods in rotation; users see p99 climb into the seconds while every dashboard says healthy. The deadlocked pods accept, then hang until the client times out.
Fixing the divergence creates a second failure mode worth naming: if /ready includes a shared dependency, that dependency's outage marks every pod unready at once and the balancer drains the whole fleet. Keep readiness to what makes this instance uniquely unable to serve, and test shared-dependency outages against the intended fallback. Also remember that established connections (HTTP keep-alive, conntrack entries) keep flowing to a removed pod until they close or drain.
To confirm rather than assume: verify /ready returns 503, verify the balancer maps 503 to unhealthy, then block the dependency port inside one staging pod and time from the first 503 to that backend disappearing from the pool. If that number exceeds the SLO burn budget, cause 2 is your answer and the fix is interval and threshold, not the handler.
Curated: · Written: · Reviewed:
Q&A-87Certificates look expired on one node and tokens fail randomly. What clock discipline does production need?
What production needs: Bound clock skew across the fleet, monitor it, and page when it exceeds a threshold. For most systems, 100 ms is a reasonable page threshold. Use monotonic clocks for timeouts. For fencing tokens, use a logical epoch, not wall clock. Treat NTP failure as a paging condition: a node that cannot sync should be drained.
Why: TLS certificate validation and token expiry both compare wall-clock timestamps. If a node's clock is 12 minutes ahead, a certificate issued 5 minutes ago appears not-yet-valid, and a token with a 5-minute TTL appears expired. I've seen a VM with 12 minutes of drift reject TLS as not-yet-valid and serve 5xx to 6% of users hashed onto that node for 2 hours. That's a Sev-2, not a curiosity.
Alternatives and trade-offs:
- chrony vs ntpd: chrony converges faster, handles virtualized clocks and intermittent connectivity better, and is the default on RHEL 8+. ntpd is older, still maintained, but slower to correct large offsets. systemd-timesyncd is lightweight but less accurate; fine for workstations, not for servers that need tight bounds.
- Step vs slew: For large offsets (>1s), chrony can step (jump) or slew (gradually adjust). Stepping is faster but can break applications that assume time moves forward. Slewing avoids jumps but takes time: a 1-second offset at 500 ppm takes ~2000 seconds. For TLS and tokens, step after draining the node; otherwise slew.
- Leap smear vs step: A leap second can cause a 1-second step. Some operators smear it over a window (e.g., Google smears over 24 hours) to avoid jumps; others step. Smearing avoids application breakage but introduces a temporary error up to 1 second. If you need sub-second accuracy, account for it.
- Logical epochs vs synced wall clock: For fencing tokens, a wall-clock timestamp is unsafe: a node with a fast clock can issue a token that appears newer. Use a logical epoch from a consensus system (e.g., Raft term) or a monotonic counter. If you must use wall clock, bound skew tightly (e.g., 10 ms) and use a lease protocol that accounts for uncertainty.
When the 100 ms threshold changes:
- Tighter (10–50 ms): distributed databases with lease-based leader election (Spanner, CockroachDB) or financial trading. These often use TrueTime or hybrid logical clocks.
- Looser (1–5 s): services that only use time for logging and coarse scheduling, and tokens with long expiry. Kerberos tolerates 5 minutes of skew by default, so 100 ms is conservative for auth.
- If you use leap smearing, adjust the threshold because the smear introduces a known error.
Failure modes and detection:
- TLS not-yet-valid: Node 12 minutes ahead rejects a certificate whose notBefore is 5 minutes in the past. Detect: monitor certificate validation errors per node; alert on spikes.
- Token expiry: Token issued at T with 300s TTL is rejected immediately if node clock is 200s ahead. Detect: track token rejection rates by node.
- Fencing split-brain: Two nodes both believe they hold the lease because clocks disagree. Detect: use a consensus system that doesn't rely on wall clock, or monitor for dual leaders.
Artefacts:
| Node | NTP offset | Symptom | Action |
|---|---|---|---|
| ip-a | 4 ms | none | — |
| ip-b | 12 min | TLS not-yet-valid on new certs | page, drain |
| ip-c | 200 ms | none | page, investigate |
Trace of token failure:
- Token issued at wall-clock T=1000, TTL=300s, expires at T=1300.
- Node clock is 200s ahead, so it reads T=1200 when token is issued.
- Token appears to expire at 1300, but node's clock reaches 1300 after only 100s of real time.
- Result: token rejected after 100s instead of 300s.
What I'd verify: Dashboard max |offset| across nodes, page at 100 ms. Game-day: skew a node by 10 minutes and watch it leave the pool. Run chronyc tracking to see offset and chronyc sources to see sync status.
Curated: · Written: · Reviewed:
Q&A-96A TLS cert expired at 00:01 and took login down. How do you make expiry a non-event?
The 90-second answer. Expiry stops being an event when renewal is continuous and the alert watches the certificate clients actually receive. Three things make that true: lifetimes short enough that renewal runs constantly (90 days, or 30), renewal with no human step in it, and an external probe that reads the served notAfter and pages at 21 and 7 days. Miss any one and you are a calendar reminder away from 00:01.
Assumptions: you terminate TLS at your own edge or a cloud LB, you control DNS for the names, and you can run a synthetic from outside the VPC.
The failure I have seen is not a missed renewal. It is a renewal that landed on disk while the load balancer kept presenting the old cert:
today: 2026-09-07
disk: notAfter = 2026-10-17 (40 days) # renewed last night
served: notAfter = 2026-09-07T00:01:00Z # LB still holds the old cert
probe: served days_left = 0 -> page
policy: page if served days_left < 21
without the probe: login 100% down at 00:01, dashboards green
ACME controller vs managed certs. The deciding question is where TLS terminates and who can write DNS, not which tool is nicer.
- cert-manager / lego / Caddy (self-managed ACME). Private key stays in your cluster as a Secret; you grant the controller DNS-01 write scope or an HTTP-01 ingress path.
renewBeforedefaults to two-thirds of the lifetime, so a 90-day cert re-issues around day 60. You own the CRDs, the upgrade path, and the failure modes. Wrong when you cannot give a controller DNS write access, or when the cert terminates on a vendor LB you do not control. - Managed (ACM, Cloudflare, GCP). ACM is free for certs attached to ALB/NLB/CloudFront, renews automatically starting 60 days out, and will not let you export the private key, so it is wrong for mTLS to your own services or for TLS terminating on your own nginx outside AWS. Cloudflare Universal SSL is free on proxied hostnames but only covers edge-to-client; origin certs are a separate problem.
- Long-lived certs, manual renewal. If you genuinely cannot automate (appliance, third party, air-gapped), the non-event is a long lifetime plus a monitored ticket. Short lifetimes make manual renewal strictly worse.
Costs are asymmetric: self-managed buys you key control and costs you a controller, DNS credentials with write scope, and its on-call. Managed buys you zero maintenance and costs you lock-in and no key export. Keep a second issuer as fallback: Let's Encrypt's published limits (50 certs per registered domain per week, 5 duplicate certs per week) are reachable during a mass re-issue after a key compromise.
Failure modes and how you catch them.
- Renewal writes disk, LB serves stale (trace above). Detect: probe served
notAfter, never the file. - DNS-01 token expired, renewal fails silently. Detect: alert on the renewal job's exit status, not just days-left. Days-left alone gives you three weeks of silence then a page.
- HSTS with
max-age=31536000means browsers will not let users click through the warning; expiry is a hard outage, not a warning. Detect: grep the config. - Clock skew on the host drifts both
notAftercomparisons and renewal schedules. Detect: NTP offset in the same probe. - One wildcard shared across forty services turns one renewal failure into forty outages. Detect: an inventory mapping cert to services.
What I would want as evidence. Last renewal timestamp, served days-left from a public probe, and a game day where you serve a one-day cert and confirm the page fires.
The certificate that matters is the one the client sees, not the one on disk.
Curated: · Written: · Reviewed:
Q&A-99Synthetics ping /healthz from inside the VPC and stay green during a DNS outage. Where should probes run?
Where the probes run
Run the synthetic from outside the VPC, on the path a user takes: public DNS resolution, TLS, edge/CDN, load balancer, and a journey that ends in a business outcome. Keep an internal probe, but demote it to a diagnostic splitter: it answers "is the origin healthy", never "can a user reach us".
Why an internal /healthz is the wrong monitor
/healthz is a liveness check for the process. It resolves no name, crosses no edge, terminates no TLS, and passes no WAF rule. Every layer it skips is a layer that can break users while it stays green. DNS is the clearest case: the record, its TTL, the delegation, geo-steering and DNSSEC all live outside the origin, so a probe that never resolves a name cannot observe any of them. A probe that shares fate with the service is a unit test, not a user.
Placement
- Nodes in each market you serve, from at least two independent networks or ASNs. One synthetic vendor's backbone is one more single point of failure.
- Assert on the journey, not the status code: DNS resolve → TCP → TLS handshake → HTTP → checkout returns an order id. A 200 carrying an error page passes a status-code assertion.
- One internal probe per service, same interval, as the splitter. When the external probe fails, the internal one tells you whether the origin is healthy (path problem) or down (service problem).
| Probe | Resolves public DNS | Crosses edge/TLS | Asserts business outcome | Blind to |
|---|---|---|---|---|
| /healthz from VPC | no | no | no | DNS, edge, TLS, WAF, LB |
| TCP connect to VIP from VPC | no | no | no | DNS, edge, TLS |
| External GET / from two ASNs | yes | yes | no | app failures that still return 200 |
| External checkout journey | yes | yes | yes | — |
Worked trace (illustrative numbers)
A deploy rotates the VIP behind the public A record at 09:14; the record's TTL is 300 s.
- 09:14–09:19: resolvers keep serving the old A record.
- Internal /healthz, every 30 s: hits the origin directly, never resolves. 200 throughout.
- External checkout journey, every 30 s: fails at TCP connect to the stale VIP, pages on-call at 09:14:30.
- Users: fail until resolver caches expire, bounded by the TTL plus resolver behaviour.
Detection latency is bounded by the probe interval, so a 30 s interval gives MTTD ≤ 30 s plus one retry; a 5-minute interval gives you a five-minute blind window on a 300 s TTL.
Failure modes to design against
- Stale record after VIP rotation (the TTL trap above): external probe fails at TCP connect, internal stays green. Alert on the external probe; an internal failure alone is a ticket, not a page.
- Probe asserts 200 but the app returns 200 with an error body: assert on content or an order id.
- Test-account or credential expiry makes the probe fail for the wrong reason. Monitor the probe itself with a dead-man's-switch heartbeat.
- Single-homed probe network: an ISP peering problem looks like your outage. Two ASNs separate them.
- Probe runs from a region you don't serve: noise that trains people to ignore the alert.
When internal-only is right
For services with no public path (internal gRPC, queue depth, connection-pool saturation) internal probes are the correct instrument, because there is no user path to walk. External journey probes cost real work: seeded test data, a test account, and cleanup after each run. Scope that work rather than skipping it; the alternative is a green dashboard during an outage users are reporting.
Curated: · Written: · Reviewed:
Q&A-82Backups are green every night. How do you know they restore?
A backup job going green proves the job ran. It says nothing about whether the artifact can be read back. The only evidence that counts is a restore that lands inside RPO and RTO, performed from the same artifacts the monitoring calls healthy, with a result you can check.
The drill. Once a month, restore the latest full plus its incremental chain into an isolated namespace: never over production. Then three checks, not one:
- Clock: time from "go" to the restored database serving reads. Compare against RTO.
- Watermark: before the drill, write a sentinel row with a known timestamp; after restore, confirm it is present and that the newest row is no older than RPO.
- Content: row counts and a checksum on two or three high-value tables, against the source.
A restore that finishes in 9 minutes but is missing the last 40 minutes of writes has failed the RPO check even though the clock passed.
Trade-offs. A monthly full isolated restore is neither free nor always the right cadence.
- Frequency vs cost. A full isolated restore of a 2 TB cluster means provisioning the environment, pulling the chain, and replaying WAL: hours of compute and a namespace someone has to own. Monthly is a floor for a system with a 15-minute RTO; weekly full restores are usually waste. Better: weekly partial (one shard, one tenant, point-in-time to a chosen timestamp) plus quarterly full.
- Full vs partial. Partial restores are cheap and catch logical corruption: a bad migration, a dropped column, a tenant's rows scrambled. They do not prove the backup chain is complete or that the cluster can be rebuilt. You need both, at different cadences.
- Automated vs manual. An automated drill proves the pipeline works. It does not prove a human can do it at 03:00 from the runbook. If the runbook is the incident path, walk it manually twice a year and time it.
- When monthly isolated restore is overkill. A 5 GB Postgres dump that restores in under a minute does not need a monthly ceremony. If the dominant risk is logical corruption rather than infrastructure loss, continuous PITR verification plus nightly row-count reconciliation buys more than a full drill.
The failure mode this catches. 14 months of green jobs hid a mysqldump run without --single-transaction. The dump was internally inconsistent: tables captured at different points in time. The first real restore applied it and failed on foreign keys after 2.1 hours. No exit code from the backup job could have shown that; only reading the artifact back could.
A drill result looks like this:
| Check | Target | Last drill |
|---|---|---|
| restore clock | <= 15 min | 11.4 min |
| watermark vs disaster time | <= 5 min | 3.1 min |
| orders row count vs source | 0 diff | 0 |
What I would ask for as evidence. The last drill's duration, its row-count diff, and the watermark time versus RPO. If those three numbers are not published, the backup is a hope.
Curated: · Written: · Reviewed:
Q&A-104Disaster recovery says RTO is 4 hours and RPO is 15 minutes, but nobody has ever timed a restore. How do you prove those numbers, and what does a green backup job actually tell you?
A green backup job proves exactly one thing: the copy process exited 0 and an artifact of roughly the expected size landed in the bucket. It does not prove the artifact restores, that it restores inside your RTO, or that the application can start against it. Those are three separate claims, and each needs its own evidence.
I'd start by pinning the numbers with the business, because "restorable" is meaningless without them. RPO is how much data you can lose: it sets backup frequency and whether you need WAL/binlog shipping. RTO is how long until service is back: it sets the restore path, not the backup path. A nightly full backup gives you an RPO of up to 24 hours; if the business says 15 minutes, no amount of backup frequency fixes that, you need continuous archiving.
Then measure. A restore drill is: restore the real artifact, into an isolated environment, using the runbook on-call would actually follow, timed end to end. The restore itself is usually a fraction of the RTO. Here's a budget for a claimed 4-hour RTO on a 2 TB Postgres cluster:
| Phase | Measured | Note |
|---|---|---|
| Detect + declare | 25 min | pager to decision |
| Provision scratch env | 40 min | IaC apply, quotas |
| Fetch artifact | 15 min | S3 standard; Glacier bulk is hours |
| Restore base + replay WAL | 2 h 50 min | 2 TB at ~200 MB/s |
| App smoke + validation | 20 min | |
| Cutover (DNS, conn strings) | 10 min | TTL |
| Total | 4 h 40 min | claim is 4 h |
That table is the deliverable. It turns "we have backups" into a number you can defend or fix.
The drill itself should be scripted, not heroic:
set -euo pipefail
# restore latest base + WAL into a scratch path, then verify
pgbackrest --stanza=prod --pg1-path=/srv/drill --delta restore
pg_ctl -D /srv/drill -o "-p 5433" start
psql -p 5433 -d app -c "SELECT count(*) FROM orders WHERE created_at > now() - interval '1 day'"
psql -p 5433 -d app -c "SELECT count(*) FROM orders o LEFT JOIN customers c ON c.id=o.customer_id WHERE c.id IS NULL"
Row counts and a referential-integrity check catch the common case where the restore "succeeds" but the data is truncated or WAL replay stopped early. Compare counts against the source, not against a hardcoded number.
Cadence: an automated restore-and-verify weekly (small, fast, catches artifact rot), a full timed drill quarterly, and a region-failover game day annually if you claim multi-region.
Failure modes worth naming:
- The backup is encrypted with a KMS key in the same account or region that failed. The artifact is intact and useless.
- Logical corruption, a bad migration or a bad DELETE, is faithfully backed up. You need PITR to before the event, and you need to know when the event was.
- Restore time scales with data. An RTO validated at 500 GB is fiction at 2 TB.
- The runbook points at a host that was decommissioned.
- The restore competes with production for IO if you run it in the same cluster.
Detect all of this by alerting on backup age and size deviation, and by tracking drill duration as a metric with a target. A drill that fails should page someone.
Not every dataset needs a 15-minute RPO. Tier by business impact: checkout and payments get PITR and quarterly drills; a log archive gets a nightly dump and an annual test.
Curated: · Written: · Reviewed: —
Q&A-81Product says "we cannot lose more than 5 minutes of orders and we must be up in 15". What do you actually build?
The place where teams go wrong on RPO and RTO is usually the step before the one they are debating.
RPO is the maximum tolerable data-loss interval, while RTO is the maximum tolerable restoration time; replication, backups, and spare capacity must be designed and drilled to meet both.
Concretely, replicate orders off-AZ within 5 minutes worst lag, keep a warm spare that can pass health checks in 15 minutes, monitor lag as an SLI, and refuse a backup-only design that snapshots hourly.
The reason for that specificity is a failure I have seen: Hourly snapshots were sold as 5-minute RPO; a 14:40 primary death lost 41 minutes of orders and 3.2 hours of restore from a cold AMI.
5-minute RPO / 15-minute RTO versus what the old design delivered.
| Design | Worst data loss | Time to first 200 | Meets? |
|---|---|---|---|
| hourly snapshot + cold AMI | 41 min | 3.2 h | no / no |
| async replica, lag SLI 90 s | 90 s | 12 min promote | yes / yes |
| sync cross-region | ~0 | 8 min | yes, + write latency |
I would not consider it settled without evidence: Show max replication lag over 30 days versus 5 minutes, and a drill clock from declared disaster to first successful checkout versus 15 minutes.
An untested RTO is a wish; an RPO shorter than the worst recoverable backup or replication interval is unmet by design.
Curated: · Written: · Reviewed:
Q&A-75Two regions cannot talk for 9 minutes. What does the service still promise, and what does it refuse?
This is one of those areas where the default answer and the right answer for consistency under partition differ.
Choose explicit behavior per journey under partition: preserve invariants by rejecting or fencing writes, or accept divergence only with a defined reconciliation rule; separately document the normal-operation latency/consistency trade-off.
Concretely, for money and inventory, fail writes that cannot reach a quorum; for feeds, serve stale with a freshness SLI; never let both sides accept conflicting unique writes.
The reason for that specificity is a failure I have seen: Both regions accepted "create account" during a 9-minute partition and minted 640 duplicate user ids, then merge was a 6-hour manual dedupe.
9-minute inter-region partition, per-journey choice.
| Journey | During partition | SLO |
|---|---|---|
| checkout pay | 503 unless quorum | availability drops, correctness holds |
| inventory decrement | 503 | no oversell |
| homepage feed | stale, 4 min old | freshness SLI burns, page still 200 |
| create account (old) | both accept | 640 duplicate ids |
I would not consider it settled without evidence: Run a partition test and show which APIs return 503, which serve stale, and that unique ids stay unique.
Hoping both sides stay consistent during a partition is a third, unnamed consistency model, and it loses.
Curated: · Written: · Reviewed:
Q&A-80You want two regions serving writes. What has to be true for that not to split inventory?
Two regions can both accept writes to inventory only if one of three things holds: the write is commutative and mergeable, the invariant is enforced through a single serialization point, or the write never touches a shared counter. If none of those is true, you don't have active-active inventory: you have two writers racing on a number that has to stay non-negative.
State the assumption up front: inventory is a bounded counter with a non-negativity invariant, and "split" means the sum of what both regions believe is available exceeds what physically exists. That is the thing that must not happen.
Why the naive version fails
Take a SKU with 4 units. Region A and Region B each serve a buy request. Both read 4, both decrement to 3, both write 3. Two units sold, stock says 3: one unit oversold. Run that across 1,140 SKUs during a 9-minute link cut and you have oversold 1,140 units. (Worked example, hypothetical numbers.) The failure isn't the partition; it's that last-write-wins on a counter is not a merge. Any read-modify-write across regions has this shape.
The three classes
Classify every write before you flip a second primary:
| Write | Class | During partition |
|---|---|---|
| page-view counter | CRDT (grow-only / PN-counter) | both accept, merge on heal |
| remaining_stock | sticky-primary | only the owning region writes; the other rejects or queues |
| capture payment | coordinated | single serialization point, never both |
| remaining_stock as naive read-modify-write | race | oversell |
A CRDT works because its operations commute and are idempotent, so both regions can accept and merge. That is fine for counters where over-counting is tolerable, like views or likes. It is wrong for stock, because a CRDT counter can over-report availability or go negative.
Sticky-primary gives each key one owning region. Writes to a non-owner are forwarded or rejected. This is the usual answer for inventory. The cost is cross-region latency for the non-owner's users, plus a failover story: what happens when the owner region is unreachable? You need a lease with a bounded TTL, and the standby must not take writes until that lease has provably expired, or you are back to two writers.
Globally coordinated means a consensus round per write: a Raft- or Paxos-backed store, or a single-region serialization point. It is correct, but it pays a cross-region round trip on every write, typically tens of milliseconds, which is usually too slow for a checkout path.
What has to be true
- Every write falls into one of those three classes, and inventory is not in the CRDT class.
- Sticky ownership is enforced with a lease whose TTL bounds the overlap window; the standby cannot accept writes for a key until the lease has expired.
- The non-owner path has a defined behavior under partition (reject, queue, or forward) not "accept locally and reconcile later."
- Payment capture is idempotent with a client-supplied key, so a retry across regions doesn't double-charge.
How I'd verify it
Run a partition drill: cut the link for 10 minutes, drive writes at both regions, and assert that stock never goes negative, no payment is captured twice, and after heal the sum of the regional views equals the true count. Document the class of each write in the schema or the service contract, not in someone's head.
If you can't say which class a write is in, you're not ready for a second primary.
Curated: · Written: · Reviewed:
Q&A-83You flip a DNS record to a spare region. What else has to be true for users to actually move?
Flipping the record is the last step, not the first. For users to actually move, three things have to be true: the spare region can carry the load, the clients will re-resolve and reconnect, and everything in front of them accepts the traffic. DNS only touches the middle one, and only partially.
The spare region has to be ready. If it's a pilot light, the flip sends traffic to a region that still has to scale out: check the autoscaling group's minimum, the instance quota, and the AMI before you flip, because a 10-minute scale-out is your real RTO, not the TTL. State is the harder half: with async replicas, the replication lag at flip time is your data-loss window, and if writes were still landing on the primary in the dead region you have to reconcile them. Session state is the quiet one, if sessions live in an in-region Redis or sticky sessions pin to the old region, users get logged out or stay pinned. Then the boring blockers: TLS certificates are regional resources, so an ALB in the spare region needs its own cert covering the hostname; secrets, parameter-store entries, feature flags, and any partner egress allowlist keyed to the old region's IPs all have to exist in the new one.
Clients have to come. TTL is a floor, not a ceiling. Resolvers clamp it, some carriers enforce a minimum well above what you publish, and clients cache beyond DNS. A JVM caches resolved addresses; the 30-second default only applies when a security manager is installed, otherwise it can be indefinite depending on the runtime, so check yours. And an established TCP connection doesn't care about DNS at all: HTTP/2 and gRPC multiplex over one long-lived connection, so a client with a 10-minute idle keepalive sits on the dead IP for 10 minutes. Your real RTO is max(TTL, resolver clamp, client DNS cache, connection lifetime), not the TTL you set.
The path in has to accept it. Target-group health checks, WAF rules, rate limits, and regional quotas (ENIs, EIPs, NAT) all have to be satisfied in the new region, or traffic arrives and gets dropped.
Trade-offs: active-active avoids the flip entirely but forces you to resolve state conflicts and pay for double capacity; warm standby is fast but drifts; pilot light is cheap but slow. Pick by RTO/RPO and by whether your state can replicate synchronously, if it can't, no DNS trick gets you under the replication lag.
Failure modes and how to catch them: replication lag spike (alert on lag, not on the flip; missing cert in the spare region) external TLS handshake probe; long-lived connections. Histogram of connection age, force reconnect with a deploy or connection draining; resolver clamping. Measure time-to-shift from real user monitors, not dig from your laptop.
| Must be true | DNS flip covers it? | Verify with |
|---|---|---|
| Spare region has capacity | No | load test at target RPS pre-flip |
| Data is current | No | replication lag metric |
| Cert valid for hostname in spare region | No | external TLS probe |
| Clients re-resolve | Partly | RUM time-to-shift |
| Clients drop old connections | No | connection-age histogram |
Worked arithmetic (illustrative): TTL 60 s, carrier clamp 300 s, JVM cache 30 s, gRPC keepalive 10 min → RTO 10 min. The DNS flip bought you nothing until the connections die.
Curated: · Written: · Reviewed:
Q&A-70Cluster A is on fire. How do you fail closed or over to cluster B without a split brain on writes?
Default to failing closed. If you cannot prove that A can no longer commit, stop accepting writes and return errors: a 503 is recoverable, two writers is not. Promotion to B is safe only when two things hold: B got its authority from a quorum that A is not part of, and every write path enforces a monotonically increasing epoch, so a resurrected A is rejected even if it still believes it is primary.
Mechanism. Run a 3-voter consensus group (etcd/Raft, ZooKeeper) with voters in three failure domains. A's write path holds a lease from a majority; the lease has a TTL, so if A is partitioned from the majority it self-fences when the lease expires. To promote, B asks the same quorum for a new lease and gets epoch 14 (A held 13). The storage layer (the replicated log, not the client) rejects any write tagged with an epoch below the highest it has seen. That is what makes promotion safe even if A is alive, reachable, and still serving.
Fencing at the client or at DNS is not fencing. A DNS flip moves traffic; it does not stop A from committing.
| Replication | RPO | Write latency | Failover behaviour |
|---|---|---|---|
| Async to B | = lag (seconds) | local | promotion loses the tail; must be an explicit, budgeted decision |
| Sync to B | 0 | + cross-region RTT (~70 ms us-east-1↔us-west-2) | B unreachable means A cannot commit; availability loss |
| Quorum (2 of 3) | 0 | + RTT to nearest quorum peer | survives one failure domain; the usual choice |
When not to fail over. If A is healthy and only the A↔B link is broken, and you cannot reach a quorum to fence A, fail closed: reject writes until the partition heals. If the workload is read-dominated, serve reads from B and leave writes on A. If your RPO budget is zero and lag is non-zero, do not promote; wait for the lag to drain or accept the outage.
Failure modes.
- Fencing not enforced on every path: a batch job or cache write bypassing the log. Detect by asserting the epoch at the storage layer and alerting on any write carrying a stale epoch.
- Lease TTL shorter than max clock drift between voters, so two leases overlap. Keep TTL well above NTP drift bounds.
- Two promotion controllers both granting leases. Route promotion through the same quorum, never through a separate orchestrator.
Trace (drill, hypothetical figures):
t=0:00 A's region loses power. B is async, lag 22 s at last ack.
t=0:40 B gets lease + epoch 14 from quorum (B + C respond; A silent).
t=0:45 B opens writes. Log rejects epoch < 14.
t=1:30 A returns, still epoch 13, writes rejected, self-fences.
t=7:00 Reads and writes fully on B. RTO 7 min.
Loss: 22 s of A's writes never reached B.
Arithmetic: at 300 writes/s, 22 s of lag is up to 6,600 writes at risk. That is your RPO exposure. The dual-writer alternative is worse: 8 minutes of both clusters accepting writes at 300/s is 144,000 writes, and any inventory decrement in that window is applied twice.
Evidence to collect in the drill: lag at promotion, the epoch bump, proof that A's post-recovery writes were rejected, and order-id uniqueness holding across the window. Time the whole path against RTO.
Curated: · Written: · Reviewed:
Q&A-92A deploy bot uses a long-lived access key in CI. What identity should automation use instead?
Give the bot a workload identity, not a key. The CI platform mints a signed OIDC token per job; the cloud provider's STS exchanges that token for credentials that live for minutes and are scoped to one repo, one environment, one role. Nothing permanent sits at rest, so "a 2-year-old key in a fork's secret store" stops being a category of incident.
The exchange is the whole mechanism. GitHub Actions, GitLab, CircleCI and Buildkite all publish an OIDC issuer; you register it once as an IAM OIDC provider, then the role's trust policy decides which jobs may assume it:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:acme/checkout:environment:prod"
}
}
}]
}
The sub condition is the security boundary. repo:acme/checkout:environment:prod means only a job running in the prod environment of that one repo gets the role: a fork's branch, or a plain PR job, presents a different subject and is refused. The job then calls aws sts assume-role-with-web-identity --duration-seconds 900 and holds 15-minute credentials.
Alternatives, and when each wins.
| Option | Credential lifetime | Use when |
|---|---|---|
| OIDC federation | minutes | platform has a verifiable issuer and you can pin the subject |
| Vault-issued dynamic creds | minutes–hours, lease-revocable | mixed clouds, on-prem runners, or you want one audit trail across providers |
| Static key rotated by automation | hours–days | platform has no OIDC issuer; the rotation job is itself monitored |
| Static key in a secrets manager | forever | last resort: better stored, still permanent |
The deciding question is whether the platform can present a claim you can verify and pin. If it can, federation wins on every axis: no rotation job, no secret to leak, revocation is "delete the trust policy". If it cannot: self-hosted Jenkins on-prem with no public issuer, or a SaaS vendor that never shipped OIDC: Vault's AWS secrets engine is the next best thing: it issues real STS credentials against a lease, and revoking the lease kills them. A rotated static key is the fallback, and it needs its own alarm on rotation failure, because a rotation job that silently stops is a permanent key again.
Failure modes. A wildcard subject (repo:acme/*, or omitting sub entirely) turns federation back into a shared secret: any repo in the org can assume the role. Not checking aud lets a token minted for another service be replayed at STS. pull_request_target workflows do receive an OIDC token, so a trust policy that matches on branch rather than environment can be triggered by a fork PR. And a 12-hour session TTL means a leaked token is a 12-hour key.
Detection. CloudTrail AssumeRoleWithWebIdentity events carry the subject claim and source IP per job; alert on any subject outside the expected set. iam:CreateAccessKey should page. The credential report lists every access key with its age: anything over 90 days on a bot account is a finding.
Curated: · Written: · Reviewed:
Q&A-93A CI role can read every secret and attach any policy. How do you shrink that?
Assumptions: the CI role is assumed by the pipeline (GitHub Actions OIDC, or a runner holding long-lived keys) and it carries two powers: secretsmanager:GetSecretValue on *, and iam:Attach* / iam:Put* / iam:CreatePolicyVersion. Both are privilege-escalation paths, not just "too much read."
Answer: shrink it in three layers, because no single one holds on its own.
-
Cut the identity policy to named resources and drop the IAM write verbs. For secrets, replace
Resource: "*"with the specific secret ARNs the job deploys, or a tag condition like"Condition": {"StringEquals": {"secretsmanager:ResourceTag/Environment": "staging"}}. For IAM, removeiam:AttachUserPolicy,iam:PutRolePolicy,iam:CreatePolicyVersionandiam:UpdateAssumeRolePolicyfrom the CI role entirely: a deploy job almost never needs them. If it must pass a role to a service, scopeiam:PassRoleto the specific role ARN with"Condition": {"StringEquals": {"iam:PassedToService": "ecs-tasks.amazonaws.com"}}. -
Put a permission boundary on the role. The boundary caps what any identity policy can grant: effective permissions are the intersection of the identity policy and the boundary. So even if the role can attach a policy, it cannot exceed the boundary. A boundary that allows only the deploy actions and the named secrets, and denies
iam:*andcloudtrail:StopLogging, neutralises the attach path directly. -
Make the credential short-lived. OIDC federation (
sts:AssumeRoleWithWebIdentity) with a trust policy conditioned on the repo and branch,sub=repo:org/repo:ref:refs/heads/main, issues session credentials that default to 1 hour. A stolen token is dead in an hour; a stolen static access key is not.
Mechanism. The trust policy decides who can assume the role; the identity policy and the boundary decide what the resulting session can do. A boundary grants nothing: it only subtracts. Critically, it does not limit resource-based policies: an S3 bucket policy that names the CI role still grants access even if the boundary denies S3. So the boundary is a backstop, not the whole answer.
Trade-offs. One-role-per-job is the cleanest but costs maintenance: with 40 jobs that churn weekly, the drift is real. Permission boundaries are cheaper and catch the "role can attach policy" case directly. OIDC short-lived credentials shrink the window but not the blast radius inside that window. Deciding criteria: can you enumerate the actions the job actually uses? If yes, per-job roles. If no, boundary plus tag conditions plus OIDC, then tighten from access logs. One-role-per-job is not worth it for a handful of near-identical jobs: a shared role with a boundary and per-environment tag conditions is less toil for the same cap.
Failure modes. iam:PassRole on * lets the role hand an admin role to Lambda or EC2: that is the escalation, not the secret read. A boundary that is not applied to the role that creates roles is bypassed the moment the CI role can create a new role without one. Secrets injected as environment variables land in logs and crash dumps.
Verification. For each role, diff allowed actions against last-used actions (IAM credential report plus CloudTrail), and run the pipeline's own dry-run to confirm it cannot DescribeDBInstances or read a payments secret.
Curated: · Written: · Reviewed:
Q&A-95A debug echo printed an API token into a public CI log. What controls stop that class of leak?
Treat the CI log as public from the first second of the job. The leak class is any secret that reaches stdout or stderr and gets persisted. Four controls matter, in order of leverage: keep secrets out of the command line and shell trace, mask known values at the log layer, detect unknown secrets after the fact, and rotate fast. None is sufficient alone.
Masking is exact-string replacement on a registered secret. In GitHub Actions, ::add-mask:: registers a value; GitLab CI masks project variables. It works for echo $TOKEN when TOKEN is registered. It fails when the secret is transformed, split across lines, or never registered. set -x is the common amplifier: it prints the expanded command, so the secret appears in the trace even if the program never prints it. Ban set -x on secret-bearing steps, or wrap the secret use with set +x. That does not stop curl -v, env, printenv, or docker inspect from leaking.
Entropy scanners catch unknown high-entropy strings in logs. They are the only control that can catch a secret you did not know was there. The cost is false positives: commit SHAs, UUIDs, base64 test fixtures, and long random-looking IDs all trip them. Tune with an allowlist, and route hits to a ticket, not a hard fail, unless the scanner is proven.
Auto-rotation limits the window. It is the control that matters after a leak. The cost is breaking in-flight jobs that still hold the old credential. Use dual credentials: issue the new one, let old jobs finish, then revoke. Without that, rotation causes outages and people disable it.
| Control | Stops | Cost | Failure mode |
|---|---|---|---|
| Runner masking | Exact registered secret in log | Must register; can mask non-secrets | Transformed or derived secret leaks |
Ban set -x | Shell trace of expanded commands | Harder debugging | Other verbose tools leak |
| Entropy scanner | Unknown high-entropy strings | False positives on hashes/UUIDs | Misses low-entropy or split secrets |
| Auto-rotate | Long-lived credential after leak | Breaks in-flight jobs | Rotation lag or cached copies |
Here is a worked trace. This shell:
set -x
TOKEN=abc123
echo "token=$TOKEN"
prints to stderr + TOKEN=abc123, + echo token=abc123, and to stdout token=abc123. If abc123 is registered with the runner masker, the log shows + TOKEN=***, + echo token=***, token=***. Now change the last line to:
echo "token=$(printf %s "$TOKEN" | base64)"
The trace becomes + TOKEN=abc123, + printf %s abc123, + base64, + echo token=YWJjMTIz, and stdout token=YWJjMTIz. The masker knows abc123, not YWJjMTIz, so the token leaks. That is the deciding criterion: masking protects the exact value you registered, not the secret.
The deciding criteria are: if the token is long-lived and high blast radius, rotate first, then add masking and scanning. If it is a short-lived OIDC token, rotation is less urgent. If scanner false positives are high, allowlist known hashes. If rotation breaks jobs, dual credentials are mandatory.
Detection: put a canary token in CI, check the log for it, and alert on scanner hits. Audit logs for token use after any suspected leak. The goal is not zero leaks; it is a short window and a loud alarm.
Curated: · Written: · Reviewed:
Q&A-61A database password lives for 18 months. How do you rotate it without downtime?
Rotate by keeping two valid credentials alive at once: create the new one, roll consumers to prefer it, prove the old one is unused, then disable it. A simultaneous swap is an outage with extra ceremony.
Why a single-shot change breaks
Authentication happens when a connection is established, not per query. Existing sessions survive a password change untouched, which is exactly why the failure is delayed and confusing. The outage comes from the connections that do reconnect mid-rollout: a pooler recycling server connections, a kubelet restarting pods, an autoscaler adding replicas. Those arrive at the database with the old password after it has already been invalidated.
A concrete example: a single-shot change at 03:00 against 40 pods. 28 pods reconnected with the new password; 12 held pooled connections that were recycled over the next 19 minutes and returned 5xx until the next kubelet sync. Nothing was wrong with the new password: the rollout had no overlap window.
The rotation, step by step
Assume a 40-pod consumer fleet and a 30-day overlap budget (the credential itself lives 18 months; the overlap is what you control).
| t (min) | Valid credentials | Pods on new | Auth failures |
|---|---|---|---|
| 0 | N | 0 / 40 | 0 |
| 5 | N, N+1 | 0 / 40 | 0 |
| 25 | N, N+1 | 40 / 40 | 0 |
| 40 | N, N+1 | 40 / 40 | 0 |
| 45 | N+1 only | 40 / 40 | 0 |
| single-shot at t=0 | N+1 only | 28 / 40 | 12 pods, 19 min of 5xx |
- Create N+1 with identical grants. Do not touch N.
- Deploy consumers that read the credential from the secret store and accept either value. Prefer N+1; fall back to N only on an auth error.
- Watch both credential ids. Auth failures must stay at zero for the whole window. If they don't, you rolled too fast: stop, don't disable anything.
- Confirm zero authentications on N over a full traffic cycle (at least one pool-recycle interval, not one minute).
- Disable N, then delete it from the store. Alert if any credential's age exceeds policy.
How the two credentials exist
This is engine-specific and worth knowing before you promise it.
- MySQL 8.0.14+ has dual passwords natively:
ALTER USER 'app'@'%' IDENTIFIED BY 'new' RETAIN CURRENT PASSWORD;then, after verification,DISCARD OLD PASSWORD;. Both work during the window. - PostgreSQL has no dual-password support. Overlap means a second role:
CREATE ROLE app_v2 LOGIN PASSWORD '...', grant it the same privileges asapp, roll consumers, thenDROP ROLE app.ALTER ROLE app PASSWORD '...'alone gives you no overlap and is the single-shot anti-pattern. - Dynamic secrets (Vault-style) sidestep this by issuing short-lived per-instance credentials, but only if every consumer can fetch and refresh them; a long-lived 18-month password usually means something in the fleet can't.
Failure modes and detection
- Pooler holds stale server connections. PgBouncer and similar authenticate at connect time; a pool restart mid-rotation is the classic trigger. Detect by counting auth failures per credential id, not in aggregate.
- One consumer reads the secret at boot only. It never picks up N+1 and dies when N is disabled. Detect by asserting every pod's reported credential version matches the current one before step 5.
- Rotation is untested because it happens every 18 months. The 18-month lifetime is the real risk: nobody has run the procedure since the last person left. Automate it and rehearse the overlap quarterly against a staging database.
If the engine genuinely cannot hold two credentials and no consumer can be changed, you have a planned maintenance window: say so, rather than calling it zero-downtime.
Curated: · Written: · Reviewed:
Q&A-65Engineers SSH to production boxes daily to "just check". What access model do you put in its place?
Assumptions first. You have an IdP your engineers already log into (Okta, Entra, Google), and prod hosts can reach a control plane: an SSM agent, or a proxy you run. If either is false the model changes; I say how at the end.
The answer: zero standing access. Delete the keys, don't rotate them. Every production session is a credential minted against a named human, scoped to one role, bound to a ticket, recorded, and expiring on its own. The default role is read-only; write is a separate, rarer grant.
| Model | Standing credential | Blast radius of a stolen laptop | Time to shell in a SEV | Breaks when |
|---|---|---|---|---|
Static keys in ~/.ssh | yes, until rotated | full, until rotation | ~5 s | never, which is the problem |
| JIT certs from an internal CA (Vault, step-ca), ticket-linked | no | ≤ cert TTL + revocation lag | 30–90 s automated, minutes if human-approved | no IdP to root identity; host can't run the agent |
| Proxy/bastion with its own CA (Teleport-style) | no | ≤ session TTL; proxy is a chokepoint | 30–60 s | proxy outage takes all access down; needs HA |
| Break-glass role, always assumable by on-call | no (assumed, not held) | ≤ role session, loudly alerted | ~10 s | when it becomes the default rather than the exception |
The exposure arithmetic is the point. A static key is live from issuance to rotation; quarterly rotation means a stolen key is valid for a mean of ~45 days, worst case 90. A 60-minute cert gives 90 days / 1 hour ≈ 2,160× less window, and needs no revocation step: expiry is the revocation.
What it costs. Labelled hypothetical: if your SEV detection-to-mitigation p99 is 12 minutes, a 3-minute human approval eats 25% of the budget. So automate approval for whoever is on call, keep human approval for everyone else, and make break-glass a role that is always assumable by on-call, alerts loudly, and requires a post-incident review within 24 hours. Break-glass is the pressure valve that stops people keeping a key "just in case".
When this model is wrong.
- No IdP. You can't root identity, so you'd mint certs from a static token: a standing credential with extra steps. Fix the IdP first.
- Hosts that can't run an agent: network gear, appliances, mainframes. You need a proxy that terminates SSH and re-originates it, with its own CA. Different design, different outage mode: the proxy is now a chokepoint, so it needs HA and its own break-glass.
- Long interactive debugging. A 60-minute TTL kills a session mid-trace. Use renewable certs with a hard maximum (say 8 hours) and re-auth, not a longer TTL.
- Teams under ~5 engineers with no 24/7 rotation. The ticket gate is one sleeping person. At that size break-glass should be the default path with post-hoc review, not the exception.
Failure modes and how you'd see them.
- Shadow standing keys. Someone keeps a key "just in case". Enumerate
authorized_keysacross prod and alert on any key not signed by the CA. Target is zero, and the alert is the metric. - Rubber-stamped approvals. Measure the distribution of approval latency. A cluster of sub-5-second approvals means the gate is theatre.
- Recording gaps. If the session recorder is down, sessions should fail closed. Compare session-start events to recorded command counts; a gap is an incident, not a logging bug.
The grant, concretely.
ssh-keygen -s ca.internal -I "sre-lee@example.com/INC-4412" \
-n sre-lee -V +60m -O no-port-forwarding id_ed25519.pub
Trace: -V +60m makes the cert valid from now until now+60m; -n sre-lee is the principal that must match the host's AuthorizedPrincipalsFile; -O no-port-forwarding blocks tunnelling out. At T+60m sshd rejects the cert: no revocation list involved. Steal the laptop at T+10m and the attacker holds 50 minutes of read-only access to hosts that trust principal sre-lee, on a session that is already being recorded.
What I'd measure before calling it done. Standing credentials in prod (target 0), median grant TTL, share of sessions carrying a ticket id, and the p99 time from "I need access" to a shell, because if that number is bad, people will route around the model.
Curated: · Written: · Reviewed:
Q&A-67Which checks belong in CI because they prevent production incidents, not because they are style?
Put in CI the gates that would have stopped your last year of change-caused outages. For most backend services, that means migration safety, IAM or policy plan diffs, secret scanning, and an end-to-end test of the SLI-critical path. Formatting, naming, and import order are style. They do not belong as blocking reliability gates unless you can point to an incident they would have prevented.
| Gate | Incident class caught | Illustrative minutes last year | Blocking? |
|---|---|---|---|
| expand-contract linter on migration files | locking ALTER | 210 | Yes, scoped to migrations |
| IAM plan diff required | overly broad policy | 88 | Yes on prod, advisory on dev |
| secret scan | tokens in repo | 40 | Yes with allowlist |
| checkout SLI e2e | auth header regression | 160 | Yes on main/release, not every PR if flaky |
| unit tests only (old CI) | none of the above | 0 caught | No, not sufficient |
Blocking gates are not free. A gate that runs on every PR adds latency to every merge. If it takes 2 minutes and you merge 200 PRs/day, that is 400 minutes/day, about 6.7 engineer-hours, before counting reruns. If it prevents one 210-minute incident per year, blocking every PR is a bad trade. Scoping it to migration files only is a good one. False positives are worse than latency because they teach people to bypass. A secret scanner that flags test fixtures will get an allowlist comment or a skip label. An IAM diff that blocks every dev change will get an exception. Once bypass is normal, the gate is decoration.
Incident-driven gating is wrong when the incident was not change-caused, such as a console change, a manual DB edit, or an expired cert. It is wrong when the gate cannot reproduce the failure deterministically. It is wrong when the false-positive rate exceeds the incident rate, when the gate would block emergency fixes, or when the fix is a process control such as approval or break-glass rather than a CI check. It is also wrong when the gate duplicates a control that already runs post-merge and the cost of blocking is higher than the cost of rollback.
Decide by mapping each change-caused incident to a missing gate. Add the gate as blocking only if it can fail deterministically, is scoped to the files that can cause the incident, and has a measured false-positive rate. Otherwise make it advisory. Measure catch rate as the share of last 12 months' change-caused Sev-1/2s that would now fail CI. Measure bypass rate as merges with a skip label divided by total merges. Measure p95 CI duration. If bypass rate rises, the gate is not working.
A green pipeline that ran unit tests but not a migration expander shipped a locking ALTER, 14 minutes of blocked writes, and a "CI was green" postmortem line. The fix is not more unit tests. It is an expand-contract linter on migration files that fails the PR before merge. That gate is worth blocking because it is deterministic, scoped, and maps to a known incident class. A checkout SLI e2e is worth blocking on main or release branches, but if it flakes on every PR, run it post-merge and alert.
A pipeline that cannot fail the class of change that pages you is a compiler, not a reliability control. But a pipeline that fails everything is a tax. Pick the few gates where the incident cost is real, the false-positive cost is low, and the bypass rate stays near zero.
Curated: · Written: · Reviewed:
Q&A-66When every control is locked down, how does on-call still mitigate a Sev-1?
What a strong candidate says first
"Every control locked down" is the scenario where the lockdown itself is the incident. On-call mitigates a Sev-1 in that state through a path designed before the lockdown: a pre-provisioned break-glass identity that needs no human approval to assume, is scoped to the failing service, expires on its own, and pages security the moment it is used. The other half of the answer is that the fastest mitigations usually need no production credentials at all: rollback through the deploy pipeline, a feature-flag flip, draining a bad node, shifting traffic, and those get exhausted first because they leave no standing access behind.
Mechanism
Break-glass is a role, not a shared password. It lives in the same identity provider as normal access, but its policy inverts the usual rule: assume is allowed, approval is not required, and the compensating controls are notification, TTL, and post-hoc review. Three properties make it work.
- Self-service assume. No approver in the loop. The gate is the audit trail, not a human.
- Short TTL. Long enough to mitigate, short enough that a forgotten session dies. 60 minutes is a common choice; anything that outlives the shift is standing access with extra steps.
- Automatic evidence. Assuming the role opens a ticket and starts session recording, and the ticket cannot be closed without a list of the actions taken.
The no-credential levers matter as much. In most architectures on-call can mitigate without touching prod: revert the last deploy (pipeline credentials, not prod), disable the flag that shipped the regression, drain the node, shift traffic off a region. If the lockdown removed those too, it removed mitigation, not risk.
Trade-offs
- Standing elevated access for on-call. Fastest, and it is the default path: every routine change goes through it, so the audit signal drowns.
- Manager approval before assume. Correct for destructive, irreversible actions; wrong for a Sev-1 where the clock is the harm.
- Two-person rule. Good for data deletion, bad at 03:00 when the second person is asleep.
- Break-glass with post-hoc review. The balance. Use it for reversible mitigation; do not use it for schema migrations or anything you cannot undo.
Failure modes and detection
- Break-glass depends on the same IdP or vault the incident took down. Detect: quarterly drill that assumes the role from a cold start, plus an alert on assume failures.
- It becomes the daily workflow. Detect: count assumes per week and page when the count exceeds a threshold.
- Sessions outlive the incident. Detect: audit of open break-glass sessions, alert on any session past TTL.
- Ticket closed with no diff. Detect: make the action list a required field on close.
- Credentials not rotated after use. Detect: rotation job with an alert on failure.
Artefacts
Illustrative drill: hypothetical numbers, not measured:
| Step | Target | Drill |
|---|---|---|
Assume checkout-break-glass | < 2 min | 71 s |
| Security page | immediate | 12 s |
| TTL | 60 min | 60 min |
| Review ticket | auto | 48 API calls listed |
| Manager approval before assume | — | rejected; that path was 45 min |
The arithmetic is the argument: 71 seconds to credentials versus a 45-minute approval gate is roughly 38× the time-to-mitigate, on an incident where every minute is customer-visible.
Break-glass that is slower than the outage is decoration; break-glass that is easier than the default path is the default path.
Curated: · Written: · Reviewed: