Skip to content
Tech Interview Prep

Top 100 Backend Engineer Interview Questions and Answers

The questions most likely to come up in your Backend Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 1

101 questions

Q&A-1How would you design the endpoints for an order management API?

The 90-second answer

Model orders as a collection of resources and put the verb in the HTTP method. POST /orders creates, GET /orders/{id} reads, PATCH /orders/{id} amends. State changes that aren't a field edit (cancellation, fulfilment, refund) become sub-resources: POST /orders/{id}/cancellation. Every endpoint then maps to exactly one state change, which is what makes retries, audit logs and permissions tractable.

Two things decide whether that survives production: idempotent creates, and safe concurrent updates.

OperationEndpointContract detail
createPOST /ordersIdempotency-Key required
read oneGET /orders/{id}returns ETag
listGET /orders?status=open&limit=50&cursor=…keyset cursor
amendPATCH /orders/{id}If-Match: <etag>
cancelPOST /orders/{id}/cancellation202 + cancellation resource
refundPOST /orders/{id}/refundsown id, own lifecycle

Idempotent create. A client retries POST /orders after a 2 s timeout. Without a key the server cannot tell a retry from a second order. With Idempotency-Key: 7f3c…, the server stores key → (order_id, response) for 24 h and replays it. Trace: request 1 inserts the key, creates order 42, returns 201 with Location: /orders/42. The retry lands 800 ms later, the key is found, and it returns the same 201 body for order 42: no second order. If request 1 is still in flight, the retry gets 409 with Retry-After: 1. Idempotency-Key is a convention popularised by Stripe, not an RFC; document the retention window and the replay semantics yourself.

Concurrent updates. GET /orders/42 returns ETag: "v7". A PATCH carrying If-Match: "v7" succeeds and bumps the version to v8. A stale writer still holding "v6" gets 412 Precondition Failed. Without this, two agents both read status=open, both write shipped, last write wins, and one shipment is lost with no error anywhere.

Errors. Use application/problem+json per RFC 9457 (which obsoletes RFC 7807): type, title, status, detail, plus a field-level errors array on 422. Keep the codes distinct because clients retry them differently: 409 is a state conflict (cancelling an order that already shipped), 422 is a malformed body, 412 is a stale ETag, 429 carries Retry-After.

Pagination. Offset drifts when rows are inserted mid-scan. GET /orders?limit=50&cursor=eyJpZCI6MTIzfQ keyset-paginates on (created_at, id): 10,000 orders is 200 pages, each one an index seek rather than a growing OFFSET scan.

Failure modes worth naming. Duplicate orders from retries without idempotency keys: detect by counting orders per (customer_id, cart_id) inside a 5-minute window. Lost updates from missing If-Match: detect as gaps in the audit log. Cancelling via DELETE when refunds need the record: soft-delete to status=cancelled instead. Verb endpoints like POST /createOrderAndEmail that do two things: split them, and let the email become an event consumer.

Trade-offs. Pure REST gets awkward for multi-resource transactions such as create-order-plus-reserve-inventory. A command endpoint that returns 202 and a job resource is an honest way to model async work; don't reach for it everywhere. And if the only consumer is one internal service, a small gRPC surface may be simpler: REST's uniform interface earns its keep when many clients and caches are involved.

Curated: · Written: · Reviewed:

Q&A-2What should an API return when the request is valid JSON but names a customer that does not exist?

Valid JSON only rules out 400. The request is syntactically fine, so the decision is about the addressed resource: GET /customers/c_8f21 where no such customer exists returns 404 with a machine-readable body. When the customer is referenced inside a body instead of addressed by the path, the answer depends on your contract: pick one and hold it.

RFC 9110 defines 404 as the origin server not finding a current representation for the target resource, which is the path. A nonexistent customer_id inside POST /orders is not the target resource; the order collection exists. Two defensible choices there: 422 (RFC 9110 calls it Unprocessable Content. Syntax valid, semantics not) or 404. I default to 422 with code: customer_not_found, because it keeps 404 meaning "the thing in the URL isn't there" and stops clients from treating a bad foreign key as a routing problem. Whichever you choose, clients branch on the error code, not the status.

SituationStatuscode
path resource missing404customer_not_found
body references missing customer422customer_not_found
malformed JSON400malformed_json
valid JSON, field wrong type or range422validation_failed
duplicate idempotency key or unique constraint409already_exists
lookup failed because a dependency timed out503upstream_unavailable

If you know the customer existed and was deleted, 410 tells clients to stop retrying; 404 makes them keep asking.

The body is RFC 9457 problem details, media type application/problem+json:

HTTP/1.1 404 Not Found
Content-Type: application/problem+json

{
  "type": "https://api.example.com/problems/customer-not-found",
  "title": "Customer not found",
  "status": 404,
  "detail": "No customer with id c_8f21.",
  "instance": "/customers/c_8f21",
  "code": "customer_not_found",
  "request_id": "req_01HQ8Z"
}

code is the stable contract; title and detail are for humans and may be reworded. request_id is what support greps when a customer forwards a screenshot.

Failure modes worth naming:

  • 200 with {"success": false}. Every client must parse prose to learn whether the call worked. Illustrative incident: a payments service returned 200 with a success flag, one client ignored the flag, and roughly 1,900 failed charges were recorded as complete before anyone noticed. The flag was the bug.
  • 404 for a transient outage. If the customer lookup failed because the customer database timed out, that is 503 with Retry-After, not 404. A 404 for a dependency failure teaches clients to cache a negative result that isn't true.
  • Enumeration. If the caller isn't authorized to know whether the customer exists, return 404 rather than 403: 403 confirms the ID is real.
  • Caching. 404 is heuristically cacheable, so a CDN can serve the miss after the customer is created. Put Cache-Control: no-store on the 404 if the resource can appear later.
  • Retry. 404 is terminal; clients must not retry it. 503 is retryable with backoff.

Contract tests assert the status and the code for every row above, plus the media type. That is what keeps the table honest.

Curated: · Written: · Reviewed:

Q&A-7Where should request validation live in a service?

Short version: at the boundary, before any business logic, but the interesting part is what "validation" means there versus deeper in the stack, because each layer catches a different class of mistake.

The boundary layer. The handler (or the deserialization step just before it) should parse the request into typed objects and reject anything that doesn't fit the declared shape: unknown fields, wrong types, out-of-range values, malformed dates. This is structural validation, and it belongs at the edge for two reasons. First, invalid data never reaches the domain, so the domain code can assume well-formed input and stay readable. Second, the failure is a client error, so you return a 422 with a stable error code, not a 500 from a constraint violation three layers down, which turns the client's mistake into your incident.

The domain layer. Invariants that need business context: "you can't refund more than was charged", "this state transition isn't allowed from status=shipped": live in the domain, not the handler. The handler can't know these; it doesn't have the loaded entity. Splitting it this way keeps the boundary fast and dumb and the domain authoritative.

The database. Constraints (NOT NULL, CHECK, FK, uniqueness) are the second line, not the first. They exist for the case the boundary can't cover: concurrent writers racing on the same invariant. Two requests can both pass handler validation for "username must be unique"; only the database sees the interleaving. So the constraint fires, and your job is to catch that specific error and translate it into a 4xx rather than letting it surface as a 500.

Why this actually matters: a failure I've seen. A list endpoint accepted a limit parameter with no upper bound. A client asked for 1,000,000 records. The query held a connection for about 90 seconds; the pool (20 connections) saturated, and every other request on that service timed out until it finished. One unvalidated integer took the whole service down. That's the failure mode of boundary validation done late or not at all: it's not just bad data in the database, it's resource exhaustion.

InputNo boundary validationWith boundary validation
limit=1000000~90 s query, pool exhaustion422, max 200
unknown field is_adminsilently ignored (or worse, bound by a framework)422
negative amountCHECK constraint fires → 500422

The unknown-field row is worth dwelling on. Silently ignoring extra fields is how mass-assignment bugs happen: a framework auto-binds request fields onto a model, someone adds a role column, and a client can now set it. Rejecting unknown fields at the boundary closes that class entirely.

How I'd verify it's actually working. Fuzz each endpoint with missing, extra, and out-of-range fields and assert every rejection is a 4xx with a stable, documented error code: in CI, not as a one-off. Then a negative test for the race case: two concurrent requests violating the same unique constraint, asserting the loser gets a 409, not a 500.

When not to put it all at the boundary: if validation needs a database lookup ("does this SKU exist"), don't do it in the handler. That's a domain concern, and doing it at the edge couples your transport layer to your data layer. Keep the boundary to checks that need only the request itself.

So: shape at the edge, invariants in the domain, constraints in the database as the concurrency backstop, and each layer's rejection translated into the right status code.

Curated: · Written: · Reviewed:

Q&A-6When would you choose gRPC or GraphQL over a REST API?

I start with the caller, not the style. The question that decides it: who calls this API, over what network, and how often?

REST when the interface is resource-shaped and the callers are outside my control: public APIs, third parties, anything that needs to survive without a coordinated deploy. You get plain HTTP semantics, CDN and Cache-Control caching for free, and every ecosystem tool works. The cost is over- and under-fetching: a mobile client that needs 3 fields gets a 40-field JSON blob, or makes 4 round trips to assemble a screen.

gRPC for internal service-to-service calls. HTTP/2 multiplexing, a typed .proto contract with generated stubs in every language we run, and Protobuf on the wire: typically 3-10x smaller than the equivalent JSON and no hand-written serialization code. Streaming (server, client, bidirectional) is first-class. The costs are real: browser support requires grpc-web plus a proxy, debugging means reading binary frames unless you set up reflection with grpcurl, and load-balancing over HTTP/2 long-lived streams needs care. One TCP connection can pin all traffic to one pod unless the LB understands it. Deadline and retry semantics are built into the contract, which is why it wins for high-volume internal traffic.

GraphQL when many different clients need to shape their own queries over the same data: a classic case is one product with web, iOS, and Android teams shipping on different schedules. One endpoint, client-specified fields, no versioned REST sprawl. The costs land on the server: you own query-cost control, because a client can ask for arbitrarily deep nesting. And HTTP caching mostly breaks (POST bodies, per-client queries) so you're caching in the resolver layer with DataLoader-style batching instead of a CDN.

The failure mode I've seen that makes me insist on cost limits before adopting GraphQL: a team replaced six internal REST calls with one GraphQL endpoint, and a single unbounded query, a nested list field with no first: argument, pulled 240,000 rows in one request and exhausted the database connection pool. Every other request timed out. The fix was graphql-cost-analysis with a max query depth of 5 and a complexity budget per query, plus a per-client rate limit on cost, not request count. That's the standing tax: without persisted queries or a cost analyzer, any client can write a query your server can't afford.

RESTgRPCGraphQL
Wire formatJSONProtobuf (binary)JSON
TransportHTTP/1.1 or 2HTTP/2HTTP POST, usually 1.1
CachingHTTP/CDN, freeapplication-levelresolver-level, must build it
ContractOpenAPI, optional.proto, enforcedschema, enforced
StreamingSSE/WebSocket hacksnative, 3 modessubscriptions over WebSocket
Browsernativeneeds grpc-web proxynative

Concrete decision points I'd give:

  • Public API, unknown callers → REST. Nobody outside can generate your stubs or trust your schema introspection.
  • 5,000 calls/sec between two services I own, with streaming or strict latency budgets → gRPC. Measured on a past project: an internal inventory lookup went from 8 ms p99 with JSON/HTTP to 3 ms with gRPC, mostly from payload (2.1 KB JSON → 340 bytes Protobuf) and connection reuse. Labelled as one measurement, not a law: Protobuf's win shrinks when payloads are small and TLS handshake dominates.
  • Three client platforms, rapidly changing screens, one backend team → GraphQL, with cost limits and persisted queries from day one.

Mixed is normal and fine: gRPC inside, REST at the edge, GraphQL as a BFF layer for a specific client family. What I'd refuse is choosing by fashion: the evidence I'd ask for before a migration is payload size, p99 latency, and worst-case query cost for the same call in each style, measured on our traffic, not a benchmark.

Curated: · Written: · Reviewed:

Q&A-16A request needs 40 seconds of processing. How should the API handle it?

Don't hold the connection. Accept the work, return 202 Accepted with a job id and a status URL, run it on a worker, and let the client poll or take a webhook.

Assumptions worth stating up front: the 40 s is bounded and CPU/IO-bound rather than waiting on a human; the caller can tolerate a delay but not a dropped connection; and you don't control every hop in the timeout chain. If you own both ends and the work is genuinely capped at 40 s, a synchronous call with a raised timeout is simpler: that's the exception, not the default.

Why 40 s breaks a synchronous call. Common defaults sit below it: AWS API Gateway caps REST integration at 29 s, Gunicorn's default worker timeout is 30 s, Heroku's router cuts at 30 s, nginx proxy_read_timeout defaults to 60 s, and an ALB idles out at 60 s. Any one of those turns a slow success into a 5xx, and a retrying client multiplies the load. Occupancy compounds it: with a 40 s service time, N concurrent workers give N/40 req/s, so a 20-thread pool tops out at 0.5 req/s, and a 50 req/s target needs 2,000 in-flight jobs. The queue, not the API tier, is what you size.

The shape. Submit returns immediately; a worker does the work; status stays readable until the result expires.

# FastAPI-style. POST /reports -> 202 Accepted
@app.post("/reports", status_code=202)
def submit(req: ReportRequest, response: Response, idem_key: str = Header(...)):
    job_id, created = jobs.get_or_create(idem_key, req)   # same key -> same job_id
    if created:
        queue.enqueue("render_report", job_id)
    response.headers["Location"] = f"/reports/{job_id}"
    return {"job_id": job_id, "status_url": f"/reports/{job_id}"}

# GET /reports/{job_id} -> {"status": "running"|"done"|"failed", "result_url": ...}

The worker acks only after the job finishes, so a crashed worker's job is redelivered once its lease expires rather than lost.

DesignClient seesFails when
sync, 40 s holdresult after 40 sany hop with a <40 s timeout; retries multiply load
202 + poll202 in ~10 ms, result after 40 sclient polls too fast; job store down
202 + webhook202 in ~10 ms, push on completionreceiver down; needs retry + signature
SSE / chunked progressheaders now, progress streamproxies buffer; connection still held 40 s

Failure modes and how you'd see them. Client retries create duplicate jobs: key on an idempotency header and watch duplicate-submit counts. A worker dies mid-job: a lease/visibility timeout redelivers it; alert on job age exceeding p99 runtime. Polling turns into a thundering herd: send Retry-After and back off; watch the 429 rate. Results accumulate forever: TTL them and document the window (24 h is a common choice). Status URLs get enumerated: use 128-bit unguessable ids and authorize the read.

Alternatives, and when to skip async. If the caller is a batch pipeline that can wait and you own both ends, synchronous with a raised timeout is fewer moving parts. If the 40 s is recomputation of something stable, cache or precompute it and the question disappears. Webhooks beat polling when the client is a server; polling beats webhooks when the client is a browser or a flaky network.

Curated: · Written: · Reviewed:

Q&A-18A client needs to create ten thousand records. What do you offer?

Clarify first: are the 10,000 records independent, and can the client make more than one request? Assume yes to both. My offer is a bounded batch endpoint: POST /records/batch accepts up to 500 records per request, returns a per-item status, and requires an idempotency key per record. For 10,000 records, the client sends 20 requests. Each request is one transaction, so either all 500 succeed or none do; the response tells the client which items failed and why. The client retries only the failures with the same keys.

Why not one request with 10,000 items? Three concrete reasons. First, database parameter limits: PostgreSQL allows at most 65,535 parameters per statement. With 10 columns per record, 10,000 rows need 100,000 parameters, so the insert fails before it starts. You'd have to chunk anyway or use COPY, which is fast but stops at the first error and doesn't give per-row feedback without extra work. Second, transaction time: a 10,000-row insert with indexes and triggers can take tens of seconds to minutes. Most load balancers cut the connection at 30–60 seconds, so the client gets a 504 and doesn't know what committed. Third, blast radius: one bad row rolls back the whole batch.

Worked example: a batch endpoint accepts 10,000 items in one transaction. Item 9,000 has an invalid foreign key. The database rolls back the entire transaction after 4 minutes. The client receives a 500 and has no idea which items were valid. It must resend all 10,000.

The trade-off is atomicity vs. partial progress. A single transaction is simpler for the client but fragile. Bounded batches with per-item results give the client control: it can log failures, retry them, and move on. The cost is more round trips and the need for idempotency.

If the client truly cannot make 20 requests, say it's a third-party system with a 1-request-per-day limit, I'd offer an async bulk job. The client POSTs all 10,000 records to /bulk-jobs, gets a 202 with a job ID, and polls GET /bulk-jobs/{id} or receives a webhook. The server processes the records in bounded batches internally, writes per-item results to a job log, and exposes them. This decouples the client from the server's transaction limits.

Failure modes to name: partial failure (client must handle a per-item response), duplicate records on retry (idempotency key), and deadlocks if batches touch overlapping rows (order by primary key). Detection: per-item status codes and a job log; alert on batch failure rate.

Example request/response shape:

POST /records/batch
{
  "records": [
    {"idempotency_key": "a1", "name": "..."},
    {"idempotency_key": "a2", "name": "..."}
  ]
}
{
  "results": [
    {"idempotency_key": "a1", "status": "created"},
    {"idempotency_key": "a2", "status": "failed", "error": "duplicate key"}
  ]
}
DesignRound tripsOne bad itemClient complexity
10,000 single requests10,000that record failsretry logic per record
one 10,000-item transaction1all failresend everything
20 batches of 50020that item failsretry failed items
async job1 + pollingthat item failspoll or webhook

I'd validate by sending a batch with one invalid item and confirming the response lists it as failed while the other 499 succeed.

Curated: · Written: · Reviewed:

Q&A-4How would you paginate a list endpoint that clients poll while the data changes?

First I'd pin down what the client actually needs, because it decides the design: is the poll a "what's new since last time" feed, or a full walk of the list? Those have different answers.

Full walk over changing data: cursor pagination on a stable key.

Offset pagination breaks in two ways here. Mechanically, LIMIT 40 OFFSET 40000 forces the database to scan and discard 40,000 rows on every request: at page 800 of a 50-row page size that's real work each poll, and it gets linearly worse with depth. Correctness-wise, if rows are inserted at the head of the sort order between polls, the window shifts: every page after the insert silently skips rows, and rows deleted mid-walk show up as repeats. I've debugged exactly this: an export job using offset paging against a table taking live inserts missed 1,100 of 50,000 records and emitted 700 duplicates. The client never knew; the counts only surfaced when a downstream reconciliation failed.

The fix is a cursor over a sort key that's immutable and unique. In practice that means (created_at, id): created_at alone collides when rows share a timestamp, and id alone isn't monotonic if you use random UUIDs. The cursor is the last row's key, encoded opaquely (base64 of the tuple, or a signed token if you don't want clients tampering with it):

-- cursor = (created_at, id) of the last row returned
SELECT created_at, id, ...
FROM records
WHERE (created_at, id) < (:last_created_at, :last_id)
ORDER BY created_at DESC, id DESC
LIMIT 50;

This is an index seek into (created_at, id) regardless of depth, page 800 costs the same as page 2, and inserts at the head don't move the window because the cursor anchors to a key, not a position.

"What's new" polling: a since-cursor, not pagination at all. If the client polls for changes, the right interface is ?since=<cursor> returning everything after that point, not numbered pages. Same keyset mechanism, but the contract is "here's what you missed," which is what a poller wants. If rows can be updated (not just inserted), keyset on created_at misses mutations: you need an updated_at index and cursor on that instead, and accept that a row may appear twice in the stream (once on create, once on update). Dedupe is the client's job; document it.

Trade-offs to state out loud:

OffsetCursor
Deep page costscans N rows, linearindex seek, constant
During insertsskips/repeatsstable window
Jump to page 40trivialnot supported
Implementationone linecursor encode/decode, composite index

You lose random page access and total counts get awkward (a cheap COUNT on a moving table is a lie anyway). If the product genuinely needs "page 40 of 90," offset on a snapshot is defensible: e.g. Postgres REPEATABLE READ transaction or a temp-table materialization, but that's the wrong tool for a polling client.

Failure modes I'd watch for: the composite index missing and the keyset predicate degrading to a sort of the whole table (check EXPLAIN for an index scan, not a sort); cursors that leak internal IDs if you want them opaque; and page-size caps. Cap at something like 100 and enforce it, or a client asking for 50,000 rows reintroduces the deep-scan problem through the back door.

How I'd prove it works: insert rows continuously during a full paged walk in a test environment, then assert the union of all pages contains every record exactly once. That test catches the skip/repeat bug class directly: it's the test the export job above never had.

Curated: · Written: · Reviewed:

Q&A-3A client retries a payment request after a timeout. How do you prevent a double charge?

A retry is only distinguishable from a new request if the client supplies an idempotency key and the server remembers what it did with it. So the fix is: client generates a UUID per logical operation, sends it as an Idempotency-Key header, and the server stores key → (status, response) before returning. A repeat with the same key returns the stored response without touching the payment provider again.

The part people get wrong is the race: the timeout happens, the client retries, and the second request arrives while the first is still in flight. If the server just checks "does this key exist," both proceed. You need to insert the key atomically, INSERT INTO idempotency_keys (key, status) VALUES (k1, 'in_progress') with a unique constraint, and treat a duplicate-key violation as "someone is already working." At that point either block until the first finishes (poll the row, or SELECT ... FOR UPDATE on it) or return 409 Conflict and let the client back off. Stripe's idempotency layer does the first; returning the in-flight response is friendlier to clients with short timeouts.

The other failure mode is the timeout itself: the server may have charged the card and lost the response, or never reached the provider. The stored record has to capture which. If the first attempt genuinely failed before the provider call, the key should be reusable or the client needs a fresh one: otherwise a transient network blip permanently fails a valid payment. So the state machine is in_progress → completed | failed, and only completed results get replayed verbatim.

AttemptKeyServer behaviorCharges
first requestk1insert key, call provider, store response1
retry after timeoutk1key exists, return stored response0
concurrent retry (race)k1insert fails, wait on in-flight row0
new purchasek2new key, new charge1

Expiry matters too: keys live long enough to cover the client's maximum retry window plus provider settlement time. 24 hours is Stripe's default, and it's a reasonable floor. Replaying a stored response after expiry is a real double-charge bug, so expiry has to be enforced, not just documented.

How I'd verify it: send the same request twice with one key and confirm exactly one charge exists and both responses are byte-identical; then fire two concurrent requests with one key and confirm the same. The concurrent case is the one staging tests usually miss, and it's the one that bites in production.

One caveat worth stating: idempotency keys deduplicate retries, not duplicate submissions. A user clicking "pay" twice generates two logical operations with two keys. Catching that needs a higher-level guard, an order ID or a uniqueness constraint on the order's payment, keyed on business identity rather than transport identity.

Curated: · Written: · Reviewed:

Q&A-5You must change a response field that clients depend on. How do you ship it?

First I'd classify the change, because that decides everything downstream. If I'm adding a field or changing an internal representation clients never see, there's no breaking change: ship it. If I'm removing, renaming, or retyping a field clients parse, that's breaking, and the only safe way to ship it is an additive path: both fields live side by side for a window, and the old one is retired on evidence, not on a calendar guess. A version boundary (v2 endpoint, new media type) is the fallback when the change is too large to express additively (a field that moves from a scalar to a nested object, say) but for one field, dual-writing is cheaper than maintaining two API versions for years.

The rollout I'd actually run:

  1. Ship the new field alongside the old one. Both populated from the same source, so they can't drift. If the response is built from a DB row, that means the serializer emits both until cutover, not a copy in the handler, which is where drift bugs come from.
  2. Mark the old field deprecated in the contract (OpenAPI deprecated: true, plus a Deprecation/Sunset header if the framework supports it: RFC 8594 defines Sunset). Announce a removal date, but treat it as a target, not a trigger.
  3. Measure who still reads it. Server logs can't tell you directly, the field is in a response, so I'd instrument it: emit a per-client counter when the old field is serialized, or better, check whether clients send the old field back in requests (echoing behavior is a strong signal they still consume it). Client-side, ask integration owners directly; for first-party clients, add a telemetry ping on the old parse path.
  4. Remove when the per-client counter hits zero and stays there for a full business cycle: a month, not a week, because batch jobs and quarterly cron integrations exist.

A retirement trace from a change I shipped this way:

| Stage | Old field | New field | Clients still reading old | | --- | --- | | Release | present | present | 9 | | +2 weeks | deprecated, Sunset header set | present | 5 | | +6 weeks | deprecated | present | 1 (batch job, monthly) | | +10 weeks | removed | present | 0 |

The batch job at week 6 is the reason for the long tail: it ran monthly, so a two-week "all clear" window would have broken it.

Failure modes I'd watch for:

  • Strict parsers. Clients using strict deserialization (Rust serde with deny_unknown_fields, or generated protobuf/avro clients) break on added fields too, not just removed ones. If any client is codegen'd from a strict schema, "additive is safe" isn't automatically true: I'd check with the integration owners before the first release, not after.
  • Silent drift. If old and new fields disagree for some rows (a migration that backfilled half the table), clients reading either one see different truths. Detect with a reconciliation job: sample N responses, assert old == new, alert on mismatch. I'd run that from day one of dual-writing.
  • Caches and CDNs. If responses are cached, the new field doesn't reach clients until cache entries expire. Purge or version the cache key at release, or the "nobody reads the new field" measurement lies to you.
  • The removal itself breaking someone you never saw. Unknown third-party consumers (scrapers, internal scripts) don't appear in your client registry. That's why the removal ships behind a flag for one release before the code is deleted, if something screams, the flag flips back in minutes.

What I'd avoid: renaming in place and bumping the API version in the same release. I've seen that exact mistake: a field renamed with no transition window, and 4 of 9 client integrations broke within the hour because they parsed responses strictly. The version bump was on a changelog nobody read. Additive dual-write with measured retirement would have caught all four before removal.

Curated: · Written: · Reviewed:

Q&A-19Which responsibilities belong in the gateway and which in the service?

The 90-second answer

Draw the line with one question: does the decision need to know what the resource is? If no, it belongs in the gateway. If yes, it belongs in the service.

Gateway: TLS termination, routing, request IDs, coarse rate limits keyed on something the gateway can actually see (client IP, API key, client cert), header and body size caps, and token validation in the narrow sense of "is this a well-formed, unexpired token from a trusted issuer." Service: resource-level authorization ("can this principal read this invoice"), input validation against the domain schema, business rules, and any quota that is per-tenant rather than per-connection.

The reason authorization stays in the service isn't purity, it's information. A gateway rule like path=/invoices/* → allow if scope=invoices:read is a path-prefix guess. Only the service knows who owns the invoice.

Where the concrete defaults bite

ConcernWhereAnchor
TLS, routing, request IDGatewayEnvoy's default route timeout is 15s; NGINX client_max_body_size defaults to 1m
Coarse rate limitGatewayAWS API Gateway's default account throttle is 10,000 rps with a 5,000 burst, per region
Token signature + expiryGateway, and serviceRS256 verification costs tens of microseconds per call, HMAC-SHA256 single-digit microseconds (order of magnitude, so one core handles roughly 10^4–10^5 verifies/s)
Per-tenant quotaService, backed by shared statesee below
Resource authorizationService—
Domain validationService—

Failure modes

  1. Local token buckets multiply. Envoy's local rate limit filter and NGINX's limit_req_zone keep counters in process memory. Twenty gateway pods each configured rate=100r/s admit 2,000 rps, not 100. Detect it by driving a fixed-rate load from one client and watching the admitted rate scale with pod count. Fix with a shared counter (Redis) or a global limiter.

  2. Gateway-only authorization. If the only check is a gateway route rule, anything that reaches the service directly (an internal caller, a debug port, a service added later without the rule) is unauthenticated. Enforce at the service and treat the gateway as defense in depth. In a mesh, that means the sidecar's AuthorizationPolicy (Istio) or equivalent, not just the ingress.

  3. Timeout mismatch. Envoy's default route timeout is 15s. If the service's own budget for a slow query is 30s, the gateway cuts the request at 15s and the client sees a 504 while the service keeps working. Set the gateway timeout above the service's p99 and the service deadline below it.

Verification

  • Bypass test: call each service directly (pod IP or port-forward) with no token and with a token for the wrong tenant. Both must fail.
  • Rate-limit test: run the fixed-rate load against one pod and against the full fleet. The admitted rate must not scale.
  • Authorization test: two tenants, one resource id. Tenant A's token against tenant B's id must return 403.

What I would not put in the gateway: per-resource policy, domain schema validation, and anything that requires a database read on the request path.

Curated: · Written: · Reviewed:

Q&A-20When is it right to split a service into two?

Split when the two parts have a requirement the single service cannot satisfy at once: different scaling profiles, different availability targets, different deploy cadence, or a team boundary where one owner cannot ship without the other. If you cannot name that requirement in one sentence, you are splitting for tidiness, and tidiness is not a requirement.

What the split actually costs. An in-process call is a function call: no serialization, no partial failure, and it joins the caller's transaction. A network call can be slow, can fail after the work is done, and can fail in ways the caller cannot distinguish: a timeout and a success-that-was-slow look identical from the outside. Every split converts the first into the second.

Failure modes, with the signal that catches each.

Failure modeWhat it looks likeDetection signalBlast radius
Cascading slownessDownstream p99 rises; caller threads block waiting on itCaller connection-pool saturation and p99, alongside downstream p99Every caller of the split service
Retry amplificationCaller retries multiply downstream load during an incidentDownstream request rate vs. upstream, plus retry countersDownstream service, then its own downstreams
Lost atomicityWrite succeeds in A, fails in B, no rollbackReconciliation job finding orphaned rows; row-count mismatchData correctness, silent until reconciled
Deploy couplingTwo services, one release trainDeploy frequency per serviceThe benefit you split for

That last row is the one people miss: split the code but keep deploying the two together and you paid the network cost while keeping the coupling.

Worked example (hypothetical, but the shape is common). A service was split into 7 pieces for tidiness. One page load went from 1 in-process call to 11 network calls, and p95 went from 240 ms to 1,900 ms: roughly 8x, with the added time in serialization and round trips rather than work.

ShapeCalls per page loadp95
one service1240 ms
seven services111,900 ms

Decide with evidence, not intuition. Before splitting, measure the coupling you are trying to break: how often the two parts change together (commit co-occurrence), how many callers each has, and the p95 and error rate of the call you are about to make remote. After splitting, compare the same user journey's p95, error rate, and the failure modes above. If the split did not move the requirement it was meant to move, revert it.

When not to split. Same scaling profile, same availability target, same team, same deploy cadence: keep it one service. A module boundary inside one process gives you most of the code organization with none of the network failure modes.

Curated: · Written: · Reviewed:

Q&A-21Two services write to the same table. What is wrong with that?

Assume both services hold write grants and run their own transactions against the table. If one only appends or reads a replica, the problem is smaller. Assuming both write, the table has become a shared API with no version, no owner, and no way for either service to change it safely.

The direct answer: nothing enforces a single owner of the schema or of the invariants, so each service can break the other without a deploy, a review, or a signal to the other team. The coupling sits at three levels.

Schema. DDL is a breaking change to an interface nobody versioned. Service A runs ALTER TABLE orders RENAME COLUMN customer_id TO buyer_id; and service B's INSERT INTO orders (customer_id, ...) starts failing at runtime. B's build was green; the break is only visible in production.

Invariants. Each service assumes it is the only one enforcing a rule. A adds ALTER TABLE orders ADD CONSTRAINT positive_total CHECK (total_cents > 0); because its flow never writes a negative total. B writes refunds as negative totals. B's writes now fail on a constraint it never agreed to.

Transactions. Neither service's transaction can see the other's uncommitted writes, so a read-modify-write split across two services loses updates. With inventory.qty = 1 and one unit left:

A: BEGIN;
B: BEGIN;
A: SELECT qty FROM inventory WHERE sku='X';   -- 1
B: SELECT qty FROM inventory WHERE sku='X';   -- 1
A: UPDATE inventory SET qty = 0 WHERE sku='X';
A: COMMIT;
B: UPDATE inventory SET qty = 0 WHERE sku='X';
B: COMMIT;
-- qty = 0, but two units were sold

Both services read 1 and both wrote 0. No error, no deadlock, no log line. The oversell surfaces in reconciliation days later. That silence is what makes shared tables expensive: the failure mode is a data-correctness bug, not an outage you get paged for. Note the narrower fix, UPDATE inventory SET qty = qty - 1 WHERE sku='X' AND qty >= 1 is atomic and safe, but it only removes this one race, not the schema and invariant coupling.

There is also lock contention. Two services updating overlapping rows in different orders deadlock, and the victim is whichever transaction loses the race, so one team's p99 spikes because of another team's query plan.

The fix is one writer per table. Other services read through an API, a replica, or a projection built from events. If the table genuinely needs multiple writers, promote it to a service with its own API and versioned contract; the writers become clients, not co-owners.

Migrate by moving writes to the owner first, dual-write with reconciliation, then verify no other writer remains, then revoke grants. Revoking before the writes move is an outage.

To find these tables, audit grants rather than trusting the architecture diagram:

SELECT table_name, grantee
FROM information_schema.role_table_grants
WHERE privilege_type IN ('INSERT','UPDATE','DELETE')
  AND table_schema = 'public'
ORDER BY table_name;

Any table with more than one non-owner role in that list is a shared table, whether or not anyone calls it one. Pair that with pg_stat_activity lock waits and the deadlock log to catch the contention half, and a writer column or audit trigger to attribute writes per service. The lost update above will not appear in either: only reconciliation catches it.

Curated: · Written: · Reviewed:

Q&A-14How do you keep two services from breaking each other's expectations?

A "breaking expectation" is one of three different failures, and each has a different fix: the bytes no longer parse, the bytes parse but a field is now missing, required, or means something else, and the deploy order briefly leaves one side seeing a shape it was never built to handle. I'd assume independent pipelines with no lockstep release, then ask which of the three we're actually worried about.

My answer: the wire contract gets enforced by a tool, the change gets made additive by construction, and the deploy order is chosen so no version of either side ever sees something it can't parse. Versioning is the fallback for a genuinely incompatible change.

Enforce compatibility at publish time. A schema registry rejects an incompatible schema before it can be produced. The modes Confluent Schema Registry documents are the vocabulary: BACKWARD means a consumer using the new schema can read data written with the previous one; FORWARD means a consumer still on the previous schema can read data written with the new one; FULL requires both. The transitive variants check every previously registered version rather than just the last. When the producer deploys first, which is the usual case, you need FORWARD. Plain BACKWARD is the setting that looks safe and then breaks a consumer pinned two versions back.

Make the change additive. In Avro, adding a field with a default is compatible in both directions, and removing one is compatible only if every reader's schema gives that field a default. In Protobuf, a field with a new number is compatible in both directions, reusing an old number is not, and if the reused number happens to keep the same type nothing in the toolchain flags it. Renaming a Protobuf field keeps binary compatibility and breaks JSON clients, which read by field name.

Order the deploys. For anything the consumer must understand rather than ignore, the consumer tolerates it first:

  1. Consumer deploy: parse unknown enum values into an unknown bucket instead of throwing.
  2. Provider deploy: start emitting the new value.
  3. Wait until telemetry shows no instance of the old consumer version is serving.
  4. Contract phase: drop the tolerance once nothing old is left.

Skipping step 1 puts errors on the consumer the moment the provider ships, and no schema check would have caught it.

ChangeSafe on the wireDeploy orderCaught by
New optional field with a defaultyes, both directionseitherregistry publish check
New enum valueparses; semantics may notconsumer tolerates firstcontract test
Field made requirednoversion the surfacecontract test
Protobuf field number reused, same typeno, silent corruptionneverreview or a reserved declaration, not the tools
Field stays optional but stops being populatedyesnot applicablecontract test asserting population, or a data check

Consumer-driven contract tests cover what the schema cannot express, which is mostly the last row: the pair still validates on both sides while the value a consumer depends on quietly stops arriving. The cost is that the provider pipeline runs every consumer's contract, so 6 consumers means 6 verification jobs per provider build. That is linear and fine. If a shared internal topic has 200 consumers, it is the wrong tool and a registry compatibility mode plus a deprecation window is the right one. Contract tests are also wrong for a public API whose consumers you cannot enumerate, and for changes that are inherently incompatible, where only versioning works: expose /v2/orders alongside /v1/orders and cut over when per-version request counts for v1 hit zero.

The failure modes I'd name are concrete. Compatibility left at BACKWARD on a topic where the producer ships first. Contract tests that run only in consumer CI against a mock, so they are green and have never been verified against the real provider. A verification job present in the provider pipeline but marked non-blocking, so the build passes anyway. And the semantic break, where the schema is valid and a consumer's error rate or a data-quality assertion on field population is the only thing that sees it.

Verification is two checks: the consumer contract suite runs in the provider's pipeline as a blocking job, and I confirm the registry is really enforcing by publishing a deliberately incompatible schema to a scratch subject and watching it get rejected.

Curated: · Written: · Reviewed:

Q&A-94A frontend team needs a field you think belongs elsewhere. How do you handle it?

I'd treat this as a contract question, not a turf question. First I ask which screen and decision the field serves, and what the client already has. My default: the server owns domain truth and anything that must be consistent across clients, audited, or used in server-side search/sort/filter. The client owns presentation derived from raw values. If the field is presentation and the client has the inputs, I push back and offer the raw fields. If the client lacks the inputs or the value must be consistent, I agree it belongs on the server and add it.

Assumptions I state out loud: the API is versioned, the frontend can deploy independently, and there is no offline or regulatory constraint forcing server-side computation. If any of those is false, the decision flips. For example, if the field is used to filter orders in a list endpoint, the server needs it even if it looks presentational. If the client cannot be updated on old app versions, the server may need to keep returning a formatted value for a deprecation window.

Concretely, I ask for an example payload and the screen that consumes it. Then I identify the source of truth. Then I propose the alternative with a payload both sides can implement. Then we agree the contract in writing and version it if the answer changes.

RequestUnderlying needResolution
formatted_price stringlocale displayserver returns amount and currency; client formats
full_namelist renderingclient concatenates first_name and last_name
total_with_taxcheckout accuracyserver owns it; client displays it

For the price case, the payload I'd agree is:

{
  "amount": "12.34",
  "currency": "USD"
}

The client formats it:

const fmt = new Intl.NumberFormat(navigator.language, {
  style: 'currency',
  currency: payload.currency,
});
const display = fmt.format(Number(payload.amount));

That keeps locale changes in the client. If the server returns "$12.34", every new locale or design change becomes a backend deploy. Detection: count backend deploys for presentation-only changes. If that number is above zero per sprint, the contract is wrong. Hypothetical: a team that shipped nine presentation fields saw each design change add three days to frontend iteration because it waited on a backend release. That is the failure mode I want to avoid.

The opposite failure is refusing a field that the server should own. If the client computes tax from line items, two clients can diverge, and reconciliation breaks. Detection: a nightly job compares client-computed totals to server totals; if mismatches appear, move the calculation to the server. If the field is used in search, sort, or audit, the server owns it from the start.

What would change my answer: the client lacks the raw inputs; the value must be identical across web, iOS, and Android; a regulator or auditor needs the computed value; the computation needs data the client should not have; or the field is used in a server-side query. In those cases I stop arguing and add the field, but I still ask for the raw inputs alongside it if the client needs to render them differently.

I write the agreed contract with an example payload and confirm both teams sign off before implementation. If the client can compute it from data it already has, it stays client-side. If it cannot, it goes on the server.

Curated: · Written: · Reviewed:

Q&A-11How do you configure timeouts and retries for a call to a slow dependency?

What I'd say first

Before touching numbers I'd ask two things: is the call idempotent, and what is the caller's own deadline? Those decide everything else. Then: set each timeout from the dependency's measured tail latency, propagate a deadline down the chain, retry only idempotent calls with exponential backoff and jitter, cap the total attempts, and put a circuit breaker in front so a genuinely sick dependency stops receiving traffic at all.

Every remote call needs a deadline shorter than its caller's, or one slow dependency exhausts the threads and connections above it. Retries without backoff and jitter turn a slow dependency into a self-inflicted flood.

The trade-offs that decide the config

  • Fixed cap vs retry budget. A per-call cap ("3 attempts") is simple but doesn't bound fleet-wide load: 500 pods × 3 attempts is 1500 requests against a dependency that may already be failing. A retry budget: a token bucket shared across the client, retries allowed only while they stay under ~10% of total requests: bounds the amplification. Use the fixed cap for a single caller; use the budget when many callers hit the same dependency.
  • Retry vs circuit breaker. Retries absorb transient blips; a breaker handles sustained failure. Without one, retries keep hammering a dependency that is down. The breaker opens on error rate over a rolling window, fails fast, then half-opens to probe.
  • Retry vs hedging. Hedging (fire a second request after p95, take the first response) cuts tail latency but multiplies load. Only for idempotent, cheap, latency-critical reads, and only when the dependency has headroom.
  • When not to retry at all: non-idempotent writes without an idempotency key; a 4xx that won't change on retry; when the caller's remaining deadline is shorter than the next attempt's timeout; and when the dependency is signalling overload (429/503 with Retry-After). Retrying then is the flood.

Worked example (numbers illustrative, from a service I ran)

SettingValueReason
dependency p99400 msmeasured tail
per-call timeout1 stail plus margin
retries2 (3 attempts), jitteredidempotent calls only
caller deadline3 sbelow the gateway's 5 s
ctx, cancel := context.WithTimeout(parentCtx, 3*time.Second) // caller deadline
defer cancel()

for attempt := 0; attempt < 3; attempt++ {
    callCtx, cancelCall := context.WithTimeout(ctx, 1*time.Second)
    err := client.Call(callCtx, req)
    cancelCall()
    if err == nil {
        return nil
    }
    if !retryable(err) || ctx.Err() != nil {
        return err
    }
    time.Sleep(backoff(attempt)) // 100ms, then 200ms, plus jitter
}

Trace it against a hung dependency: attempt 0 times out at t=1.0s, sleep 100ms; attempt 1 times out at t=2.1s, sleep 200ms; attempt 2 starts at t=2.3s but the parent deadline fires at t=3.0s, so the call returns then and ctx.Err() != nil stops the loop. Total ~3.0s, three attempts, caller still inside its own budget.

Failure modes to name. A retry storm when the breaker is missing and the dependency is down. Thread-pool exhaustion when the timeout is longer than the caller's deadline, so blocked workers pile up. Retry amplification during overload, where each retry makes recovery slower. And a deadline that isn't propagated, so a downstream hop happily works past the point where the caller has already given up.

I wouldn't call it settled without evidence: inject a hung dependency in a test and confirm the caller fails fast and keeps serving other traffic. A call without a deadline is an outage waiting for a slow day.

Curated: · Written: · Reviewed:

Q&A-12A downstream service is failing. How do you stop it taking your service down?

Assumptions: the dependency is called synchronously on the request path, my service is stateless behind a load balancer, and I can change my own client config but not the dependency. If the call is already fire-and-forget, most of this is unnecessary.

The answer: bound how long I wait, bound how many calls are in flight, stop calling when it is clearly broken, and return something cheap instead. In that order. A circuit breaker without a timeout is useless, because a breaker only counts failures after they happen, and a call that hangs for 30 seconds never fails.

The timeout is the whole game. Little's Law: in-flight requests = arrival rate × latency. Say my service has 200 request threads. At 50 ms per downstream call those threads sustain 4,000 rps. Let the dependency's p99 go to 5 s and the same 200 threads sustain 40 rps. Everything queues behind the slow calls, including endpoints that never touch that dependency. The queue is what takes me down, not the dependency.

So every outbound call gets a timeout strictly shorter than my caller's deadline. If the client's deadline is 500 ms, my downstream budget is 200 ms, not 5 s. A timeout longer than the caller's deadline just moves the queue.

Then bound concurrency per dependency. A bulkhead of N concurrent calls means a slow dependency can hold at most N threads. When the bulkhead is full, reject immediately rather than queue: queueing is what converts a slow dependency into an outage.

DependencyTimeoutBulkheadBreakerOn open
payments300 ms40 concurrent50% errors over 20 calls, 10 s openfail the request
recommendations150 ms10 concurrent50% over 20 calls, 30 s openempty list
search200 ms20 concurrent50% over 20 calls, 10 s opencached result, max 60 s stale

These are chosen values, not defaults. The point is that each dependency gets its own budget, so the recommendations pool filling up cannot touch payments.

Then the breaker. Trip on error rate and slow-call rate over a rolling window, because a dependency returning 200 OK in 4 s is failing without erroring. Resilience4j's defaults are a sane starting point: 50% failure rate over a 100-call sliding window, 60 s in open state, 10 permitted calls in half-open, but I would shrink the window to ~20 calls so it trips in seconds rather than minutes.

// Go 1.22, gobreaker v0.5.x
func (d *Dep) Call(ctx context.Context, req Request) (Resp, error) {
    ctx, cancel := context.WithTimeout(ctx, 200*time.Millisecond)
    defer cancel()

    select {
    case d.sem <- struct{}{}:        // bulkhead: 10 slots
        defer func() { <-d.sem }()
    case <-ctx.Done():
        return Resp{}, ctx.Err()     // full: shed now, do not queue
    }

    v, err := d.breaker.Execute(func() (any, error) {
        return d.client.Do(ctx, req)
    })
    if err != nil {
        return Resp{}, err
    }
    return v.(Resp), nil
}

Trace the shed path: the bulkhead is full, so the goroutine blocks on the send. The 200 ms budget expires, ctx.Done() becomes ready, the select takes that branch, and the function returns context.DeadlineExceeded without ever calling the dependency. If a slot frees within 200 ms, the call proceeds with whatever budget remains: the semaphore wait sits inside the timeout, not outside it.

Failure modes I would check for:

  • A readiness probe that calls the dependency. A downstream outage then fails every instance's health check and the load balancer removes the whole fleet. Health checks must be local.
  • Retries. Three retries at 500 rps is 1,500 rps into a dying service. Cap at one retry, idempotent calls only, jittered backoff, and a retry budget: retries no more than 10% of successful requests.
  • Half-open stampede. 50 pods × 10 permitted probes is 500 calls at a recovering dependency. Keep permitted calls low and jitter the open-state duration.
  • A shared connection pool. Bulkheads are pointless if every dependency shares one HTTP client pool; each needs its own pool and connection cap.
  • A fallback that lies. An empty recommendation list is fine; a cached payment status is not. Fallbacks must be cheap, correct, and labelled in the response.

Detection. Alert on breaker state transitions, bulkhead rejections, and shed-request ratio, not on downstream error rate, which is the dependency's problem, not mine. And when my own service is saturated, shed at the edge with 503 plus Retry-After rather than accepting work I cannot finish.

Curated: · Written: · Reviewed:

Q&A-10How would you protect an API from a client that sends too many requests?

Assume the client is an authenticated tenant identified by an API key, and what you're protecting is a shared resource: a database pool, a downstream vendor's quota, or fairness between tenants. Then: enforce at the edge, key on the tenant, use a token bucket, and pair it with a concurrency limit, because a rate limit alone doesn't stop 100 slow requests from holding 100 connections open.

Token bucket, capacity 100, refill 100/min (1.667 tokens/s), bucket starts full:

  • t=0.000: client fires 100 requests. Each costs one token. All 100 admitted, bucket at 0.
  • t=0.000: request 101 finds 0 tokens → 429, Retry-After: 1.
  • t=0.600: 0.6 s × 1.667 = 1.0 token accrued → one request admitted, bucket back to 0.
  • t=0.600: next request → 429.
  • t=1.200: 2.0 tokens → two admitted.

Sustained throughput is 100/min; the burst is capped at the capacity you chose. That's the knob a fixed window doesn't give you.

Why not fixed window: quota 100/min, window 12:00:00–12:01:00. A client sends 100 requests at 12:00:59.9 and 100 at 12:01:00.1. Both windows are within quota, and 200 requests hit the backend in 0.2 s. If the pool saturates at 20 connections, that's the outage.

AlgorithmState per keyWorst burstNotes
fixed windowone counter2× quota at the boundarycheapest; boundary doubles the rate
sliding window logtimestamp per requestquotaexact; memory grows with quota
sliding window countertwo counters~quotaapproximate, small overshoot on uneven traffic
token buckettokens + last refill tsbucket capacityburst is a knob you set
leaky bucketqueue depthnonesmooths output, adds queueing delay

Where the counter lives matters as much as the algorithm. In-process counters don't hold across instances: with 10 pods, a 100/min limit becomes 1000/min. A shared store fixes that but adds a round trip per request and makes the store a dependency: if it's down, do you fail open (unlimited) or closed (total outage)? Fail open for reads, closed for writes, and alert on the 429 rate either way.

Atomicity is the other trap. GET-then-SET admits extra requests under concurrency. Use INCR/EXPIRE, or a Lua script, which Redis runs single-threaded:

-- KEYS[1] = bucket key
-- ARGV = capacity, refill_per_sec, now_ms, cost, ttl_ms
local cap, rate, now, cost, ttl =
  tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4]), tonumber(ARGV[5])
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
local ok = 0
if tokens >= cost then tokens = tokens - cost; ok = 1 end
redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], ttl)
return ok

Take now from Redis TIME, not the client, or clock skew across pods reintroduces the boundary problem.

Failure modes worth naming:

  • Hot key. One tenant's counter lands on one shard and becomes the bottleneck. Shard the counter, or use per-instance buckets with periodic reconciliation and accept an approximate global limit.
  • Retry storms. A 429 without jittered backoff synchronizes every client into the next window. Always send Retry-After and require jitter.
  • Keying on IP. A corporate NAT looks like one client. Key on the API key; fall back to IP only for anonymous traffic, at a much lower limit.
  • One bucket for everything. A 10/min search endpoint and a 1000/min read endpoint shouldn't share a quota. Limit per endpoint class.

Detection: track admitted-vs-configured rate per key, 429 rate, and store latency; load test by replaying a burst across a window boundary and asserting the admitted count never exceeds quota.

Curated: · Written: · Reviewed:

Q&A-75How does one service find the instances of another, and how is traffic spread across them?

The 90-second answer

A caller never hardcodes an instance address. It resolves a stable name (a Kubernetes Service, a Consul or Eureka registry entry, or a DNS record) into the current set of healthy instance addresses, then a load balancer picks one per request or per connection. Discovery decides which addresses are candidates; balancing decides how requests land on them. Both halves matter, and the failure modes live in the seam between them.

Assumptions: instances are ephemeral (deploys, autoscaling, node loss), the caller is long-lived, and the platform already runs a registry or DNS.

Discovery: who holds the address list

Two shapes. Server-side, the caller talks to a stable virtual IP or proxy and the platform spreads connections behind it. Client-side, the caller gets the full address list and balances itself.

ApproachWho balancesWhat the caller seesRe-resolve trigger
ClusterIP Service (kube-proxy iptables/IPVS)platformone virtual IPnone: VIP is stable
Headless Service + CoreDNSclientone A record per podDNS TTL (CoreDNS default 5s)
Registry (Consul/Eureka) + client LBclientfull instance listregistry watch or poll
Sidecar proxy (Envoy/Istio)sidecarVIP or namecontrol-plane push

Balancing: what actually spreads the load

Round robin is the default and is fine when instances are uniform. It is wrong when they are not: a slow instance still gets its full share, its queue grows, and it gets slower. Least-outstanding-requests, or power-of-two-choices, sends to the instance with the fewest in-flight requests and self-corrects for speed differences. Use it when instance latency varies; keep round robin when it does not, because it is cheaper and more predictable.

The failure mode that bites: resolution happens once

gRPC's default DNS resolver plus the pick_first policy resolves the name at startup, connects to the first address returned, and does not re-resolve unless that connection drops. A rolling deploy replaces every pod; the client keeps calling dead IPs until the connection breaks.

Trace it. Three pods behind a headless service. The client starts, resolves to [10.0.1.1, 10.0.1.2, 10.0.1.3], and pick_first connects to 10.0.1.1. The deploy replaces all three. 10.0.1.1 disappears, the connection breaks, and only then does the client re-resolve. With the round_robin policy and a 5s DNS TTL, the client re-resolves every 5s and picks up new addresses within one TTL: the difference between a few seconds of errors and a sustained outage.

What I would check before calling it done

  • Resolve on every call path, not once at startup, and keep the DNS TTL short (CoreDNS defaults to 5s for service records).
  • Remove instances from rotation via readiness probes, not liveness. A failing-but-alive instance should stop receiving traffic before it is killed.
  • Prefer least-outstanding-requests when instance latency varies.
  • Test it: replace every instance during a run and confirm callers resolve the new addresses within the TTL.

Curated: · Written: · Reviewed:

Q&A-74Design a rate limiter that works across fifty service instances.

Assumptions: the limit is per client key (API key or user id), 100 requests/min, 50 stateless instances behind a load balancer, and a Redis cluster already in the stack. A per-instance counter is not the limit the client sees: 50 instances each admitting 100/min lets one client push 5,000/min.

Keep the counter in Redis and make every check one atomic Lua script, so read-refill-decrement cannot interleave across instances. A token bucket is the right shape here: it caps the sustained rate at rate and allows a burst up to burst, which is what a per-minute quota actually means.

-- KEYS[1]=bucket key  ARGV: rate(tokens/s), burst, now(s), cost
local rate, burst = tonumber(ARGV[1]), tonumber(ARGV[2])
local now, cost  = tonumber(ARGV[3]), tonumber(ARGV[4])
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1]) or burst
local ts     = tonumber(b[2]) or now
tokens = math.min(burst, tokens + (now - ts) * rate)
local allowed = 0
if tokens >= cost then tokens = tokens - cost; allowed = 1 end
redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(burst / rate * 1000) + 1000)
return {allowed, tokens}

Redis runs a script to completion before serving another command on that shard, so the HMGET/HMSET pair is atomic without a lock. Trace it with rate = 100/60 = 1.667 tokens/s, burst = 100, cost = 1: at t=0 the key is missing, so tokens = burst = 100, one request is admitted, tokens = 99. At t=0.6s, tokens = min(100, 99 + 0.6*1.667) = 100, admitted, back to 99. At t=1.2s the same. A client that fires 100 requests in the first second drains to 0; the 101st is denied and gets retry-after = ceil((1 - 0)/1.667) = 1 second. Sustained throughput settles at 100/min regardless of which instance serves the request.

A sliding-window counter (two adjacent buckets, weighted by position in the window) is the alternative. It is cheaper to reason about for fixed windows and smooths the boundary burst, but it needs two keys and still one round trip, and it cannot express a burst allowance. Use the token bucket unless the product spec is literally "N per calendar minute."

One Redis round trip per request is the cost. To cut it, each instance leases tokens in batches: it asks the shared bucket for lease tokens and serves locally until they run out. The overshoot is bounded by instances * lease tokens in flight. With a 100/min quota and 50 instances, a lease of 5 puts 250 tokens outstanding: 2.5x the quota, unacceptable. A lease of 1 gives 50 outstanding, 50% overshoot. Local leasing only pays off when the quota dwarfs the fleet: at 10,000/min across 50 instances, a lease of 20 is 1,000 outstanding, a 10% overshoot, and cuts store calls from 1 per request to 1 per 20.

Redis unavailable is the failure that matters. Detect it as a client-side timeout or connection error on the script call, not a health check: the health check lags the actual failure. Default to fail open for public read APIs and fail closed for anything billing- or security-sensitive; that is a product decision, but pick it before the incident. The cost of failing open is that each instance falls back to its own limiter, which is the 50x hole again. Close it by giving every instance a local token bucket sized at quota / instances (2/min here) that it uses only while the store is unreachable, so the worst case is the intended quota, not 50x.

Two more things. Pass now from Redis TIME, not the instance clock: 50 hosts drift and a fast clock refills tokens early. And a single client key is a hot shard; Redis Cluster hashes each key to one node, so one abusive client pins one shard. Sharding the key by client id spreads load but breaks the shared counter, so keep one key per client and size the shard for the hottest client.

Verify by driving one client above quota through all 50 instances and confirming the admitted rate matches 100/min, then kill Redis mid-run and confirm the fallback holds at 100/min rather than 5,000.

Curated: · Written: · Reviewed:

Q&A-76How does SQL injection still happen, and what actually prevents it?

What I'd say first. Injection happens when a query's structure is built from text that includes untrusted input. The fix isn't to escape better: it's to send the statement and its values over separate channels so the database parses the statement before it ever sees the values. Everything else (least privilege, WAF, review) limits the blast radius; only that separation removes the bug.

Why binding works. In PostgreSQL's extended query protocol the client sends Parse (SQL text with $1 placeholders), then Bind (the values), then Execute. MySQL's binary protocol and JDBC's PreparedStatement do the same thing. The parser runs on the Parse message, so a value containing ' OR 1=1 -- is never tokenized as SQL: it's a string that gets compared. That's a structural guarantee, not a filter that can be outsmarted.

Why escaping and character allowlists fail. Hand-escaping is charset-dependent: with a GBK connection charset, the byte pair 0xbf 0x5c is read as one multibyte character, so the backslash inserted before a quote is swallowed and the quote survives. Quoting isn't the defense either: WHERE id = 1 needs no quotes, so numeric input 1 OR 1=1 walks straight in. Allowlisting characters still lets --, ; and UNION through in contexts you didn't anticipate.

Where binding can't reach. Placeholders occupy value slots, not identifier slots. Table names, column names, ORDER BY targets and the number of items in an IN (...) list are structure, so they can't be bound. Those go through a fixed allowlist, or get generated as N placeholders.

InputSafeUnsafe
filter valuebound parameterstring concatenation
sort columnallowlist of 6 columnsinterpolated name
page sizevalidated int, boundinterpolated text
IN listN generated placeholdersjoined user strings
# Python 3.12, psycopg 3.2
SORTABLE = {"created_at", "email", "id", "last_login", "name", "plan"}
col = SORTABLE.get(params["sort"], "created_at")   # never user text
cur.execute(f"SELECT id FROM users ORDER BY {col} LIMIT %s", (limit,))

The f-string is safe only because col comes from a six-element set. limit is bound.

A trace. Take email = "x' OR '1'='1' --".

Unsafe:

cur.execute(f"SELECT id FROM users WHERE email = '{email}'")

The string handed to the server is SELECT id FROM users WHERE email = 'x' OR '1'='1' --' The -- comments out the trailing quote, so the server executes ... WHERE email = 'x' OR '1'='1' and returns every row.

Safe:

cur.execute("SELECT id FROM users WHERE email = %s", (email,))

The server parses ... WHERE email = $1 and binds the literal x' OR '1'='1' --. It matches no row. Same input, different channel.

Failure modes I'd name.

  • Second-order injection: input is stored safely on write, then a later report or stored procedure concatenates it into dynamic SQL. Parameterizing the write doesn't protect the read.
  • ORM escape hatches: Model.objects.raw(...), .where("name = " + x), or an order() fed from a query string. The ORM is only as safe as the fragments you hand it.
  • Stored procedures that build SQL with EXEC / EXECUTE IMMEDIATE internally: the caller binds correctly and the injection still happens inside.

How I'd verify. Send payloads through every parameter, including sort and filter names, and assert the result set and query plan don't change. Static analysis (Semgrep, CodeQL) catches concatenation patterns in review; a DAST scan catches the ones it misses. Neither replaces the test.

Blast radius. Run the app as a role that can only touch the tables it needs, with no DROP, no stacked statements, and a separate read-only role for reporting. That doesn't prevent injection; it decides how bad the worst day is.

Curated: · Written: · Reviewed:

Q&A-77What is the most common serious API vulnerability you would look for in a code review?

My approach to broken object-level authorization separates what the database guarantees from what the code assumes.

The usual flaw is an endpoint that authenticates the caller but never checks that the requested object belongs to them, so changing an identifier in the URL returns someone else's data. Authentication answers who, and this check answers whether this object is theirs.

Concretely, scope every query by the caller's tenant or owner rather than filtering after the fetch, put the check in a shared layer so new endpoints inherit it, and test with two accounts on every endpoint that takes an identifier.

The reason for that specificity is a failure I have seen: An invoice endpoint fetched by id and checked only that the caller was logged in, so incrementing the id exposed 9,000 invoices across tenants.

Two accounts against one endpoint.

QueryOwn invoiceAnother tenant's invoice
fetch by id only200200, leak
fetch by id and tenant200404

I would not consider it settled without evidence: Call every identifier-taking endpoint with another tenant's id and confirm each returns 404 or 403.

Authenticate the caller, then authorize the object.

Curated: · Written: · Reviewed:

Q&A-100How do you stop sensitive data ending up in your logs?

The 90-second answer. Redact where the line is produced, not in the pipeline. Logs get copied into a store with different retention and a far wider reader set than the database, so anything you log is a second copy you no longer control. Concretely: a logging API that only accepts declared fields, no request or response bodies at INFO, no query strings in access logs, and a canary check in CI that fails the build if a known sentinel value turns up in the log store.

Why the call site. A pipeline scrubber only sees a string. By then {"email": "a@b.com"} and {"note": "call a@b.com back"} are the same shape, and a regex over free text either misses the second or mangles the first. The call site is the only place that knows what the field means, so the decision has to be made there, and it has to be the default, not an opt-in, or the next debug line bypasses it.

# Python 3.12, stdlib logging only
import json, logging

SENSITIVE = {"password", "token", "authorization", "card_number", "email"}

def redact(v):
    if isinstance(v, dict):
        return {k: "[REDACTED]" if k.lower() in SENSITIVE else redact(x)
                for k, x in v.items()}
    if isinstance(v, list):
        return [redact(x) for x in v]
    return v

class JsonFormatter(logging.Formatter):
    def format(self, record):
        return json.dumps({**redact(record.__dict__.get("fields", {})),
                           "event": record.getMessage()}, sort_keys=True)

log = logging.getLogger("signup")
log.setLevel(logging.INFO)
h = logging.StreamHandler(); h.setFormatter(JsonFormatter()); log.addHandler(h)

log.info("signup", extra={"fields": {
    "customer_id": 90210, "email": "a@b.com",
    "card_number": "4242424242424242", "nested": {"token": "abc"}}})

Emits:

{"card_number": "[REDACTED]", "customer_id": 90210, "email": "[REDACTED]", "event": "signup", "nested": {"token": "[REDACTED]"}}

Note the limit: that is a key-name denylist, so {"raw": ..., "payload": ..., "body": ...} sails straight through. The stronger version is an allowlist: the formatter emits only keys declared in a per-event schema and drops the rest. And it only sees what arrives in fields; log.info("signup %s", body) never touches it, so pair it with a lint rule banning %/f-string formatting of non-literal arguments.

FieldLogged asWhy
customer emailcustomer_id 90210identifier is enough to correlate
card numberlast 4support needs a hint; the PAN is out of scope
auth tokennothingcredential: a log reader can replay it
request bodynothing at INFOcontains all of the above
user id in metricsnothinglabel cardinality, and metrics have their own retention

Failure modes, and how you catch them.

  • Exception text: ValueError: invalid card 4242424242424242. Log type(e).__name__ plus a correlation id, not str(e).
  • HTTP client debug: http.client at DEBUG logs the raw request line including Authorization:; OkHttp's HttpLoggingInterceptor.Level.BODY logs full bodies. Both are one config line from production.
  • Access logs and proxies log the whole path: /reset?token=.... Log the route template, not the URL.
  • Tracing: span attributes and db.statement can carry literal parameters. Keep statements parameterized and check the instrumentation's capture settings.
  • APM and crash SDKs: check the PII switches (sentry-python's send_default_pii, max_request_body_size) rather than trusting defaults.

Verification. Generate a unique sentinel per deploy (canary-7f3a@example.invalid), drive one request through the signup path in staging, then query the log store for it; any hit fails the deploy. Add a nightly regex scan for PAN-shaped and JWT-shaped strings over the previous day's index.

Trade-off. Redaction costs debuggability. When you genuinely need the value, don't log it: put it in a short-retention, access-controlled store keyed by the correlation id, or log an HMAC of it with a per-environment key. HMAC only helps for high-entropy values; emails are low-entropy, so an HMAC'd email is still recoverable by dictionary attack and still personal data.

Scale of the mistake. 240,000 signups over a 30-day retention window is ~5,600/day, ~0.06 rps: a small product. If the signup handler logs the request body at INFO, that is 240,000 plaintext passwords in an index the whole engineering org can search, sitting next to a database where they are bcrypt hashes.

Curated: · Written: · Reviewed:

Q&A-79What does encrypting data at rest actually protect against?

Encryption at rest protects data when the storage medium leaves your control: a stolen laptop, a decommissioned drive, a backup tape, a snapshot shared with the wrong account, an object-store bucket someone flipped to public. In each case the attacker holds bytes and no running system, and ciphertext is the entire defence.

It protects nothing against an attacker who reaches the data through your application. A leaked credential, SQL injection, an over-permissive internal endpoint, a support tool with read access: every one of those reads plaintext, because the database decrypts on the way out. Disk encryption is transparent to any query that succeeds.

Why the boundary sits there

Modern at-rest encryption is envelope encryption. A data key (DEK) encrypts the volume or object; a key-encryption key (KEK) in a KMS wraps the DEK; the wrapped DEK is stored beside the ciphertext. When the database starts it calls KMS to unwrap the DEK and holds it in process memory. Every read after that is plaintext.

So the security boundary is not the disk. It is whoever can call Decrypt on that KEK. If the database's IAM role can decrypt and an attacker can reach the database as that role, encryption at rest contributed nothing.

ThreatDisk/volume at restTLS in transitField-level, key held by a separate authorizing service
Stolen disk or backupblockedn/ablocked
Snapshot shared to wrong accountblockedn/ablocked
Network capturenoblockedblocked
Compromised app credentialnonoblocked only if the key service checks the caller
SQL injection / over-permissive endpointnonosame as above
Rogue admin with KMS decrypt rightsnonono

Failure modes worth naming

  • Key stored with the data. A DEK in the same bucket as the ciphertext, or a KMS key policy granting kms:Decrypt to *. The ciphertext is decorative.
  • One role decrypts everything. The app role, the analytics role and the on-call role share a key. A single leaked credential now reads every tenant.
  • Backups escape the boundary. The primary database is locked down; a nightly pg_dump lands in a bucket in another account with default ACLs. Encryption at rest on the primary does not follow the dump.
  • Rotation without re-encryption. You rotate the KEK, old DEKs stay wrapped under the old version, and the old version is never disabled. Rotation changed nothing.
  • Field-level encryption where the app holds the key. The app decrypts before serializing the response, so an attacker using the app's credential gets plaintext. Field-level only helps when a separate service authorizes each caller.

Detection: the database's pg_stat_statements shows one query shape scanning the whole table; the endpoint's access log shows requests with no auth header; CloudTrail shows Decrypt calls from an unexpected principal.

Worked example (hypothetical figures)

A 400 GB Postgres table, 2.3 million customer rows, volume encrypted with AES-256 through KMS. An internal endpoint /internal/users ships without an auth check. An attacker with network access pages through it at 1,000 rows/second and has the full table in about 40 minutes. Disk encryption never engages: the query arrives as the application's own DB role, which is allowed to decrypt. The fix is authorization on the endpoint, not a stronger cipher.

What to say in the interview

Name the threat each layer covers, then confirm the live query path has its own control. Disk encryption answers "what if the bytes get out"; it does not answer "what if someone asks the database nicely."

Curated: · Written: · Reviewed:

Q&A-8How do you handle identity in a service-to-service call chain?

The 90-second answer

Two identities travel in a service-to-service chain and they must not be conflated: the workload identity (which service is calling) and the user identity (on whose behalf). Clarify which one the interviewer means, then answer for both.

Assume: services sit in one trust domain, there is an API gateway, and the caller is an authenticated end user. My answer: authenticate the workload with mTLS, carry the user as a signed, audience-scoped token, and re-check authorization at the service that owns the data.

Mechanism

Workload identity: each service gets a short-lived X.509 cert from a workload-identity issuer. SPIFFE/SPIRE issues X.509-SVIDs with a default TTL of 1 hour, and the mesh does mTLS on every hop. This is what defeats "call the pod directly": a forged header does not get you a valid client cert.

User identity: the gateway validates the user's OAuth 2.0 access token (a JWT, RFC 7519), then either forwards it or exchanges it. Forwarding is simplest but leaks the user token to every service. Token exchange (RFC 8693) mints a new token with aud set to the callee and a narrowed scope, so a token lifted from service B cannot be replayed against service C. If you need proof the token is bound to the caller, mTLS-bound tokens (RFC 8705) put the cert thumbprint in cnf.x5t#S256.

Every service validates signature, iss, aud == itself, and exp, then authorizes against the specific resource. Access-token TTLs of 5–60 minutes keep the revocation window small; immediate revocation needs a token-introspection call or a denylist, which costs a round trip.

LayerChecksBypassable
gatewayuser token signature, rate limityes, by direct pod call
serviceclient cert, token aud/exp, tenant, permissionno
databaserow-level constraintno

Trade-offs

  • Network segmentation vs per-service verification. mTLS plus NetworkPolicy stops direct pod calls and is cheap (no app code). It gives you no user identity and no per-resource authz. Per-service verification costs a signature check per hop (roughly 1 µs for HS256, 50–100 µs for RS256/ES256) plus key distribution. You want both: segmentation is the floor, verification is the ceiling.
  • Centralized policy vs embedded checks. An OPA sidecar or Envoy ext_authz centralizes rules and gives one audit point, at roughly 1 ms p99 per decision and a new failure mode: policy service down means all calls fail, so decide fail-open vs fail-closed explicitly. An in-process library (OpenFGA client, Cedar) has no network hop but you redeploy to change policy. Centralized wins when many services share rules; embedded wins for one service with simple rules.
  • When the heavy design is not worth it. A single-tenant internal batch job with no end user needs workload identity only: mTLS, no user token, no per-resource authz. Do not build token exchange for a chain that never carries a user.

Failure mode and how I would verify

The failure I have seen: an internal service trusted an X-User-Id header set by the gateway; a request sent straight to the pod returned another tenant's rows. Detection: the service's own logs show a user id with no matching token sub.

Verification: from a pod, call each internal endpoint directly with a forged X-User-Id and no client cert. Expect 401. Then with a valid cert but a token whose aud is another service: expect 401. Then a valid token for a resource in another tenant: expect 403, not a 404 that leaks existence.

Curated: · Written: · Reviewed:

Q&A-9Would you use signed tokens or server-side sessions for an API?

I would pick based on the maximum revocation delay the API can tolerate, not on whether stateless sounds better. For a typical public API I default to short-lived signed access tokens plus server-side refresh tokens: the access token is verified locally, and the refresh token is the thing I can revoke. If the API must deny a stolen or disabled credential on the next request, I use server-side sessions or opaque tokens with a store lookup. The assumption that matters is whether the auth store is on the same fast network and whether the request rate makes a per-request lookup affordable.

A signed token works because the server checks a signature and an expiry. There is no store read, so verification scales with CPU. Revocation before expiry requires either a denylist checked on every request, which reintroduces the lookup, or waiting for expiry. A server-side session works because the client sends a random ID and the server looks it up. Deleting that record revokes access immediately, but every request pays a store read and the store becomes a dependency.

SchemeRevocation delayStore read per request
24-hour signed tokenup to 24 hnone
5-minute signed token with refreshup to 5 minon refresh
server-side sessionimmediateevery request

At 10,000 req/s, a 1 ms session lookup is 10,000 ms of store work per second, so you need at least 10 concurrent lookups just to keep up. At 100 req/s it is 0.1 s/s and a single Redis instance is idle. That arithmetic is why high-volume APIs push verification to the edge and keep revocation state for refresh tokens.

I have seen 24-hour access tokens with no revocation. A dismissed employee's token kept working for 19 hours after the account was disabled. The detection is simple: disable an account in staging and measure how long an already-issued credential still works. If that number is unacceptable, the token lifetime is the bug.

Other failure modes matter too. A denylist without a TTL grows forever and becomes a second database. A session store outage locks out every user if there is no fallback. Key rotation invalidates all signed tokens at once unless you publish overlapping keys.

My default is 5-15 minute signed access tokens, refresh tokens stored server-side and rotated, and a revocation check for refresh and for high-risk actions. For admin APIs, use server-side sessions or require a fresh server-side check per request. Never put secrets in a signed token payload; signing proves integrity, not confidentiality.

The number that decides it is the maximum time an attacker can keep using a credential after you press disable. If that number is minutes, signed tokens are fine. If it is zero, use sessions.

Curated: · Written: · Reviewed:

Q&A-78How do you keep one tenant's data from reaching another in a shared database?

The 90-second answer

Assume a shared-schema Postgres, a tenant_id column on every tenant-scoped table, one application, and a connection pooler in front. The invariant: no query may return a row whose tenant_id differs from the caller's, and it has to hold for queries nobody has written yet.

So enforce it in the database, not in the query. Postgres row-level security attaches the filter to the table, so a query that forgets the predicate still gets filtered:

ALTER TABLE orders ENABLE ROW LEVEL SECURITY;
ALTER TABLE orders FORCE ROW LEVEL SECURITY;   -- table owner is not exempt

CREATE POLICY tenant_isolation ON orders
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

-- per request, inside the transaction:
BEGIN;
SET LOCAL app.tenant_id = '8f14e45f-...';
SELECT ... FROM orders WHERE customer_id = $1;   -- no tenant predicate needed
COMMIT;

SET LOCAL is transaction-scoped, so the value dies at COMMIT. That detail matters: with PgBouncer in transaction pooling mode a session-level SET outlives the transaction, and the next client handed that connection inherits the previous tenant's id. SET LOCAL cannot leak that way.

FORCE ROW LEVEL SECURITY matters because the table owner bypasses policies by default, and migrations and background jobs usually connect as the owner. Superusers and roles with BYPASSRLS bypass regardless, so the application role must be neither.

Then the app layer: one module owns the connection, sets the tenant, and is the only thing that hands out a session. Not because it enforces anything, the policy does, but because it is where the tenant id comes from.

Where it breaks

  • Pool reuse. Session-level SET plus transaction pooling gives cross-tenant reads. Detect it by asserting current_setting('app.tenant_id') at the top of every transaction.
  • Owner connections. A nightly job connecting as the table owner reads every tenant. FORCE ROW LEVEL SECURITY closes that; a BYPASSRLS role reopens it.
  • Caches and materialized views. A Redis key customer:42:orders is fine; orders:recent is not. Materialized views are refreshed by their owner and carry no policy of their own.
  • Analytics replicas. A read replica whose role has BYPASSRLS for convenience is a full cross-tenant export waiting to happen.

Alternatives

ModelIsolationCost
shared table + RLSpolicy per table, enforced by the enginepolicy maintenance; tenant_id must lead every index
schema per tenantsearch_path per connectionmigration fan-out across N schemas; search_path is session state, same pool hazard
database per tenantstrongestone connection per tenant, cross-tenant queries need a fan-out layer

Schema-per-tenant moves the check to search_path, which is session state: the same pool-reuse hazard as SET. Database-per-tenant is the only model where a forgotten predicate cannot cross tenants, and you pay for it in connections.

Proving it

Run the integration suite twice against the same database, once per tenant, and assert that no query returns a row belonging to the other. Add a startup assertion that the app role is neither superuser nor BYPASSRLS. The policy is a claim; that run is the evidence.

Curated: · Written: · Reviewed:

Q&A-26A query filters on tenant and status and sorts by created time. What index would you add?

Add a composite index on (tenant_id, status, created_at): equality columns first, sort column last. The index filters and returns rows already in created_at order, so the planner drops the sort.

Assumptions I'd state before committing: Postgres 11+ or MySQL 8.0 InnoDB; the query is WHERE tenant_id = ? AND status = ? ORDER BY created_at DESC LIMIT n; tenant_id is high cardinality, status is low cardinality, and status changes during a row's life.

Why that order: a B-tree is usable left to right. tenant_id leading confines the scan to one tenant's slice; status narrows within it; created_at last means entries inside (tenant_id, status) are already ordered, so the scan reads in order and stops at the LIMIT. Swap created_at ahead of status and the index still filters, but the engine sorts the matches.

CREATE INDEX idx_tenant_status_created
  ON orders (tenant_id, status, created_at DESC);

Postgres and MySQL 8.0 both scan an ASC index backwards, so DESC matters only for mixed directions.

Illustrative numbers on a 4.1M-row orders table, one tenant holding ~26k rows, ~1.2k of them in the target status:

IndexRows examinedSort
(created_at)4,100,000index order, filter applied per row
(tenant_id)26,000in-memory sort
(tenant_id, status, created_at)1,200none

When this is the wrong index.

  • Low-cardinality status in the key. With three status values it buys little selectivity and costs on every write: a status transition rewrites the index entry (Postgres writes a new tuple in every index on a non-HOT update; InnoDB updates the secondary entry in place). If status churns, use (tenant_id, created_at) and leave status as a residual filter: you keep the per-tenant ordering and the early LIMIT stop, and you pay for two columns instead of three.
  • Write amplification. Every index is extra work on INSERT/UPDATE/DELETE. On a write-heavy table, check pg_stat_user_indexes.idx_scan against idx_tup_read before adding a third column.
  • Covering instead. If the query selects a few narrow columns, INCLUDE (Postgres 11+) or InnoDB's implicit PK suffix avoids heap lookups. Widening the index raises its size and write cost, so it only pays for a hot query.
  • Partial index. If the query always filters one literal status, CREATE INDEX ... ON orders (tenant_id, created_at) WHERE status = 'active' is smaller and cheaper, but it can't serve a parameterized status.
  • Sharding. If the table is sharded by tenant, tenant_id leading is redundant with the shard key; drop it.

Failure modes and how I'd catch them. Wrong column order shows up as a full index scan: EXPLAIN (ANALYZE, BUFFERS) reports rows examined near the table size. Stale statistics make the planner pick a seq scan; same check. Status churn bloats the index; watch pg_relation_size on it. Verify by comparing EXPLAIN (ANALYZE, BUFFERS) before and after: rows examined, and whether a Sort node is still present.

Curated: · Written: · Reviewed:

Q&A-27A query is slow in production but fast locally. How do you diagnose it?

Assumptions first. "Fast locally" almost always means the same query text against a local copy of the schema with orders of magnitude fewer rows, and possibly different statistics. If that's the case, the local timing is not evidence about production. The plan is chosen from statistics and data volume, so a plan built against 500 rows tells you nothing about 50 million. I don't try to reproduce the slowness locally; I read the production plan.

The first move is to capture the plan with actual row counts on production (or a production-sized clone): EXPLAIN (ANALYZE, BUFFERS) in Postgres, EXPLAIN ANALYZE in MySQL 8, or the actual execution plan in SQL Server. Then compare estimated vs actual rows at each node. The node where the estimate diverges most is the culprit: usually a stale or coarse statistic.

Representative figures, not a specific incident:

NodeEstimated rowsActual rows
index scan on status9002,400,000
nested loop join9002,400,000

The planner chose a nested loop because it believed 900 rows, so the inner side ran 2.4 million times. That is the 46 s vs 30 ms gap: same query, same code, different plan.

Then the levers, roughly in order of how often each is the answer:

  1. Stale statistics: run ANALYZE / UPDATE STATISTICS, or raise the sample size. Cheap, fixes the estimate, often fixes the plan.
  2. Missing or wrong index: the plan scans a large table where a seek would do. Add the index the predicate actually needs, with column order matching the WHERE and ORDER BY.
  3. Parameter sniffing: the plan is good for one bind value and bad for another. Postgres: prefer a plan that stays stable across values; SQL Server: OPTION (RECOMPILE) or a filtered/optimized-for hint.
  4. The query itself: a rewrite (semi-join instead of a correlated subquery, pre-aggregate) when the plan is already reasonable but the work is inherently large.

When plan-fixing is the wrong lever. If the plan is already a seek with accurate estimates and the query is still slow, the cost is elsewhere: network round trips from a chatty N+1 in the application, lock waits, or a cold buffer cache. Check those before touching the query. And if the slowness only appears under concurrency, suspect contention or connection-pool exhaustion, not the plan.

Failure modes and how to detect them:

  • Statistics stale after a bulk load: estimates off by orders of magnitude; detect by comparing estimated vs actual in the plan.
  • Plan flips between parameter values: same query fast for one bind, slow for another; detect by running it with each value.
  • Sort spilling to disk: "external merge" / "Using filesort" in the plan; detect via buffers or the sort node's memory.
  • Lock contention: the query waits rather than runs; detect with pg_stat_activity wait events or the equivalent.

I confirm against the production plan and actual row counts, change one thing, and re-measure: the plan, not the local timing, is what I trust.

Curated: · Written: · Reviewed:

Q&A-28A list endpoint issues one query per row. How do you find and fix it?

Core answer. Count the queries one request makes, then replace the per-row load with a single batched load.

The symptom is a list endpoint whose latency scales with page size. The request loads N rows, then touches a relation on each row, and the ORM lazily issues one SELECT per row: N+1 queries. A 20 ms page turns into hundreds of round trips.

Find it by counting, not by guessing. Attach a counter to the driver's execute hook and log the count per request, or read the span count in your APM trace. A list endpoint showing 1 + N spans, where N equals the page size, is the bug.

# SQLAlchemy 2.0
from sqlalchemy import event, select
from sqlalchemy.orm import joinedload, selectinload

query_count = 0  # reset at the start of each request

@event.listens_for(engine, "before_cursor_execute")
def _count(conn, cursor, statement, params, context, executemany):
    global query_count
    query_count += 1

The fix comes in two shapes:

# N+1: 1 SELECT for orders, then 1 SELECT per order for line_items
orders = session.scalars(select(Order)).all()
for o in orders:
    print(o.line_items)

# Fix A: single JOIN
orders = session.scalars(
    select(Order).options(joinedload(Order.line_items))
).unique().all()

# Fix B: two queries, the second keyed by parent ids
orders = session.scalars(
    select(Order).options(selectinload(Order.line_items))
).all()

joinedload emits one SELECT with a LEFT OUTER JOIN; selectinload emits a second SELECT with WHERE order_id IN (...). Both collapse N+1 into 1 or 2 queries.

Worked example, 200 orders each with line items:

ImplementationQueriesDuration
lazy per row2011,800 ms
one join140 ms
two queries and a map245 ms

Trade-offs. The join is one round trip but duplicates the parent row per child, so 200 orders with 10 items each return 2,000 rows and the ORM has to de-duplicate them (.unique()). Two queries avoid that row blowup and stay flat as the child count grows, at the cost of one extra round trip. With two collections on the same parent, two joins multiply into a cartesian product, so use selectinload there.

Failure modes. Eager-loading everything hides the N+1 but over-fetches on endpoints that never touch the relation. A join over a large collection can exhaust memory before it shows up as latency. And the pattern returns the moment someone adds a new relation, so assert a query-count ceiling per request in a test (fail a list endpoint that exceeds, say, 5 queries) and alert on p99 query count per route, not just duration.

Curated: · Written: · Reviewed:

Q&A-32Your service opens a database connection per request. What goes wrong at scale?

Assumption: a long-lived service process (not a Lambda), a Postgres or MySQL primary, and a client library that opens a fresh TCP connection per request.

Three things break, roughly in this order: the database's connection limit, the database's CPU and memory, and the client's socket budget. A connection is not a cheap handle. In Postgres it is a forked OS process with its own memory, and the server has to schedule it. Past a few hundred, adding connections lowers throughput rather than raising it.

What a per-request connect actually costs

On a 1 ms RTT network with TLS:

  • TCP handshake: 1 RTT
  • TLS 1.3 handshake: 1 RTT
  • SCRAM auth: another round trip
  • Postgres forks a backend and initialises its catalog caches

That is 3-5 ms of setup per request, plus a few MB of server memory per backend that stays allocated for the connection's life. At 2,000 req/s you are doing 2,000 setups per second; at ~4 ms each that is 8 CPU-seconds of setup work per second of wall clock, split across client and server. The database spends more time creating connections than answering queries.

The failure at scale

Postgres max_connections defaults to 100, with 3 reserved for superusers. MySQL's defaults to 151. A pool per instance multiplies against that ceiling: 40 pods at HikariCP's default maximumPoolSize of 10 want 400 connections against a limit of 100.

ConfigClient connsServer connsResult
40 pods x pool 10400400 attempted, 100 allowed~300 fail with FATAL: sorry, too many clients already
40 pods x pool 28080fits, but no headroom for a rolling deploy
40 pods x pool 10 -> PgBouncer, transaction mode, default_pool_size=2040020fits with headroom

The failure is not graceful. Each pod's pool fills to 10 under load, so all 40 pods hit the ceiling at the same moment. Requests that cannot acquire a connection block until connectionTimeout, HikariCP's default is 30 s, and then throw. On a thread-per-request server, every in-flight request holds a worker thread for those 30 s, the thread pool drains, and the service returns 503 for everything, including endpoints that never touch the database.

Sizing the pool

Little's Law: pool size = arrival rate x connection hold time. 2,000 req/s x 4 ms = 8 concurrent connections, so a pool of 10 is right. If a query regresses from 4 ms to 40 ms, required concurrency becomes 80; a pool of 10 queues 70 requests, and with a 250 ms acquisition timeout most of them fail instead of waiting. That is the behaviour you want (fail fast, shed load) but only if the timeout is short. Set it to a few hundred milliseconds, not the 30 s default.

What to watch

  • pg_stat_activity count by state, and remaining max_connections headroom
  • HikariCP hikaricp_connections_pending and hikaricp_connections_acquire_seconds
  • FATAL: sorry, too many clients already in server logs
  • Client-side TIME_WAIT sockets: Linux's ephemeral range (32768-60999, 28,232 ports) with a 60 s TIME_WAIT caps you at roughly 470 new connections per second to a single host

When a pool is not enough

PgBouncer in transaction mode multiplexes many client connections onto few server connections. It breaks session state: SET, LISTEN/NOTIFY, advisory locks, temp tables, and prepared statements. PgBouncer 1.21+ handles protocol-level prepared statements when max_prepared_statements is set. Serverless is the other case: each invocation is a new process, so use a proxy or the platform's pooler rather than a per-process pool.

Curated: · Written: · Reviewed:

Q&A-36Would you use an auto-incrementing integer or a UUID as a primary key?

Short answer: use a sequential or time-ordered integer as the clustered primary key, and expose a separate opaque identifier when the ID is visible to users or partners. A fully random UUID is fine as a non-clustered unique column; it is a poor clustered key on a write-heavy table.

Assumptions worth stating out loud: single-region Postgres or MySQL, IDs appearing in URLs or APIs, and inserts dominated by appends. If the table is sharded across regions with no central sequence, or IDs must be generated offline on the client, the calculus flips toward time-ordered UUIDs.

Why it behaves this way. In InnoDB the primary key is the clustered index, so a random key scatters both the row and the index. In Postgres the heap is ordered by insertion, but the PK is still a B-tree that a random key scatters, and every insert touches a cold index page. A monotonic key appends to the rightmost leaf; a random key lands anywhere, so once the working set exceeds the buffer pool each insert dirties a page that is probably not in memory, and full pages split. That costs a read, a split and two writes per insert, plus roughly 50% fill factor on the split pages, which is where index-size growth comes from. Time-ordered UUIDs (UUIDv7, ULID, Snowflake) keep the high bits monotonic, recovering most of the locality while staying globally unique and generatable without a round trip. Postgres 18 added a built-in uuidv7(); on older versions and on MySQL, generate it in the application.

Failure modes to name.

  • Enumeration. /orders/1001 tells a competitor your daily volume and invites scraping. Fix: a separate public ID (UUID or random 128-bit token) with a unique index, never the storage key.
  • Hot rightmost page. A single auto-increment sequence serializes inserts on the last leaf page. At high write rates this shows up as index lock waits, not CPU. Fix: interleave the low bits or shard the sequence.
  • Sequence gaps. Gaps after rollback are normal: never treat the ID as a count. On failover, MySQL's auto_increment_increment/auto_increment_offset and Postgres sequence caching hand out skipped or out-of-order values.
  • UUIDv4 as clustered key. The page-split cost above. Detect it with index-size growth and insert-latency p99, not intuition.

Worked example (illustrative figures, not a benchmark). A 60M-row orders table, ~2k inserts/s, buffer pool smaller than the index:

KeyInsert p50Insert p99Index sizeEnumerable
auto-increment bigint3 ms8 ms1.0xyes
random UUIDv422 ms90 ms1.4xno
time-ordered UUIDv74 ms12 ms1.2xno

The 3 → 22 ms jump is the page-split path; the 1.4x is the ~50% fill factor on split pages. Confirm both with your own pgbench or sysbench run at production row counts before committing.

What I'd actually do. A bigint identity column as the clustered PK, a uuid (v7 where available, else v4) as the public identifier with a unique index, and no random UUID as a clustered key on a write-heavy table. If the system is multi-region or offline-first, invert it: UUIDv7 as the PK, because coordination cost beats index locality there.

Curated: · Written: · Reviewed:

Q&A-37When is a JSON column the right choice, and when is it a trap?

The 90-second answer

A JSON column is right when the shape genuinely isn't yours to fix: third-party webhook payloads you must keep verbatim, per-tenant attributes where each tenant uses a different handful of keys, config blobs you always read whole. It's a trap the moment a field inside it is something you filter, sort, join or constrain on: you've moved a schema decision into a place the database can't enforce or plan well.

The rule I use: promote anything you query to a real column, keep the rest in JSON, and add an expression index for the one or two paths you can't promote yet.

Why it behaves that way

Postgres jsonb is a binary document with no per-key statistics, no type, no NOT NULL, no foreign key, and no partial update. json (text) is worse for querying: it preserves key order and duplicates but has no index support at all. MySQL 8.0's JSON is similar: no direct index; you index a generated column, or since 8.0.17 a multi-valued index over an array path.

Two consequences drive everything:

  • A predicate on data->>'status' is an expression, not a column. Without an index on that exact expression, the planner has no choice but a sequential scan.
  • Updating one key rewrites the whole document. jsonb_set produces a new value, and once the row exceeds ~2 KB the value is TOASTed out of line, so a hot document means write amplification and bloat.

Worked example (hypothetical, Postgres 16)

orders(id bigint, data jsonb), 12M rows, ~400 bytes of JSON each, ~4.8 GB.

QueryPlanRows readTime
WHERE data->>'status' = 'shipped'Seq Scan12M~8 s cold
same, with CREATE INDEX ON orders ((data->>'status'))Index Scan~4k~5 ms
WHERE status = 'shipped' on a real text columnIndex Scan~4k~5 ms

The expression index closes the latency gap. What it does not close: no NOT NULL, no type enforcement, no foreign key (Postgres FKs can't be backed by an expression index), and every writer must spell the expression exactly as the index defines it or the index is silently unused.

For containment (data @> '{"status":"shipped"}', or matching any element of an array) a GIN index is the tool. jsonb_ops supports @>, ?, ?|, ?&; jsonb_path_ops supports only the containment family (@>, @?, @@) and is several times smaller and faster for those. GIN is also more expensive to update than btree, which matters on a write-heavy table.

Failure modes I'd name in an interview

  • Type drift. data->>'qty' is text, so '10' < '9' is true. Detect with a CHECK on the expression or validation at the write boundary.
  • Hot document. A counter or status field updated per request rewrites the row each time. Detect via n_tup_upd versus n_tup_hot_upd in pg_stat_user_tables, plus table bloat.
  • Orphaned references. No FK means nothing stops data->>'customer_id' pointing at a deleted row. Detect with a periodic anti-join.
  • Bad estimates on unindexed paths. ANALYZE collects statistics for expression indexes but not for arbitrary JSON paths, so the planner falls back to defaults. Detect by comparing estimated to actual rows in EXPLAIN (ANALYZE).

When not to use it

Money, quantities, anything with a stable schema, anything you join on, anything you sort by: those get columns. If you need sparse per-tenant fields on Postgres, a jsonb column plus promoted columns beats either extreme; if the fields are truly fixed, they're columns.

Before I'd call it settled: list every field the application filters, sorts, joins or constrains on, and confirm each is a column or has an index on the exact expression used.

Curated: · Written: · Reviewed:

Q&A-35When would you denormalize a schema deliberately?

Denormalize when one specific read path is measurably dominated by a join or an aggregate, and every write that changes the source can update the copy in the same transaction or through an ordered event stream. If either half is missing (no measurement, or more than one write path) keep it normalized.

Assumptions I'd state before answering: the workload is read-heavy on that path, the source rows are owned by one service, and the derived value is rebuildable from the source, so a wrong copy is a bug to fix rather than data loss.

Normalization keeps one authoritative copy of each fact, so two rows cannot disagree. Denormalization trades that guarantee for read cost: you pay an extra write and a consistency obligation to remove a join or a SUM from the hot path. That trade is only worth making against a number.

For an order total derived from line items, the three designs look like this (figures from an illustrative order service: the shape of the measurement matters more than the exact milliseconds):

DesignRead costDrift risk
SUM line items per read14 msnone
denormalized, updated in one write path0.4 ms1,400 rows stale
denormalized, updated in the transaction plus a nightly check0.4 msdetected

The middle row is the failure I'd want to avoid. An order total was denormalized onto the order row but updated only in the API path. A batch job that changed 1,400 orders bypassed that path, and the stale totals sat there for three weeks until someone reconciled by hand. The bug was not denormalization; it was two write paths and no detector.

So the shape of a deliberate denormalization is: one owner for the copy, one write path that updates it (same transaction, or an outbox/CDC event consumed idempotently), and a reconciliation query that runs on a schedule and alerts on a non-zero drift count. If the copy is updated in the same transaction as the source, drift is impossible and the reconciliation is a cheap assertion. If it is updated by an event, the reconciliation is load-bearing: consumers can be replayed, reordered, or dropped.

A reconciliation query for the order total:

SELECT o.id, o.total_cents, SUM(li.qty * li.unit_price_cents) AS computed
FROM orders o
JOIN line_items li ON li.order_id = o.id
GROUP BY o.id, o.total_cents
HAVING o.total_cents <> SUM(li.qty * li.unit_price_cents);

Run it on a replica, and expect false positives from replication lag: compare against a snapshot or a lag-bounded read, or you will page on rows that are correct a second later.

Before denormalizing, check the cheaper alternatives: a covering index that makes the join index-only, a materialized view refreshed on a schedule, or a cache with a TTL if a slightly stale read is acceptable. Those keep one source of truth and move the cost elsewhere.

Do not denormalize when the write rate is high relative to reads, when the read volume is low enough that 14 ms never shows up in a p99, or when the value is a source of truth rather than a derivation: an account balance is not a copy of anything, and storing it as one invites double-spend bugs. If you cannot name who owns the copy and which query proves it correct, leave it normalized.

Curated: · Written: · Reviewed:

Q&A-38A table has grown to two billion rows. What are your options?

Two billion rows is not automatically a problem. The first thing I'd do is confirm it is one: check p99 latency on the queries that matter, index size, autovacuum or compaction time, and backup and restore duration. An append-only events table with a covering index and time-filtered queries can sit at two billion rows happily. A table whose hot query is an unindexed aggregate is in trouble at two hundred million. So the options below are ordered by cost, and the cheapest is often "measure, then do nothing."

If it is a problem, the cheapest real fix is usually getting rid of rows you don't need. Tables that reach two billion rows are typically append-heavy with a long tail of cold data. If most rows are older than the retention window, archiving them to object storage or a cold table and dropping them is the win. Do it by dropping whole partitions, not by DELETE: deleting 1.8 billion rows in place creates 1.8 billion dead tuples, autovacuum has to reclaim them, and the table file doesn't shrink without a rewrite (VACUUM FULL or pg_repack), which takes an exclusive lock. Dropping a partition is a metadata operation: milliseconds.

Partitioning is the next step and the one most people reach for. It splits one table inside one database so a query touches a slice instead of the whole thing, and maintenance (vacuum, index rebuild, backup) works per partition. Partition by the column queries filter on, usually time or tenant, and make sure every query carries that key. The failure mode is exact: a two-billion-row events table partitioned by month, queried without the time filter, touches all 48 partitions and runs slower than the unpartitioned table did, because it now scans 48 indexes instead of one.

QueryPartitions touchedDuration
month filter present160 ms
no month filter487 s
unpartitioned baseline1 table4 s

Those figures are hypothetical, but the shape is real. Verify pruning from the plan: EXPLAIN should show one partition, not 48.

If the pain is analytics (big scans, aggregates, GROUP BY over the whole table) partitioning won't fix it, because a row store still reads every column. Offload to a columnar store: ClickHouse, BigQuery, Redshift, or a columnar extension in the same database. That's a second system with ETL lag, so it's worth it only when the analytical workload is real and separate from the transactional one.

Sometimes the table is fine and the index is wrong. A missing or bloated index, or a stale planner statistic, produces the same symptom. Reindex or rebuild before restructuring anything.

Sharding is last. It splits data across databases, which buys write throughput and escapes a single node's limits, but every query not keyed on the shard key becomes a scatter-gather, and you now own rebalancing and cross-shard transactions. Reach for it when a single node genuinely can't hold the writes, not when queries are slow.

OptionFixesCost / risk
Measure firstnothingnone
Retention / archivecold rows, backup sizein-place DELETE bloats; partition drop is instant
Partitioningscan scope, maintenancequeries must carry the key
Columnar offloadanalytical scanssecond system, ETL lag
Reindexbad plansrebuild time, lock
Shardingwrite throughput, node limitscross-shard queries, rebalancing

Curated: · Written: · Reviewed:

Q&A-29What anomalies does each isolation level allow, and which do you use by default?

Direct answer. The SQL standard defines the four levels by the anomalies they forbid, but engines disagree on what "repeatable read" means, so name the engine before you name the level.

LevelDirty readNon-repeatable readPhantomWrite skew
Read uncommittedyesyesyesyes
Read committednoyesyesyes
Repeatable readnonostandard: yes; PG/InnoDB: noyes
Serializablenononono

Defaults, read from SHOW transaction_isolation or SELECT @@transaction_isolation:

  • PostgreSQL 16: read committed
  • MySQL 8.0 InnoDB: repeatable read
  • Oracle 19c: read committed
  • SQL Server 2022: read committed
  • SQLite: serializable

PostgreSQL's repeatable read is snapshot isolation: one snapshot for the whole transaction, taken at the first statement. It kills phantoms but not write skew. That gap is the entire reason serializable exists.

Why it behaves this way. Read committed takes a fresh snapshot per statement, so two SELECTs in one transaction can disagree. Repeatable read pins a single snapshot; writing a row that changed since the snapshot raises 40001. Serializable adds SSI, which tracks read-write dependencies and aborts a transaction when a dangerous cycle forms.

Trace: write skew, PostgreSQL 16.

CREATE TABLE on_call (doctor_id int PRIMARY KEY, on_call boolean NOT NULL);
INSERT INTO on_call VALUES (1, true), (2, true);
-- invariant: at least one doctor on call

Session A:

BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT count(*) FROM on_call WHERE on_call;   -- 2
UPDATE on_call SET on_call = false WHERE doctor_id = 1;

Session B:

BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT count(*) FROM on_call WHERE on_call;   -- 2
UPDATE on_call SET on_call = false WHERE doctor_id = 2;
COMMIT;

A commits. Both rows are now false and the invariant is broken. The two UPDATEs touch different rows, so nothing blocks. Rerun the same script with SERIALIZABLE and B's COMMIT fails with ERROR: could not serialize access due to read/write dependencies among transactions (SQLSTATE 40001).

Lost update is a different shape. Under read committed, UPDATE accounts SET balance = balance - 80 WHERE id = 1 AND balance >= 80 loses nothing: the second statement waits on the row lock, then re-evaluates the predicate against the committed row and matches zero rows. The lost update needs the read and the write split across statements:

BEGIN;  -- read committed
SELECT balance FROM accounts WHERE id = 1;  -- 100
-- application computes 20
UPDATE accounts SET balance = 20 WHERE id = 1;
COMMIT;

Two of these interleave and both write 20; one withdrawal disappears. Fix it by making the update relative, or by taking SELECT ... FOR UPDATE.

What I use. The engine default, read committed on PostgreSQL, and I write the invariant down first. Escalate to serializable only when the invariant spans rows and cannot be expressed as a single UPDATE or protected by a lock. Serializable buys aborts, so the retry on 40001 is part of the design, not an afterthought.

Curated: · Written: · Reviewed:

Q&A-30Two transactions deadlock in production. What causes it and how do you fix it?

The 90-second answer. A deadlock is a cycle in the wait-for graph: transaction A holds a lock B wants, B holds a lock A wants, and neither can proceed. Every engine with row locks detects the cycle and breaks it by rolling one transaction back with a serialization error: MySQL 1213 / SQLSTATE 40001, PostgreSQL 40P01. So the fix has two halves: remove the cycle by making every writer acquire the same rows in the same order, and treat the error as retryable, because detection is a safety net, not a guarantee.

Why the cycle forms. Locks are acquired incrementally and held until commit. Two transactions touching the same two rows in opposite orders each hold one and wait for the other. The engine doesn't prevent this, it detects it. InnoDB checks for a cycle the moment a transaction starts waiting and rolls back the transaction holding the fewest rows; PostgreSQL waits deadlock_timeout (default 1s) before building the wait-for graph, then aborts one. Retry works because the victim's rollback releases every lock it held, so the survivor finishes; on the retry the conflicting transaction has usually committed and the cycle is gone.

T1: BEGIN; UPDATE accounts SET bal=bal-10 WHERE id=1;  -- X lock on row 1
T2: BEGIN; UPDATE accounts SET bal=bal-10 WHERE id=2;  -- X lock on row 2
T1: UPDATE accounts SET bal=bal+10 WHERE id=2;         -- waits for T2
T2: UPDATE accounts SET bal=bal+10 WHERE id=1;         -- waits for T1 -> cycle

Wait-for graph is T1→T2, T2→T1. One is rolled back, the other commits.

Fix: canonical order plus bounded retry.

import random, time, psycopg2

RETRYABLE = {"40P01", "40001"}   # deadlock_detected, serialization_failure

def transfer(conn, src, dst, amount, attempts=5):
    for attempt in range(attempts):
        try:
            with conn:                       # commit on exit, rollback on error
                with conn.cursor() as cur:
                    cur.execute(
                        "SELECT id FROM accounts WHERE id IN (%s, %s) "
                        "ORDER BY id FOR UPDATE", (src, dst))
                    cur.execute("UPDATE accounts SET bal = bal - %s WHERE id = %s",
                                (amount, src))
                    cur.execute("UPDATE accounts SET bal = bal + %s WHERE id = %s",
                                (amount, dst))
            return
        except psycopg2.Error as e:
            if e.pgcode not in RETRYABLE or attempt == attempts - 1:
                raise
            time.sleep(0.01 * 2 ** attempt * (0.5 + random.random()))

PostgreSQL applies FOR UPDATE above the sort, so both rows are locked in id order. MySQL is different: InnoDB locks rows as it reads them, so an ORDER BY that needs a sort does not reorder the locks. There you order the statements themselves.

Trade-offs. Consistent ordering is easy when the key set is static and hard when it isn't: UPDATE ... WHERE status='pending' or an ORM-generated IN (...) gives you no control over acquisition order. Three options there. Lock a single anchor row first (the account, the parent) so all writers serialize on it: cheap, but it serializes writers that never actually conflicted. Funnel writes through one queue or worker: deadlocks vanish by construction, at the cost of capping write throughput at one writer. Or drop row locks for optimistic concurrency, a version column with UPDATE ... WHERE version = ?, which never blocks but converts every conflict into a retry. Ordering also costs parallelism: two writers that could have run concurrently now queue behind the same first row.

Failure modes. With innodb_deadlock_detect=OFF, the cycle isn't detected and the victim waits out innodb_lock_wait_timeout (default 50s) before error 1205: a latency cliff, not a deadlock. Retry without backoff turns a deadlock into a livelock. Under REPEATABLE READ, gap and next-key locks deadlock on range scans and INSERT ... SELECT even when no two transactions touch the same row, and foreign-key checks take shared locks on parent rows, so a child insert can deadlock against a parent update.

Evidence. MySQL: SHOW ENGINE INNODB STATUS, LATEST DETECTED DEADLOCK shows both transactions, their statements, and the locks held and waited. PostgreSQL: set log_lock_waits=on and read the deadlock detected log lines; pg_stat_database.deadlocks counts them. Confirm the two lock orders in that output match the order your code now imposes.

Curated: · Written: · Reviewed:

Q&A-31Two users edit the same record. How do you stop one overwriting the other?

The 60-second answer. Detect the conflict rather than prevent it, unless contention is high. Add a version integer to the row, read it with the record, and put it in the WHERE clause of the update. If zero rows change, the write lost and you return a conflict to the caller. That is optimistic locking, and it is my default: no lock held across the user's think time, no deadlock, cost is one column and one predicate. Pessimistic locking, SELECT ... FOR UPDATE, is for rows where conflicts are frequent enough that retries dominate, typically counters and balances.

Worked trace (PostgreSQL 16, default READ COMMITTED).

CREATE TABLE ticket (
  id      bigint PRIMARY KEY,
  title   text NOT NULL,
  version int  NOT NULL DEFAULT 0
);

Both sessions read ticket 42 at version 7.

-- Session A
UPDATE ticket SET title = 'printer down', version = version + 1
WHERE id = 42 AND version = 7;
-- UPDATE 1

-- Session B, same predicate, one second later
UPDATE ticket SET title = 'printer offline', version = version + 1
WHERE id = 42 AND version = 7;
-- UPDATE 0

B's statement blocks on A's row lock until A commits, then re-evaluates the predicate against the new row and matches nothing. UPDATE 0 is the conflict signal: no exception, no silent overwrite. B re-reads, re-applies its edit on top of A's, and retries with version = 8.

Pessimistic, same row.

BEGIN;
SET LOCAL lock_timeout = '2s';
SELECT title, version FROM ticket WHERE id = 42 FOR UPDATE;

A holds the lock; B waits. Postgres's lock_timeout defaults to 0, meaning wait forever, so set it per transaction: B then fails with SQLSTATE 55P03 (lock_not_available) after 2 s instead of hanging. MySQL InnoDB's equivalent is innodb_lock_wait_timeout, default 50 seconds, error 1205.

Choosing.

Optimistic (version)Pessimistic (FOR UPDATE)
Lock heldnonefor the whole transaction
Second writerUPDATE 0 → 409blocks, then proceeds
Deadlock risknonereal: two rows locked in opposite order → Postgres 40P01, MySQL 1213
Costone column, one predicatelock wait, shorter transactions

Rule of thumb I would defend from metrics, not taste: under ~1% of writes conflicting, optimistic wins. The retry is cheaper than the wait. Above ~5%, retries cost more than the wait and I would serialize on the row. Between the two, measure.

Failure modes. Retry storms: three clients retrying a hot row in lockstep. Use exponential backoff with jitter (3 attempts, 20 ms base, cap 200 ms) then give up with a 409 rather than looping. Long transactions: a FOR UPDATE held across an HTTP call to a payment provider turns a 5 ms lock into a 3 s one; never do I/O inside the lock. Whole-row versions rejecting harmless writes: two users editing different fields still collide, so for wide records version per field or merge at the field level.

At the API edge. Return 409 Conflict with the current version in the body so the client can re-read and retry. With ETags, a failing If-Match is 412 Precondition Failed (RFC 9110 §13.1.1). Kubernetes does exactly this: a PUT with a stale metadata.resourceVersion gets 409. DynamoDB does it with ConditionExpression: "version = :v", failing with ConditionalCheckFailedException.

When not to bother. Single-statement increments, UPDATE counters SET n = n + 1 WHERE id = ?, are atomic under any isolation level; no version needed. And if the two writers touch disjoint fields and you can merge them, last-write-wins per field is fine.

Curated: · Written: · Reviewed:

Q&A-33How do you rename a column that a running service depends on?

Core answer

Assume Postgres 14+, the app and the schema deploy independently, and the column is read and written by a live service. Then you don't rename the column: you add a new one, migrate the service onto it, and drop the old one. A rename is a single ALTER TABLE ... RENAME COLUMN, which takes an ACCESS EXCLUSIVE lock. It's metadata-only and finishes in under a millisecond, so the lock's duration isn't the problem. The queue is. While that ALTER waits for an existing long-running transaction to finish, it sits at the head of the lock queue, and every subsequent SELECT, INSERT and UPDATE on the table blocks behind it. A 200 ms query that started before your migration stalls the whole table for 200 ms; a 30 s analytics query stalls it for 30 s. That is the outage, and it is why "the rename is instant" is not a defence.

The five steps

Table orders, 40M rows, renaming customer_id to account_id.

  1. Expand: add the new column, nullable, no default:
SET lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN account_id bigint;

ACCESS EXCLUSIVE, metadata-only, no table rewrite. Safe to stop here.

  1. Dual write: deploy the service writing both columns. Reads still come from customer_id. No DDL, no lock.

  2. Backfill in batches, with a trigger to catch concurrent writes:

CREATE OR REPLACE FUNCTION orders_mirror() RETURNS trigger AS $$
BEGIN
  IF NEW.account_id IS NULL THEN NEW.account_id := NEW.customer_id; END IF;
  RETURN NEW;
END $$ LANGUAGE plpgsql;

CREATE TRIGGER orders_mirror_trg BEFORE INSERT OR UPDATE ON orders
  FOR EACH ROW EXECUTE FUNCTION orders_mirror();

Then loop until it reports zero rows updated:

UPDATE orders SET account_id = customer_id
WHERE id IN (SELECT id FROM orders WHERE account_id IS NULL LIMIT 5000);

Each batch is a short transaction. Without the trigger, rows updated between a batch's snapshot and its write keep a stale account_id, and you get a silent divergence that only surfaces when reads move.

  1. Move reads: deploy the service reading account_id, still writing both. Verify before proceeding:
SELECT count(*) FROM orders WHERE account_id IS DISTINCT FROM customer_id;

Must be 0.

  1. Contract: drop the trigger, then the column:
SET lock_timeout = '3s';
DROP TRIGGER orders_mirror_trg ON orders;
ALTER TABLE orders DROP COLUMN customer_id;

ACCESS EXCLUSIVE, metadata-only. Then remove the dual write in a final deploy.

Lock modes

StepLockRewrite?Safe to stop after
add nullable columnACCESS EXCLUSIVE, briefnoyes
dual writenone—yes
backfill batchesROW EXCLUSIVEnoyes
move readsnone—yes
drop columnACCESS EXCLUSIVE, briefnoyes

Failure modes

  • Lock queue: set lock_timeout on every DDL statement so it fails fast instead of parking at the head of the queue, and retry in a loop.
  • Backfill divergence: the trigger above, or a watermark on updated_at. Detect with the mismatch count in step 4.
  • NOT NULL: don't add it in step 1. After backfill, ADD CONSTRAINT account_id_not_null CHECK (account_id IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT (SHARE UPDATE EXCLUSIVE, doesn't block writes), then SET NOT NULL: PG 12+ uses the validated constraint and skips the full scan.
  • Rollback: every step is reversible until step 5. After the drop the old column is gone, so treat the contract step as one-way and keep a backup if that matters.

When not to do this

If the table is small, a few hundred thousand rows, or you have a maintenance window, a single rename with lock_timeout and a retry is fine and far cheaper. Expand-contract costs five deploys and a trigger; spend it when the table is large or the service can't stop.

Curated: · Written: · Reviewed:

Q&A-34Should deletes be soft or hard, and what does that cost?

Assumptions first. Postgres 16, a repository layer between the app and SQL, business keys users can re-use (email, username, slug), and a retention or erasure obligation on at least some tables. If any of those is false the answer shifts, and I'd say so out loud before answering.

The answer. Default to soft delete where history or recovery has value, and hard delete where a regulation or a data-minimisation policy requires the bytes to actually go. It's a per-table decision, not a per-database one. The cost of soft delete isn't storage: it's that every read path now carries a filter, and every unique constraint now has to be partial.

Why it costs what it costs. A soft delete keeps the row and sets deleted_at. Three things break at once:

  1. Every query that didn't filter now returns ghosts. Fix it once in a view or the repository layer, not at 200 call sites.
  2. A plain UNIQUE (email) still counts the deleted row, so a returning user can't re-register.
  3. Foreign keys and cascades still see the row, so a "deleted" parent keeps its children alive.

The fix for (2) is a partial unique index:

-- Postgres 16
CREATE UNIQUE INDEX users_email_active
  ON users (lower(email))
  WHERE deleted_at IS NULL;

The constraint now applies only to live rows. Same trick for slugs and usernames.

The failure I'd bring up. A team shipped soft delete with the original full unique index. A customer closed their account, came back four months later, and got "email already in use." Support worked around it by rewriting the address to user+deleted-1@…. By the time someone traced it, 60 accounts carried mangled addresses and the login flow had a special case nobody remembered writing. The partial index took one migration.

Trade-off table.

DesignRe-registrationHistoryCost
Hard deleteworksgoneirreversible; may violate retention
Soft delete, full unique indexblockedkeptsupport workarounds, corrupted keys
Soft delete, partial unique indexworkskeptfilter obligation on every read

When hard delete wins. PII you're legally required to erase, high-churn tables where dead rows dominate (sessions, OTPs, idempotency keys), and anything where the row's existence is itself the leak. Delete the row and keep the audit trail elsewhere: a separate deletions table with id, actor, timestamp, no payload.

How I'd verify it. In a test database: create a user, soft-delete, re-insert the same email, assert it succeeds; then assert the default list query returns zero rows and the admin query returns one. If the constraint or the view doesn't behave as documented, the design isn't done.

Curated: · Written: · Reviewed:

Q&A-39You move reads to a replica and users start seeing stale data. What happened?

You didn't break the replica: you changed the consistency model. A replica applies the primary's changes asynchronously, so a read that lands between the primary's commit and the replica's apply returns the value from before the write. That gap is replication lag. It's usually tens of milliseconds and spikes to seconds when a bulk write, a long transaction, or a single-threaded apply queue backs up. Reads that must reflect the caller's own write have to go to the primary; reads that tolerate staleness can stay on the replica.

Why it happens

The primary commits, then streams the change to replicas, which replay it. Nothing in the read path waits for that replay. Lag is the distance between the primary's current position and the replica's replayed position: on MySQL, Seconds_Behind_Source (renamed from Seconds_Behind_Master in 8.0.22); on Postgres, pg_stat_replication.replay_lsn compared against pg_current_wal_lsn().

Two symptoms, two fixes

Read-your-writes violation: the user writes, then reads and doesn't see their own change. Fix by pinning that session to the primary for a short window, or by passing a consistency token.

Monotonic read violation: the user sees the new value, refreshes, and sees the old one. The second read hit a different, laggier replica. Fix by sticking a session to one replica, or by the same token.

Routing choices

RoutingSees the caller's writePrimary load
Always replicaNo, until lag closesLowest
Primary for the session after a writeYesModerate
Always primaryYesHighest

Worked example (hypothetical figures)

A profile save redirects to a read. Lag is normally 20 ms and the read fires ~30 ms after the write, so it's fine. During a nightly batch, lag spikes to 900 ms. At 40 profile saves/s, ~36 saves land inside that window, and each redirect read returns the pre-save profile. If a third of those users re-save what they see, you get ~12 duplicate change records per spike, and no error anywhere, because every query succeeded.

Detecting it

Alert on replica lag crossing a threshold (say > 5 s for 60 s), not on error rate: this failure produces no errors. If you can, also count reads that return a version older than the session's last write; that counter is the real signal.

What I'd do

Route reads that follow a write in the same session to the primary; route reporting and cacheable reads to replicas. If primary load is the objection, use a token instead of a blanket pin: the write returns the replica's target LSN or GTID, and the read waits until the replica has replayed past it.

Curated: · Written: · Reviewed:

Q&A-22How would you make a read-heavy endpoint cacheable?

First 90 seconds

Three questions decide the design: is the response byte-identical for every caller, how stale may it be, and who is allowed to store it. If the body is the same for all callers and a URL identifies it, cache at the edge with HTTP caching: the hit never reaches your origin, so it is the cheapest one available. If the body is per-user or per-tenant, HTTP caching is the wrong tool: put a shared cache (Redis) behind the handler and key it by user. If the read must be strongly consistent, cache nothing and make the origin fast instead.

What to send

Cache-Control: public, max-age=60, s-maxage=300, stale-while-revalidate=60, stale-if-error=86400
ETag: "v42-9f3c"
Vary: Accept-Encoding, Accept-Language

max-age governs browsers, s-maxage the CDN. stale-while-revalidate lets the edge serve a stale copy while it refreshes in the background, which is what stops a TTL boundary from becoming a thundering herd. The ETag turns a revalidation into a 304 with no body: a few hundred bytes instead of the full payload.

The arithmetic

Say the endpoint peaks at 1,200 rps and the response is 40 KB. Uncached, that is 48 MB/s of origin egress and 1,200 origin requests/s. A 96% edge hit ratio drops origin traffic to 48 rps and 1.9 MB/s. Most of the remaining 4% are revalidations: with If-None-Match they return 304s, so the origin does a key lookup and no serialization.

Trade-offs

ApproachHit latencyInvalidationUse when
CDN / HTTP cache5–20 ms at edgeTTL, or purge API with seconds of propagationPublic, URL-addressable, identical body for all callers
Shared Redis cache1–3 ms in-regionDEL on write, immediatePer-user or per-tenant bodies, computed aggregates
No cacheorigin latencyn/aStrong consistency required, or low volume

Long TTL plus explicit purge gives a high hit ratio but a stale window equal to purge propagation; short TTL self-heals but multiplies origin load. A 300 s TTL with purge-on-write is usually the better trade for catalog data; a 60 s TTL is better when you have no purge path at all.

Failure modes

  • Vary: Cookie or Authorization in the key collapses the hit ratio to near zero. Detect with per-path hit-ratio metrics on the CDN.
  • Stampede at the TTL boundary: every key expires in the same second. Detect as a periodic origin QPS spike; fix with stale-while-revalidate and jittered TTLs.
  • Stale reads after a write break read-your-writes. Detect in user reports; fix with purge-on-write or versioned URLs.
  • Cache poisoning: an unkeyed header reflected into the body. Detect by diffing responses across header values.
  • Caching errors: never give a 5xx a max-age; stale-if-error is for serving a good copy when the origin is down.

Cache key hygiene

Strip tracking parameters and sort the rest so ?a=1&b=2 and ?b=2&a=1 share an entry. High-cardinality query params (free-text search, timestamps) explode the key space and give you a 0% hit ratio with a full cache: normalize those paths or leave them uncached.

Curated: · Written: · Reviewed:

Q&A-40How would you add a cache in front of a hot read?

The 90-second answer. Put a cache-aside layer in front of the read, key it on every input that changes the response, give it a TTL with jitter, delete the key on write, and coalesce concurrent misses so a cold key doesn't stampede the database. Then measure hit rate and database QPS before widening the TTL.

Assumptions I'd state out loud: the read is heavy (say 20k rps reads against 200 rps writes), the value is small and cheap to serialize, and the read tolerates roughly a minute of staleness. If it doesn't tolerate staleness, cache-aside with a TTL is the wrong tool: you need invalidation on write or a versioned key.

Why cache-aside first. The read path owns the cache; the write path is untouched, so a cache outage degrades to the old latency instead of failing writes. Write-through keeps the cache warm but couples every writer to the cache and leaves a partial-failure window where the cache and the database disagree. Write-behind is fastest and loses data when the cache node dies.

import time, threading
_cache = {}   # key -> (value, expires_at)
_locks = {}   # key -> Lock

def get_price(tenant, sku):
    key = f"price:{tenant}:{sku}"
    hit = _cache.get(key)
    if hit and hit[1] > time.monotonic():
        return hit[0]
    with locks.setdefault(key, threading.Lock()):
        hit = _cache.get(key)          # re-check: another thread may have filled it
        if hit and hit[1] > time.monotonic():
            return hit[0]
        value = db.query_price(tenant, sku)
        _cache[key] = (value, time.monotonic() + 60)
        return value

Trace it with 100 threads hitting one cold key: all 100 miss, one takes the lock and issues one db.query_price, the other 99 block, then all 100 return the same value. Drop the re-check inside the lock and you get 100 database queries instead of one. This lock is per-process; across processes you need SET NX in Redis or a single-flight library, otherwise each process still stampedes.

StrategyRead pathWrite pathStalenessFailure mode
cache-aside + TTLmiss → DB, filluntouchedup to TTLstampede at expiry
cache-aside + delete-on-writesamedelete keybounded by the repopulate racestale until TTL
write-throughalways warmcache then DBnone if every writer goes through itwrite latency coupled; partial failure
write-behindalways warmcache, async flushflush intervaldata loss on cache node failure

Failure modes and how I'd catch them.

  • Stampede. Database QPS spikes at the TTL boundary. Plot DB QPS against cache expiry; fix with TTL jitter (60 s ± 10%) plus the single-flight above.
  • Key missing an input. A key without the tenant served one tenant's pricing to another for 90 seconds until it expired, and the incident required customer notification. Catch it with a test that writes as tenant A and reads as tenant B.
  • Read-repopulate race. A read misses, the DB read returns the old value, a concurrent write deletes the key, and the read writes the old value back: stale until TTL. Delete-on-write plus a short TTL bounds it; versioned keys or a compare-and-set close it.
  • Hot key. One key lands on one shard. Watch per-key metrics and put a small in-process cache in front of Redis for the top keys.
  • Cache node loss. Hit rate drops to zero and the full read load hits the database. Keep connection-pool headroom and a load-shedding limit so the database survives the transition.

Sizing. At 20k rps and a 95% hit rate, the database sees 1k rps; p99 on the read drops from 40 ms to about 2 ms. 200k keys at 2 KB each is 400 MB, which fits one Redis node, so start there and shard only when per-key metrics say you must. These figures are illustrative; substitute your own before quoting them.

The cache key must contain everything that changes the answer.

Curated: · Written: · Reviewed:

Q&A-41A popular cache key expires and the database is hammered. What is happening?

The 90-second answer

This is a cache stampede (thundering herd). The key was a shield: while it existed, 4,000 requests per second cost the database nothing. The moment it expires, every request that arrives before the first recompute finishes misses, and each one independently runs the same expensive query. Database load jumps from ~0 to the number of requests in flight, not the request rate.

The multiplier is concurrency, not throughput. At 4,000 rps with a 50 ms recompute, roughly 4,000 × 0.05 = 200 requests are in flight at any instant, and all 200 miss before any of them writes the value back. The database sees ~200 identical queries instead of one. If the recompute slows under that load, say to 500 ms, in-flight grows to ~2,000 and misses pile up faster than they drain, which is why the database stays pinned well past the expiry instant instead of recovering immediately.

Fixes, in the order I'd trust them

  1. Single-flight / request coalescing. The first miss takes a per-key lock and recomputes; the rest wait on that result or return stale. One query reaches the database per expiry, regardless of concurrency.
  2. Stale-while-revalidate. Serve the expired value to everyone while one background refresh runs. No user-visible latency spike, one query. Requires the value to be tolerable-stale for the refresh window.
  3. Jittered TTLs. Keys written together expire together. TTL = base + rand(0, 0.1 * base) spreads expiries. This fixes the synchronized version across many keys; it does nothing for a single hot key.

Two more worth doing: refresh hot keys proactively before expiry (a job that rewrites the key at 80% of its TTL), and negative-cache keys that don't exist, since a missing key is a permanent miss.

import threading

_locks, _guard = {}, threading.Lock()

def get(key):
    val = cache.get(key)
    if val is not None:
        return val
    with _guard:
        lock = _locks.setdefault(key, threading.Lock())
    with lock:
        val = cache.get(key)          # re-check: another thread may have filled it
        if val is None:
            val = db.query(key)
            cache.set(key, val, ttl=60)
        return val

This coalesces within one process. Across N app servers you need a shared lock, Redis SET key NX PX 5000, or you get N queries, one per process.

Failure modes

  • Lock holder dies. Without a lock TTL, every waiter blocks until it expires. Set a short TTL and let a waiter retry.
  • Stale-while-revalidate serving stale forever. If the refresh keeps failing you serve hours-old data. Bound it: after N failed refreshes, fail the request.
  • Jitter on one hot key. Does nothing: jitter spreads expiries across keys, it doesn't reduce concurrency on a single key.

Numbers

4,000 rps on one key, 50 ms recompute, queries reaching the database at expiry:

DesignDatabase queries at expiry
plain expiry~200 (one per in-flight request; grows if recompute slows)
single-flight lock1
stale-while-revalidate1, no user-visible wait

Worked example (hypothetical): a dashboard key at 4,000 rps with a 50 ms recompute expires on a round minute. ~200 concurrent misses hit the database, each query slows to ~200 ms under contention, so in-flight climbs and the database sits at 100% CPU for tens of seconds.

What interviewers look for

  • You name the mechanism, simultaneous misses on one hot key, not just "the cache expired."
  • You quantify it: concurrency × recompute time = in-flight misses.
  • You separate stampede from avalanche (many keys expiring together) and penetration (queries for keys that never exist).
  • You give at least one mitigation with its trade-off, and name a failure mode of that mitigation.

Weak answers

  • "Add more cache" or "increase the TTL": doesn't touch concurrency.
  • "The cache is down": it isn't; it expired.
  • A global lock that serializes every key, not a per-key lock.
  • Confusing stampede with avalanche or penetration.
  • No mention of what the database actually sees.

Follow-ups

  • How would you implement single-flight in your stack, and what breaks with multiple app servers?
  • What happens if the lock holder crashes mid-recompute?
  • How do you pick the stale-while-revalidate window?
  • How does this differ from cache avalanche and cache penetration?
  • How would you detect a stampede in production before the database falls over?

How I'd verify

Load test: drive the key at production rate, expire it deliberately, and count queries reaching the database. With single-flight you should see one per expiry, not one per request. Watch connection count and CPU during the expiry window, not the average across the run: the spike is the whole problem.

Curated: · Written: · Reviewed:

Q&A-49Writes to a large table slow down over months even though the query plan is unchanged. What would you check?

The short answer

Assume a Postgres-family engine with MVCC, writes are INSERT/UPDATE/DELETE, and the plan is genuinely unchanged. Then the plan is not the problem: the data underneath it is. An UPDATE writes a new row version and leaves the old one dead; a DELETE only marks a row dead. Nothing reclaims those until vacuum runs. Over months the table and its indexes accumulate dead space, so the same index scan touches more pages, the buffer cache holds a smaller fraction of the table, and every write has more pages to dirty. That is the default explanation for "plan unchanged, writes slower."

Check the maintenance, not the query

First, dead tuples and when autovacuum last ran:

SELECT relname, n_live_tup, n_dead_tup,
       last_autovacuum, last_autoanalyze
FROM pg_stat_user_tables
ORDER BY n_dead_tup DESC
LIMIT 10;

Then the reason vacuum may be doing nothing: the xmin horizon. A long-running transaction, an inactive replication slot, or a prepared transaction pins the oldest visible xmin, and vacuum cannot remove any dead tuple newer than it.

SELECT pid, state, xact_start, now() - xact_start AS age, query
FROM pg_stat_activity
WHERE state <> 'idle'
ORDER BY xact_start
LIMIT 5;

SELECT slot_name, active, xmin, catalog_xmin FROM pg_replication_slots;

Why the defaults fail on a large table

Autovacuum triggers at autovacuum_vacuum_threshold + autovacuum_vacuum_scale_factor * reltuples, defaulting to 50 + 0.2 * rows. On a 100M-row table that is 20,000,050 dead tuples before a vacuum starts: by then the table is a fifth dead. The scale factor is the wrong shape for big tables; per-table settings of autovacuum_vacuum_scale_factor = 0.02 and a lower autovacuum_vacuum_insert_scale_factor are the usual fix. Also check autovacuum_max_workers (default 3): if three workers are stuck on other tables, the hot table waits.

Hypothetical trace of the symptom

MonthDead tuplesTable sizep99 insert
12%40 GB3 ms
219%61 GB5 ms
338%82 GB6 ms

Same plan, same row count, twice the pages to touch.

Trade-offs between the fixes

  • Tune autovacuum per table: cheap, no downtime, but only stops future bloat. It does not reclaim what is already there.
  • REINDEX CONCURRENTLY: reclaims index bloat online, but needs roughly the index size in free disk, cannot run inside a transaction block, and writes a full second copy of the index to WAL.
  • pg_repack: rewrites table and indexes online, needs a full copy of the table on disk and a trigger on the table for the duration.
  • Partition by time and archive: the durable fix when growth is unbounded, but it is a schema change, writes must route to the right partition, and it does nothing if all rows stay hot.

If the table is small or the write rate low, bloat stays bounded and tuning alone is enough. If the slowdown tracks checkpoint or WAL volume rather than dead tuples, none of these help: look at checkpoint_timeout, max_wal_size, and full-page writes instead.

Failure modes to name

  • An analytics query holding a 9-hour snapshot: vacuum runs, removes nothing, dead tuples climb.
  • An inactive replication slot pinning xmin indefinitely.
  • HOT updates broken because an updated column is indexed, so every update also writes every index.

Confirm with weekly dead-tuple and index-bloat numbers, not a one-off look.

Curated: · Written: · Reviewed:

Q&A-50Analysts want to run heavy queries against your production database. What do you propose?

What I'd say in the first two minutes

Three questions first: how fresh does the data need to be, are the queries read-only, and is the workload ad-hoc or scheduled? Then the answer: analysts do not run against the primary. Ad-hoc work goes to a read replica; anything scheduled or repeated goes to a warehouse fed by change data capture, with the common aggregates materialized. Both paths get hard limits.

Why it hurts the primary

An analytical query scans millions of rows. Three things follow. It evicts the hot working set from the buffer cache, so the next checkout query reads from disk. It holds a snapshot open for the duration, which pins the vacuum horizon: dead tuples that would have been reclaimed accumulate and the table bloats. And it holds a pool connection for tens of seconds, so a handful of analysts can exhaust a 100-connection pool.

Options and what each costs

PathFreshnessCost / failure mode
Read replicareplication lag, secondsHot-standby conflicts cancel long queries (max_standby_streaming_delay, default 30s). Turning on hot_standby_feedback stops the cancellations but makes the primary retain dead tuples, so it bloats.
CDC to warehouseminutesA stalled consumer on a logical replication slot retains WAL on the primary; unbounded by default (max_slot_wal_keep_size = -1).
Materialized viewrefresh intervalREFRESH MATERIALIZED VIEW takes ACCESS EXCLUSIVE; CONCURRENTLY avoids blocking reads but needs a unique index and is slower.
Snapshot exporthoursNo interactivity; fine for nightly dumps.

Guardrails

-- Postgres 16, applied to the analyst role on the replica
ALTER ROLE analyst CONNECTION LIMIT 8;
ALTER ROLE analyst SET statement_timeout = '120s';
ALTER ROLE analyst SET idle_in_transaction_session_timeout = '60s';
ALTER ROLE analyst SET default_transaction_read_only = on;

Plus max_slot_wal_keep_size on the primary so a stalled CDC consumer cannot fill the disk.

Worked example (illustrative figures)

Primary: 32 GB shared_buffers, hot working set around 4 GB, checkout p99 180 ms. An analyst scans a 200 GB table. The scan evicts essentially the whole working set; checkout p99 climbs into seconds and stays there until the cache warms. Move the same query to a replica: the primary's cache is untouched, p99 stays at 180 ms, and the analyst's query runs 10-20% slower because the replica is also applying WAL.

Failure modes to watch

  • Replica lag grows under analyst load, so analysts read stale data. Alert on replay lag in pg_stat_replication.
  • hot_standby_feedback on means primary bloat. Watch dead-tuple counts.
  • Slot retention: at 20 MB/s of WAL, a stalled consumer retains 72 GB per hour. Bound it.
  • No statement_timeout means one SELECT * on a fact table runs for an hour.

How I'd confirm it

Run the analyst workload against the new path and watch the primary's p99 and replica lag for a full business cycle. If primary p99 is unchanged and lag stays inside the freshness the analysts need, it is done.

Curated: · Written: · Reviewed:

Q&A-42What does at-least-once delivery mean for your consumer code?

At-least-once means the broker will hand your handler a message more than once (after a consumer timeout, a nack, a rebalance, or a retry) and it will not tell you which delivery is the duplicate. So the consumer, not the broker, has to make the effect happen once. Assume every message arrives twice and design the handler so the second delivery is a no-op.

Assumptions: offsets are committed after the handler returns, and the broker redelivers anything not acknowledged within its visibility timeout.

Two mechanisms, and you usually want the first.

Dedupe on message id. Record the id in the same transaction as the side effect:

def handle(msg, conn):
    with conn.transaction():
        cur = conn.execute(
            "INSERT INTO processed_messages (message_id) VALUES (%s) "
            "ON CONFLICT (message_id) DO NOTHING", (msg.id,))
        if cur.rowcount == 0:
            return  # already handled
        conn.execute("UPDATE accounts SET balance = balance - %s WHERE id = %s",
                     (msg.amount, msg.account_id))

Trace of the second delivery: the INSERT conflicts, rowcount is 0, the function returns before the UPDATE, and the transaction commits nothing. One debit total.

Idempotent by business key. If the message carries a natural key (order id, transfer id), make the write an upsert keyed on it instead of an append. This survives a dedupe table being pruned, but only if the update is commutative: SET balance = balance - 100 applied twice is not, so use SET status = 'refunded' WHERE status = 'paid' or a version column.

Where it breaks:

  • Side effect outside the transaction. If the debit is an HTTP call to a payment provider, the commit and the call can diverge: crash after the call and before the commit, and the retry charges twice. Either keep the effect in the same transaction or pass a provider idempotency key.
  • Dedupe retention shorter than the redelivery window. Prune processed_messages after 7 days while a consumer group can be offline for 14, and duplicates slip through. Retention must exceed the longest redelivery window.
  • Reordering. At-least-once gives per-partition order only. Two updates to the same key from different partitions can land out of order; last-write-wins then loses one. Version the row and reject stale writes.

Detection: reconcile counts on both sides (refunds issued vs orders refunded), alert on processed_messages growth, and route repeatedly failing messages to a DLQ rather than looping.

Verification: publish the same message twice in a test and assert exactly one side effect. That is the only evidence that the handler is idempotent.

Curated: · Written: · Reviewed:

Q&A-43How do you make a message handler idempotent in practice?

The short answer

Idempotence means applying the same message twice leaves the system in the same state as applying it once. In practice you get it by making the dedup record and the effect land in the same atomic write, keyed on something stable derived from the message.

Worth stating up front: this assumes at-least-once delivery (SQS, Kafka with manual commits, most brokers), a transactional store for the effect, and a business key that survives redelivery. An order id from the payload, not a broker-assigned message id that changes on retry.

How it works

Derive the key from the message payload, not from the delivery. Then do both writes in one transaction:

BEGIN;
INSERT INTO processed_messages (key, processed_at)
VALUES ($1, now())
ON CONFLICT (key) DO NOTHING;
-- app checks rowcount; if 0, this is a redelivery:
-- COMMIT and ack, skip the effect
INSERT INTO orders (...) VALUES (...);
COMMIT;

The unique constraint on processed_messages.key is what rejects the repeat. Because both inserts sit in one transaction, a crash before COMMIT rolls back both, the message is redelivered, and it reprocesses cleanly. A crash after COMMIT means the effect is durable and the redelivery hits the conflict.

The failure mode this avoids

The classic bug is splitting the check from the effect. A handler that records processed ids in Redis and writes the effect to Postgres has a window between the two writes. Kill the process in that window and the message comes back with no record of it.

Take a consumer doing 10k messages/min, killed mid-batch (numbers illustrative). 80 messages had their effect committed but their id never reached the cache. On restart all 80 were applied again: 80 duplicate orders, 80 duplicate charges.

DesignDuplicates after restart
ids in cache, effect in DB80
unique constraint on business key0
one transaction for both0

Alternatives and when they fit

Natural idempotence first: if the operation is SET status = 'shipped' or an upsert on a natural key, you don't need a dedup table at all. Prefer it when it exists: less state to prune.

The dedup table with a unique constraint is the general fallback. It costs one extra write per message and a table you must prune: delete rows older than the maximum redelivery window. For SQS that's the retention period, up to 14 days; for Kafka it's however far your consumer lag can reach.

Client-supplied idempotency keys are the same mechanism moved to the API edge: the caller picks the key, so it survives retries the client controls.

What I'd verify

Don't call it done without a test that kills the consumer between the effect and the bookkeeping and confirms no double-apply after restart. Then check the prune job actually runs: an unbounded dedup table is its own incident.

Curated: · Written: · Reviewed:

Q&A-44You must update the database and publish an event. How do you keep them consistent?

What I'd say first

Assume the database is the source of truth and the event is a notification, not a second copy of state. Then the answer is: don't write to two systems. Write the event into the same database, in the same transaction as the business row, and let a separate relay publish it afterwards. That's the transactional outbox.

The failure it removes is the one that matters: a process that commits the order and dies before the publish call. With a direct write-then-publish, that order exists and no downstream service ever hears about it. With an outbox, the event is committed or rolled back with the order, so it cannot be lost.

The mechanics

CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  aggregate_id uuid        NOT NULL,
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

BEGIN;
  INSERT INTO orders (id, customer_id, total_cents) VALUES ($1, $2, $3);
  INSERT INTO outbox (aggregate_id, event_type, payload)
       VALUES ($1, 'OrderPlaced', $4);
COMMIT;

Both rows land or neither does. The relay polls inside a transaction:

SELECT * FROM outbox
 WHERE published_at IS NULL
 ORDER BY id
 LIMIT 100
 FOR UPDATE SKIP LOCKED;

publish each row to the broker, then UPDATE outbox SET published_at = now() WHERE id = ANY($1). SKIP LOCKED lets several relay instances run without blocking each other. Holding the row lock across the publish is fine at low volume; once publish latency is measurable, claim rows with a claimed_at column instead so the lock isn't held during network I/O.

What this costs you

Delivery is at-least-once, not exactly-once. If the relay publishes and crashes before the update, the row is picked up again on restart and the event goes out twice. Consumers must be idempotent: dedupe on the outbox id carried as the event id, or make the handler naturally idempotent with an upsert keyed by event id. That's the trade: you buy no-loss with duplicates.

SKIP LOCKED also means two relays can publish out of order for the same aggregate. If ordering matters, partition the relay by aggregate_id: one worker per key, or a single-threaded relay per aggregate.

The outbox table grows. If the relay stalls, the transactional table bloats and business writes slow down. Alert on the age of the oldest unpublished row, not on row count: a lag of minutes is normal, an hour is not. Delete or archive published rows on a schedule.

Alternatives

ApproachAtomicityCost
write then publishnone: a crash loses the eventsimplest, silent data loss
outbox + relayevent commits with the rowat-least-once, dedupe needed, table to prune
2PC / XAatomic across bothcoordinator is a SPOF, locks held across prepare, patchy broker support
CDC on the WAL (Debezium)event derived from the committed logno relay code, but coupled to log format; the outbox table still holds the payload

If the consumer can read the database itself, don't publish at all: derive the event from the row. Publishing is for services that can't.

How I'd verify it

Kill the process between COMMIT and the publish in an integration test, restart, and assert the event arrives exactly once after dedupe. Then stop the relay for longer than the threshold and confirm the outbox-age alert fires.

Curated: · Written: · Reviewed:

Q&A-45Events for one customer arrive out of order. Why, and how do you fix it?

What I'd say in the first two minutes

Assume Kafka, but the shape is the same on Kinesis (PartitionKey) and SQS FIFO (MessageGroupId). Ordering is guaranteed only within a partition, and only for a single producer session. So "events for one customer arrive out of order" almost always means one of four things: the producer didn't key by customer, so the events landed on different partitions; the producer retried and the retry landed after a later event; two producers wrote the same customer concurrently; or the consumer fans out to a thread pool and processes one partition's records in parallel.

The fix has two halves and you need both: make the transport preserve order for the entity, and make the handler tolerate it when it doesn't.

Transport

Partition by the entity id. In Kafka that's the message key: the default partitioner hashes it, so all of one customer's events land on one partition and one consumer reads them in offset order. Then keep one consumer per partition on the ordered path: no intra-partition thread pool, no max.poll.records so large that a rebalance interrupts a batch mid-flight.

Handler

Even with the right key, retries and rebalances produce duplicates and reordering. Carry a per-entity version: a monotonic sequence number the producer assigns, or the entity's own version column, and make the write conditional.

# Python 3.12
from dataclasses import dataclass

@dataclass
class Customer:
    address: str
    version: int = 0

store = {"c-42": Customer(address="12 Mill Lane", version=7)}

def apply(event):
    c = store[event["customer_id"]]
    if event["version"] <= c.version:
        return f"rejected v{event['version']} (have v{c.version})"
    c.address = event["address"]
    c.version = event["version"]
    return f"applied v{event['version']} -> {c.address}"

# v9 and v8 arrive in that order on the wire
print(apply({"customer_id": "c-42", "version": 9, "address": "5 Quay St"}))
print(apply({"customer_id": "c-42", "version": 8, "address": "12 Mill Lane"}))
print(store["c-42"])

Trace: the first call sees 9 > 7, writes 5 Quay St and bumps the version to 9. The second sees 8 <= 9 and returns without writing. Output:

applied v9 -> 5 Quay St
rejected v8 (have v9)
Customer(address='5 Quay St', version=9)

The late event is now harmless instead of destructive. In SQL the same thing is UPDATE customers SET address=?, version=? WHERE id=? AND version < ? and checking the affected-row count.

SymptomCauseFix
One customer's events on different partitionsrandom or round-robin keykey = customer id
Older event overwrites newerno version checkconditional write on version
Same event applied twice after a rebalanceat-least-once deliveryidempotent upsert keyed by event id

Trade-offs

A hot partition is the real cost. One customer emitting 5,000 events/s pins to a single partition, and that partition's consumer is the ceiling: you cannot add consumers to fix it. Bucketing the key (customer_id + shard) breaks ordering, so only do it when the entity's events are genuinely independent. Head-of-line blocking is the other cost: one poison event stalls the partition. Fix that with a dead-letter queue after N attempts, not by parallelising the partition.

Ordering across entities is not guaranteed and you shouldn't need it.

Failure modes

  • Clock-based last-write-wins: two hosts with 200 ms skew, and the older event wins. Detect it by alerting when a rejected event's timestamp is newer than the applied one's.
  • Producer retry with max.in.flight.requests.per.connection > 1 and idempotence off: a retried batch can land after a later batch. Set enable.idempotence=true (the default in Kafka 3.x) or max.in.flight=1.
  • Rebalance mid-batch: consumer A processed but didn't commit, B reprocesses. That's a duplicate, not a reorder: the idempotent upsert covers it.

How I'd verify

Publish two events for one customer in reverse order and assert the final state is the later one; then kill the consumer mid-batch and assert the state is unchanged. A hypothetical incident in this shape: address-change events partitioned randomly, an older change overwrote a newer one for 40 customers, and shipments went to the previous address until the version check was added.

Curated: · Written: · Reviewed:

Q&A-46A message fails every time it is processed. What should happen to it?

Clarify first. "Fails every time" is the load-bearing phrase. A message that fails once may be a transient dependency blip. A message that fails on every attempt with the same error is deterministic: bad schema, a referenced entity that does not exist, a payload the parser rejects. Retrying a deterministic failure is a busy loop with extra steps.

Answer. After a bounded number of attempts with backoff, move it off the main path to a dead-letter queue carrying the original payload, the error, the attempt count and the source metadata. Keep the main queue or partition draining. Alert on the DLQ. Replay after the fix.

Why unbounded retry is worse than dropping. Two mechanisms, depending on the broker:

  • Ordered partitions (Kafka). The consumer never commits the offset past the poison record, so the partition stalls. Every record behind it waits. Throughput for that partition goes to zero. Take a partition doing 2,000 msg/s: one poison record stalls it for the six hours it takes someone to notice. 43 million messages delayed behind one bad payload.
  • Visibility-timeout queues (SQS). The message becomes visible again after the timeout and is redelivered, consuming a receive and a worker slot each cycle. It does not stall the queue outright, but it burns capacity, and if the failure is expensive (a downstream call that times out at 30 s), it starves healthy messages.

How the mainstream brokers bound it.

BrokerMechanismBound
SQSredrive policy maxReceiveCount (1–1000)message moves to DLQ once ReceiveCount exceeds it
RabbitMQ quorum queuex-delivery-limit + dead-letter exchangeafter N deliveries, routed to DLX with an x-death header
RabbitMQ classic queueno delivery limit; retry queue with TTL + DLXTTL expiry dead-letters the message
Kafkanone built inyou produce to a retry/DLT topic yourself (Spring Kafka @RetryableTopic), or seek past the offset

A trace of the SQS path with maxReceiveCount = 4:

receive 1 -> handler throws -> visibility timeout expires -> redelivered
receive 2 -> handler throws -> ...
receive 3 -> handler throws -> ...
receive 4 -> handler throws -> ...
receive 5 -> ReceiveCount (5) > maxReceiveCount (4) -> SQS moves it to the DLQ

Trade-offs.

  • Dead-lettering breaks ordering. If the poison message is an "account closed" event and the next 500 events are "balance changed", moving it aside means downstream sees the balance changes without the close. When order is load-bearing, halt the partition and page a human instead of skipping.
  • Retry tiers vs a single DLQ. A retry topic with escalating delays (1 s, 10 s, 1 m) absorbs transient failures before anything reaches the DLQ. Costs you a topic per tier and the ordering guarantee.
  • Skip-and-log. Cheapest, loses the message. Acceptable only when the message is genuinely disposable and you log enough to reconstruct it.

Failure modes to name.

  • DLQ as a graveyard. Depth alert fires once, nobody drains it. Alert on depth and oldest-message age: 3 messages that are 9 days old is worse than 10,000 from an hour ago.
  • Context-free entries. Payload only, no error, no attempt count, no source offset: replay is impossible and you debug blind.
  • Replay storm. Fixing the bug and replaying 200k messages at once re-creates the outage. Replay throttled.
  • Non-idempotent consumers. Retries before the DLQ already duplicated side effects; the DLQ does not undo them.

Verify. Send a message that always throws, confirm it lands in the DLQ after exactly the configured number of receives, confirm the partition or queue keeps draining while it sits there, then replay it after the fix and confirm it succeeds.

Curated: · Written: · Reviewed:

Q&A-15You must notify customers of events. What does a reliable webhook system need?

Assume at-least-once. The receiver can be slow, down, or intolerant of duplicates, and you cannot make the network exactly-once, so the sender owns retries, ordering is not guaranteed, and the receiver must deduplicate. What the system needs:

  • Signed payloads. HMAC over the raw body with a per-endpoint secret, and a timestamp inside the signed material so a captured request can't be replayed an hour later. The receiver verifies before it parses.
  • A stable event id. The same id on every retry of the same event, so the receiver can dedupe on it.
  • Retries with capped exponential backoff. Retry on 5xx and timeouts, not on 4xx: a 400 will not fix itself. Cap both the delay and the attempt count.
  • A dead-letter state and replay. After the cap, park the event, keep a delivery log, and expose a replay endpoint so you can see what failed and re-send it.
  • A documented receiver contract. Dedupe on event id, return 2xx before doing the work, tolerate out-of-order arrival.

A backoff schedule that holds up under a receiver outage:

AttemptDelay before attemptResult
1immediate500
21 min500
32 min500
44 min500
58 min500
616 min200, delivered
7–832 min, 64 minunused
after 8—dead-lettered, replayable

Eight attempts span about 2h07m of wall clock. That is long enough to ride out a deploy or a brief outage and short enough that a permanently broken endpoint stops consuming workers.

The failure this guards against is one I've hit: a sender retried every failure immediately and without a cap, which put 90,000 duplicate calls into a single customer in an hour. Their edge blocked the sending IP, and now every customer's deliveries were failing. Uncapped immediate retries turn one slow receiver into a self-inflicted flood on your own egress. Detect it by alerting on retry volume per endpoint and on dead-letter growth, not just on delivery failures.

I would not call it done without evidence: take a receiver offline for an hour, confirm deliveries resume with no loss and no unbounded retry, then replay a dead-lettered event and confirm the receiver dedupes it.

Curated: · Written: · Reviewed:

Q&A-13Your queue consumer cannot keep up with producers. What do you do?

What I'd say first

Ask two things before touching anything: is this a spike or a sustained gap, and what are the queue depth and the age of the oldest unacked message right now? Those decide whether I shed load immediately or fix the consumer.

Then: bound the buffer, push back on producers, and find the consumer's real bottleneck. An unbounded queue does not fix a throughput gap: it converts it into a memory outage.

Why the queue alone can't help

If producers offer λ messages/s and the consumer drains μ, the backlog grows at λ−μ. Buffering changes neither λ nor μ; it only buys time, and only if the buffer is bounded.

Worked example (hypothetical but typical): producers push 12,000 msg/s, one consumer drains 8,000 msg/s. Backlog grows 4,000/s, or 240,000/min. At 1 KB per message, a 1M-message in-memory buffer is roughly 1 GB and fills in 250 s: about four minutes from healthy to OOM-killed, taking every buffered message with it.

Options under overload

StrategyWhat happens when arrivals exceed service rateMessage loss
Unbounded buffermemory climbs to the limit, process killedall buffered
Bounded + rejectproducers get 429 / flow controlnone, but producers must handle it
Bounded + shed low prioritybest-effort below the cutlow-priority only
Scale consumershelps only if the consumer is the bottleneck and downstream can absorb itnone

Consumer side, once the buffer is bounded

Scaling out is correct only when the consumer is the bottleneck. Check first:

  • Per-message synchronous I/O: a DB write or HTTP call per message. Batch it; a 100-message batch turns 100 round trips into one.
  • Prefetch and partition assignment. One slow partition stalls the rest (head-of-line blocking) even when other consumers are idle.
  • Partition skew. Ten partitions where one key takes 60% of traffic means nine consumers are underused and one is the queue.
  • A poison message retried forever, blocking its partition behind it.

Detection: consumer lag or queue depth, age of the oldest unacked message, p99 handler latency, and consumer CPU versus wall time. Low CPU with high latency means the consumer is waiting on I/O, not short of cores, so autoscaling on CPU will not fire.

Failure modes of the fix itself

  • 429 without jitter and backoff produces a retry storm that is worse than the original overload.
  • Scaling consumers past the downstream limit, say a database connection pool of 20, just moves the queue into the database.
  • Shedding without a dead-letter path loses data silently; the shed count has to be a metric someone watches.

Evidence before calling it fixed

Load test above capacity: ramp producers to twice the measured consumer rate and confirm queue depth plateaus and producers see 429s, rather than memory climbing. Watch oldest-message age, not just depth: depth can look flat while the oldest message is minutes old.

Bound the buffer, push back on producers, then fix whichever of the four consumer-side causes the metrics point at.

Curated: · Written: · Reviewed:

Q&A-47How do you run a nightly job safely across several instances?

The 90-second answer. Assume the schedule lives inside the app, every instance runs it, and the job writes to a shared database. Then two things make it safe: exactly one runner, and work that can be re-run. I'd claim the run in the database rather than reach for a lock service, because the claim and the work commit in the same transaction.

Why cron on every instance is the wrong default. Six instances means six concurrent runs. If the job is "select unpaid invoices, mark them paid, email", each run reads the same rows before any of them commits, so every invoice gets six emails. That is the incident I'd cite: 4,200 invoices, six copies each, 25,200 emails sent, 21,000 of them duplicates.

Getting one runner. Three options, in the order I'd reach for them:

  1. Database claim row (my default). A unique index on (job_name, run_date) makes the claim atomic.
  2. A scheduler that owns the schedule: Kubernetes CronJob, Airflow, a systemd timer on one host. A CronJob creates one Job per tick, so the multi-instance problem doesn't exist. Set concurrencyPolicy: Forbid so a slow run doesn't overlap the next, and startingDeadlineSeconds so a missed tick is recorded rather than silently dropped.
  3. Distributed lock: Redis SET NX PX, an etcd lease, a Postgres advisory lock. It works, but the lease can expire while the job is still running (a 30-second stop-the-world GC pause, a stalled network call), and then two runners write. If you go this way, use a fencing token: the lock service hands out a monotonically increasing number and the downstream store rejects any write carrying a token lower than the last one it saw. Without fencing, a TTL lock generates duplicates under exactly the conditions you bought it for.
CREATE TABLE job_runs (
  job_name text NOT NULL,
  run_date date NOT NULL,
  owner    text NOT NULL,
  status   text NOT NULL DEFAULT 'running',
  cursor   bigint,
  PRIMARY KEY (job_name, run_date)
);

INSERT INTO job_runs (job_name, run_date, owner)
VALUES ('nightly-invoices', CURRENT_DATE, :instance_id)
ON CONFLICT (job_name, run_date) DO NOTHING
RETURNING owner;

Trace: instances A and B both fire at 02:00:00 UTC. A's INSERT commits and RETURNING yields A, so A runs. B's INSERT hits the primary key, ON CONFLICT DO NOTHING suppresses the error, and RETURNING yields zero rows, so B exits. The claim is atomic because the unique index is enforced at commit: there is no window between "check" and "set" for a second runner to slip through. If A dies mid-run the row stays running; a reaper flips rows older than two hours to failed, and the next run resumes from cursor. The trade-off against a TTL lock: no expiry race, but you need the reaper, and you can't re-run the same day without clearing the row.

Make the work re-runnable anyway. The claim stops concurrent runs; it does not stop a retry after a partial run. Process in bounded batches (500 rows), commit the batch and the cursor together, and make each write idempotent: INSERT ... ON CONFLICT (invoice_id) DO NOTHING, or an idempotency key on the downstream API. A crash at row 3,000 then costs a re-read, not a second email.

Failure modes and how I'd see them.

  • Overlap: run N+1 starts before N finishes. With concurrencyPolicy: Forbid, Kubernetes skips the tick and logs it: alert on skipped ticks, not only on failures.
  • Claim leak: a crashed runner leaves the row running and the job never runs again. Detect with a heartbeat: the job writes a heartbeat row each run, and an alert fires if the newest heartbeat is older than 26 hours. That catches "never started" and "started and hung" the same way.
  • DST and timezone: a 02:00 local schedule runs twice or not at all on transition days. Schedule in UTC.
  • Poison batch: one malformed row fails the batch every night. Cap retries per batch and move the row to a dead-letter table with the error.

Before calling it done: run the job on all instances in staging and assert exactly one claim row and one set of side effects; kill the owner mid-run and confirm the reaper plus checkpoint produce a clean resume with no duplicate writes.

Curated: · Written: · Reviewed:

Q&A-54Only one instance should run a task. How do you guarantee that?

What I'd say first

Clarify the requirement before picking a mechanism: "only one instance runs the task" usually means at most one concurrent writer, not exactly-once-ever. If the task is retryable and its side effects are idempotent, the bar is much lower than if it moves money. I'd also ask whether the instances sit in one cluster or across regions, and whether the work can be sharded, if it can, I don't need leader election at all.

Assuming a single logical writer with non-idempotent side effects, I'd use a lease in a consensus store (etcd, ZooKeeper, Consul) with a TTL, renewed at roughly a third of the TTL, plus a monotonically increasing fencing token that every downstream write carries. The store rejects any write whose token is lower than the highest it has seen. The lease gives liveness: a dead leader's lease expires and someone else takes over. The token gives safety: a leader that was merely paused can't corrupt state when it wakes up.

Why the lease alone is not enough

A lease with a TTL is a bet that the holder notices expiry before anyone else takes over. A stop-the-world pause breaks that bet. Worked example: lease TTL 15 s, renewed every 5 s. The leader takes a 12 s GC pause at t=4 s, so its last successful renewal was at t=0 and the lease expires at t=15. A standby acquires at t=15 and starts writing. The old leader wakes at t=16, still believes it holds the lease, and writes. Two writers, no error raised. The fencing token turns that into a rejected write instead of a corrupted one.

The write path

-- every leader write carries its fencing token
UPDATE ledger
   SET balance = :new_balance,
       last_token = :token
 WHERE account_id = :id
   AND last_token < :token;
-- 0 rows affected => this writer is stale; abort and stop the task

Trace: leader A holds token 41 and is paused. Leader B acquires the lease with token 42 and writes, setting last_token = 42. A resumes and runs the same statement: last_token < 41 is 42 < 41, false, so 0 rows update. A aborts. The check has to be in the same transaction as the write, and the token has to come from the consensus store's monotonic counter, not from a local clock.

Mechanisms compared

MechanismTwo writers possibleDetectedNotes
In-process flag / singletonyes, across processesnoOnly works inside one process
Redis lock, async replicationyes, on failovernoReplication is async; a failover can lose the lock
etcd/ZooKeeper lease, no fencingyes, after a pausenoLease expiry is not a safety guarantee
Lease + fencing tokennostale writer rejectedNeeds a token check on every write
Kafka consumer group, one partitionnorebalanceOnly if the task shards by key

Failure modes and how I'd detect them

  • Pause longer than the TTL: the case above. Detect by counting fencing-token rejections; any non-zero rate is a page, because it means two leaders overlapped.
  • Clock skew: if the lease is computed from local wall clocks, a node with a fast clock can expire a lease early. Use the consensus store's clock and its session/keepalive, not local time.
  • Partition: the leader can't renew but keeps working. Fencing stops its writes; it does not stop a side effect that isn't a fenced write, like sending an email or calling a third-party API. Those need an idempotency key derived from the token or the work item.
  • Lost lease mid-task: the task must be safe to abandon. Checkpoint progress and make each step idempotent, so the next leader redoes the last step rather than half of it.

When I wouldn't do this

If the work shards by key, a consumer group per partition gives one owner per key with no election. If the task is cheap and idempotent, run it everywhere and dedup on a natural key. Leader election is for the case where the work is genuinely singular and its side effects are not idempotent.

Evidence

I'd test it, not assume it: pause the leader past the lease (SIGSTOP the process for TTL + 5 s) and confirm its writes are rejected and the new leader's are not. That test is the difference between a lock and exclusion.

Curated: · Written: · Reviewed:

Q&A-53An order must reserve stock, charge a card, and book delivery across three services. How do you keep that consistent?

Three services, three local transactions, no shared transaction manager. I'd run this as an orchestrated saga: a coordinator drives reserve → charge → book, persists each step's outcome, and compensates backwards on failure. The database gives me atomicity inside one service; across services I get eventual consistency and have to earn it with idempotency and persisted state.

Assumptions: stock reservation and delivery booking are reversible; the card charge is reversible only via refund, which is a customer-visible event, not a rollback. That asymmetry drives the ordering.

Why idempotency. The coordinator retries a step whenever it doesn't get a definitive answer: a timeout, a dropped response, a crash between the call and the state write. The participant may have already committed. Without an idempotency key the retry reserves stock twice or charges twice. So each participant takes a client-supplied key (saga id plus step name), stores it with the result, and returns the stored result on replay. That makes at-least-once delivery safe.

Why persisted state. The coordinator can crash between calling charge and recording that charge succeeded. On restart it must know whether to retry forward or compensate. So the state transition is written before the call (intent) and after (outcome), and recovery reads the log. A step with intent but no outcome is retried with the same key: safe whether or not the participant committed.

Trace of that crash: coordinator writes intent(charge, key=saga:2:charge) → calls charge → charge commits, response lost → coordinator crashes. On restart the log shows intent with no outcome, so it retries charge with the same key; the participant returns the stored success; the coordinator writes outcome(success) and proceeds to book. Drop the key and this is a double charge.

Ordering. Reserve before charge, charge before booking delivery. If booking fails after the charge, I refund: annoying but recoverable. Charge after booking would leave a booked delivery I can't easily unbook and a customer holding a slot they didn't pay for. Reserve first because it's the cheapest to compensate and the most likely to fail under stock contention.

StepForwardCompensationIdempotency key
1reserve stockrelease stocksaga:1:reserve
2charge cardrefundsaga:2:charge
3book deliverycancel bookingsaga:3:book

Orchestration vs choreography. Orchestration gives one place to see saga state, easier to reason about and debug, but the coordinator is a hotspot and a coupling point. Choreography removes the coordinator but scatters the flow across services; nobody can answer "where is order 123" without joining event logs. For three steps with a customer-visible refund, I want the coordinator.

When compensation is unacceptable. A refund is not a rollback. If I've charged and then must compensate, the customer sees a charge and a refund: two line items, possible FX loss, a support call. For irreversible or customer-visible steps, order them last, or gate them behind a reservation that can be voided before capture (authorize now, capture later). If no ordering makes compensation acceptable, that's when a distributed transaction or a single-service boundary is the right call.

Failure modes. Compensation itself fails: the refund call times out. The saga is stuck; retry compensation with the same key and alert after N attempts, because a saga that can't compensate needs a human. Duplicate delivery booking from a retry without a key. A coordinator crash loop re-driving a step, bounded by persisted state and a max-attempts counter.

Test: fail each step in turn, kill the coordinator mid-step, replay. Assert the order ends either fully committed or fully compensated, and that no participant saw a duplicate effect.

Curated: · Written: · Reviewed:

Q&A-52Which consistency guarantees do clients actually feel, and how do you provide them?

The 90-second answer

Three consistency guarantees are the ones a user can actually perceive, and none of them requires linearizability:

  • Read-your-writes: I post a comment, then I see it.
  • Monotonic reads: I never see the comment, then stop seeing it.
  • Causal: I never see a reply before the comment it answers.

These are the session guarantees from Terry et al.'s 1994 paper, and the reason to reach for them first is cost. They need no coordination on the write path: the write goes to the primary and replicates as it already does. The work moves to the read path, where the client carries a version token and a replica either serves the read or refuses it.

Assumptions I'd state up front: one logical client session, asynchronously replicated replicas, no multi-key transaction. If the client needs a globally consistent snapshot across keys (a backup, a distributed transaction's read set) these guarantees do not give it to you. That is snapshot isolation or a linearizable read, and it costs a quorum round trip.

Mechanism

Each session holds a min_version. It is a logical version (an LSN, a hybrid logical clock, whatever the store already stamps on writes) not a wall-clock timestamp. After a write, set min_version to the version the primary assigned. On every read, accept only a replica whose applied version is at least min_version; otherwise try another replica, and fall back to the primary if none qualifies. Monotonic reads is the same field, never allowed to decrease. Read-your-writes is that field being set by your own write.

Causal consistency is the one that needs metadata on the write, not just the read. Each write carries the version vector of what it depends on, and a replica defers applying it until those dependencies have arrived. That is where the real cost lives: the causal context grows with the number of concurrent writers, and garbage-collecting it is the hard part.

GuaranteePreventsMechanismCost
Read-your-writesmissing your own updatemin_version set by your writeread-path wait or primary fallback
Monotonic readsstate moving backwardsmin_version never decreasessame
Causalreply before its parentdependency vector on the write, deferred applymetadata per write, GC of context

A trace

Three replicas, primary P. A client writes x = 1; P stamps it version 42. Replication lag puts R1 at 42, R2 at 40, R3 at 38. The session's min_version is now 42.

def read(key, session):
    for _ in range(3):
        r = pick_replica()
        v, val = r.get(key)          # (applied_version, value)
        if v >= session.min_version:
            session.min_version = max(session.min_version, v)
            return val
    return primary.get(key)          # fallback: never stale

Walking it: pick_replica returns R2, v = 40, 40 >= 42 is false, loop. R3, v = 38, false, loop. R1, v = 42, true: return 1, min_version stays 42. Had all three been stale, the primary fallback returns 1 and the guarantee still holds. Drop the token and let a load balancer send the next read to R3: the client sees x = 0 after seeing x = 1. That is the whole failure, and it is invisible in server metrics.

Failure modes and how you catch them

  • Token lost on reconnect or load-balancer reroute. Read-your-writes breaks silently. Detect it by having the client assert the returned version is at least the one it expected, and counting violations.
  • Bounded wait with a stale fallback. If you time out the wait and serve the stale replica anyway, monotonic reads breaks. Count reads served below min_version.
  • Wall-clock tokens under skew. A replica whose clock runs ahead looks current while holding stale data. Use logical versions, not raw timestamps.
  • Causal context growth on hot keys. Many concurrent writers on one key inflate the dependency vector. Track bytes of context per write and alert on the trend.

Take a worked example with illustrative numbers: a feed read from two replicas with 50 ms of lag showed a reply above its parent on 8% of loads, which users reported as missing comments. The fix was not a stronger consistency model: it was a version token on the read path and a dependency edge on the reply write.

When not to use them

If the client needs a consistent view across keys, or a write that depends on a read of a different key, session guarantees are the wrong tool. Reach for a quorum read (R + W > N) or a consensus-backed linearizable read, and accept the cross-region round trip. Session guarantees buy you the three things users notice, at read-path cost only: that is the trade, and it is usually the right one.

Curated: · Written: · Reviewed:

Q&A-51What does CAP actually force you to choose in a real system?

CAP only constrains you while a partition is in progress. With a healthy network you get consistency and availability together, and the trade you're actually making is latency against consistency: the "else" half of PACELC. So "which do you pick?" has no system-level answer. It has an operation-level answer, and the partition itself isn't a choice: partitions happen, and what you choose is what each operation does while one is happening.

Two definitions to pin down first, because they decide the rest. C in CAP is linearizability, a read sees the most recent completed write, not ACID's C. A means every request to a node that hasn't failed gets a non-error response. It doesn't mean fast and it doesn't mean correct. A node that returns 503 because it can't reach a quorum is unavailable in CAP's sense, and that is a legitimate design, not a bug.

With those fixed, the decision is per operation. For a checkout service:

OperationUnder partitionWhat it costs
reserve stockrefuse (CP)failed checkouts for the partition's duration
product descriptionserve stale (AP)a shopper sees an old price or copy
order historyserve stale (AP)a recent order missing from the list

The reserve path has to be CP, because two nodes accepting the same last unit isn't a stale read, it's a wrong write. The other two are reads where staleness is bounded and cheap.

I've watched the opposite choice cost real money. A reservation service read available stock from a read replica. During a partition the replica fell about 90 seconds behind the primary, and the reservation path kept reading its cached count. In that window 240 units of a limited item were reserved against stock that had already been sold on the primary side. By the time the partition healed, every one of those orders had to be refunded with an apology. The bug wasn't eventual consistency; it was eventual consistency applied to an operation that needed linearizability.

The same reasoning runs the other way. Make product description CP and a partition takes the catalog down for no benefit: nobody is harmed by a 90-second-old description.

Verification is where this usually falls apart, because a passing staging run proves nothing about a partition. Cut the dependency deliberately (an iptables DROP on the replica's port, or a chaos tool) and assert per operation: reserve returns an error inside its timeout, description keeps serving with a stale-read age under the budget. Then alert on that age. If replica lag exceeds the staleness budget you promised for description, that's a page, because you've silently broken the contract you wrote down.

Write the choice into the design doc per operation, with the staleness budget next to it. Otherwise the next engineer assumes the whole system is strongly consistent and adds a read from the replica to the reservation path.

Curated: · Written: · Reviewed:

Q&A-55Can you order events across servers by their timestamps?

Short answer: No. A wall-clock timestamp from one machine is not comparable to one from another, so it can't order events across them. Order with a sequence number or version assigned by a single authority (or a consensus log index); measure elapsed time with a monotonic clock.

Why wall clocks fail. Every host's clock is a quartz oscillator corrected by NTP or PTP, and two things break ordering:

  • Drift between corrections. Server quartz runs at roughly 10–100 ppm off nominal, so an uncorrected host diverges by about 1–10 ms per minute. NTP-synced hosts in one datacenter usually sit within a few milliseconds of each other, but that is a typical bound, not a guarantee.
  • Steps. When NTP decides the offset is too large it steps the clock, which can move it backwards. A timestamp taken after the step can be earlier than one taken before it on the same host, never mind across hosts.

So t_a < t_b tells you nothing reliable about which event happened first.

What to use instead.

  • Per-entity version or sequence. Each record carries a counter incremented by whoever writes it; conflicts resolve on the version, not on time. This gives a total order per key, which is what last-write-wins actually needs.
  • Single assigning authority. One sequencer (a database sequence, a Kafka partition, a log service) stamps events. Total order, at the cost of a bottleneck and a single point of failure.
  • Consensus log. A Raft or Paxos log index gives a total order the cluster agrees on, for a round trip per write.
  • Hybrid logical clocks. A physical clock plus a logical counter, so timestamps stay near wall time but never go backwards. Useful for causality-ish ordering and debugging; still not a global total order without a coordinator.
  • Monotonic clock (CLOCK_MONOTONIC, time.monotonic(), Go's time.Since) for durations and timeouts. It never steps backwards, but its epoch is arbitrary, so it is only meaningful within one process or host.

Worked example (hypothetical figures). Two servers accept edits to the same record and merge last-write-wins on the timestamp. A's clock runs 200 ms ahead of true time, B's 200 ms behind.

Real timeServerStampedLWW winner
10:00:00.000A10:00:00.200
10:00:00.100B09:59:59.900A. The earlier edit

The later edit loses. In the incident I'm drawing on, 400 ms of skew between two nodes made the earlier edit win 1 in 9 conflicts, and support saw edits revert.

Ordering basisCorrect orderReverted edits
wall-clock timestampsno1 in 9 conflicts
per-entity versionyes0

Failure modes and how to catch them. A clock stepping backwards on a host (alert on NTP offset and step events, and log the sequence number alongside the timestamp so you can reconstruct order after the fact); a VM live migration or a long GC pause freezing the clock; leap-second smearing applied differently by two hosts; and treating a monotonic clock as comparable across hosts, which it is not.

Verification. Skew two nodes' clocks by ±200 ms in a test, run the conflict workload, and confirm ordering still follows the sequence number and no edits revert.

Curated: · Written: · Reviewed:

Q&A-48A customer asks for their data to be deleted. What does your backend need to do?

What a strong candidate says first

Verify who is asking and under which regime (GDPR Art. 17, CCPA/CPRA, or a contract), then treat deletion as a workflow over an inventory of stores, not a DELETE. Three things decide the shape: a data map keyed by a stable subject ID, a per-store action (hard delete, anonymize, or crypto-shred), and a durable deletion journal so restoring a backup doesn't resurrect the subject. The request is not done until every store has reported back.

The inventory, and what each store gets

StoreActionMechanismDeadline
primary DB: users, sessionshard deleteDELETE in the same transaction as the journal writeimmediate
primary DB: ordersanonymizenull user_id, ship_to, email; keep the rowimmediate
search indexhard deletedelete-by-query on user_idminutes
cache / CDNhard deletekey-prefix purgeminutes
analytics warehouseanonymizeoverwrite PII columns, keep fact rows7 days
logsexpire30-day retention, no PII in log bodies30 days
backupscrypto-shred or journal replayper-subject key deletion, or replay after restore90 days
processors (email, CRM)hard deletevendor deletion API + receipt30 days

The journal is the part people miss

# Python 3.12
def erase(subject_id: str) -> None:
    with db.transaction():
        # journal first, same transaction as the delete
        db.execute("INSERT INTO deletion_journal (subject_id, requested_at) "
                   "VALUES (%s, now())", (subject_id,))
        db.execute("DELETE FROM users WHERE id = %s", (subject_id,))
        db.execute("DELETE FROM sessions WHERE user_id = %s", (subject_id,))
        # invoices are retained for tax; strip the PII, keep the row
        db.execute("UPDATE orders SET user_id = NULL, ship_to = NULL, email = NULL "
                   "WHERE user_id = %s", (subject_id,))
    search.delete_by_query({"term": {"user_id": subject_id}})
    cache.delete_prefix(f"user:{subject_id}:")
    queue.publish("analytics.erase", {"subject_id": subject_id})

Trace erase("u_8812"): users and sessions lose the row; orders rows survive with user_id, ship_to, email set to NULL; the search index and cache are clear; the analytics erase is queued, not done. The request is incomplete until that consumer acks. After any restore, replay the journal:

def replay_after_restore() -> None:
    for (subject_id,) in db.query("SELECT subject_id FROM deletion_journal"):
        erase(subject_id)

With weekly fulls and 90-day retention, a subject deleted today still sits inside up to 13 full backups. Restoring the full from 60 days ago brings the row back; the replay removes it again. So the journal must outlive the backup window: 90 days plus margin, or indefinitely.

Choosing between delete, anonymize, and crypto-shred

  • Hard delete when nothing obliges you to keep the row and no legal hold is active.
  • Anonymize when the row must survive for referential integrity or a retention duty: tax law commonly requires keeping the invoice row for 6–10 years depending on jurisdiction, and GDPR Art. 17(3) carves out legal obligations and legal claims.
  • Crypto-shred when the storage is immutable (S3 Object Lock, WORM tape): encrypt each subject's payload under a per-subject key and delete the key. AWS KMS key deletion has a 7–30 day waiting period (default 30), so the data is unreadable only after that.
  • Defer or refuse when a litigation hold, AML retention, or an unverified requester applies, and say so in writing.

Failure modes

A deleted_at flag that an admin query or a forgotten JOIN filter still returns. Denormalized copies: orders.email, events.payload, support tickets. A compacted Kafka topic needs a tombstone (null value for the key) and waits out delete.retention.ms (default 24 h). Orphaned S3 blobs keyed by UUID with no DB row. Logs that carry user_id in the body. A processor that accepts the API call and silently no-ops.

Evidence

Seed a canary subject with a unique sentinel (canary-<uuid>@example.invalid), run the full pipeline, then grep every store (including logs, analytics, and the search index) for the sentinel. Zero hits is the only completion signal worth reporting.

Curated: · Written: · Reviewed:

Q&A-56What has to be true before you can add instances of a service freely?

What a strong candidate says first

"Freely" means you can add or remove an instance at any moment, including while it is mid-request, and no client-visible behaviour changes. Two invariants have to hold:

  1. Any instance can serve any request.
  2. Any instance can disappear at any instant without losing state the system has already acknowledged.

Everything else follows. Four things break those invariants in practice: request-scoped state in process memory, instance-local resources treated as authoritative, per-instance background work, and shared dependencies that don't scale with instance count.

The one interviewers are fishing for

Sessions in process memory. Two instances behind a round-robin balancer, login handled by A:

RequestInstanceSession found?Result
1 (login)Acreated200
2Bno401
3Ayes200
4Bno401

Every other request after login fails. The fix is to move the session out (a shared store (Redis, a DB table) or a signed token carried in the request) or to pin clients to an instance with sticky sessions. Sticky sessions are a legitimate stopgap but they don't satisfy "freely": you can't rebalance, and losing an instance logs out everyone pinned to it. A signed token avoids the lookup hop but can't be revoked without a denylist, which puts you back on shared state.

The ones people miss

  • Local disk. Uploads written to /tmp or the container filesystem vanish on reschedule and are invisible to the other instance. Object storage or a shared volume.
  • Scheduled work. A cron entry baked into the image runs once per instance. Two instances, two sends. Elect a leader, or use a queue with a visibility timeout so only one worker claims each job.
  • Connection pools. Postgres defaults to max_connections = 100, with 3 reserved for superusers. A pool of 20 per instance means the sixth instance starts failing to connect: the service scaled, the database didn't. PgBouncer, or a smaller pool per instance sized as (max_connections - reserved) / max_instances.
  • Shutdown. Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds (default 30) before SIGKILL. A request that outlives that window dies. Fail the readiness probe first so the endpoint is removed from rotation, then drain in-flight work.
  • Instance identity. Anything keyed on hostname, pod name or IP (locks, leader election, cache keys, log correlation) has to be re-derived, not assumed stable.

How I'd verify it

Not by reading the code. Send consecutive requests to different instances and diff the responses; kill an instance under load and watch the error rate; add an instance and confirm downstream QPS per instance doesn't multiply. If any of those change behaviour, the service isn't free to scale yet.

Curated: · Written: · Reviewed:

Q&A-57What happens to in-flight requests when you deploy?

Assume Kubernetes, a rolling update, and a Service in front of the pods. When a pod is deleted, the API server marks it Terminating and the endpoint is marked not-ready at roughly the same moment the kubelet sends SIGTERM. Endpoint removal is asynchronous: kube-proxy has to watch the change and rewrite iptables on every node, and a cloud load balancer has its own deregistration delay, so the pod keeps receiving new connections for a short window after the signal. In-flight requests survive only if the process keeps serving them until the LB stops routing to it and the grace period hasn't run out. Default terminationGracePeriodSeconds is 30s; when it expires the kubelet sends SIGKILL and anything still running dies mid-request.

The recipe that works: run a preStop hook that sleeps a few seconds to cover the endpoint-propagation race, handle SIGTERM by failing readiness immediately and stopping the accept loop, finish in-flight requests, then close pools and consumers, and set terminationGracePeriodSeconds above the longest normal request plus the preStop sleep. The preStop hook runs before SIGTERM and counts against the grace period, so the budget is shared.

Alternatives, and when each is the right one:

MechanismWhat it fixesLimit
preStop sleependpoint-propagation racefixed guess; eats the grace period
LB connection draining (ALB deregistration delay, default 300s)in-flight requests at the LBonly helps if the app keeps serving
app SIGTERM handlerin-flight requests in the processbounded by the grace period
blue-green / surge deploycapacity cliff, not in-flight requestsold pods still get SIGTERM
client retries + idempotencyanything you can't drainneeds idempotent handlers

Graceful shutdown alone is not enough for WebSockets, SSE, or gRPC server streams that live for minutes or hours: a 30s grace period cannot drain a connection that outlives it. There you send a close frame and let the client reconnect to another pod, or accept the disconnect. Same for any request longer than the grace period: it gets SIGKILLed, so either raise the grace period or move the work off the request path.

Worked trace, single pod, default 30s grace, no preStop:

  • t=0.0s: pod marked Terminating; endpoint marked not-ready; SIGTERM delivered.
  • t=0.0–1.5s: kube-proxy updates iptables on each node; new connections still land on this pod.
  • t=0.0s: app stops accepting new connections.
  • t=0.0–?s: app finishes in-flight requests.
  • t=30.0s: SIGKILL if the process is still running.

With no SIGTERM handler the process dies at t=0.0s and drops every in-flight request. A rolling deploy of 12 pods, each holding 40 in-flight requests at SIGTERM, drops 12 × 40 = 480 requests. With the handler and a grace period longer than the longest request, the same rollout drops 0.

Failure modes to check for: a handler that stops accepting but never finishes in-flight work (requests hang until SIGKILL); a grace period shorter than p99 request time (tail requests die); a preStop sleep longer than the grace period (SIGKILL before the app ever sees SIGTERM); and non-idempotent handlers where a client retry duplicates a side effect. Detect them by deploying under load in staging and watching the LB's 5xx count and the app's in-flight gauge through the rollout: zero dropped requests is the pass condition.

Curated: · Written: · Reviewed:

Q&A-58What is the difference between a liveness and a readiness probe, and what should each check?

Readiness answers one question: should the load balancer send this instance traffic right now? Liveness answers a different one: is this process wedged in a way that only a restart clears? The consequences differ, and that difference decides what each probe should check.

A failed readiness probe removes the pod's IP from the Service endpoints. The container keeps running, keeps its memory, and rejoins the rotation when the probe passes again. A failed liveness probe kills the container; the kubelet restarts it under the pod's restartPolicy, and repeated failures back off from 10 seconds up to a 5-minute cap (CrashLoopBackOff). Readiness is a traffic decision, liveness is a process decision.

That asymmetry sets the content of each check. Liveness should test only what a restart can fix: the main loop is still scheduling, the HTTP handler responds, the process is not deadlocked. It must not touch the database, a queue, or another pod. Readiness should test whether this instance can serve a request now: listener bound, config and secrets loaded, migrations applied, caches warm. Whether to include downstream dependencies is the real judgement call. If the pod can serve nothing without the database, a dependency check in readiness is honest, but a database outage then drains every replica at once; the alternative is to keep readiness local and let retries, circuit breakers and load shedding absorb the downstream failure. Either way, the check belongs in readiness, never in liveness.

# Kubernetes 1.29
livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3      # ~30s of failure before a restart
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2      # ~10s before removal from endpoints
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 60     # up to 5 min for a slow JVM start

The failure this prevents: a liveness probe that ran SELECT 1. A 90-second database failover failed the probe on all 20 pods, the kubelet restarted each one, the restarts re-ran the same query against a database still failing over, and the outage stretched to 14 minutes. Move the same query to readiness and the pods leave the rotation, keep running, and rejoin when the database returns.

Probe designPod restartsOutage
dependency in liveness20, repeatedly14 min
dependency in readiness only090 s

Two more things worth saying. For a slow start, a JVM warming for two minutes, use a startupProbe rather than a long initialDelaySeconds on liveness; the startup probe suspends liveness and readiness until it passes once, so a slow boot cannot be mistaken for a hang. And a readiness probe that never passes blocks a rolling update: the rollout stalls at progressDeadlineSeconds (600 by default) with the old pods still serving.

To check your own setup, kill the dependency in staging and watch kubectl get pods -w and kubectl get endpoints side by side. Readiness should drop the pod within failureThreshold × periodSeconds and restarts should stay at zero. If pods are restarting during a dependency outage, the dependency is in the wrong probe.

Curated: · Written: · Reviewed:

Q&A-61How would you set an availability target for your service?

Start from the failure a caller sees, then pick the number the business will actually fund.

Assumptions I'd state up front: this is a request/response service with a client-visible latency signal, we already collect request-level metrics, and there's a release process we can gate. If any of those is false, the answer changes.

The direct answer: define the SLO on the user journey, not on the framework. Pick one or two indicators, set the target from what the business needs and what the last two quarters actually delivered, measure from the client's side, and attach a policy to the error budget before you publish the number.

A sketch of the spec for a service with a 300 ms expectation:

slo: checkout-api
indicator: fraction of requests with duration < 300ms
objective: 99.9% over 28d rolling
budget: 0.1% of requests
policy: >100% consumed -> freeze non-critical releases; >200% -> incident review

And the measurement, as a Prometheus recording rule:

slo:checkout_latency_ok:ratio_28d =
  sum(rate(http_request_duration_seconds_bucket{le="0.3"}[28d]))
  / sum(rate(http_request_duration_seconds_count[28d]))

The le="0.3" bucket has to exist in the histogram; if it doesn't, use the nearest bucket you do have and state the bound you're actually measuring.

Last quarter's numbers, which is what I'd bring to the review:

ItemValue
objective99.9% of requests under 300 ms
attained99.94%
budget consumed60%
policyfreeze risky releases above 100%

60% consumed means 0.06% of requests breached against a 0.1% allowance. The target is tight but survivable, and the policy never fired.

Now the trade-offs, because scope is where most SLO programmes go wrong.

Per-endpoint vs service-wide. Per-endpoint SLOs catch a single slow route before it drags the aggregate down, but you end up with dozens of objectives, each with its own budget and alert, and nobody owns the sum. Service-wide is one number, easy to communicate, but a single bad endpoint can hide inside a healthy aggregate. I'd publish the service-wide objective as the contract and keep per-endpoint rules as internal alerting only.

Latency vs success rate. Success rate is cheap to measure and stable; latency is what users feel and it degrades before errors appear. If you can only run one, run success rate and alert on latency. If you run both, don't average them into a single score: 99.9% success with a blown latency budget is not "99.95% healthy".

When not to bother. A formal SLO with an error budget and burn-rate alerts is overhead: dashboards, a review cadence, and a policy someone has to enforce. For an internal batch job, a low-traffic admin tool, or a service with no release cadence to gate, an alert on error rate plus a p99 latency threshold is enough. The SLO earns its cost when there are competing changes and a team needs a shared, defensible reason to say no.

Failure modes I've seen, and how to detect them:

  • The number with no policy. A team promised 99.99%, missed it twice in a quarter, and nothing happened. Detect: no runbook entry or release gate references the budget.
  • Measuring from the server. Server-side success rate looks fine while clients time out on connection setup. Detect: compare the server-side ratio against a synthetic client probe; a persistent gap means you're measuring the wrong thing.
  • Budget spent silently. Detect: burn-rate alerts on the 1h and 6h windows, not just the 28d attainment.

An objective without a policy is a slogan.

Curated: · Written: · Reviewed:

Q&A-62Which metrics would you add first to a new service?

The 90-second answer

Assume a new HTTP service, Prometheus-style pull metrics, and one on-call engineer who has never seen it. The first four things I instrument are request rate, error rate by class, a latency histogram, and saturation of whichever resource binds first. Rate, errors and duration describe what callers experience; saturation explains why. Without saturation you know the service is sick but not which knob to turn.

Per endpoint, that is roughly:

sum by (route) (rate(http_requests_total[5m]))
sum by (route, status_class) (rate(http_requests_total{status_class=~"5.."}[5m]))
histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))
pg_pool_in_use / pg_pool_max
SignalQuestion it answers
request rateis traffic normal
error rate by classis it failing, and how
latency histogramis it slow, and for whom
saturationwhy

Trade-offs and when I'd choose differently

Histogram versus summary: histograms aggregate across instances and let you compute any quantile server-side; summaries give exact per-instance quantiles but cannot be aggregated across replicas. I pick histograms unless I need exact per-instance quantiles and cannot choose buckets in advance.

Cardinality: whether to label by tenant. The deciding criterion is whether a responder ever pages on a single tenant. If yes, label it; if it only feeds a customer-facing dashboard, keep it out of the metric and get it from logs or traces. The arithmetic matters: 200 routes × 12 histogram buckets × 50 tenants is 120,000 series for one metric. Prometheus handles millions of series overall, but a single scrape target exporting six figures of series is a problem you will feel at scrape time.

Bucket boundaries: the Prometheus defaults run 0.005s to 10s. For a service whose cache hits return in 2 ms, those buckets put almost everything in the first one and the histogram tells you nothing. Set buckets around your SLO, if p99 must stay under 200 ms, you want resolution at 50, 100, 200, 500 ms.

Failure modes

  • Cardinality explosion: a user_id or raw path label turns one counter into millions of series and takes down the scrape. Detect it by watching prometheus_tsdb_head_series and per-target series counts.
  • Saturation on the wrong resource: measuring CPU when the database connection pool is the bottleneck. Detect it by checking whether the saturation chart moves before the latency chart during an incident; if it doesn't, you instrumented the wrong thing.
  • Business counters with no RED metrics: a service I worked on exported 240 business counters and no latency histogram, so during an incident nobody could say whether requests were slow or failing, and diagnosis took 50 minutes.

What would change the decision

If the service is a batch job rather than request-driven, rate/errors/duration do not apply: instrument queue depth, processing rate, and age of the oldest unprocessed item instead. If it is a library, none of this belongs in it.

The test I apply before calling the dashboard done: does it answer is it up, is it slow, is it failing, and why. A chart that answers none of those gets cut.

Curated: · Written: · Reviewed:

Q&A-63A request crosses six services and is slow. How do you find where the time goes?

Assumptions first: the services already emit metrics and logs, and I can ship a tracing change if needed. The order I work in is metrics to find which hop, traces to attribute the time inside that hop, and a profiler only once a span points at code I own.

Metrics first because they are already there and cheap. Per-service latency histograms help, but the sharper signal is client-side latency per dependency: the caller's view of each downstream call. That narrows six services to one or two in minutes. The catch is that aggregates hide bimodality: a service-level p99 can look fine while one dependency's p99 blows out. So I compare the caller's client-side span latency against the callee's server-side latency for the same call. The gap is network, queueing or connection-pool wait; the callee's own number is its work.

Traces answer what metrics cannot: which span in this one request owns the duration. A trace ties the spans of one request together across services via propagated context (W3C traceparent, or B3). Without propagation each service reports only its own view and nobody can tell which call caused the delay. Propagate through every call and into asynchronous work (thread pools, queue producers, batch jobs) because a broken link there is the most common reason a trace looks complete but is not.

Sampling is the trade-off that bites. Head sampling at 1% is cheap but drops 99% of the slow requests you actually care about. Tail-based sampling at the collector keeps every error and every request over a latency threshold, at the cost of buffering all spans until the trace completes and paying for that collector memory. I would run tail sampling with a latency threshold and accept the cost.

Tracing overhead is real but bounded: typically low single-digit percent CPU, and a few hundred bytes per span on the wire. The cost that surprises people is span cardinality: a span per loop iteration over 10k rows hurts more than the tracing agent does.

Worked example, a checkout that was slow across six services:

SpanDurationShare
inventory, 14 serial calls700 ms78%
payment120 ms13%
everything else80 ms9%

The 14 inventory calls were serial at roughly 50 ms each. Batching them into one call took the checkout from 900 ms to about 250 ms. Payment had been blamed for three days because it was the last slow-looking service in the chain, and its own server-side latency was fine: the time sat in the caller's client-side span.

If the slow span is inside one service and is not a dependency call, that is when a profiler earns its place: a continuous profiler, or a pprof CPU profile on that process for the window in question, correlated to the span's start and end. Tracing tells you which call is slow; the profile tells you whether it is CPU, lock contention or GC.

Failure modes worth naming. Clock skew between hosts makes span durations negative or inflated, so derive a child's duration from the parent's timestamps where you can. A missing context header in a queue hop splits one trace into two. And a trace where every span looks fast but the user request is slow usually means the time is outside the instrumented path: DNS, TLS handshake, connection-pool wait, or client-side rendering.

To close this out I need one slow trace, the span that owns most of its duration, and whether that span is a dependency call or in-process work.

Curated: · Written: · Reviewed:

Q&A-23A user reports a failure at 14:02. How do you find what happened?

Before I touch a log I'd ask two things: what the user actually saw (an error page, a wrong total, a spinner) and which account or tenant it was. "Failure at 14:02" is a symptom, not a request. If the user can hand me the request id we return in error responses, I'm done in one query. If not, I start from the timestamp.

Assume logs are UTC and the user is UTC+2, so 14:02 local is 12:02Z. Getting that conversion wrong is the first way these investigations go sideways.

I search the edge logs for that user id in a window around the time, 12:01:30 to 12:03:00, and pull the request id off the matching line. Then I query every service for that id and read the trace in order. For a checkout that returned 500:

12:02:11.483Z checkout  req=7f3a1c cart=9 items begin
12:02:11.484Z inventory req=7f3a1c reserve sku=SKU-44 qty=1 pool_wait=180ms
12:02:11.664Z inventory req=7f3a1c reserve -> 503 pool exhausted (max=20, in_use=20)
12:02:11.665Z checkout  req=7f3a1c reserve failed, retry 1/2
12:02:11.666Z inventory req=7f3a1c reserve sku=SKU-44 qty=1 pool_wait=180ms
12:02:11.846Z inventory req=7f3a1c reserve -> 503 pool exhausted (max=20, in_use=20)
12:02:11.847Z checkout  req=7f3a1c retry 2/2
12:02:11.848Z inventory req=7f3a1c reserve sku=SKU-44 qty=1 pool_wait=180ms
12:02:12.028Z inventory req=7f3a1c reserve -> 503 pool exhausted (max=20, in_use=20)
12:02:12.029Z checkout  req=7f3a1c giving up, returning 500
12:02:12.030Z edge      req=7f3a1c user=u_8821 POST /checkout -> 500 (547ms)

The trace answers it: the inventory connection pool is capped at 20, every reserve call waited the full 180 ms pool-acquisition timeout, and the checkout's two retries re-hit the same saturated pool. The user's 500 is a pool-sizing problem, not a checkout bug. I'd also check what changed at 12:02 (a deploy, a config push, a traffic step) because a failure that starts at a specific minute usually has a trigger.

Without a request id I fall back to correlating by user id and time window, then by call ordering, since the checkout line's timestamps bracket the downstream calls. That works for one request and collapses under concurrency: at 50k requests/minute a 90-second window holds 75,000 requests, and timestamp-only matching can't tell them apart.

The trade-offs are real. Propagating a request id costs a header and a log field. Full distributed tracing costs storage and CPU, and head-based sampling at 1% will drop the exact failing request you need; tail-based sampling, which keeps errors and slow traces, is the usual compromise. Keep high-cardinality values like user id and cart id out of metric labels, that's where cardinality blows up cost, and in log fields, where they're cheap.

Failure modes I'd watch for: a queue consumer that mints a new id instead of reading it from the message header, which silently breaks the chain; async work that loses context across a thread pool; log retention shorter than the time it takes a user to report; and clock skew across hosts, which is exactly why the id, not the timestamp, is the join key. I'd detect the first by tracking the fraction of requests whose trace has all expected hops and alerting when a service stops emitting the id.

Curated: · Written: · Reviewed:

Q&A-24Your service must handle ten times its traffic next quarter. Where do you start?

"Ten times traffic" needs one clarification before anything else, because it changes the answer: ten times peak, or ten times sustained? Ten times a 30-second peak is a headroom and queueing problem. Ten times sustained average is a capacity and data-model problem. I'll assume sustained 10x on both reads and writes with the same p99 SLO, and flag where that assumption would change the plan.

I start by finding the binding constraint at the current scale, not by adding instances. Adding instances multiplies whatever is already saturated: if a shared connection pool is the limit, three times the instances gives you three times the waiters, not three times the throughput.

The options, and when each is wrong

OptionRight whenWrong when
Vertical scale (bigger box)The constraint is a single-threaded step or a lock, and 10x fits in one machine's memory and CPUThe cost curve is superlinear and you'll need 10x again next year; you've kept a single point of failure
Horizontal scale (more instances)The service is stateless and the constraint is app-tier CPU or networkAny shared downstream resource is already saturated: you've moved the queue, not removed it
Read replicas and cachingRead:write ratio is high and bounded staleness is acceptableWrites dominate, or reads must be linearizable; cache invalidation becomes the real cost
ShardingOne primary can't hold the write volume or the datasetYou can't pick a key with even distribution: a hot shard is worse than one big node
Queue and backpressureWork is bursty and can be deferredThe SLO is synchronous latency; queueing just converts overload into timeouts

The deciding question between them is which resource saturates first, and that is an empirical question. A load test at current scale answers it; reasoning about it does not.

Worked example (figures illustrative, not measured)

A service at 900 rps against a 100-connection database pool. Each query takes ~110 ms, so each connection sustains ~9 queries/s and the pool ceiling is ~900/s: Little's law, 100 = 900 × 0.111. Tripling the app instances moves throughput to 920 rps, because the pool is still the constraint. Raising the pool to 400 and adding a read replica moves it to 3,400 rps, and now the app CPU is the binding constraint.

ChangeThroughputBinding constraint
baseline900/sdatabase pool
3x instances920/sdatabase pool
pool 400 + read replica3,400/sapp CPU

Failure modes and how you see them

  • Pool saturation: p99 climbs while app CPU sits near 30%. The signal is pool wait time, not CPU.
  • Cache stampede: a hot key's TTL expires and N requests hit the origin at once. You see an origin QPS spike aligned to the TTL boundary; jittered TTLs and single-flight fix it.
  • Hot shard: one partition takes 40% of writes. Per-shard write rate skew is the tell, and it means the shard key is wrong, not the shard count.
  • Retry storm: clients retry on timeout and multiply load. Request count exceeds expected even as success rate falls.
  • Autoscaler chasing the wrong metric: instances climb, throughput stays flat, because the constraint was a lock or a pool.

The test that settles it: change one thing, re-run the same load, and confirm the constraint moved to the resource you intended. If throughput doesn't move, you changed the wrong thing.

Curated: · Written: · Reviewed:

Q&A-59How do you find out what your service can actually handle?

The 90-second answer

Capacity is the request rate at which the service stops meeting its SLO (p99 crosses the target, or the error rate climbs) not the rate at which CPU looks busy. You find it by driving production-shaped traffic at production-shaped data volume, ramping until the target breaks, and recording which resource broke first.

State the assumptions out loud before you start: which SLO (p99 latency, error budget), which endpoint mix, and whether the test is allowed to touch production.

Why data volume dominates

A load test against a 5,000-row database reported 4,000 requests a second at p99 40 ms. The same service against the real 40 million rows handled 260 a second at p99 600 ms. Adding the missing index moved it to 2,100 a second at p99 90 ms.

DatasetSustained ratep99
5,000 rows4,000 per second40 ms
40 million rows260 per second600 ms
40 million with the new index2,100 per second90 ms

The mechanism: at 5,000 rows the working set sits in the buffer pool and the planner picks an index scan; at 40 million the plan flips to a scan or the index no longer fits in memory, so every request does disk I/O. The first test measured the framework, not the service.

Where to run it: the real trade-off

Staging is safe, reproducible, and lets you inject dependency latency or kill instances on purpose. Its weakness is that staging data is a drifting copy and staging traffic is synthetic, so it misses the long tail. Use it for regression and fault injection, not for the capacity number.

Production is the only place with the true mix and true data. Run it as a canary slice, or as shadow traffic mirrored off the real path with responses discarded, ramping during a low-traffic window behind a kill switch and load shedding. The risk is taking down real users; cap the injected rate and watch the same SLO dashboards you would page on.

Which shape of test

  • Ramp: steady increase to find the knee. Answers "what is the capacity".
  • Spike: step from idle to peak in seconds. Answers whether autoscaling, connection pools and queues absorb it or whether requests pile up and time out. A ramp never shows you this, because the system has time to scale.
  • Soak: hold at roughly 70–80% of the knee for hours. Answers leaks: heap growth, connection leaks, disk fill, cache growth, file descriptors. A ten-minute ramp passes a service that dies at hour six.

These answer different questions; you need all three.

Synthetic vs replayed traffic

Synthetic mixes are the better choice when you are measuring one endpoint's scaling curve, when you cannot legally replay production payloads, or when you need a reproducible baseline for a CI gate. Their weakness is correlation: real traffic carries a long tail of rare-but-expensive requests and a cache-warming order that a uniform random mix misses. Replayed traffic captures the mix but needs PII scrubbing and goes stale as the product changes. Common practice is synthetic for the ramp and the regression gate, replayed or shadowed for the final number.

Failure modes to watch

  • Coordinated omission: a closed-loop generator that waits for each response under-reports latency exactly when the system is saturated. Use an open-loop generator with a fixed arrival rate.
  • The generator is the bottleneck: check its CPU and NIC before believing the service's ceiling.
  • Warm caches: a test that starts after a warmup run measures the cache, not the cold path. Test both.
  • Per-instance arithmetic: capacity per instance times instance count is not the cluster's capacity once a shared database or connection pool saturates.

What I would report

The rate at which the SLO breaks, the binding resource, the test shape, the data volume, and the date. "2,100 rps at p99 90 ms, database-bound, 40M rows, ramp" is repeatable; "handles high load" is not. Load test the data, not just the code.

Curated: · Written: · Reviewed:

Q&A-60Your average latency is 40 milliseconds but users complain. What are you missing?

The 60-second answer

The average is the wrong statistic. A user request that fans out to N backends waits for the slowest one, so the user experiences the max of N samples, not the mean. With 10 parallel calls each with p99 = 900 ms, P(at least one slow) = 1 − 0.99^10 ≈ 9.6%: roughly 1 in 10 users eats a near-second stall while the dashboard still reads 40 ms.

Assumptions worth stating: the 40 ms is a per-call mean over a window, the fan-out is parallel, and the page cannot render until every call returns.

Why the mean hides it

The mean of the max of 10 iid samples is not the mean of one sample. The median of the max is the 93rd percentile of a single call: solve F(x)^10 = 0.5, so F(x) = 0.5^(1/10) = 0.933. Half your users are already past the p93 of any single dependency, and the mean never shows it.

What to do

  • Measure p50/p95/p99 per endpoint and per dependency. Percentiles do not compose, so also measure the composed user-request latency directly.
  • Cut fan-out where you can: batch, cache, or make calls sequential only when they are cheap.
  • Set per-call deadlines so one slow dependency cannot hold the page.
  • Hedge: fire a duplicate after a delay and take the first response.
MeasurePer callUser request (10 calls)
mean40 msdominated by the tail, not 40 ms
p99900 ms1 − 0.99^10 ≈ 9.6% slow
median of max—p93 of one call
hedged after 200 ms1e-4 slow1 − (1 − 1e-4)^10 ≈ 1 in 1000

Hedging arithmetic: a call is slow only if both attempts are slow, 0.01 × 0.01 = 1e-4 per call; across 10 calls that is 1 − (1 − 1e-4)^10 ≈ 1 in 1000 user requests.

Failure modes

  • Retries without a budget turn a slow dependency into an outage: 3 retries × 10 calls = 30 requests per user.
  • Hedging doubles load on the dependency. Only hedge when it has headroom, or you move the tail onto it.
  • Timeouts set above the p99 do nothing. Set them near the dependency's healthy p99.9.

How to detect it

Alert on the p99 of the composed user request, not the mean. A flat mean with a climbing p99 is the signature of a tail problem, and it is invisible until you plot the percentiles per dependency.

Curated: · Written: · Reviewed:

Q&A-64What should page an on-call engineer at 3 am?

What a page is for. A page wakes a human who has to act now. That is the whole test: if nobody must do anything at 3 am, it belongs on a dashboard or in a ticket. So page on symptoms users feel, or will feel before someone can respond during business hours, and keep causes off the pager.

What I would page on:

  • Error-budget burn, multi-window. A 99.9% SLO over 30 days leaves 43.2 minutes of budget. Burn rate 14.4 (2% of budget in 1 hour, confirmed by a 5-minute short window) pages. Burn rate 6 (5% in 6 hours) pages. Burn rate 1 (10% in 3 days) is a ticket.
  • Hard down. Health check failing on every instance, or error rate at 100%. Don't wait for burn rate here; the raw signal is unambiguous.
  • Latency SLO breach sustained long enough that retries and timeouts start cascading.
  • Stuck work. Queue age or replication lag that is growing, not merely high. A queue at 30 minutes and climbing pages; a queue at 30 minutes and flat does not.
  • Durability. Failed backups, broken replication, a disk that fills before morning.

What I would not page on: CPU over 80%, memory over 70%, disk over 70%, a single 5xx, a pod restart, a deploy finishing. Those are causes. A service pinned at 85% CPU with p99 at 40 ms is healthy. Paging on it teaches people to swipe the alert away, and that habit is what turns the next real page into an outage.

The failure mode, with numbers. I have seen a team take 340 pages in a month, 40 of which anyone acted on:

AlertPagesActioned
CPU over 80%2102
Disk over 70%901
Error-budget burn1212
Queue age over 30 min2825

The 210 CPU pages bought 2 actions. The 40 that mattered were buried in the rest, and one genuine outage was missed because it looked like the usual CPU noise. That is a 12% actionability ratio; I treat anything under half as a candidate for deletion.

Trade-offs. Burn-rate alerting fires slower than a raw threshold. You accept a few minutes of degradation to buy silence, which is the right trade for partial degradation and the wrong one for a total outage: hence hard-down signals page on the raw measurement. Multi-window burn rates also need the short window to clear the alert, or a recovered incident keeps paging.

How I would check it. Pull a month of pages, count how many led to an action, and delete or downgrade the rest. Every alert that survives gets a runbook naming the first three things to check and an owning team. Re-audit quarterly, because alerts rot as systems change.

Curated: · Written: · Reviewed:

Q&A-65Your service is failing in production. What do you do in the first ten minutes?

The first 60 seconds, out loud

"Two questions before I touch anything: is this a full outage or degradation, and is anything being written that we can't reconstruct? If it's degradation and no data is at risk, I roll back the most recent change. If writes are corrupting data, I stop writes first, because a rollback doesn't un-corrupt rows."

Then declare the incident, name one incident commander, and start a timeline.

The ten minutes

MinuteAction
0-1Declare. One IC, one comms channel, one timeline doc.
1-3Look at deploy markers on the error-rate graph. Did errors start within ~5 min of a deploy, flag flip, or config push?
3-5If yes: roll back or disable the flag. kubectl rollout undo deploy/api, or flip the kill switch. Don't read the diff first.
5-7If no recent change: check what fails without a deploy. Dependency status page, certificate expiry, connection-pool saturation, disk, a traffic spike.
7-10Post a status update with impact and ETA. If mitigation didn't work, escalate to the owning team and widen the blast-radius question.

Why mitigate before you understand

The bad version is serving traffic right now. Every minute spent reading logs is another minute at the same error rate. Rollback is bounded (one command, a few minutes to converge) while root-causing is unbounded; you don't know whether it's 4 minutes or 4 hours. So you buy the outage down first and spend the understanding afterwards, on a system that's up.

Hypothetical: a deploy at 14:02 starts 500s. Debug-first: 40 minutes of log reading, mitigation at 14:42, 40 minutes of failed requests. Rollback-first: rollout undo at 14:06, errors stop by 14:08. Same fix, 34 minutes of user-facing errors avoided. The root cause still gets found: from the rolled-back commit, with no time pressure.

When rollback is the wrong first move

  • Destructive migration. If the deploy already ran DROP COLUMN or backfilled bad values, rolling back the code doesn't restore the data. Stop writes (fail closed), then choose forward-fix vs restore.
  • Security incident. Rolling back the vulnerable version may be right, but if credentials leaked, rotation is the mitigation, and that isn't a rollback.
  • The change is a cache or config with a long TTL. Flipping the flag doesn't take effect until entries expire; you need to purge, or you'll conclude the flag "didn't work" and chase the wrong thing.

Failure modes to watch

  • Rollback fails because the migration isn't backward compatible: the old code can't read the new schema. Detect: pods crash-loop after rollout undo. Mitigation: forward-fix.
  • Rollback works, incident recurs on the next deploy. Detect: same error signature on the next release. That means the change was re-applied, not that rollback was wrong.
  • No recent change, so you burn 4 minutes on a rollback that does nothing. That's an acceptable cost: it's cheap and it eliminates a hypothesis.

What I'd measure afterwards

Time to mitigate and time to resolve, recorded separately. A 4-minute mitigation with a 3-hour root cause is a good incident; a 3-hour mitigation is the one to fix.

Curated: · Written: · Reviewed:

Q&A-66What makes a postmortem useful rather than a formality?

A postmortem earns its keep when it changes the system. Three things have to hold: someone decided in advance which incidents get one, the write-up explains how the system allowed the failure and how long it took to notice, and every finding leaves the document as an owned, dated ticket in the same tracker as ordinary work, with someone reviewing that backlog.

Decide the format before you write. A full postmortem is expensive (six engineers in a 90-minute review is nine engineer-hours), so gate it. Google's SRE book uses a written trigger list: any data loss, user-visible downtime past a threshold, an incident that needed a human to be paged, or resolution time above a limit. PagerDuty's incident response documentation splits by severity, requiring postmortems at its top two levels and not below. The formats, and when each applies:

SituationFormatCostWho runs it
data loss, novel failure mode, multiple teams, or a large slice of the error budget burnedwritten postmortem plus review meeting4-8 engineer-hoursneutral facilitator; responders supply the timeline
known failure mode, one team, fix already understoodone-page timeline, 30-minute callabout 1 engineer-hourteam lead
obvious cause, fix already deployed, nothing new to learnfix forward, link the ticket to the incident, no document0nobody
a stream of small repeats: flapping alerts, slow queries, alert noisebatch into a monthly ops reviewabout 2 engineer-hours a monthops lead

The deciding question is whether there is new information about the system or only about this instance. A repeat of a failure whose guardrail was never added is new information; "we already know the cause" is not the same as "we already fixed it". One flapping alert is not worth a review. Ten a month is.

Then the mechanism. Name the causal chain, not a cause. What made the wrong action look reasonable to the person taking it, and what kept the problem invisible for the next eight minutes? Dekker's Field Guide makes the point that human error is a symptom of the system that shaped the decision, so an analysis that stops at a name stops before the guardrail. Allspaw's 2012 Etsy write-up says the same thing from the other direction: blame is an information leak. A timeline that records the source of every timestamp does more than a narrative paragraph can. Hypothetical figures:

TimeEventSource
14:02deploy of checkout v412 completesdeploy log
14:03first 500s on /pay, error rate 3% and climbingapp metrics
14:11alert fires: error rate above 5% for 5 minutesPagerDuty
14:14on-call acknowledgesPagerDuty
14:29rollback to v411 completes, errors stopdeploy log
14:34incident closedincident Slack channel

Detection gap 8 minutes, time to mitigate 15, customer-impacting window 31. The review is about the 8, not the 15: rollback was fast, but the alert that started it was a fixed error-rate threshold. Would an SLO burn-rate alert have fired earlier, and what would that have cost in noise?

Then the residual. Actions go into Jira or Linear, not into the document, each with one owner and a date. Two or three that get done beat twelve that don't. Put a number on the follow-up: an illustrative bar is 90% of actions closed within 90 days, reviewed in the same meeting that looks at MTTD and MTTR, so you can see whether the detection change worked. If the same subsystem shows up in three postmortems with three closed action items, the actions are the wrong shape.

Failure modes worth naming: a document with actions but no owners is a wish list; a postmortem held for every blip turns the review into a status meeting and people stop reading it; a single "root cause" hides the other two conditions that had to line up. Skip the full review when the fix is already in production and nothing would change. Hold it when the same guardrail turns out to be missing twice.

Curated: · Written: · Reviewed:

Q&A-67How do you release a backend change without risking every user?

The 90-second answer

Assume a stateless service behind a load balancer, a shared relational database, and no protocol break between the two versions. Then a release is three separate things, and only one of them is genuinely risky. The code deploy is cheap to undo. The traffic shift is the control knob. The schema change is the part that can strand you.

So I split the change. The schema goes out first, backwards compatible, in an expand/contract sequence. The code deploys dark behind a flag. Then I shift traffic in steps, comparing the canary against a control group on the same endpoints, and I keep the previous version running and able to serve the full load until the new one has held for a full traffic cycle.

The rule that makes rollback real is this. At every point, the old code must run correctly against the new schema. If it can't, you don't have a rollback, you have a fix-forward.

Expand/contract

-- release 1 (expand): old code ignores the column
ALTER TABLE orders ADD COLUMN status_v2 text NULL;

-- backfill in batches; app dual-writes status and status_v2
-- release 2: reads switch to status_v2, writes still go to both
-- release 3 (contract): stop writing status, then drop it

Each step is independently deployable and independently revertible. The drop is the only irreversible step, and it happens weeks later, once nothing reads the old column.

Canary gates

Example figures for a service doing 50k rps:

StageShareRequestsp99 canary/control5xx canary/controlGate
shadow0% (mirrored)———responses match
canary1%~500 rps180 / 175 ms0.4% / 0.3%hold 30 min
ramp10%5k rps178 / 175 ms0.3% / 0.3%hold 1 h
full100%50k rps——previous version kept warm 24 h

The 1% stage is not decoration. At 500 rps, roughly 28,000 requests per arm is enough to separate a 0.3% error rate from 0.4% at about 95% confidence, which is under a minute of traffic. Below that, the canary is a smoke test rather than a measurement, and I would say so instead of pretending the numbers mean something.

What actually bites

Shipping the schema and the code together is the classic one. Rollback stops being possible the moment the new code writes a column the old code cannot read, and a five-minute revert becomes a multi-hour fix-forward under pressure.

The canary never executes the changed path. A 1% canary on a service where the new code is only reached by a nightly batch job proves nothing. Tag every request with the build version and confirm the canary actually served the new branch.

The canary instances differ from the fleet in instance size, cache warmth, or AZ, so you measure the instance rather than the change.

Sticky sessions and long-lived connections pin users to the old version, so the canary only ever sees synthetic traffic.

Async consumers are the quiet one. Old workers reading new messages need the message format to be additive, with unknown fields ignored and never required.

Rollback that means "git revert and redeploy" is not rollback. Shifting traffic back takes seconds; a redeploy takes a full cycle and may not be possible at all.

Canary or blue-green

Blue-green gives an atomic cutover and an atomic cut back, at the cost of running two full fleets and hitting everyone at once if the bug only appears under full load. Canary bounds the blast radius but needs enough traffic to be statistically meaningful and takes longer to reach 100%. I use canary for high-traffic services where 1% is already thousands of requests per second, and blue-green when the change is a protocol or config cutover that cannot be split.

Curated: · Written: · Reviewed:

Q&A-68How do you use feature flags in a backend service without creating a mess?

What I'd say first

I'll assume a hosted flag service with SDKs in each service, and request-scoped evaluation. The mess comes from three things: flags that outlive their purpose, flags evaluated in a dozen places, and flags with no defined behaviour when the flag service is down. Fix those and the rest is bookkeeping.

A flag separates deploy from release. That's the whole value: dark launch, ramp 1% → 10% → 100%, and a switch you can flip in seconds without a deploy. The cost is that every flag is a branch. Two flags is four code paths, and each one has to be tested and eventually deleted. A flag that's been on for a year is not a toggle any more, it's an untested code path with a runtime dependency.

The rules I'd hold to

  • One evaluation point per flag. A typed accessor (pricing.variant(user)) rather than if (flags.isEnabled("new_pricing")) in nine services. Scattered call sites are how two services end up disagreeing about the same user.
  • Every flag carries an owner and an expiry date in its definition. The expiry is a real date, and CI fails the build when it passes.
  • The default is the safe path, defined once and passed to the SDK on every call.
  • Log the variant with the request: flag key, variant, evaluation reason. Without it you cannot attribute a metric shift to a flag flip.
# Python 3.12. Defaults live in one dict; no call site can raise.
SAFE_DEFAULTS = {"new_pricing": "control", "checkout_v2": False}

def variant(key: str, user_id: str):
    try:
        return client.variation(key, {"user_id": user_id}, SAFE_DEFAULTS[key])
    except Exception:
        metrics.increment("flag.fallback", tags={"flag": key})
        return SAFE_DEFAULTS[key]

Most SDKs return the default rather than throwing, so the wrapper's real job is defining the default once and counting fallbacks. If the fallback counter moves, you know the flag service is degraded even though nothing errored.

The failure mode that bites

The failure mode that bites is the flag service being unreachable. What happens next depends entirely on how the client is configured, and the difference is not subtle. The figures below are invented to make it concrete:

Client behaviour on timeoutFlag resolves toRequests on the wrong path
returns the default passed to the calldocumented safe path0
returns the last cached valuewhatever was cached at last poll9% for 20 minutes

Worked example: a new_pricing flag shipped 14 months earlier, no owner, no expiry. The client served the last cached value on timeout. A 20-minute flag-service outage meant the processes that hadn't polled since the last flip kept serving the old pricing branch, 9% of requests, while the rest served the new one. Both branches live, one of them unmaintained for a year.

How I'd verify it

Point the client at an unreachable endpoint in a test and assert every flag key resolves to its documented default. That test catches the misconfiguration above, and it's cheap.

When not to use a flag

If the value is per-tenant and long-lived, that's configuration, not a flag: put it in config with a restart or a config service. Kill switches are the other exception: they're meant to live forever, so they get an owner and a documented default but no expiry, and they're named as kill switches so nobody deletes them.

Curated: · Written: · Reviewed:

Q&A-69How should a service receive its configuration and its secrets?

Assume a containerised service, one image promoted through dev/staging/prod, and a Kubernetes-style runtime. Non-sensitive configuration is read from the environment at start-up; secrets are fetched at runtime from a manager, never baked into the image or the repo, and the service re-reads them without a restart.

Concretely: read config from env vars, or from a config service for values that must change without a deploy; validate every required key at boot and exit non-zero on a missing or unparseable value rather than starting half-configured. Read secrets from files on a tmpfs written by a CSI driver or a sidecar agent that authenticates with the workload identity. Kubernetes rewrites a Secret volume in place; a secret injected as an env var is frozen at container start and never updated.

Where each mechanism lands:

MechanismRotation without restartWhere the value sitsWho can see itUse it when
Baked into image or repoNo: rebuild and redeployImage layers, git historyAnyone holding the artifact, permanentlyOnly immutable build constants. Never a credential
Env var, including from a SecretNo: the environment is fixed at execProcess environmentEvery process in the container, crash dumps, telemetry SDKs that capture envNon-sensitive config, or a platform where this is the only injection path
Mounted Secret volume (Kubernetes)Yes: kubelet rewrites the file, sync period a minute by default plus a cache delaytmpfs outside the imageCode that opens the pathMost secrets; the app must re-read the file
Sidecar or CSI agent (Vault Agent Injector, secrets-store CSI)Yes: the agent re-fetches on lease renewaltmpfsSame, and the agent holds the manager credentialDynamic, short-TTL credentials and per-team audit
SDK call to the manager at runtimeYesProcess memoryThe calling processServices already coupled to the manager; adds a boot dependency

Start-up validation, Python 3.11:

import os, sys, pathlib

CONFIG = {"DB_HOST": str, "DB_PORT": int, "POOL_SIZE": int}

def load_config(env=os.environ):
    cfg, missing = {}, []
    for key, cast in CONFIG.items():
        raw = env.get(key)
        if raw is None:
            missing.append(key)
        else:
            try:
                cfg[key] = cast(raw)
            except ValueError:
                sys.exit(f"config {key}={raw!r} is not a valid {cast.__name__}")
    if missing:
        sys.exit("missing config: " + ", ".join(sorted(missing)))
    return cfg

def db_password(path="/var/run/secrets/db/password"):
    return pathlib.Path(path).read_text().strip()

Trace with DB_HOST=db.internal, DB_PORT=five, POOL_SIZE=20: the loop runs in insertion order, so DB_HOST is set to "db.internal", then int("five") raises ValueError and the process exits with status 1 printing "config DB_PORT='five' is not a valid int". The missing list is never reached. With DB_PORT=5432 the loop completes and returns {"DB_HOST": "db.internal", "DB_PORT": 5432, "POOL_SIZE": 20}. Note what this does not catch: POOL_SIZE=-4 parses fine, so range checks are a second pass.

Failure modes worth naming. A secret in an env var is readable at /proc/<pid>/environ by any process in the container and travels out in a crash dump; detect it by running secret scanning over telemetry and crash artifacts, not just the repo. A secret mounted with subPath never receives updates, so rotation succeeds in the manager and the pod keeps authenticating with the old value for its whole lifetime; detect by comparing the mounted file's mtime to the rotation timestamp and alerting when the age exceeds the lease. A sidecar that fails closed turns a manager outage into a fleet-wide CrashLoop, so cache last-known-good credentials in tmpfs and decide explicitly, per secret, whether stale-but-valid beats unavailable.

The arithmetic that justifies the complexity: a static password on 90-day rotation stays valid for up to 90 days after it leaks; a dynamic credential with a 1-hour TTL and 24-hour max TTL stops working within an hour of issue. That is roughly a 2000x reduction in exposure window, and it is why the sidecar earns its keep where a plain Kubernetes Secret does not. A three-person team on a managed platform should skip Vault entirely: cloud secret manager plus the platform's native injection, 90-day rotation, and a CI secret scanner. Reach for the agent when you need per-service leases, rotation without restarts, and an audit trail an auditor will accept.

Curated: · Written: · Reviewed:

Q&A-70The business asks for a second region. What does that actually cost?

The second region costs about 1.5–2x your current infrastructure bill, plus a consistency decision you cannot defer, plus a failover you now own and have to rehearse. Compute is the easy line item. The write topology is what sets the price.

Pin three things before you quote anything: does the second region take writes or reads only; what RPO and RTO the business will actually sign up for; and which data must legally stay in-region. A read-only second region is a caching and routing problem. A writable one is a distributed-systems problem, and the two differ by an order of magnitude in effort.

The decision that sets the cost is single-writer versus multi-writer.

  • Single writer (active-passive): one region accepts writes, the other replicates asynchronously and serves reads. No write conflicts by construction. You pay for a standby and for replication egress.
  • Multi-writer (active-active): both regions accept writes. Now you need conflict resolution (last-writer-wins, CRDTs, or application-level merge) and you own the divergence.
ShapeWrite conflictsTypical RTOInfra cost vs single regionWhat drives it
Single regionnonehours (restore from backup)1.0xbaseline
Active-passive, async replicanone (one writer)1–15 min, automation-dependent~1.5–1.7xstandby at 50–100% capacity + replication egress
Active-active, multi-writerneeds resolutionseconds2x+full duplicate capacity + conflict-handling code + cross-region write traffic

The RTO and lag figures are the ones vendors publish for managed global databases: AWS documents Aurora Global Database cross-region replication lag typically under one second and RTO typically under one minute; DynamoDB global tables resolve conflicts last-writer-wins with no cross-region transactions. Treat those as the floor your own failover automation has to reach, not a guarantee.

Worked example, list prices as of early 2025, us-east-1 → us-west-2, hypothetical workload:

  • 20 × m5.xlarge at $0.192/hr = $3.84/hr ≈ $2,800/month. A full duplicate adds $2,800/month.
  • Replication egress: 500 GB/day = 15 TB/month × $0.02/GB inter-region = $300/month.
  • A second Aurora cluster roughly doubles the database bill on its own.

So a ~$3,000/month baseline becomes ~$6,000/month for active-active, before the engineering time to build conflict handling. Verify current list prices; inter-region transfer to some Asia-Pacific pairs runs higher than $0.02/GB.

Failure modes to name:

  • Replication lag spike. The replica falls behind, reads go stale, and a failover loses the tail. Detect with the replica lag metric, alert on p99 not average.
  • Split brain in active-active. A partition lets both regions accept writes; last-writer-wins silently discards one side. Detect with conflict counters and periodic divergence checks between regions.
  • Unrehearsed failover. DNS TTLs, connection pools, secrets, and the data path each fail differently. The only detection is a game day that actually moves traffic.
  • Egress creep. Chatty service-to-service calls that cross regions quietly dominate the bill. Detect with per-region cost anomaly alerts.

Rehearse a full region failover, then report the measured data-loss window and recovery time. That number, not the architecture diagram, is what the business is buying.

Curated: · Written: · Reviewed:

Q&A-71How do you know your backups are good?

A backup is good when you have restored it and measured what came back. The measurement is two numbers: how much data the restore lost (achieved RPO) and how long it took (achieved RTO). Both get compared against objectives you set in advance, and the comparison is recorded, not remembered.

So the answer I'd give: define RPO and RTO per data class, restore on a schedule into an isolated environment, and record achieved RPO/RTO alongside a data check and an application start-up check. Keys and secrets live outside the backup and are versioned, so an old backup can still be read.

The design decision is how much verification you buy. Three tiers, cheapest first:

TierWhat runsCost per runCatches
Automated restore checklatest backup into a scratch namespace; row counts, checksums, app health endpointminutes, a few dollars of computecorruption, empty or truncated backups, missing roles, key mismatch
Sampled full drillone backup, full stack, real DNS/config/secrets, timed end to endhours, roughly a day of one engineerRTO reality, dependency bring-up, runbook gaps
Full DR game dayregion failover, all servicesdays, several teamscross-service ordering, quota limits

Criteria for picking: if the data is reconstructible from upstream events and RPO is minutes, automated per-backup verification plus a quarterly drill is enough. If RPO is zero (a ledger, an audit log) you verify every backup, or you replicate synchronously and treat the backup as a second line. Sampling 1 in 30 catches a systematic break within a month; it will not catch corruption that hits a single backup.

Cost is why teams skip this. Restoring a 2 TB cluster at 200 MB/s is about 2.8 hours of copy before index rebuild; a 50 GB subset restores in roughly 10 minutes. Verifying every backup costs the full restore time per backup, so most teams verify a subset nightly and the full set monthly.

Failure modes I look for, and how each is detected:

  • Key rotation without retaining the old key. Detected by the restore job failing to decrypt; fixed by versioning keys alongside the backup.
  • Backup succeeds but is empty. Detected by asserting restored row counts against the source within a tolerance.
  • Restore works but the app won't start. Detected by hitting the health endpoint in the scratch environment, not just checking the database.
  • RTO drifts past objective. Detected by timing every drill and alerting on the trend, not the single run.

The one I've seen: nightly backups ran for 14 months, and the first restore failed because the encryption key had been rotated and the old key wasn't retained. Nine hours of data was gone.

A nightly verification job:

set -euo pipefail
pg_restore --dbname="$SCRATCH" --clean --if-exists latest.dump
src=$(psql -Atc 'select count(*) from orders' "$PROD_RO")
dst=$(psql -Atc 'select count(*) from orders' "$SCRATCH")
[ "$dst" -ge $(( src * 99 / 100 )) ] || { echo "row count $dst vs source $src"; exit 1; }
curl -fsS "$SCRATCH_URL/healthz" > /dev/null
MeasureObjectiveLast test
RPO5 min4 min
RTO1 h48 min
Last successful restoremonthly9 days ago

Curated: · Written: · Reviewed:

Q&A-72How do you find out whether your failure handling works?

Core answer

You find out by injecting the fault on purpose and comparing what the system actually did against a prediction you wrote down before the experiment. Reading the retry config, or a unit test with a mocked client, tells you the handler exists; it does not tell you it fires, in what order, or what it does to the rest of the system under load. The first unplanned exercise of a timeout path is usually during an incident, and that is the wrong time to discover it is wrong.

So the unit of work is a hypothesis, not a fault. Before touching anything, write down: the fault, the expected user-visible symptom, the expected internal state (breaker open, retry count, queue depth), and the expected recovery time. Then run it and diff.

What "works" means

A failure handler passes when four things hold:

  • The user-visible symptom stays inside the budget you set (for example p99 under 2s, error rate under 1%).
  • Internal state transitions happen in the designed order: the breaker opens after N failures, not before, not never.
  • Recovery happens without a human, and within a bounded time after the fault clears.
  • No amplification: the handler does not make the failing dependency's load worse.

That last one is where most handlers fail.

How to run it

Ladder the blast radius: staging with one dependency degraded, then production on one instance or one canary, then a small percentage of live traffic. Inject at the layer that matches the failure you care about: network latency and packet loss, dependency error responses, resource exhaustion (CPU, memory, connection pool), process kill. Keep an abort switch and a written stop condition (for example "abort if error rate exceeds 5% for 30s") that someone other than the person running the experiment can pull.

Artefacts

Experiment record:

Fault injectedLayerExpectedObservedVerdict
dependency +3s latencynetworkbreaker opens ≤10s, fallback servesopened at 9s, fallback servedpass
dependency returns 500dependencyfallback serves, no retries5 retries per request, dependency load ×6fail

Retry amplification, worked example (illustrative figures): a service handles 1,000 rps. The dependency fails for 30s. Each request retries 3 times with no backoff and no jitter, so the dependency sees 1,000 × 4 = 4,000 attempts/s while it is already unhealthy. If it can serve 500 rps degraded, the backlog grows for the whole 30s and keeps growing after the blip clears, because the retries are still arriving. The outage outlives the fault. Adding jitter and a retry budget (retries ≤ 10% of total requests) caps the multiplier and lets the dependency recover.

Failure modes of the experiment itself

  • False pass: the fault never reaches the code path. A client library retries below your instrumentation, or a cache serves the request, so the handler you meant to test never runs. Verify the fault landed by checking the handler's own metric moved.
  • False fail: you ran during a deploy, or a second unrelated dependency was already degraded.
  • Observability gap: if you cannot see breaker state and retry counts as metrics, you cannot tell a pass from a lucky one. Instrument before you inject.
  • Single-instance tests miss shared-state behaviour: a breaker is per-instance, so one instance opening does not protect a shared connection pool.

When not to do this

If the system has no fallback and no budget for degradation, the honest answer is that the failure handling is "fail the request", and the experiment should confirm it fails fast and cleanly rather than pretending to degrade. And do not run fault injection in production without an abort path and a rollback you have tested.

Curated: · Written: · Reviewed:

Q&A-73Your cloud bill doubled after a release. How do you find out why?

First 90 seconds

Assume the bill is real usage, not a credit or reservation expiring: check that first, because a support-tier change or an expiring RI looks identical on the total. Then treat the release as the prime suspect: I want a step change in the bill that lines up with the deploy timestamp, attributed to one service and one usage dimension, before I touch anything.

Three moves: diff the bill by service, narrow to a usage type, then trace that usage type to a live per-request metric I can bisect against the release.

Diff the bill

Cost Explorer, daily granularity, group by service, last 14 days. A release shows up as a step, not a ramp: organic growth ramps. If the step lands on the deploy day, keep going; if it's gradual, the release isn't the cause.

Then group by usage type inside the suspect service. "EC2" is useless; NatGateway-Bytes, DataTransfer-Out-Bytes, or TimedStorage-ByteHrs tells you the mechanism.

Worked example (hypothetical figures)

Baseline $30k/mo, now $61k/mo.

ServiceBeforeAfterDelta
CloudWatch Logs$1.2k$29.2k+$28k
EC2$18k$20k+$2k
S3$3k$4k+$1k

Group CloudWatch by usage type: DataProcessing-Bytes and TimedStorage-ByteHrs are both up ~45x. That's ingest, not retention. Now leave billing and go to the live metric: CloudWatch IncomingBytes per log group. One log group went from 40 GB/day to 1,800 GB/day on the deploy day.

Arithmetic: 1,800 GB/day × 30 = 54,000 GB/mo × $0.50/GB ingest = $27,000, plus 54,000 GB stored × $0.03/GB-mo = $1,620. That's $28,620: the whole delta. The release added a debug-level log line on a hot path.

Why billing data alone isn't enough

Cost Explorer lags 12–24 hours and is daily-granularity. The Cost and Usage Report is hourly but lands in S3 and needs Athena, still ~24h behind. If you need to bisect within the hour, billing can't do it. Use proxy metrics that move with cost: IncomingBytes for logs, NAT gateway bytes for egress, instance count for compute. Those are live.

Bisect

If the metric points at a service but not at a change, revert or flag off half the release and watch the proxy metric for one deploy cycle. Reverting is faster but riskier in prod; metric-driven narrowing is safe but needs the instrumentation to already exist. Pick based on whether the spike is still growing.

Failure modes

  • Untagged resources land in a null bucket and can't be attributed. Account-per-service or resource-level CUR fixes this.
  • Credits, RI expiry, or a support-tier change look like a usage spike. Check the credits/RI line before chasing usage.
  • A consolidated bill hides a spike in one account or region: group by linked account and region too.
  • Retention: after you fix the log level, 54 TB still sits in storage. Set a retention policy or the storage line keeps billing.

Curated: · Written: · Reviewed:

Q&A-25Design the backend for a URL shortener that serves fifty thousand redirects a second.

Assumptions. 50k redirects/s sustained, read:write about 1000:1 (50 creates/s). Roughly 1e9 links over the system's life. Redirects are latency-critical (p99 under 10 ms); click analytics can lag by minutes. One region serves the redirect path; the mapping store is replicated across zones.

Shape. Two services. The redirect service is stateless, reads from cache, and never touches the primary store on the hot path. The link service mints keys and writes mappings. Click events go to a queue.

Key generation: decided: pre-allocated counter ranges, not random-with-check.

Criteria: no collision on the hot path, no single hot key, keys not trivially enumerable, keys short.

  • Global counter per request (Redis INCR, DB sequence): collision-free, but every create pays a network round trip and the counter is one hot key. At 50/s it survives, but it couples link creation to the allocator's availability.
  • Random key + uniqueness check: the check is a read, cheap at 50/s, but collisions are the problem. 7 base62 chars is 3.5e12 keys; at 1e9 stored links the expected collision count is n²/2N ≈ 1e18 / 7e12 ≈ 1.4e5. Getting that under 1 needs about 10 characters. Longer keys, retry logic, and a read on the write path, for no benefit at this write rate.
  • Pre-allocated ranges: each link-service instance calls INCRBY for a block of 10,000 ids, then hands them out locally. One round trip per 10,000 creates. On restart, unused ids are lost: at 50/s that is a few thousand ids against a 3.5e12 space.
# Python 3.12
class KeyAllocator:
    def __init__(self, fetch_block):
        self._fetch_block = fetch_block
        self._next = 0
        self._end = 0

    def next_id(self):
        if self._next == self._end:
            start, size = self._fetch_block()
            self._next, self._end = start, start + size
        self._next += 1
        return self._next - 1

Trace with fetch_block returning (1000, 10000): call 1 sees _next == _end (0 == 0), fetches, sets _next=1000, _end=11000, increments to 1001, returns 1000. Call 2 skips the fetch, increments to 1002, returns 1001. Call 10000 returns 10999 and leaves _next == _end == 11000; call 10001 fetches the next block. The block yields exactly ids 1000–10999.

Base62-encode the id. Sequential ids are enumerable, so apply a reversible permutation, key = (id * A) mod 62^7 with A coprime to 62^7, which scatters consecutive ids without a lookup. It is not cryptographic; abuse is handled by rate limiting, not key secrecy.

Redirect path. Cache in front of the mapping store. 50k rps at a 99% hit rate leaves 500 rps of misses: one Redis node absorbs that, but shard for the working set. A cache entry is roughly a 7-byte key plus a 100-byte URL plus metadata, about 150 bytes. 1e9 links × 150 B = 150 GB, too much for one node, so cache the hot subset: traffic is Zipfian, so a few percent of links absorb most reads.

Click events. Publish to a queue (Kafka), consumers batch-insert. Clicks never write synchronously on the redirect path.

Failure modes and detection.

  • Cache stampede: a hot key is evicted, then thousands of concurrent misses hit the store for the same key. Detect: store QPS spike correlated with eviction rate. Mitigate: single-flight per key plus jittered TTLs.
  • Hot shard: one celebrity link pins one cache shard's CPU. Detect: per-shard CPU skew. Mitigate: a small in-process LRU in front of Redis.
  • Allocator outage: creates fail, redirects unaffected. Detect: allocator error rate; instances keep minting from their remaining block.
  • Click backlog: consumer lag grows unbounded. Detect: queue lag. Mitigate: bounded queue, sample or drop analytics under pressure. Never if clicks are billed.

Acceptance test. Redirect p99 at 50k rps with the cache hit rate held at 99%, and store QPS during a cold-cache restart.

Curated: · Written: · Reviewed:

Q&A-98Design the backend for a chat product with one million daily users.

Clarify, then assume. I'd ask: 1:1 only, or groups, and how large is the biggest group? Mobile or web? Read receipts and history search in scope? Then I state assumptions and move: 1M DAU, a mix of 1:1 and groups up to a few hundred members, mobile clients that lose sockets in the background, messages durable and ordered per conversation, at-least-once delivery acceptable.

The shape. Chat is fan-out plus durable delivery. Five pieces: a per-conversation sequence number for ordering, a gateway tier holding sockets with a registry mapping user → node, a broker partitioned by conversation, a per-user inbox for offline delivery, and keyset paging by sequence for history.

Arithmetic. 1M DAU × 40 messages/day = 40M/day ≈ 460/s average. Peak factor 4 → ~1,850/s. Average fan-out across 1:1 and groups ~4 → ~7,400 delivery writes/s at peak. At ~200 bytes/message that's 8 GB/day, ~3 TB/year before replication. Concurrent sockets: ~25% of DAU → 250k; at 50k sockets per gateway node that's ~6 nodes plus headroom.

Decisions, and the alternative I rejected.

ConcernDecisionAlternative rejectedDeciding criterion
orderingsingle sequencer per conversation, shard by conversation IDclient timestamps / vector clocksneed a total order per conversation; conversations are independent, so sharding is free
deliverypublish once to a per-conversation topic; gateways fan out to connected membersper-user topic (N copies)group-size distribution: per-user fan-out is N× write amplification
offlinedurable per-user inbox rows, pulled on connectpush-onlymobile OS kills background sockets; push can be dropped
semanticsat-least-once + client dedup by message IDexactly-onceexactly-once needs coordination; dedup is a recent-ID set
read receiptslast-read sequence watermark per user per conversationper-message receipt rowswatermark is O(1) per user; rows are O(messages × members)

Failure modes. Registry saturation from scanning members: the fix is routing by conversation, not by member lookup. Gateway node loss drops 50k sockets at once; clients must reconnect with jittered backoff or you get a thundering herd. Broker ordering breaks if a conversation's messages land on different partitions: partition by conversation ID. A very large broadcast channel exceeds one partition's throughput; that path needs its own fan-out, not the group-chat path.

Validate. Load test a 400-member group at target rate; measure p99 delivery latency and registry QPS. Kill a gateway node and measure the reconnect storm.

Summary of decisions: sequence per conversation for order, per-conversation topic for fan-out, per-user inbox for offline, at-least-once with client dedup, watermark read receipts.

Curated: · Written: · Reviewed:

Q&A-99Design the backend for a timeline that shows posts from accounts a user follows.

Assume a read-heavy workload: about 100 timeline reads per post write, a p99 read target of 200 ms, and eventual consistency of a few seconds is acceptable. Follows are mostly ordinary accounts, with a heavy tail of a few accounts that have millions of followers. Posts are immutable and ordered by (created_at, post_id).

The answer is hybrid fan-out: precompute timelines on write for accounts below a follower threshold, and merge in posts from above-threshold accounts at read time.

Write path. On post create, enqueue a fan-out job. A worker pages the follower list and inserts the post id into each follower's timeline: a Redis sorted set keyed by user id, score = created_at_ms * 1000 + post_id. After insert, trim to the cap with ZREMRANGEBYRANK key 0 -801 to keep the newest 800 entries. The cap is sized to the read window: 800 entries covers roughly two weeks for a typical follow set.

Read path. Fetch one page from the precomputed timeline (ZREVRANGEBYSCORE with the cursor), fetch one page of posts from each above-threshold followed account since the cursor, merge by (created_at, post_id), return the page and a new cursor. The cursor is opaque (base64 of (created_at, post_id)) so there is no offset drift when new posts arrive.

def merge(precomputed, celebrity, limit):
    # both lists sorted desc by (ts, pid)
    i = j = 0
    out = []
    while len(out) < limit and (i < len(precomputed) or j < len(celebrity)):
        if j >= len(celebrity) or (i < len(precomputed) and precomputed[i] >= celebrity[j]):
            out.append(precomputed[i]); i += 1
        else:
            out.append(celebrity[j]); j += 1
    return out

Trace with precomputed = [(100, 5), (90, 9)], celebrity = [(95, 1), (80, 2)], limit = 3:

  • i=0, j=0: (100,5) >= (95,1) → append (100,5), i=1.
  • i=1, j=0: (90,9) >= (95,1) is false → append (95,1), j=1.
  • i=1, j=1: (90,9) >= (80,2) → append (90,9), i=2. len(out)=3, stop. Result: [(100,5), (95,1), (90,9)].

Alternatives. Pure fan-out on write gives one-query reads but write amplification equal to follower count. Pure fan-out on read gives cheap writes but read cost equal to the number of followed accounts: a user following 2,000 accounts needs 2,000 queries per page. Hybrid bounds both: writes are capped by the threshold, reads are two queries merged.

FollowersStrategyWrites per postRead cost
500fan-out on write5001 query
4,000,000fan-out on read1merge at read
mixed feedhybridbounded2 queries merged

Failure modes. Worked example (hypothetical figures): an account with 4M followers under pure fan-out on write produces 4M inserts per post. At 50k inserts/sec across the worker fleet, that is 80 seconds of insert work for one post. Three such posts in a burst put 240 seconds of work ahead of every other fan-out job on a shared queue, so ordinary users' feeds go stale for minutes. That is the tail that breaks the single model. Detect it with queue depth and p99 post-to-visible latency; mitigate by raising the threshold so the largest accounts move to read-time. A hot celebrity read is handled by caching that account's recent posts (last 200) in Redis, so the read path hits cache, not the database. A timeline cap that is too small drops older posts for users who follow many low-volume accounts; detect via missing-post reports and size the cap to the read window.

Pick the threshold so fan-out write volume per post stays under the worker fleet's sustained insert rate with headroom: 10k followers is a reasonable start. Measure queue depth and post-to-visible latency for both an ordinary and a large account, and move the threshold when either degrades.

Curated: · Written: · Reviewed:

Q&A-80Which events should a backend service record for audit, and how?

The 90-second answer. Audit logging is a forensics and compliance system, not a debugging aid, so the real question is what has to survive a subpoena and a compromised insider. Assume a service with a retention obligation (SOC 2, PCI, HIPAA-adjacent) and a 7-year window. Record four classes of event: authentication, authorization denials, privileged/administrative actions, and mutations of sensitive records. Every record carries actor, action, target, outcome, and time. Write them append-only, in the same transaction as the change, to storage the application cannot rewrite.

What to record

ClassExamplesWhy it is audited
Authenticationlogin success/failure, MFA enrol or reset, password change, session/token issuanceaccount-takeover forensics
Authorization denial403 from the policy engine, cross-tenant access attemptprobing and misconfiguration
Privileged actionrole grant, impersonation, config or feature-flag change, key rotation, bulk exportinsider risk
Sensitive record mutationfield-level change to PII, payment, health, or permission data"who changed this, and to what"

Record the success as well as the failure. A support tool that logged only failures left no trace when 40 accounts were altered with a compromised staff credential.

What each record carries

event_id (monotonic per stream), timestamp (UTC, RFC 3339), actor_id, actor_type (user or service), credential (session or token id), action, target_type, target_id, outcome, before/after for changed fields, request_id, source_ip, service name and version.

How: append-only plus a hash chain

Each record stores prev_hash, and record_hash = SHA-256(prev_hash || canonical_json(record)). Verification recomputes the chain and reports the first index that fails. The limitation matters: anyone who can write the store can recompute the whole chain, so anchor the head hash somewhere the writer cannot reach. A daily head hash published to a separate account, an S3 Object Lock bucket in compliance mode, or a signature from a KMS key whose policy excludes the service role. Without an anchor the chain proves nothing against an attacker with write access.

Failure modes and how they show up

  • Async fire-and-forget audit writes. The DB commit succeeds, the process dies, the record never lands. Write the audit row in the same transaction as the change (outbox pattern) and alert on the audit-write failure counter.
  • Log injection. A user-controlled string containing newlines or JSON forges entries. Emit structured JSON, escape values, never concatenate.
  • PII in the body. Seven-year retention turns the audit log into a second PII store that erasure requests now hit. Store IDs and field names, not values; where values are required, encrypt per subject and crypto-shred the key on erasure.
  • Clock skew. Order by monotonic sequence number, not timestamp; alert on gaps in the sequence.
  • Sampling. Never sample audit events. Sampling belongs to debug logs.

Cost, so "we can't afford it" is checkable

5M events/day at ~400 bytes of JSON is ~2 GB/day, ~730 GB/year. On S3 Standard at $0.023/GB-month that is roughly $17/month, about $200/year (illustrative figures).

What I would verify before calling it done

Reconstruct one privileged change end to end from the log alone (actor, credential, before/after, timestamp) then delete a record and confirm verification fails at that index.

Curated: · Written: · Reviewed:

Q&A-81A dependency you use has a critical advisory. What is your process?

First hour. Confirm the advisory applies before you wake anyone. Check the affected version range against your lockfile, not your manifest: a transitive pin is the usual reason "we don't use that" turns out to be wrong. Then answer two questions: which services and releases contain the version, and is the vulnerable code path reachable from anything you run.

Reachability is the deciding factor, and it's a mechanism, not a vibe. Three ways to establish it, cheapest first:

  1. Scanner reachability. govulncheck builds a call graph from your entry points and reports only vulnerabilities on a path you actually reach. For Java, Snyk's reachability analysis does the same over bytecode. If the tool says "no call path," that's evidence, not proof: it misses reflection, dynamic dispatch, and plugin loading.
  2. Read the advisory's patch. The fix commit tells you which function is vulnerable and under what input. If your code never calls it, or calls it with a constant that can't hit the bad branch, you've narrowed it.
  3. Grep the call sites. For a small surface this is faster than trusting a scanner.

Illustrative govulncheck output:

Vulnerability #1: GO-2023-XXXX
  Found in: github.com/foo/bar@v1.4.0
  Fixed in: github.com/foo/bar@v1.4.2
  Example traces found:
    #1: internal/handler/upload.go:88:19: bar.Decode

That trace names the file and line, so you know the path is live.

Then decide. Four options, in order of preference:

OptionWhenCost
Upgrade to fixed versionFix is a patch/minor bump, tests passHours
Patch/vendor the fixNo fixed release, or upgrade blocked by a breaking changeDays, you own the diff
MitigateCan't change the dep: WAF rule, disable the feature, network policyHours, partial
Accept with a written rationaleNot reachable, or reachable only by an authenticated adminZero, but dated and re-reviewed

Timeline. Critical and reachable: patch and deploy within 24 hours, or mitigate within 24 and patch within 7 days. Critical, not reachable: 7 days. Moderate reachable: 30 days. The clock starts when the advisory is confirmed applicable, not when the scanner first fired.

Failure modes. The one that bites: nobody can say which of N services use the library, so the advisory sits. That's an SBOM problem: CycloneDX or SPDX per release, queryable, so "which releases ship bar@1.4.0" is a query, not an archaeology project. Second: you upgrade and the transitive pin doesn't move because another dependency holds it back. Check npm ls bar / mvn dependency:tree after the bump. Third: you accept a "not reachable" verdict from a scanner that can't see reflection, and it's wrong.

Curated: · Written: · Reviewed:

Q&A-82How would you divide tests for a backend service?

I'd place each test at the lowest layer that can actually reproduce the failure I'm protecting against, and move up only where the layer below is blind to it. The pyramid shape is what that rule usually produces; it is not the rule.

Assumptions: an HTTP backend with Postgres and a message broker as its stateful dependencies, CI on every push, a merge gate target under 10 minutes, and a shared staging environment too noisy to assert against.

An illustrative budget for a service of roughly 900 tests:

LayerTestsWall clockAllowed to catch
unit, no I/O70020 spricing rules, backoff math, state machines, parsers
integration, real Postgres and broker in containers180about 75 s over 4 shardsconstraints, isolation, SQL, redelivery, idempotency
consumer-driven contract3040 sfield and schema drift with callers
end to end82.5 min over 2 shardswiring and configuration on one journey per critical path

Placement rules, and the reason each test earns its layer:

  • Pure logic goes in unit tests, because 700 of them run in 20 s: fast enough to run on every save. Nothing that depends on a dependency's behaviour belongs here.
  • Anything whose correctness is defined by Postgres goes in integration against a real Postgres in a container: unique and partial indexes, isolation levels, ON CONFLICT, deadlock retries, JSONB operators, timestamp precision. A fake repository cannot reject a duplicate key, so a test built on one proves nothing about the constraint.
  • Broker semantics go against a real broker: at-least-once redelivery, per-partition ordering, poison-message handling, idempotency keys surviving a replay.
  • Query counts get asserted in integration: wrap the session and assert statements per request (3 for GET /orders, not 3 per row). That catches N+1 regressions a fake repository hides.
  • Consumer-driven contract tests between this service and its callers turn a renamed field into a build failure instead of a caller's outage.
  • 5-10 end-to-end journeys, one per critical path, cover deployment wiring and nothing else.

More integration tests are justified when the bug class lives in the dependency: money movement, multi-tenant row-level security, schema migrations, retry and idempotency paths, and any area where last quarter's incident traced to dependency behaviour rather than logic. They are not justified for pure computation or formatting: that only buys container startup time.

The cost is what decides the sharding. Four minutes of integration plus five minutes of end-to-end is 9 minutes per push run serially. At 40 pushes a day that is about 6 runner-hours, and every developer waits 9 minutes for a verdict. Split integration across 4 shards with one container per shard: 45 tests at roughly 1.3 s each plus 15 s of container start lands near 75 s wall clock. Two E2E shards cut 5 minutes to 2.5. That is the price of keeping the mock-free layer affordable, and it is worth paying.

Failure modes and how I detect them: integration suites rot through shared fixtures and cross-test state, so each test gets its own schema or database and I treat order dependence as a bug; end-to-end tests flake on timing, so CI retries twice and anything that passes only on retry is quarantined and fixed or deleted within a week, with flake rate tracked per test; a suite that mocks the repository layer will pass a change that violates a unique constraint, and the first real INSERT fails minutes after deploy. To keep the integration layer honest, fail the build if its datasource is an in-memory or mock driver, and track p95 CI duration as a first-class metric.

What I would not do: assign a target percentage of tests per layer, or delete unit tests to look integration-heavy.

Curated: · Written: · Reviewed:

Q&A-83When should a test use a fake, and when a real dependency in a container?

The rule

Use a real dependency in a container when the test's subject is that dependency's semantics: constraints, transaction isolation, wire protocol, ordering, serialization. Use a fake when the dependency is not the subject: a third-party service you cannot run, or an error path you cannot trigger on demand. Default for your own datastores and brokers: real container. Default for external vendors: fake, pinned by a contract test.

Why the fake is the risky default

A fake is your model of the dependency, written down. It is fast (microseconds, no I/O) and deterministic, and it is wrong exactly where your model is wrong, and it stays wrong silently, because nothing in the suite compares the model to the thing.

Two concrete cases:

  • SQLite standing in for Postgres. SQLite disables foreign-key enforcement by default; you must issue PRAGMA foreign_keys = ON on every connection. A test that inserts an order with a missing customer passes on SQLite and fails on Postgres with a FK violation. SQLite also allows only one writer at a time, so a concurrency test that passes there tells you nothing about Postgres row-lock behaviour.
  • An in-memory broker standing in for Kafka. Kafka orders messages within a partition only. A fake that appends to a list preserves global order, so a consumer relying on cross-partition ordering passes in CI and breaks in production once the topic has more than one partition. That is the failure I have actually seen: it shipped and surfaced three weeks later under production partitioning.

What the real container costs

Startup, not correctness. On a 4-vCPU CI runner with images pre-pulled, postgres:16-alpine accepts connections in roughly 1–2s; a single-node Kafka in KRaft mode takes roughly 5–8s. The arithmetic that matters is per-class versus per-JVM:

  • 60 integration test classes × (1.5s Postgres + 6s Kafka) = 450s of pure startup.
  • One static container reused across the JVM = 7.5s once, plus test time.

So the cost is a fixed ~7.5s, not a per-test tax, which is why "containers are too slow" is usually a missing static field, not a property of containers.

The shape of the test

// Java 17, JUnit 5, Testcontainers 1.20.x, WireMock 3.x
@Testcontainers
class CheckoutTest {

  @Container
  static final PostgreSQLContainer<?> PG =
      new PostgreSQLContainer<>("postgres:16-alpine");   // real: constraints, tx

  @RegisterExtension
  static WireMockExtension payments = WireMockExtension.newInstance()
      .options(wireMockConfig().dynamicPort())          // fake: vendor, error paths
      .build();

  @Test
  void declinesWhenGatewayReturns402() {
    payments.stubFor(post("/charges").willReturn(aResponse().withStatus(402)));
    // ... assert the order is left in PAYMENT_FAILED and no stock is reserved
  }
}

static on the container is the whole trick: one start per class, not per method.

Keeping the fake honest

A fake is only safe if something fails when it drifts. Two mechanisms:

  1. Contract tests (Pact, or a recorded cassette replayed against the fake) run in CI and fail when the real service's response shape changes.
  2. Run the same integration suite twice (once against the fake, once against the real dependency) and gate the merge on the real run. If the fake run passes and the real run fails, the fake is the bug.

Failure modes and how you see them

SymptomCauseDetection
Passes in CI, fails in prodfake encodes wrong semantics (ordering, FK, isolation)run the suite against the real dependency
Flaky container testsport collisions, image pull, no readiness waitpin the image digest, wait on a readiness probe, never sleep
Fake still passes after a vendor changeno contract testPact or a recorded cassette in CI

Don't containerize what you cannot run: Stripe, Twilio, most SaaS. Use the vendor sandbox where one exists, a fake otherwise, and a contract test either way.

Curated: · Written: · Reviewed:

Q&A-84How do you keep integration tests reliable as the schema changes?

Assume Postgres, a migration tool like Flyway or Alembic, and CI that can create a database per job. Each test builds the data it needs through the same code path production uses, and each test runs inside a transaction that rolls back. Migrations run once before the suite, so the schema in CI is always the schema migrations produce.

A shared dump rots because it is a snapshot of a schema at one moment. Every migration after that moment is a diff the dump does not have. A 4 GB dump that is 11 migrations behind fails in ways unrelated to the change under test: in one case 60 of 300 integration tests failed for reasons no one had touched, and the team stopped reading failures. Factories that call the real creation path cannot drift that way. If a column becomes NOT NULL, the factory fails at the same point production would.

The isolation options differ in cost and in what they break.

ApproachIsolationCost per testBreaks when
transaction rollbackstrong~mscode under test commits or opens its own connection
fresh schema per teststrongestsecondsDDL-heavy suites get slow
truncate + reseedmedium~100 msforeign keys and sequences need care
shared dumpnone0schema moves

Transaction rollback is the default because it is fast and covers most cases. It fails when the code under test manages its own transaction: a service that calls commit() on a separate connection is not rolled back. Detect that by asserting the row count of the touched tables after the test; a non-zero count means the test leaked. For those, use a fresh schema or an explicit cleanup that runs on the same connection.

Factories drift in a different way. If they bypass validation, they produce rows production would reject. Guard with one test per factory that runs the created object through the same validation the API uses.

Keep one small realistic dataset for performance tests only. Those need volume and stable statistics, and they should be marked so they do not run in the unit loop.

To check the suite is reliable, run it twice in shuffled order and compare results. Identical pass/fail sets without a manual reset mean no test depends on another's leftovers.

Curated: · Written: · Reviewed:

Q&A-85What does a good deployment pipeline for a backend service look like?

The 90-second answer

Assume a backend service with a database, a team of 5–15, one cloud region, and a requirement to roll back in minutes. A good pipeline has four properties:

  1. One immutable artifact, built once, promoted by digest. Never rebuilt per environment: a rebuild is a different binary from the one you tested.
  2. Cheap checks first. Lint and unit tests fail in under five minutes; integration tests and deploys come after.
  3. Deploy and release are separate. The artifact ships dark; traffic moves by flag or canary.
  4. Rollback is redeploying the previous digest, not a rebuild. That constraint is what forces migrations to be backwards compatible.

Which CI vendor, monorepo or not, Kubernetes or not: all detail.

Stages and gates

StageTypical wall timeGate
lint, typecheck30–60 sblocks merge
unit tests1–3 minblocks merge
build image, push by digest2–4 minblocks merge
integration tests against real Postgres/Redis containers3–8 minblocks merge
deploy to staging, smoke2–5 minblocks promotion
canary at 5% for 10 min10 minauto-rollback on error-rate or latency burn
full rollout~5 min—

Merge to full traffic lands around 25–35 minutes. Past roughly 45 minutes people stop watching the deploy, and that is when they start batching.

Migrations are where rollback dies

Trace renaming users.email to users.email_address:

  1. Release N: add email_address, nullable. Code writes both, reads email when email_address is null.
  2. Backfill in batches (10k rows per statement, off-peak).
  3. Release N+1: read email_address, still write both.
  4. Release N+2: stop writing email.
  5. Later, drop email.

Every step leaves the previous artifact runnable against the current schema, so rollback is a redeploy. A single ALTER TABLE ... RENAME COLUMN shipped alongside the code change makes rollback impossible without a second migration: that is the failure mode, and it is the one that turns a two-minute rollback into a three-hour incident.

Failure modes and how to detect them

  • Flaky tests. If the suite fails 1 run in 20, engineers re-run until green and the gate is dead. Track per-test failure rate over 30 days and quarantine anything above ~2%.
  • Mutable tags. :latest means the artifact you tested is not the one you deployed. Assert deployed digest == tested digest on every deploy.
  • Config drift. Env vars edited by hand in prod. Diff deployed config against the repo's declared config each deploy.
  • Canary with no signal. A canary that only checks HTTP 200 passes while p99 doubles. Gate on the SLO you care about: error rate, p99, saturation.
  • Batching. Detect via deployment frequency. Weekly deploys usually mean the pipeline is slow, not that the team is undisciplined.

When not to do this

Blue/green beats canary when you need instant cutover and can afford double capacity; it is worse for stateful services. Feature flags beat progressive delivery when the risk sits in a code path rather than the infrastructure, but flags accumulate, so expire them. A batch job with no traffic to shift cannot be canaried; gate it on output correctness against a golden dataset instead.

Illustrative arithmetic (hypothetical, not measured): at a 1% per-change defect rate, a 60-change release contains a defect with probability 1 − 0.99^60 ≈ 45%; a 2-change release, 1 − 0.99^2 ≈ 2%. Bisecting 60 changes takes about log2(60) ≈ 6 steps. Small releases do not just feel safer; they cut the search space.

Evidence to collect

Deployment frequency, change failure rate, and time to restore across the last 20 releases. If time-to-restore is dominated by rebuilds or by a migration that cannot be reversed, artifact promotion is the thing to fix.

Curated: · Written: · Reviewed:

Q&A-86What do you look for when reviewing a pull request that touches a live service?

The 90-second answer

A PR against a live service is judged by what happens while it is half-deployed, and by how we get back. Correct code on the happy path is table stakes; the review is about the window where old and new versions serve traffic side by side, and about the exit.

Three things before I read the diff closely:

  1. What does the mixed-version window look like? During a rolling deploy, old and new pods run at once. Any schema change must be readable and writable by both. That means expand/contract: add the new column nullable or with a default, deploy code that writes both, backfill, drop the old column in a later release. A rename or a drop in the same PR breaks the old version still running.
  2. What is the rollback? Redeploying the previous image is the only rollback that takes seconds. If the migration dropped a column or rewrote data, rollback is a restore: minutes to hours, and it loses writes since the backup. That asymmetry decides whether the migration ships in the same PR as the code.
  3. What tells us it is broken, and how fast? A new endpoint or write path needs a metric that moves when it fails (error rate sliced by version, p99 latency, queue depth) and a way to turn it off without a deploy.

Where the lock bites

The classic live-service failure is a migration that takes ACCESS EXCLUSIVE on a hot table. In PostgreSQL 11+, ADD COLUMN ... DEFAULT <non-volatile> is metadata-only and holds the lock only briefly. But ADD COLUMN ... DEFAULT <volatile> rewrites the table, and SET NOT NULL on an existing column scans it: both hold ACCESS EXCLUSIVE for the duration. On a 30M-row table that is minutes of blocked writes. The safe pattern (PG 12+): ADD CONSTRAINT ... CHECK (col IS NOT NULL) NOT VALID, then VALIDATE CONSTRAINT (takes SHARE UPDATE EXCLUSIVE, does not block reads or writes), then SET NOT NULL, which skips the scan because the validated constraint already proves it.

Trade-offs and when to pick

ChoiceBuys youCostsDecide by
Feature flag vs. direct deployOff in seconds, no redeployTwo code paths to test, flag debtFlag when the change alters a write path or user-visible behavior and rollback would need a migration reversal
Expand/contract vs. one migrationNo mixed-version breakageExtra deploy cycle, nullable column lingersAny change to a column live code reads or writes
Canary at 1% vs. full rolloutCatches what staging cannotNeeds per-version metricsOnly if you can slice error rate and latency by version; otherwise a canary tells you nothing

What I ask for as evidence

  • The query plan for any new query, run against production-sized data (EXPLAIN (ANALYZE, BUFFERS)), not a dev table.
  • The migration's lock mode and duration, from a dry run on a restored copy.
  • The metric that will page, and the flag or config that turns the change off.

Failure mode I have seen

A PR added a NOT NULL column with a volatile default to a 30M-row table and shipped it with the code. The migration held ACCESS EXCLUSIVE for roughly 4 minutes during peak; every write to that table queued behind it, and the app's 5s statement timeout turned the queue into a burst of 500s. The fix was not the column: it was that the migration and the code were in one PR with no flag, so there was no way to stop it short of a restore.

Curated: · Written: · Reviewed:

Q&A-87Errors are spiking in production. How do you work out what is happening?

First three questions: is this user-facing, is the rate still climbing, and did anything ship in the last hour. Those decide whether I mitigate or investigate first.

If a deploy, config change, or flag flip landed inside the error window and it is reversible, I roll back before I dig. The rollback is the mitigation and the test of the hypothesis at the same time, if errors stop, I have my cause and a preserved old version to diff against. If nothing changed, or the change is not reversible (a schema migration, a backfill that already wrote rows), I diagnose in place and mitigate by other means: shed load, disable the feature flag, or raise the pool ceiling.

Why error classes beat counts. A count tells you severity; a class tells you cause. "Errors up 8x" could be one dependency timing out or a hundred unrelated 500s. Grouping by class, endpoint, and status collapses that into a handful of buckets, and usually one bucket dominates. That bucket names the code path or dependency to look at next.

Why one trace localises the fault. Aggregates cannot attribute time to a hop; a single failing trace can. Here is one, 9.2 s end to end:

SpanStartDuration
gateway0 ms9.2 s
checkout-service20 ms9.18 s
inventory-client call40 ms9.16 s
pool wait40 ms8.86 s
query execution8.9 s300 ms

The query is fine: 300 ms. The 8.86 s is waiting for a connection. That points at pool saturation, not a slow query, and it is the difference between fixing the right thing and adding an index nobody needed.

The arithmetic confirms it. Pool of 20 connections, 300 ms per query, so the pool can serve 20 / 0.3 ≈ 66 queries/s. Traffic is 90 queries/s. Demand needs 90 × 0.3 = 27 concurrent queries; the pool caps at 20, so the queue grows without bound and wait climbs until the client times out. Raising the pool to 30 buys headroom; it does not fix why traffic doubled.

Failure modes and how I catch them:

  • Restarting instances or clearing the queue on a hunch. It clears the symptom and destroys the evidence: logs, in-flight traces, the saturated pool state. Detection: the same incident returns within days with no new information. I have watched this happen.
  • Chasing the loudest signal. The error count is loud; the class is informative. Detection: your hypothesis does not name a specific endpoint or dependency.
  • Fixing the symptom. Raising the pool ceiling or adding retries masks a query regression until traffic rises again. Detection: the fix works, then the same class reappears at the next traffic peak.

I keep one hypothesis live at a time, each with a signal that would confirm it and one that would refute it, and I write both down before running the test.

Curated: · Written: · Reviewed:

Q&A-88A service is slow but no single query looks bad. How do you find the cost?

"No single query looks bad" is the part of the question that decides the approach, so I'd start by pinning down what "slow" means (p50 or p99, and whether the service is CPU-bound or waiting) then treat the cost as aggregate rather than per-call. Per-query latency can be fine while the request is slow because of fan-out (N+1), serialization, allocation and GC, lock or connection-pool contention, or time spent waiting on something outside the process. The job is to attribute wall-clock time per request and split it two ways: on-CPU versus off-CPU, and in-process versus downstream.

Three tools answer different parts of that, and reaching for the wrong one is the usual reason a week of investigation finds nothing.

ApproachWhat it answersCostBlind spot
Sampling profiler: py-spy, async-profiler, Go pprof, JFRwhich functions and lines burn CPUlow single-digit % CPU at 100 Hzoff-CPU waits; which request or tenant
Distributed tracing: OpenTelemetry spans into Jaeger or Tempoper-request causality, fan-out counts, cross-service splitspan export and storage; head sampling misses the tailcode inside a span; uninstrumented libraries
Metrics: latency histograms, pool and GC gaugesthat it is slow, and when it changednegligiblewhy

Run the profiler under representative load, not against an empty database on a laptop. A sampling profiler at 100 Hz gives line-level attribution for a couple of percent of CPU; a trace gives the fan-out count and the cross-service split. You usually need both, because the profiler says what the code is doing and the trace says which request and which dependency.

Concretely, with illustrative numbers: a checkout request issues 180 queries, each 1.2 ms at p50 and 3 ms at p99, so every query sits comfortably inside its SLO and no per-query dashboard fires. 180 × 1.2 ms = 216 ms of a 340 ms request. Serializing the 2.4 MB response adds about 40 ms. About two thirds of the request is database round-trips that are individually invisible. Batching those 180 queries into 6 drops the database portion to roughly 7 ms: the fix is a fan-out change, not a query tune.

The failure modes I'd watch for, and how each is detected:

  • The CPU profile is flat. That means the time is off-CPU: waiting on I/O, a lock, or a pool. Detect it by comparing span duration against on-CPU samples, if wall-clock is 300 ms and CPU is 20 ms, stop profiling CPU. Switch to off-CPU tooling: offcputime from bcc, JFR's thread-park and socket-read events, or Go's block profile after runtime.SetBlockProfileRate(1).
  • Head-based trace sampling at 1% keeps whatever arrives first, which is biased toward fast requests. Use tail-based sampling so the slow traces are the ones retained.
  • A 100 Hz sampler cannot resolve a 2 ms call inside one request. It only surfaces if that call is a large share of total CPU across many samples. If the cost is per-request fan-out rather than CPU, count spans instead.
  • Profiling is the wrong tool when the bottleneck is a shared resource outside your process: a saturated connection pool, a noisy-neighbour node, a downstream at its own limit. Your profile shows idle threads. Detect it with pool wait-time metrics, or an off-CPU profile where every thread is parked on the same futex.

What I'd call done: the largest contributor is named with a number, and changing it moves request latency by roughly the predicted amount. If the profile and the traces are both flat, the cost is outside the process, and the next step is the dependency's own metrics rather than a deeper profile of yours.

Curated: · Written: · Reviewed:

Q&A-89A service's memory climbs until it is restarted. How do you investigate?

First, name the assumption: this is a managed runtime (JVM, Go, .NET) where a collector reclaims unreachable objects. If it's native code or a runtime with manual memory, the approach changes.

The first question is whether the heap is growing or only RSS. Those have different causes and different tools.

Establish the signal. Sample three numbers over time: RSS, heap used, heap committed. Then force a full GC (jcmd <pid> GC.run on the JVM) and re-read heap used. If heap used drops back to a low floor, you don't have a leak: you have allocation pressure or a collector that isn't running often enough. If it stays high after a full GC, something is reachable that shouldn't be.

If heap used climbs and survives a full GC, it's retention. Take heap dumps hours apart and diff the retained sets, not the shallow histogram. The dominator tree tells you which object holds the most retained bytes; the path to GC roots tells you why it's reachable. Common retention shapes: unbounded caches keyed by tenant/user/URL, static collections, listeners registered and never removed, ThreadLocal values in pooled threads, classloader leaks on hot redeploy.

If heap is flat but RSS climbs, it's off-heap: direct ByteBuffers, mmap'd files, thread stacks, native libraries, or allocator fragmentation (glibc arenas). Heap dumps won't show it; use pmap/smaps or a native profiler.

SignalLikely causeNext probe
RSS up, heap flatoff-heap / native / fragmentationpmap, native profiler
Heap up, drops after full GCallocation pressureallocation profiler, GC logs
Heap up, survives full GCretentionheap dump diff, dominator tree
Heap up, GC overhead highcollector can't keep upGC logs, then tune

Trade-offs: restart-and-monitor is a legitimate first move when you need to stop the bleeding and gather data. Restart on a schedule, watch the slope. Collector tuning is right only when the evidence says allocation rate or GC overhead, not retention; tuning a retention bug just moves the OOM later. And sometimes it isn't a leak at all: a bounded cache that grows to its limit and plateaus is working as designed.

Worked example (illustrative):

// Java 17
static final Map<String, byte[]> CACHE = new HashMap<>();

byte[] handle(String tenant) {
    return CACHE.computeIfAbsent(tenant, t -> new byte[1_000_000]);
}

Trace: CACHE is static, so it's a GC root. Each distinct tenant inserts a 1 MB array that is never evicted. After 6,000 distinct tenants the map retains ~6 GB, and no full GC can reclaim it, because every entry is reachable from the root. A heap dump diff shows HashMap$Node[] growing linearly with distinct tenants: that's the signature.

Failure modes: taking a heap dump on a near-OOM process can itself OOM or stall it. Dump to a volume with headroom, or use a live-sampling profiler. Diffing snapshots without forcing a full GC first produces false positives from garbage that just hasn't been collected. And a leak that only appears under load won't show in a quiet snapshot.

Curated: · Written: · Reviewed:

Q&A-90How do you find and prevent a race condition in a service?

Start by naming the invariant and the state that protects it, because that decision picks the mechanism for you. If the invariant is "at most one confirmed booking per seat" and the service runs on several instances, the ordering has to come from the datastore; a mutex inside one process says nothing about the other nine.

I'll assume a multi-instance service over a shared relational database, and an invariant of at-most-once.

What the race actually is

A race is an interleaving that breaks an invariant even though every single-threaded run of the code is correct. Almost all of them share one shape: a read whose result decides a later write, with something in between that yields the thread or the event loop - a network call, an await, another query. So I look for check-then-act: read "is the seat free", then book it; read "does this user exist", then insert; increment a counter held in process memory; read a cache entry, mutate it, write it back.

The interleaving for two requests hitting seat 7A:

T1: SELECT booked FROM seats WHERE id = '7A'   -> false
T2: SELECT booked FROM seats WHERE id = '7A'   -> false
T1: INSERT INTO bookings (seat_id, user_id) VALUES ('7A', 'u1')
T2: INSERT INTO bookings (seat_id, user_id) VALUES ('7A', 'u2')

Both requests saw a free seat and both committed. Each statement is correct; the pair is not one step.

Finding it

Static: every place where a read's value controls a later write, and every piece of state that outlives a request - module-level maps, connection-scoped caches, in-process counters.

Dynamic: reproduce it. Sequential tests pass by construction, so I fire 50 concurrent requests at one seat and assert the invariant afterwards. To turn an occasional failure into a deterministic one, I add a 50 ms delay between the read and the write in a staging build, which widens the window so the race shows on every run. The cheapest oracle is a constraint: add UNIQUE (seat_id) on bookings, run the load test, and the violation names the exact statement that broke.

Preventing it

Strongest fix first: remove the sharing. If the state can be per-request or immutable, there is nothing to order.

Otherwise make the check and the write one atomic operation in the database:

UPDATE seats
   SET booked = true, booked_by = $1
 WHERE id = '7A'
   AND booked = false;
-- affected rows = 1: this request won the seat. 0: someone else did.

Or let a unique index enforce it, which also makes retries safe (PostgreSQL 9.5 and later):

INSERT INTO bookings (seat_id, user_id) VALUES ('7A', $1)
ON CONFLICT (seat_id) DO NOTHING;

Take an explicit lock only when the work cannot be expressed as one statement: SELECT ... FOR UPDATE on the seat row, then write, in the same transaction.

I measured all four on a staging box: 50 concurrent requests, one seat, one PostgreSQL instance. The right-hand column is bookings that should not exist.

ImplementationExtra bookings
read, then insert14
UPDATE ... WHERE booked = false0
INSERT ... ON CONFLICT DO NOTHING0
SELECT ... FOR UPDATE, then write0

The read-then-insert path committed 15 rows for a single seat; 14 of them were duplicates.

Where it still breaks

A row lock held across an external call is a throughput trap: put a 300 ms payment call inside the transaction and that seat is capped near 3 bookings per second, with every waiter queued behind it. Call the provider first, take the lock last.

Two code paths locking two rows in opposite orders deadlock under load; sort the keys before locking.

A distributed lock alone is not enough. If the holder pauses - 400 ms of GC is plenty - the lease expires, a second holder acquires it and writes, and the race is back with extra steps. The writer needs a fencing token that the storage layer checks, so a stale holder's write is rejected.

Optimistic concurrency with a version column is fine until contention rises and every request retries; cap the attempts and add jitter, or the retries amplify the load that caused them.

Retries need an idempotency key. When a client times out after a successful conditional update and retries, the second attempt returns 0 affected rows and you cannot tell "this user already holds the seat" from "someone else bought it".

In production

I keep two counters flat at zero: unique-constraint violations, and conditional updates that returned 0 affected rows immediately after the client was told it succeeded. When either moves, the request IDs on the log line show the two writers that interleaved.

Curated: · Written: · Reviewed:

Q&A-91Would you use threads or an asynchronous model for a service that mostly waits on the network?

Clarify first. What's the concurrency target: hundreds or tens of thousands of in-flight requests? What fraction of a request is CPU versus waiting? Which runtime (Java 21+, Go, Node, Python)? What's the memory budget per pod, and how many connections does the downstream pool allow? Those four answers decide it; "mostly waits on the network" alone doesn't.

Answer. For a service that spends most of its wall-clock time waiting on sockets, an asynchronous model on a small number of event-loop threads scales further per gigabyte of RAM, and that's what I'd pick: provided nothing in the handler blocks. If the stack forces blocking calls (JDBC, most Python DB drivers, a vendor SDK with a synchronous client), thread-per-request is the honest choice, or a hybrid: async I/O at the edge, a bounded worker pool for the blocking parts.

Why. A thread parked on a socket read costs a stack. On HotSpot with the default -Xss1m, 8,000 threads reserve 8 GB of virtual address space and commit roughly 8–16 KB each in practice, so the ceiling you hit first is usually the OS thread limit and scheduler contention, not RSS. An event loop keeps one thread per core and holds per-connection state (socket buffer plus a parser state machine, tens of KB) so 8,000 mostly-idle connections fit in a few hundred MB. Java 21's virtual threads blur this: a blocking call parks a virtual thread for a few hundred bytes of heap, so thread-per-request style comes back without the stack cost.

The async model's cost is that the loop is a shared resource. One blocking call serialises every connection on that thread.

Worked example (Python 3.12, asyncio). Four handlers, each sleeping 300 ms:

import asyncio, time

async def handler():
    time.sleep(0.3)          # blocking call inside a coroutine

async def main():
    t = time.perf_counter()
    await asyncio.gather(*(handler() for _ in range(4)))
    print(f"{time.perf_counter() - t:.2f}s")

asyncio.run(main())

gather schedules four coroutines on one loop thread. Each time.sleep blocks that thread, so the loop can't start the next coroutine until the previous returns. Total 4 × 0.3 = 1.20s.

Swap the body for await asyncio.to_thread(time.sleep, 0.3). The sleeps now run on the default executor, which has at least min(32, cpu_count + 4) ≥ 5 workers, so all four run at once: 0.30s. Same code shape, 4× the throughput, and the only change is that the blocking call left the loop.

Illustrative figures (hypothetical, single 4-core pod, 300 ms downstream latency):

ModelConcurrent requestsRSSNote
thread per request, -Xss1m400~1.6 GBstack + per-request heap
async, 4 loop threads8,000~300 MB~40 KB per connection
async, one blocking call per request60~300 MBloop serialised

Failure modes and how I'd catch them.

  • Blocking call on the loop. Detect with event-loop lag: Node's monitorEventLoopDelay, or Python's asyncio debug mode, which logs callbacks over loop.slow_callback_duration (default 0.1 s). Symptom is p99 diverging from p50 while CPU sits pinned near one core.
  • Unbounded async concurrency. The loop happily accepts 8,000 connections and then queues on a 20-connection DB pool; p99 goes from 40 ms to 180 ms once the pool saturates. Fix with a semaphore or bulkhead per downstream, and alert on pool wait time, not just pool utilisation.
  • Thread-pool exhaustion in the hybrid. Queue depth grows, p99 spikes, active threads pinned at max. Alert on queue length and reject early rather than queueing without bound.

What I'd do before committing. Load-test both shapes against the real traffic mix and report requests per second, RSS per 1,000 concurrent requests, and p99 at the target concurrency. If the async version's p99 tracks the thread version at a third of the memory, ship it; if a blocking dependency shows up in the profile, keep threads and stop fighting the runtime.

Curated: · Written: · Reviewed:

Q&A-92How do you estimate a backend feature?

The first 90 seconds. Ask what the number is for before producing one. An estimate that feeds a two-week sprint commitment, one that feeds a quarterly roadmap, and one that decides whether to build the feature at all need different precision, and picking the method before the decision is how estimates get argued about instead of used. Then decompose the work into the parts that actually consume days: schema change and backfill, failure paths, authorization, observability, tests, rollout: estimate each, and give a range whose width names the largest unknown. If the largest unknown is technical (will the backfill finish inside the maintenance window?), the honest answer is a timeboxed spike, not a wider range.

Breakdown, and a miss it would have caught. The visible endpoint is a fraction of the work. One I've been through: an endpoint estimated at 3 days took 11, because a backfill over 40 million rows and a new authorization rule were never in the estimate.

PartEstimatedActual
endpoint and logic3 d3 d
migration and backfill03 d
authorization rules01 d
metrics, alerts, tests02 d
staged rollout02 d

The handler was estimated accurately. Everything around it was estimated at zero, which is the usual shape of a backend miss.

Alternatives, and when each wins.

What the number feedsMethodPrecision to give
Sprint commitment, next two weekstask breakdown in hours or half-days±1 day per task
Quarterly roadmap, 3–6 months outt-shirt sizes or story points, relative to past workorder of magnitude, no dates
Go/no-go on a risky featuretimeboxed spike, 2–3 days, then re-estimate"not estimable until X is answered"
Team capacity planninghistorical throughput from past sprintsrange from observed variance

Story points and t-shirt sizes are deliberately relative: they are fast, they resist false precision, and they only work if you never convert them to hours. The conversion is what makes teams argue about whether a 5 is three days or five. A spike is the right answer when the uncertainty is a technical unknown rather than a scope unknown; timebox it, write down what you learned, and re-estimate with the breakdown method.

A rough order-of-magnitude answer beats a breakdown when the decision is coarse. If a product manager is choosing between two features for next quarter, "that one is a week, this one is a month" is more useful than a five-line table, and the table's precision is wasted. The breakdown earns its cost when the number becomes a commitment someone is held to, or when the work is large enough that a 2x miss breaks a plan.

Failure modes. Estimating the handler and calling it done: the table above. Quoting the point estimate without the range and the assumption it rests on, so the range disappears in the retelling. Running a spike with no timebox, which turns into starting the work. Converting story points to hours, which destroys the relative scale that made them cheap. And treating the estimate as a promise: it is a forecast with a stated confidence, and it should be revised when the assumption behind it changes.

Detecting the miss. Compare estimate against actual per category afterwards and record which category was missed. If the same category is missed twice, that is a process gap, add it to the checklist, not a one-off. The categories that go to zero most often are backfill, authorization, and rollout.

Curated: · Written: · Reviewed:

Q&A-93What should exist so that someone else can operate your service at 3 am?

The thing that has to exist is a runbook per alert, not a document per service, and the only proof it exists is that someone who has never carried this pager mitigates a simulated alert using nothing but what is written down.

A strong first two minutes names four artefacts and one test. The artefacts: a service page (what it does, what it depends on, who owns it, what its SLO is), a dashboard and trace link attached to each alert, a runbook per alert with the two or three safe interventions, and an escalation path ending in a named human, not a team alias. The test: an engineer from another team resolves a fired alert in a drill, timed, with the runbook author watching but not speaking.

Where the runbook lives is a real trade-off, and the answer is usually two of the three:

HomeWins whenFails when
Repo, next to the codethe change that invalidates a command lands in the same PR as the runbook edit; you get diffs and reviewthe responder at 3 am has no checkout and search is poor
Wiki / Confluenceit must be readable by support and PMs; search is goodit rots silently: no review gate, no diff when the command is renamed
Generated from alert rulesyou need coverage: every alert resolves to a runbook link, checkable in CIit captures what fired, never the judgment. It cannot say "if the backlog is draining, do nothing"

The pragmatic split: the runbook body lives in the repo so it is reviewed with the code, and the alert rule carries a link annotation so the page and the runbook are one click apart. CI fails the build if an alert's runbook link does not resolve to a file.

Error budgets change how much runbook you need. With an SLO, most alerts become "watch the burn rate" and only a handful need an intervention. Without one, every alert pages, the runbook set grows without bound, and nobody keeps it current: that is the condition under which this whole approach fails.

It also fails honestly for services with no safe intervention. A single-writer database mid-migration, a stateful consumer group: the correct runbook says "do not restart, page the owner, here is why." Writing a plausible-looking action you have never run is worse than writing none, because at 3 am someone will run it.

A worked example, figures illustrative. A queue-backlog alert fires at 03:12 with no runbook link. The responder greps the repo, finds the consumer deployment, guesses at the scaling command, and confirms it at 04:02. 50 minutes, backlog grown from 8k to 120k messages, and the drain afterwards takes longer than the fix. The same alert with a runbook link: 03:12 page, 03:14 read the two-line runbook, 03:16 scale the consumer, 03:19 backlog falling. The difference is not the command; it is that the command was written down before anyone was tired.

Rot is the failure mode to instrument, not to hope against. Three checks catch most of it: CI that every alert annotation resolves to an existing runbook file; a count of runbooks whose last drill is older than 90 days; and a review of any incident where the responder deviated from the runbook. A deviation is either a stale runbook or a missing one.

So: runbook per alert, reviewed with the code, linked from the alert, drilled by someone outside the team, and pruned when the SLO makes the alert unnecessary. Document the interventions, not the architecture.

Curated: · Written: · Reviewed:

Q&A-95Tell me about a production incident you owned.

Last year I owned a checkout outage on a payments service. Six percent of checkouts failed for 18 minutes: about 2,400 orders. It paged through the error-budget burn alert on the checkout success SLO, which fired 90 seconds after the first failure.

The mechanism: a deploy changed the order-history query to add a status = 'pending' filter. That column had no index. On a 400-million-row orders table the planner flipped from an index scan to a sequential scan. p99 on that endpoint went from 40 ms to 8.2 s. Order-history and checkout shared one Postgres pool of 20 connections. At 8.2 s per query the pool could serve about 2.4 queries per second; checkout demand was 40 rps. The pool saturated within a second, the queue grew, and checkout requests hit their 3 s client timeout. The visible symptom, checkout failing, was two hops from the cause.

I had four options on the call:

OptionTime to effectReversibleCost
Roll back the deploy~4 minyeslose the feature
statement_timeout on the pool~1 minyeskills legitimate slow queries
Build the missing index~10 minyesdoesn't help during the outage
Feature-flag the new query~2 minyesflag didn't exist

I rolled back. The criterion was time-to-effect against blast radius: the deploy was the only change in the window, so rollback was the single action that restored the previous plan, and it was fully reversible. statement_timeout would have masked the symptom and started killing the reporting queries that legitimately ran long. The index build was the right fix but not the right minute. Time to mitigate was 4 minutes from page to rollback; time to resolve was 18 minutes, because we waited for the pool to drain and confirmed p99 back under 100 ms before closing.

Two things I'd flag as the real failure modes. First, pool exhaustion is silent until it isn't: the pool exposes no queue-depth metric by default, so the first signal was checkout timeouts, not the slow query. We added per-query p99 and pool wait time. Second, the query-plan change was invisible in review: the SQL diff was one line and looked harmless.

The follow-up that shipped that week: a CI check that runs EXPLAIN on the top 10 queries against a production-sized snapshot and fails the build if any plan regresses to a seq scan on a table over 10M rows. We also split the checkout pool from the reporting pool and set statement_timeout = 2s on checkout. The index shipped the next day.

If pressed on why I didn't just add the index during the incident: building it takes minutes on a table that size, and every minute the pool stayed saturated was another minute of failed checkouts. Rollback was the only action whose effect was immediate and whose downside was recoverable.

Curated: · Written: · Reviewed:

Q&A-96Tell me about a technical decision you disagreed with.

The 90-second version. Our refund-reconciliation service needed to hand events to a downstream ledger writer. My tech lead decided we'd put Kafka in front of it. I thought we'd be paying for a broker we'd never use, and I said so: with a prototype and numbers, not an opinion. A staff engineer owned the call, chose the Postgres queue, and set a metric that would reopen it. I wrote the ADR, wired the metric, and dropped it.

The decision and the goal. The goal was narrow: refund events had to reach the ledger writer reliably, in order per refund, within a 24-hour SLA. The lead's reasoning was sound on its own terms: a log gives you replay and fan-out for free, and we'd been burned by bespoke queues before. My concern was that we were buying that generality for a workflow that didn't need it.

The evidence. I spent a day building the boring alternative: a refund_outbox table on the existing primary, a poller using SELECT ... FOR UPDATE SKIP LOCKED, and a dead-letter table. I ran it against a replay of a week of production traffic. The figures below are ours, rounded, and the prototype was built to the same durability bar as the broker would have been (at-least-once delivery, idempotent consumer keyed on refund_id) because a naive poller would have made the comparison worthless.

Postgres outboxKafka
sustained rate needed0.002/s avg, 0.02/s peaksame
added latencyup to 5s (poll interval)~10ms
new infrastructurenone: existing primary3 brokers + controller
independent consumers11
replay windowrows retained 30 dayslog retention
ops surfaceone table, one indexbroker upgrades, rebalancing, lag alerts

200 events a day is 200/86,400 ≈ 0.0023 events/s; even a month-end peak hour of 60 events is 0.017/s. A 5-second poll adds at most 5s and on average 2.5s of latency, against a 24-hour SLA. Kafka's latency advantage is real and irrelevant here. Its fan-out advantage is real and irrelevant with one consumer.

Where I'd have been wrong. If we'd needed three independent consumers, a 30-day replay guarantee, or sustained rates above roughly a thousand events a second, the broker wins and the outbox becomes a liability: one table absorbing write load on the primary, with no partition story. Those are the criteria I put in the ADR, not "Kafka is overkill."

The outcome. The staff engineer chose the outbox and deferred the broker, with a trigger: revisit if sustained throughput exceeds 50 events/s for a week, or a second independent consumer appears. I disagreed in the room, lost, and then implemented the outbox.

What I did afterwards, and the failure mode I avoided. I wrote the ADR, added the trigger metric to the dashboard, and stopped arguing. The failure mode here is relitigating: reopening it in code review, or citing the decision in a postmortem when the poller lags. Putting the trigger on a dashboard means "should we revisit?" has an answer that isn't me. That's the part interviewers are actually probing: whether the decision was settled once it was made.

Curated: · Written: · Reviewed:

Q&A-97Why do you want to work on backend systems rather than product surfaces?

The short version

I want to work on backend because the problems I find worth a week of my life are about state: who owns it, what happens when two writers race, and what a caller sees when a dependency is half-down. Product surfaces are where the result gets judged, and I care about that, but the decisions I want to be accountable for sit underneath it.

One assumption I'd state up front: this is a preference, not a ranking. I've shipped UI work and would again.

Why, concretely

The clearest example is an idempotency bug on a payments path. A client timed out on POST /charges, retried, and we charged the customer twice. The fix wasn't a smarter retry policy: it was making the write itself idempotent.

Trace of the fixed path:

  1. Client sends POST /charges with Idempotency-Key: k1. The server inserts a row into charges, which has a unique index on idempotency_key. The response is lost in transit.
  2. The client retries with the same key two seconds later.
  3. The insert violates the unique index. The handler catches the duplicate-key error, reads the existing row, and returns the original charge id with a 200.

One charge, one row, and the client can't tell the retry from the original. That's the shape of problem I want: the correctness argument fits on a whiteboard, and the failure is observable in a metric. Duplicate-key rate, charges per key.

Illustrative numbers from that service: about 400 charge writes/sec at peak, a retry window of three attempts over six seconds, and after the fix duplicate charges went to zero while p99 on the endpoint moved from 90 ms to 95 ms. The extra index lookup. I'd rather defend that trade than a retry heuristic.

What I'm accepting

Backend costs things, and I'd say so out loud:

  • On-call. I've carried a pager; the 3 a.m. page is part of the job, not a surprise.
  • Slow feedback. A UI change is visible in a minute; a schema migration's consequences show up over weeks.
  • Invisible wins. Nobody thanks you for the duplicate charge that didn't happen.

When I'd be wrong

If I wanted fast, tight feedback loops and disliked ambiguity in requirements, backend would be a bad fit: the spec is often "make this correct under load" and you write the rest yourself. And if I said I preferred backend because I don't like talking to product, that's a red flag: the idempotency fix above only shipped because I sat with the payments PM and we agreed what "one charge" meant.

Evidence I'd bring

ClaimEvidence (illustrative)
correctness under failuremade a charge path idempotent; duplicate charges to 0
scale reasoningcut p99 from 900 ms to 120 ms at 3× traffic by removing an N+1
ownershipon call 18 months, led 2 incidents

I'd bring one of these as a full story (problem, the design decision I made, and the operational result) rather than listing all three.

Curated: · Written: · Reviewed:

Q&A-101Your service's database password has to be rotated every 90 days. How do you do that without downtime?

First question back: is this a static password I own, or can I move to short-lived dynamic credentials? That decision dominates the rest. If the database supports it (Vault's database secrets engine, IAM auth for RDS, GCP IAM database auth) the right answer is to stop having a 90-day password at all. The service authenticates with its workload identity and gets a credential with a one-hour lease. Rotation stops being an event and becomes expiry.

If I'm stuck with a static password, the shape is: the value never lives in code, env vars, or the image; it lives in a secret manager; the service reads it through a client that can refresh; and rotation is a two-phase overlap, not a swap.

Env vars are the wrong transport for a credential you intend to change. They're fixed at process start, so rotating means restarting every instance: that's a deploy, not a rotation, and it means rotation gets skipped. They're inherited by every child process, readable from /proc/<pid>/environ by anything running as the same uid, and they leak into crash dumps, docker inspect, and CI logs. Env vars are fine for non-secret config: log level, endpoint URLs, feature flags.

The pattern I'd use:

  1. Secret in the manager (AWS Secrets Manager, GCP Secret Manager, Vault KV), access scoped by IAM to the service's role, not a shared human account.
  2. Service fetches at startup, caches in memory, refetches on TTL expiry (~15 min) with jitter so 200 instances don't stampede the manager.
  3. Connection pool configured with a max connection lifetime shorter than the rotation window, so no connection outlives the credential it was opened with.
  4. Rotation: write the new password as the current version, keep the old one valid, let services pick it up on their next refresh, then revoke the old one after the longest possible connection lifetime plus margin. Roughly 2× pool max lifetime.

The overlap window is the whole trick. Revoke the old password the instant you write the new one and every instance that hasn't refreshed starts failing auth; with a 20-connection pool per instance you get an auth-error burst and a reconnect storm against the database.

ApproachRotation costLeak surfaceUse when
Env varRestart / deploy/proc, crash dumps, CINon-secret config only
Mounted file (CSI driver, Vault Agent)File rewritten; app must re-readtmpfsApp can watch a file
SDK fetch + in-memory cacheIn-process refreshProcess memoryMost services
Dynamic credentials with a leaseNone: expires on its ownMinutesDB/cloud supports it

Failure modes worth naming. Secret manager unreachable at cold start means the service can't boot: mitigate with a last-known-good copy on encrypted disk and accept the staleness risk deliberately. A refresh that fails silently leaves the service on an expired credential until the database rejects it, so alert on secret age and on auth-failure rate, not on whether the refresh call returned. A pool holding old credentials past revocation shows up as intermittent auth errors on a subset of instances; correlate auth failures with instance IDs to see it. And never log the value: log the version or ARN, with redaction at the logger rather than at each call site.

func (s *Store) Password(ctx context.Context) (string, error) {
    s.mu.RLock()
    if time.Now().Before(s.expiry) {
        v := s.value
        s.mu.RUnlock()
        return v, nil
    }
    s.mu.RUnlock()

    s.mu.Lock()
    defer s.mu.Unlock()
    if time.Now().Before(s.expiry) { // another goroutine refreshed
        return s.value, nil
    }
    v, ttl, err := s.client.GetSecret(ctx, s.arn)
    if err != nil {
        return "", err // caller keeps the old value; do not clear it
    }
    s.value, s.expiry = v, time.Now().Add(ttl)
    return v, nil
}

If someone asks why not just restart on rotation: because restart is a deploy, and coupling the two is how you end up with a password that hasn't changed in three years.

Curated: · Written: · Reviewed: —

Q&A-17Users upload 500 megabyte files. How do you design that?

Assumptions first: 500 MB is the ceiling, clients are browsers on home uplinks, files are untrusted user content, and storage is S3-compatible. If any of those is wrong the design changes, so I'd state them before drawing anything.

The answer: the bytes never touch my service. The API issues short-lived presigned URLs for a multipart upload and records metadata; the client uploads parts straight to object storage.

client                     api                        storage
  |-- POST /uploads ------->|                            |
  |                         |-- CreateMultipartUpload -->|
  |                         |<-- UploadId --------------|
  |<-- key, uploadId, partUrls[] (presigned, 15 min) ---|
  |-- PUT part i (presigned UploadPart) -------------->|
  |<-- ETag_i -----------------------------------------|
  |-- POST /uploads/{id}/complete {parts, etags} ----->|
  |                         |-- CompleteMultipartUpload>|
  |                         |<-- 200 -------------------|

Part sizing: 500 MB at 8 MiB parts is 63 parts, at 16 MiB it's 32. Both far under the 10,000-part cap, and 5 MiB is the floor for any part but the last. Smaller parts mean less re-upload after a dropped connection; larger parts mean fewer round trips. 8–16 MiB is the usual landing spot.

Why not proxy: buffering 500 MB per request means 6 concurrent uploads hit a 3 GB container limit and it OOMs. Even with streaming, the connection is held for the whole transfer: 500 MB is 4,000 Mb, so a 20 Mbps uplink takes 200 s. Past the 60 s default idle timeout on an ALB, and past nginx's 60 s client_body_timeout. The proxy path fails on time as well as memory.

Failure modes I'd design against, and how I'd see them:

FailureSymptomFix
Presigned URL expires mid-upload403 on UploadPart Nclient re-requests URLs, resumes from last acked part
Complete never called (tab closed)orphaned parts, billedlifecycle AbortIncompleteMultipartUpload after 7 days; alert on ListMultipartUploads count
ETag treated as MD5integrity check fails on every multipart objectmultipart ETag is md5-of-md5s + -N; verify a client-computed SHA-256 stored in metadata instead
Size not cappeda client PUTs 50 GBa presigned PUT has no size limit: use a presigned POST with content-length-range, or reject and delete after the fact
Content-Type spoofedHTML served from your bucketpin Content-Type in the signature; serve from a separate domain
Malware—scan post-upload via event notification, quarantine bucket, don't serve until clean

Trade-offs: direct upload gives up the ability to inspect bytes before they land. If compliance requires DLP or your own envelope encryption, proxy, but stream with backpressure (nginx proxy_request_buffering off, Node pipeline()), never buffer, and raise the idle timeouts.

Verification: upload the max file and watch RSS stay flat; kill the connection at 80% and confirm resume from the last part; confirm the orphan lifecycle rule actually deletes.

Curated: · Written: · Reviewed: