- Home
- Prep by Job Role
- Full-Stack Developer
- Top 100 Q&A
Top 100 Full-Stack Developer Interview Questions and Answers
The questions most likely to come up in your Full-Stack Developer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-3Should the browser call each microservice itself, or should a product API sit in front and return a page-shaped payload?
Short answer: put a product API (a BFF) in front when the page needs data from more than a couple of services, or when those services are not same-origin, not browser-safe, or not versioned for a client. Call services directly from the browser when there are only a handful, they share an origin and an auth model, and their contracts are already public.
Assumptions I'd state out loud: one product team owns the page, the services are internal and behind a private network, and the page needs five or more downstream calls.
Why the BFF wins by default here
The browser is a hostile, high-latency client. Every extra origin costs a DNS lookup, a TLS handshake, a CORS preflight, and its own connection pool. Every extra service is another auth audience, another version to pin, and another place a schema change breaks the page. A BFF moves all of that server-side: one origin, one session cookie or token, one response shape, and credentials that never reach the client. The client stops stitching six schemas together and just renders.
What the BFF costs: this is the part that decides it
- Extra hop. Browser → BFF → service adds one round trip. Intra-region that's roughly 1–5 ms; cross-region or through a cold function it's 30–100 ms. Fine for a page load, wrong for per-keystroke autocomplete.
- The BFF is now on the critical path for every widget. One slow downstream can stall the whole response unless you fan out concurrently with per-call deadlines and return partial results.
- Ownership hotspot. Six teams contributing fields to one aggregation layer means one deploy queue, one on-call rotation, and merge conflicts on the response schema.
- Deploy coupling. Adding a field to the page can require a BFF release, which is slower than the client shipping.
- It rots into a distributed monolith if business logic leaks into it.
When direct browser calls are fine
- Two to four services behind one gateway, same origin, same auth, stable contracts.
- Third-party browser SDKs built for it: Stripe.js, the Auth0 SPA SDK, a CDN.
- Public APIs with correct CORS and rate limits where the browser is the only consumer.
Failure modes and how you see them
- BFF as bottleneck: its p99 tracks the slowest downstream. Detect by comparing BFF p99 against per-downstream p99 and watching its thread and connection pool saturation.
- Timeout cascade: one downstream hangs and every widget fails. Fix with per-call deadlines, concurrent fan-out, and a per-widget error field in the payload.
- Ownership drift: nobody owns the BFF and PRs stall. Detect via review latency and the gap between teams with write access and teams on call for it.
Worked example (illustrative numbers)
A dashboard needs 16 internal services.
| Caller | Browser-origin requests on first paint | p95 first paint |
|---|---|---|
| browser → 16 services | 16, plus preflights | 735 ms |
| browser → 1 BFF | 1 | 210 ms |
| BFF → 16 services | 16, server-side, concurrent | hidden from CORS |
Arithmetic: 16 sequential browser calls at ~45 ms each is ~720 ms. The BFF fans out concurrently, so wall clock is the slowest call (~180 ms) plus one hop (~30 ms), ~210 ms. The win is concurrency and one origin, not the hop, which is exactly why the hop cost matters when the fan-out is small.
Decision rule: if the page needs more than a few services, or any of them are not same-origin and browser-safe, put one product API in front. If it needs two same-origin services with public contracts, call them directly and skip the layer.
Curated: · Written: · Reviewed:
Q&A-18A profile screen is slow. How do you tell overfetch from underfetch at the client-server boundary, and which half should shrink the payload?
Short answer. Overfetch is bytes on the wire the screen never reads; underfetch is round trips the screen needs and doesn't have. They are different metrics, payload size versus critical-path request count, so measure both before touching either. And the payload only shrinks on the server: the client can narrow what it asks for, but the bytes are the server's decision.
Diagnosis. Two measurements, both taken at the boundary.
- Bytes per response, and bytes actually used. Resource Timing gives transfer size per request:
// Chrome 120+, Resource Timing Level 2
performance.getEntriesByType('resource')
.filter(e => e.initiatorType === 'fetch')
.forEach(e => console.log(e.name, e.transferSize, e.startTime, e.responseEnd));
transferSize is 0 for cross-origin responses without Timing-Allow-Origin, so for a third-party API read Content-Length from the response or measure at your own proxy.
Then compare that against what the render path reads. A Proxy over the parsed JSON records the keys the component tree actually touches:
const read = new Set();
const track = (o) => new Proxy(o, {
get(t, k) {
if (typeof k === 'string') read.add(k);
const v = t[k];
return v && typeof v === 'object' ? track(v) : v;
}
});
If 180 kB arrives and the screen reads 31 kB of it, that is overfetch: 83% of the payload is dead weight.
- Critical-path request count. Count the requests that must resolve before first meaningful paint, and their depth. Five requests where each starts only after the previous resolves is a waterfall: the cost is 5 × RTT, not 5 × bytes. On a 120 ms RTT link that is 600 ms of pure latency before anything renders.
The two signatures are opposite and do not overlap: overfetch is one large response with low field utilization; underfetch is several small responses with a long dependency chain.
Which half shrinks the payload. The server. The client's lever is the ask: a GraphQL selection set, a REST sparse fieldset (?fields=id,name,avatar), or a BFF endpoint shaped to the screen. But if the server ignores the selection and serializes the full resource, nothing shrinks. So the split is: the client narrows the request, the server enforces it and stops writing unused fields. Underfetch is not a payload problem at all: shrinking bytes there makes it worse, because the fix is fewer round trips (aggregate into one BFF call or one query), not smaller responses.
Worked example (hypothetical figures). Profile screen, mid-range Android, 120 ms RTT, ~1.5 Mbps effective:
| Boundary | Requests | Payload | Read by screen | TTI |
|---|---|---|---|---|
| 5 REST resources | 5, sequential | 28 kB | 28 kB | 2.4 s |
| Uncapped GraphQL | 1 | 180 kB | 31 kB | 2.8 s |
| BFF page slice | 1 | 31 kB | 31 kB | 1.1 s |
The REST version is underfetch-bound: 600 ms of its 2.4 s is RTT on the waterfall. The GraphQL version fixed the round trips and introduced overfetch: 149 kB of unused edges is ~1.19 Mbit, roughly 800 ms of extra transfer at 1.5 Mbps. The BFF slice is one request carrying only the fields the screen renders.
Failure modes. A server-side query cost cap (depth and complexity limits) stops one client pulling the whole graph, but it does not tell you whether this screen overfetches: that is the utilization measurement. And a BFF that returns the union of every screen's fields reintroduces overfetch one layer down; keep the slice per screen and check its utilization the same way.
Curated: · Written: · Reviewed:
Q&A-27A catalog paints 48 cards and each card then loads its seller. How do you fix that loop on the client and the API?
Core (first 90 seconds)
Clarify one thing before picking a fix: are the 48 cards pointing at 48 distinct sellers, or a handful reused? That decides join versus batch. Assume a catalog page of 48 cards, each rendering seller name and rating, sellers N:1 and reused across cards.
Fix both ends. The list endpoint returns the seller fields the card renders, so the client never issues a second call. The card reads card.seller from the list payload; the per-card fetch is deleted, not cached.
If the list endpoint belongs to another team or a third party, batch on the client instead: collect the distinct seller ids on the page, issue one request for all of them, hand each card its row. One list call plus one batch call replaces 49.
Join or batch
A join is one query, with seller columns riding along on each card row. It wins when the relation is N:1, the seller projection is narrow, and the list query already exists. It hurts when the seller row is wide, a description or JSON blob repeated 48 times, or when the relation is 1:N, where the join fans out and 48 cards become 480 rows needing DISTINCT or a lateral.
A batch (DataLoader-style) is two queries: the list, then one IN over the distinct seller ids. It wins when sellers are reused (dedupe shrinks the second query), when the seller lives behind a service you cannot join, and when the card needs only a few fields. It costs a second round trip and needs a per-request cache.
// Node 20 — batch seller lookups within one render pass
const pending = new Map(); // id -> [{resolve, reject}, ...]
let scheduled = false;
function sellerById(id) {
return new Promise((resolve, reject) => {
const waiters = pending.get(id) ?? [];
waiters.push({ resolve, reject });
pending.set(id, waiters);
if (!scheduled) { scheduled = true; queueMicrotask(flush); }
});
}
async function flush() {
scheduled = false;
const batch = pending; pending.clear();
const ids = [...batch.keys()]; // 48 calls, 9 distinct ids
const rows = await db.query(
'SELECT id, name, rating FROM sellers WHERE id = ANY($1)', [ids]);
const byId = new Map(rows.map(r => [r.id, r]));
for (const [id, waiters] of batch) {
const row = byId.get(id) ?? null;
for (const { resolve } of waiters) resolve(row);
}
}
Trace: 48 cards render in one pass and call sellerById with ids [7,7,3,7,12,...]. All 48 land in pending before the microtask runs, so ids is the 9 distinct ids and one query carries 9 parameters. Each of the 48 promises resolves from byId. 48 calls, 1 query.
GraphQL is the same shape: a seller field resolver without a loader reintroduces the N+1; with a per-request loader it collapses to one query and returns only the fields the query selected.
Failure modes
- Batch size: Postgres caps bind parameters at 65535, so a 100k-row page breaks the
INlist. Chunk into 1000s. - Fan-out: joining 1:N multiplies rows; 48 cards become 480 and you need
DISTINCTor a lateral join. - Cache scope: memoize per request. A process-wide cache serves a stale seller name after a rename; key by seller id and invalidate on write or TTL.
- Slow downstream: one batch call to a seller service at p99 200 ms now blocks the whole page instead of one card. Time it out and render a placeholder.
Verification
Wrap the db client and count queries in a test: a full first page asserts at most 2 queries and at most 1 list fetch. That is the regression guard; the network panel is the manual check.
Curated: · Written: · Reviewed:
Q&A-14The UI compiles a TypeScript type and the API validates a schema. How do you keep those from drifting, and who fails first when they do?
Core answer (first 90 seconds). Make the schema the single source of truth and derive the client's TypeScript from it, so a breaking change fails the build before it can reach a browser. If the API is TypeScript, z.infer (or TypeBox's Static) gives you the type with no generation step at all. If the API is Go or Python, emit OpenAPI from the validator and generate the client types in CI, then fail the build when the committed output differs from a fresh generation.
Who fails first: with generation wired into CI, the build fails first. The regenerated types diff, and tsc rejects any client code still using the old field: before merge, before deploy. Without it, the API fails first: it either rejects the old shape with a 400 or, worse, accepts it and writes a default.
// schema.ts — the API validates this at runtime; the client type is derived from it
import { z } from "zod";
export const Payout = z.object({
id: z.string(),
amountCents: z.number().int().nonnegative(),
});
export type Payout = z.infer<typeof Payout>; // no hand-written mirror
# CI: regenerate and fail if the committed client types are stale
npm run gen:types
git diff --exit-code src/generated/api.d.ts # exit 1 => build red
Trace a rename of amountCents to amount: the schema changes, gen:types rewrites api.d.ts, git diff --exit-code exits 1, and any client file reading payout.amountCents fails tsc. Two independent gates, both before a request is sent.
| Change | Generated types + CI | Hand-written mirrors |
|---|---|---|
| Add optional field | compiles; client ignores it | compiles; same |
| Rename a required field | CI red at the regen diff; tsc error | runtime 400, or a silent default |
Retype int → string | CI red | runtime |
Trade-offs. Schema-first (a hand-written OpenAPI file) makes the spec the source, but nothing forces the handler to match it unless a validator reads the same file: otherwise the spec drifts from behaviour. Code-first (Zod, Pydantic, TypeBox) makes the validator the source, so the spec can't drift from validation; the risk moves to the emitted spec not being regenerated in CI, which the diff check above closes. Runtime client-side validation, parsing responses with Zod in the browser, catches drift even when you can't regenerate, but it's a runtime failure and adds bundle weight; keep it for third-party APIs you don't control. Generation isn't worth it for a single-consumer internal endpoint, a prototype, or a loose spec that emits any or all-optional types; there a contract test (Pact) or a shared types package is the cheaper guard.
Failure modes. Additive drift is invisible and harmless: an extra optional field compiles and is ignored. The dangerous case is a silent default: if the server ignores an unknown field and the column defaults to 0, types never see it. Detect that with a DB constraint (amount_cents > 0) or a canary assertion, not with the type system. A spec missing required generates all-optional types that compile against anything; assert required fields in the spec and add a test that a missing field fails validation. And a generated type is only as good as the spec's accuracy, if the handler and the spec disagree, generation faithfully propagates the wrong contract.
Curated: · Written: · Reviewed:
Q&A-51How do you ship a frontend change that depends on a new API field?
Assume you can deploy the API and the bundle independently: that is the case worth designing for. The answer is expand/contract: the API adds the new field while the old one keeps working, the bundle ships reading the new field, and the API removes the old field only after old clients have drained. Flipping both artifacts in one window is what breaks.
A browser tab, a service worker cache, a CDN edge node, or an iOS webview can hold the previous bundle for days. The API cannot tell which bundle is calling it unless you tell it, so the API has to be a superset of what every live bundle needs, for as long as any of them are live.
| Criterion | Lockstep (both at once) | Expand/contract |
|---|---|---|
| Old bundle still live | 400s or NaN on the new field | keeps working; API serves both fields |
| Rollback | must revert both, in order | revert either side alone |
| Cost | one deploy window | two deploys plus a drain wait |
| Acceptable when | no cached clients: internal tool with a forced reload, native app with a mandatory update gate | any public web surface with long-lived sessions |
The check that makes this safe is a contract test in the pipeline that runs the currently deployed bundle against the candidate API, and blocks the release if it fails:
// contract-check.mjs — CI step, Node 20, run against the candidate API
const res = await fetch(`${API}/api/orders/42`);
const order = await res.json();
// the field the live bundle reads today must still be present and typed
if (typeof order.legacyTotalCents !== 'number') {
console.error('live bundle would render NaN for total');
process.exit(1);
}
Illustrative numbers for the failure this prevents: an API removed a field 11 minutes before the matching bundle shipped. 18% of sessions were still on the old bundle, those sessions got 400s on checkout, and checkout success sat at 82% until the web deploy finished. The same change done expand/contract would have held 99.7%.
Detect the drain rather than guessing it. Have the bundle send its build id as a header, then dashboard 4xx rate split by build id and alert when a build older than a few hours is still above ~0.5% of traffic. If p99 session length is 40 minutes but 2% of sessions run past 12 hours, you wait for the tail, not the median: that tail is what decides when the old field can go. Lockstep is fine only when no cached client can exist: an internal tool that forces a reload, or a native app with a mandatory update gate. An additive-only change where old bundles ignore unknown JSON keys needs neither dance.
Curated: · Written: · Reviewed:
Q&A-7The bundle and the API ship on different days. How do you change a response so open tabs and a new deploy both keep working?
The 90-second answer. Ask two things before touching anything: which side is changing, and how long a tab can live. Then hold one invariant: never remove or retype a field the oldest live bundle reads. Ship additive changes beside the old shape, version the path only when semantics must change, and gate removal on measured client-version traffic, not a calendar.
Assumptions: the bundle is JS cached by a CDN; tabs stay open for hours to days; there is no forced reload; the API is the only writer of the response.
Why additive works. A JSON parser ignores keys it does not know. The break is never "a new field appeared": it is "a key the old bundle reads disappeared, changed type, or changed meaning." So the response must stay a superset of what the oldest live bundle reads. Emit total_cents beside totalCents and keep both until telemetry shows no v1 client is calling. Three ship states, one field: additive (both keys, both bundles work), rename in place (new key only, old bundle breaks), sunset after zero old traffic (new key only, old bundle gone).
When additive is not enough. A type change (number to string), a semantic change (cents to dollars), or a newly required request field. Then version:
| Mechanism | Cache-friendly | Visible in logs | Cost |
|---|---|---|---|
Path /v2/orders | yes | yes | two code paths to maintain |
Header X-API-Version: 2 | only with Vary | only if logged | easy to drop in a proxy |
Media type Accept: application/vnd.acme.v2+json | yes with Vary | no | hardest to debug |
Path versioning is the default for breaking changes; header versioning only when URLs must stay stable. Do not version for additive changes: version sprawl is its own outage.
Tolerant client. Read the field with a fallback so the rename window is a non-event:
function price(order) {
const cents = order.total_cents ?? order.totalCents;
return cents == null ? "—" : `$${(cents / 100).toFixed(2)}`;
}
Trace: {total_cents: 1999} → 1999 → $19.99. {totalCents: 1999} → 1999 → $19.99. {} → undefined → —. The old bundle renders through the rename instead of throwing.
Failure modes and how they surface.
- Rename in place: old bundle reads
undefined→ blank UI or 422 on write. Detect via a client-version header in server logs; alert on 4xx from versions older than the current deploy. - New enum value: old
switchfalls to default. Detect by counting unknown-value branches client-side. - Required request field added: old bundle omits it → 400. Same version-tagged 4xx signal.
- CDN serving the old bundle after deploy: hash bundle filenames, long
max-ageon hashed assets,no-cacheonindex.html.
Sunset gate. Do not remove on a timer. Emit a client-version header, count sessions still sending the old field, and require zero for a window longer than the p99 tab lifetime. If p99 tab lifetime is 26 hours, a 24-hour quiet window can still miss a tab: use 7 days, or force-refresh clients below a minimum version.
Worked example (hypothetical). A rename shipped at noon; 1,420 tabs still ran the previous bundle; checkout returned 422 for 27 hours until users hard-refreshed. Additive shipping plus a version header would have made the old bundle a non-event.
Stale tabs are callers, so version for them.
Curated: · Written: · Reviewed:
Q&A-54How do you change a response field without breaking users still on the previous bundle?
The 90-second answer
The contract is: every bundle we still allow to run can parse every response we send. So the real question is which bundle is the oldest one in the wild, and how we cross from here to there without a window where it can't parse. The default answer is expand/contract: ship the new shape alongside the old, migrate readers, then remove the old shape once the oldest allowed bundle is gone.
Clarify two things before picking a mechanism: is this a first-party client you can force-update, or third-party integrators you can't? And is the field read by clients, or only written?
Mechanism
Expand: the API emits both fields. Old bundles read the old one, new bundles read the new one with a fallback. Nothing breaks. Migrate writers and readers. Contract: drop the old field once telemetry says no allowed bundle reads it.
// bundle v41 — reads either shape; v40 only knows total_cents
function cartTotal(cart) {
return cart.totalCents ?? cart.total_cents;
}
Trace: {total_cents: 1999} → totalCents is undefined → returns 1999. {totalCents: 0} → returns 0, no fallback, because ?? only falls through on null/undefined, not on 0. That matters if zero is a legal total.
| Phase | API emits | v40 (old bundle) | v41 (new bundle) |
|---|---|---|---|
| expand | total_cents + totalCents | reads total_cents | reads totalCents, falls back |
| migrate | same | same | same |
| contract | totalCents only | parse fails | reads totalCents |
Alternatives, and when expand/contract isn't worth it
- Versioned endpoints (
/v2/cart). Clean cut, old bundle untouched. Cost: two handlers, two test suites, and you still have to drain v1. Versioning moves the drain, it doesn't remove it. Worth it when the change is structural (different resource shape or auth model), not one field. - Dual-read in the new bundle only. The new bundle tolerates both shapes, so the server can flip in a single deploy. Cost: the tolerant client must ship and reach the fleet before the server flips, the same drain problem in reverse, and it does nothing for third parties.
- Server-side mapping keyed on client version. One canonical internal model, a translation layer at the edge. Cost: reliable version detection (header, UA, token claim) and a mapping table that grows with every version you support. Worth it when there are many consumers and you want one internal schema.
- Not worth expand/contract: internal service-to-service where both sides deploy atomically, or a field with no reader (write-only, or read only by a job you control). Just change it.
Failure modes
- A canary that only sees new-bundle traffic. A rename shipped to 16% of the fleet looks clean because the dashboard averages over sessions that all have the new bundle; the 84% on v40 fail to parse and their errors get attributed to the old release. Detect by segmenting error rate by client bundle version and alerting on parse failures per version, not fleet-wide.
- "Additive is safe" is false for strict deserializers. Jackson with
FAIL_ON_UNKNOWN_PROPERTIES, or a JSON Schema withadditionalProperties: false, rejects an added field. Check consumers' parsers before assuming expand is free. - Old field kept populated but with changed meaning: units, nullability, casing. Old bundles parse it and render wrong numbers, with no error rate to alert on. Detect with contract tests that replay captured old-bundle traffic against the new API.
Drain arithmetic: if you support the last two app versions and telemetry shows v40 at 0.4% of sessions on day 45 and flat after, the tail is users who never update. Set a floor, force-update below v41, rather than waiting for zero.
Curated: · Written: · Reviewed:
Q&A-25Web uses cookies and a bundle you can ship daily. Mobile uses a bearer header and a binary you cannot. What must the shared API accept so neither client papers over the other?
One authorization layer that resolves both credential types to the same principal, one response shape, and a versioning scheme that neither client's release train controls.
Concretely, the shared API must accept:
- Auth:
Cookie: session=...andAuthorization: Bearer <token>on the same routes, both resolving to the same principal before any handler runs. CSRF protection applies to the cookie path only: a native client sends no ambient credential to forge, and applying CSRF to bearer requests breaks mobile for no security gain. - Errors: one envelope, e.g.
{"error": {"code": "...", "message": "..."}}, with the same status codes. No HTML error pages, no mobile-only{"success": false}wrapper. - Pagination: opaque cursors, identical for both. If web gets
?page=3and mobile gets?cursor=..., one client is papering over the other. - Versioning: a version in the path or a header, so a store binary pinned for months and a bundle shipped daily are both served. Deprecation windows sized to the slowest client's release cycle, not the fastest.
// Node 20, Express 4, cookie-parser mounted above this
async function authenticate(req, res, next) {
const bearer = req.get('authorization')?.replace(/^Bearer /i, '');
const token = bearer ?? req.cookies?.session;
if (!token) return res.status(401).json({ error: { code: 'unauthenticated' } });
const principal = bearer ? await verifyJwt(token) : await lookupSession(token);
if (!principal) return res.status(401).json({ error: { code: 'unauthenticated' } });
req.principal = principal;
next();
}
| Caller | Credential | Result |
|---|---|---|
| web bundle | cookie session | 200, JSON page slice |
| mobile binary | bearer header | 200, same JSON |
| mobile against cookie-only API | none sent | 401, blamed on the app |
That is the shared-contract answer. It is not always the right one.
Alternatives. A backend-for-frontend puts one API per client shape over a shared domain service. Right when read patterns genuinely diverge: web renders a page slice, mobile wants a compact sync payload. Cost: two surfaces to version, two test suites, and the divergence grows. Separate gateways in front of one domain API let each terminate its own auth and reshape responses; right when the mechanisms differ in kind (mTLS or request signing for mobile, cookies for web). GraphQL with client-specific queries is right when over-fetching is the actual problem, wrong when you need per-client caching and rate limits.
When not to force one contract. If the mobile binary ships quarterly and web ships hourly, a single contract means every field you ever shipped stays forever, or you break the store binary. A versioned facade or a BFF is then cheaper than a union-shaped response that over-fetches for both.
Failure modes. Cookie-only assumption: mobile sends nothing, every call 401s. Detect by splitting 401 rate by client id. CSRF middleware mounted globally: mobile POSTs 403 with no cookie present. Error-envelope divergence: web shows a toast, mobile crashes parsing HTML. Detect with contract tests asserting content-type and schema on every error path. Cursor format leaking into client code: one client cannot resume. Detect by replaying the same cursor from both clients in CI.
A representative version of this: an API required a cookie the mobile client never stored, 2,160 logins returned 401 over 53 hours, and each team blamed the other. The fix was not a new endpoint; it was one auth layer.
Verify by running the same write from a cookie browser session and a bearer mobile session and asserting status, body schema, and pagination token match, then again with the cookie stripped and the bearer expired, to confirm both fail identically.
Curated: · Written: · Reviewed:
Q&A-98You are handing a full-stack feature to another team. What contract documentation do they actually need?
The contract doc the next team needs is the wire-level artifact: versioned example payloads for success and error, field-level compatibility rules, status-code semantics, and one consumer test they can run on day one. Screenshots, table listings, and diagrams of your internals are context they can't act on: leave them out.
Payloads first. Publish literal JSON for a request, a success response, and an error response, each carrying the contract version. That version field is what lets the receiving team tell which rules apply to a payload they pull out of a log at 2am.
{
"contract_version": "order.v2",
"id": "ord_8812",
"status": "shipped",
"total_cents": 4599,
"currency": "USD"
}
Then the rules that decide disputes. A table beats prose here, because the receiving team will grep it during review:
| Change | Breaking? | Consumer obligation |
|---|---|---|
| Add an optional response field | No | Ignore unknown fields |
| Add an enum variant | No | Map unknown values to a fallback |
| Rename or remove a field | Yes | Version bump |
| Make an optional field required | Yes | Version bump |
| Add a required request field | Yes | Version bump |
The enum row is the one teams skip and regret. If the consumer's switch is exhaustive over status, a new variant from the producer throws at runtime in production, not in CI.
Status and error semantics. Say which codes mean retry, which mean the request was malformed, and which mean the caller's token is stale. "4xx is your fault" is not enough when the receiving team is writing retry logic.
One consumer test. Ship a fixture and a test that runs against it, so the receiving team's CI fails the moment the contract drifts:
# Python 3.12, pytest
import json, pathlib
FIXTURE = json.loads(pathlib.Path("contract/order.v2.json").read_text())
def normalize(payload):
known = {"pending", "shipped", "cancelled"}
status = payload["status"]
return {**payload, "status": status if status in known else "unknown"}
def test_tolerates_unknown_enum_variant():
payload = {**FIXTURE, "status": "partially_refunded"}
assert normalize(payload)["status"] == "unknown"
def test_ignores_added_field():
payload = {**FIXTURE, "loyalty_tier": "gold"}
assert normalize(payload)["id"] == FIXTURE["id"]
Failure modes to name in the doc. Missing versus null: an absent field and a field that is null are different, and consumers that conflate them break the first time the producer starts sending explicit nulls. Strict deserializers: a consumer that rejects unknown fields breaks on the first additive change. Exactly the change the contract calls safe. Version negotiation: if the payload carries no version, neither side can tell which rules it was built against.
How I'd know the handoff worked. Have the receiving team add one new field using only the contract doc, then confirm web, mobile, and API agree on the same fixture. If they have to read your source to do it, the doc is not done.
Curated: · Written: · Reviewed:
Q&A-4For a browser SPA talking to your API, when does a cookie session beat a bearer token the page must store, and what does logout have to clear on both sides?
Same-site SPA and API: use a cookie session. The browser attaches it, page script can't read it, and logout is one server-side delete that kills every tab and every device at once. Reach for a bearer token only when the API is genuinely cross-origin, when you're delegating access to third parties, or when you have non-browser clients, and even then keep the long-lived credential in an httpOnly cookie and the short-lived access token in memory, never in localStorage.
Where each wins
| Situation | Cookie session | Bearer in page |
|---|---|---|
| API same-site, or behind a BFF proxy on the same origin | yes | works, no benefit |
| API on a different site | unreliable: Safari and Firefox block third-party cookies by default | yes |
| Third-party / delegated access | no | yes |
| Immediate revocation required | yes, one delete | only with a denylist |
| XSS on the origin | credential unreadable; attacker can still call the API from the page | credential exfiltrated and replayable off-site |
The XSS row is the real decision. HttpOnly does not stop XSS from making authenticated requests: it stops the attacker from carrying the credential to their own machine and using it after the tab closes.
Cookie session, concretely
The API sets Set-Cookie: __Host-sid=<id>; Secure; HttpOnly; SameSite=Lax; Path=/. The __Host- prefix forces Secure, Path=/ and no Domain attribute, so a compromised sibling subdomain can't overwrite it. SameSite=Lax means cross-site POSTs don't carry the cookie, which kills form-based CSRF; keep state-changing routes off GET, and require a custom header (say X-CSRF-Token) on mutations, since a cross-site form can't set one without a preflight.
The SPA can't set this cookie itself: the API does, and the fetch call needs credentials: 'include' with Access-Control-Allow-Credentials: true and an exact Access-Control-Allow-Origin (never *).
Server side, <id> maps to a Redis key sess:<id> holding the user id and expiry. Every request is one GET, sub-millisecond against a local Redis. That lookup is what buys instant revocation.
Bearer token, concretely
Access token 10 minutes, refresh token 30 days, rotated on every use with reuse detection. The access token lives in a JS variable; the refresh token lives in an httpOnly cookie so script can't read it either. localStorage is the wrong home for either: any injected script reads it, and it survives tab close.
Logout, both sides
Cookie:
POST /logout Cookie: __Host-sid=abc123
DEL sess:abc123
204 + Set-Cookie: __Host-sid=; Max-Age=0; Path=/; Secure; HttpOnly; SameSite=Lax
Any tab still sending __Host-sid=abc123 gets 401, because the server finds no sess:abc123. Cookies are per browser profile, so Max-Age=0 clears every tab at once; the DEL is what kills the phone and the laptop.
Bearer:
POST /logout Authorization: Bearer <access> {"refresh_token": "..."}
DELETE refresh_tokens WHERE hash = ...
SET deny:<jti> 1 EX <seconds until access exp>
204
Client sets accessToken = null and removes any storage key. Without the deny:<jti> write, the old access token keeps working for up to 10 minutes after logout: that's the gap people miss.
Failure modes
- Cookie: session fixation if the ID isn't rotated on login; Redis down means every request 401s, so decide whether that's an accepted outage or you keep a signed fallback.
- Bearer: logout that only clears client storage leaves the token valid until expiry; refresh tokens without rotation let a stolen one be replayed silently.
- Both: a token or session ID in a URL leaks through Referer and access logs.
How to check it
Console: document.cookie shows no session ID. A cross-site POST returns 403. After logout, redis-cli GET sess:abc123 returns nil, and curl -H "Authorization: Bearer <old>" returns 401.
Curated: · Written: · Reviewed:
Q&A-6A fetch works in a terminal and fails in the browser. What should the API advertise, and what must the page stop doing as a workaround?
For CORS as a contract not a workaround, I would name the contract both halves must keep before choosing a library.
CORS is the server telling the browser which origins may read a response, not a client bug to proxy away. Echoing an unvalidated Origin header is an open door. A star with credentials is blocked by the browser and is not the usual leak.
Concretely, return an exact allowlist origin that you compared to the request Origin. Handle the preflight on the server. Never copy an unknown Origin into the allow-origin header, and never use a star when the client sends cookies.
The reason for that specificity is a failure I have seen: An API echoed any Origin it was given, 312 unknown sites could read account JSON with the user cookie, and a later proxy workaround added 47 milliseconds to every browser call without closing the hole.
What the browser actually checks.
| Server header | Cookie request from product origin | Cookie request from unknown origin |
|---|---|---|
| star, credentials on | browser blocks | browser blocks |
| exact origin, credentials on | allowed | blocked |
| exact origin, credentials off | cookies omitted | blocked |
I would not consider it settled without evidence: Inspect the preflight and the GET from the real origin and from a foreign origin, and confirm only the product origin receives the body.
Name the origin on the API rather than hiding the browser rule behind a proxy.
Curated: · Written: · Reviewed:
Q&A-97The same POST succeeds in curl and fails in the browser with a CORS error while the API returns 200. Which request did the browser make first, what must that response carry, and what changes once the fetch also sends cookies?
I would start CORS preflight and credentialed requests from the screen the user sees and the row the database actually wrote.
A CORS error is the browser refusing to hand your page a response it is not allowed to read; the API accepted the request and logged a normal 200. curl, Node and Python have no same-origin policy, so they never send a preflight and never see the failure, and any call with a JSON body, a custom header, or a verb other than GET or POST is preceded by an OPTIONS preflight the API must answer correctly.
Concretely, trace the OPTIONS round trip in the network panel before the real request, and answer it with the allowlisted Origin echoed back, Access-Control-Allow-Methods and Access-Control-Allow-Headers covering exactly what the page sends, and an Access-Control-Max-Age so the check is not repeated per call. Then repeat the same origin on the real response with Vary: Origin, and when the fetch uses credentials, send Access-Control-Allow-Credentials: true and a concrete origin instead of a wildcard.
The reason for that specificity is a failure I have seen: A team silenced a preflight failure by returning Access-Control-Allow-Origin: * together with Access-Control-Allow-Credentials: true; the browser still blocked the read because a wildcard is never valid with credentials, so the next patch echoed whatever Origin arrived. Every curl check and every server test stayed green for three weeks while any site could read authenticated /orders responses out of a logged-in browser session: the fix for a blocked read became a cross-origin data leak.
What each request must carry.
| Page request | Preflight? | Response headers required |
|---|---|---|
| GET /orders, no custom header | none | ACAO: https://app.example.com, Vary: Origin |
| POST /orders, Content-Type: application/json | OPTIONS first | + Allow-Methods: POST, OPTIONS; Allow-Headers: content-type, authorization |
| same call with credentials: 'include' | OPTIONS first | + Allow-Credentials: true, exact origin, never * |
Hypothetical, 80 ms RTT: the preflight adds 80 ms before the POST lands. Access-Control-Max-Age: 600 turns that into 80 ms per 10 minutes per browser instead of 80 ms per call.
I would not consider it settled without evidence: Show the OPTIONS response and the real response side by side and confirm both hit the same allowlist middleware, then replay the request with Origin: https://evil.example.com and confirm no allow header comes back for it.
CORS is the browser's permission slip to read the response; curl never needed one.
Curated: · Written: · Reviewed:
Q&A-5If the session is a cookie, how do the page and the API stop another site from firing a write the browser will happily credential?
The 90-second version. Two independent gates, and you need both. First, scope the cookie so the browser won't attach it to a cross-site write: SameSite=Lax (or Strict) on the session cookie. Second, require a value the attacking origin cannot read or set: a per-session CSRF token echoed into the page and returned in a header, plus a server-side Origin check. SameSite alone fails the moment you need SameSite=None for a cross-site embed; a token alone fails if you forget it on one endpoint. Neither is a substitute for the other.
Why each gate works. SameSite=Lax means the cookie rides along only on same-site requests and top-level GET navigations. A cross-site POST from evil.example arrives with no session cookie, so the handler sees an anonymous request and rejects it. The token gate covers the case where the cookie is sent: the attacker's page can make the browser send the cookie, but it cannot read the token out of your HTML (same-origin policy) and cannot set a custom header on a cross-origin request without triggering a CORS preflight your server won't approve.
// Node 20, Express 4. Gate every cookie-authenticated write.
import crypto from 'node:crypto';
const SAFE = new Set(['GET', 'HEAD', 'OPTIONS']);
const ALLOWED_ORIGINS = new Set(['https://app.example.com']);
function csrfGate(req, res, next) {
if (SAFE.has(req.method)) return next(); // state changes are never GET
const origin = req.get('origin');
if (!origin || !ALLOWED_ORIGINS.has(origin)) {
return res.status(403).json({ error: 'bad_origin' });
}
const sent = req.get('x-csrf-token') || '';
const expected = req.session.csrfToken || '';
const a = Buffer.from(sent), b = Buffer.from(expected);
if (a.length !== b.length || !crypto.timingSafeEqual(a, b)) {
return res.status(403).json({ error: 'bad_csrf' });
}
next();
}
// Session cookie: explicit SameSite, __Host- prefix (requires Secure, Path=/, no Domain).
res.cookie('__Host-session', sid, {
httpOnly: true, secure: true, sameSite: 'lax', path: '/'
});
The token is generated per session, stored server-side against the session id, and rendered into the page (<meta name="csrf-token" content="...">); the client copies it into X-CSRF-Token on every write.
Trace: a cross-site form POST to /settings/email.
| Step | Browser sends | Server does |
|---|---|---|
<form action="https://api.example.com/settings/email" method="POST"> on evil.example | Origin: https://evil.example, no session cookie (Lax) | 401, no session |
Same, but cookie is SameSite=None | Origin: https://evil.example, Cookie: __Host-session=…, no X-CSRF-Token | 403 bad_origin |
fetch() with X-CSRF-Token from evil.example | OPTIONS preflight; server returns no Access-Control-Allow-Origin for that origin | Browser blocks; the POST is never sent |
Failure modes worth naming. A state-changing GET (/email/change?to=…) is exploitable under Lax, because top-level navigation does send the cookie: keep writes on POST/PATCH/DELETE. A token that isn't bound to the session lets an attacker fetch their own token and pair it with the victim's cookie. Double-submit cookies without an HMAC signature fall to cookie tossing from a sibling subdomain. A CORS policy that reflects Origin with Access-Control-Allow-Credentials: true collapses the custom-header gate entirely. And Chromium sends Lax-by-default cookies on cross-site top-level POSTs for two minutes after they're set if SameSite was never specified: set it explicitly.
Trade-offs. SameSite=Strict logs users out on inbound links; Lax is the usual default. SameSite=None is unavoidable for cross-site iframes, and then the token is your only gate. Per-session tokens are simpler than per-request; per-request needs server-side storage and rotation.
Verify it. curl -X POST https://api.example.com/settings/email -H 'Origin: https://evil.example' -H 'Cookie: __Host-session=<real>' must return 403, and the same call from the first-party page with the token must return 200.
Curated: · Written: · Reviewed:
Q&A-71Users look logged in on www and logged out on data. What is wrong with the session cookie?
What a strong candidate says first
The session cookie is host-only on www. A cookie set without a Domain attribute is stored with the host-only flag and sent only to the exact host that set it (RFC 6265 §5.3–5.4). So www.example.com receives it and data.example.com never does. The page renders from the cookie and looks signed in; every request to the data host goes out anonymous and the API answers 401. Two fixes, and one trap.
Mechanism
Scope is fixed at Set-Cookie time, not at request time. The browser compares the request host against the cookie's domain and host-only flag before attaching it.
Set-Cookie line | Sent to www.example.com | Sent to data.example.com |
|---|---|---|
sid=abc; Secure; HttpOnly (no Domain) | yes | no |
sid=abc; Domain=example.com; Secure; HttpOnly | yes | yes |
__Host-sid=abc; Secure; Path=/ | yes | no |
The trap is the third row. The __Host- prefix requires Secure, Path=/, and no Domain attribute, so a Host-prefixed cookie can never be shared across hosts. You cannot have both.
The second cause people miss
Even with Domain=example.com, a cross-origin fetch() from www to data defaults to credentials: 'same-origin', so the browser strips cookies from the request. You need credentials: 'include' on the client and, on the server, Access-Control-Allow-Credentials: true with an exact Access-Control-Allow-Origin (never *). SameSite is not the culprit here: www.example.com and data.example.com share the registrable domain example.com, so they are same-site and a Lax cookie is still sent. It becomes the culprit only if the data host sits on a different registrable domain.
Trace
GET https://www.example.com/→ request carriesCookie: sid=abc123. Page renders "Signed in as Dana".- Page JS runs
fetch('https://data.example.com/me')with default credentials → request carries noCookieheader. Server returns401. UI flips to "logged out". - After
Set-Cookie: sid=abc123; Domain=example.com; Secure; HttpOnly; SameSite=Laxandfetch('https://data.example.com/me', {credentials: 'include'})→ step 2 now carriesCookie: sid=abc123and returns200.
Trade-off
A parent-domain cookie widens the blast radius. HttpOnly stops JS from reading it, but every subdomain can still set a cookie for example.com (cookie tossing, session fixation) and every subdomain's server receives the cookie on inbound requests. A compromised marketing or user-content subdomain can therefore harvest sessions from its own access logs. Widen only when the hosts genuinely must split.
Verification
DevTools → Network → select the document request and the XHR, and read the Cookie request header on each. Confirm the Domain attribute in the Set-Cookie response header. From a shell: curl -v -H 'Cookie: sid=abc123' https://data.example.com/me and check whether the response is 200 or 401.
When not to widen
If you control both hosts, put the session behind a same-origin BFF: the browser talks only to www.example.com, the BFF holds the token server-side and calls the data service internally. No Domain attribute, no CORS, no credentials: 'include', and the cookie stays host-only and __Host--prefixed.
Curated: · Written: · Reviewed:
Q&A-82How do clickjacking and cookie-based auth interact on a full-stack site?
The 90-second answer. Cookie auth makes clickjacking cheap, because the attacker never has to read your session cookie: only to make the victim's browser send it. A framed page is a same-origin request from the browser's point of view, so the cookie rides along and the side effect lands even though the attacker cannot read the response. The control is a response header on the HTML document: Content-Security-Policy: frame-ancestors 'none'. SameSite is a CSRF control, and it is scoped to the registrable domain, not the origin, so it is not the framing control.
Why the two interact. Clickjacking is not a forged request; the user really does click. That is why CSRF tokens alone do not cover it, but they do help, because the attacker cannot read your token out of the framed page (same-origin policy), so a token-protected POST cannot be assembled inside the overlay. The dangerous shape is a state-changing GET or a form the attacker can pre-fill. SameSite=Lax (Chrome's default since Chrome 80, Feb 2020) stops the cookie on cross-site subresource and iframe requests, but app.example.com framed by evil.example.com is same-site, so the cookie is sent. SameSite=None; Secure, required for legitimate cross-site embedding, sends it everywhere.
Headers, on every HTML response.
// Express 4.x — mount before routes so error pages get it too
app.use((req, res, next) => {
res.setHeader('Content-Security-Policy', "frame-ancestors 'none'");
res.setHeader('X-Frame-Options', 'DENY'); // legacy browsers only
next();
});
app.use(session({
name: '__Host-sid', // Secure + Path=/ + no Domain, enforced by the prefix
cookie: { httpOnly: true, secure: true, sameSite: 'lax', path: '/' }
}));
frame-ancestors is ignored in a <meta> tag, and when both headers are present the CSP directive wins and X-Frame-Options is ignored, so a stray frame-ancestors * silently re-enables framing. X-Frame-Options: ALLOW-FROM is dead in Chrome.
| Control | Cookie sent in cross-site iframe | Frame renders | Gap |
|---|---|---|---|
SameSite=Lax | no | yes | same-site subdomain frames still send it |
SameSite=None; Secure | yes | yes | needed for real cross-site embedding |
frame-ancestors 'none' | yes: request already made | no | must be on every HTML response |
X-Frame-Options: DENY | yes | no | ignored when frame-ancestors is present |
top !== self JS check | yes | until script runs | defeated by sandbox without allow-top-navigation |
Trace. Victim signed in on app.example.com, visits evil.com, which frames /settings.
GET /settings HTTP/1.1
Host: app.example.com
Cookie: (absent — SameSite=Lax, cross-site subresource)
Sec-Fetch-Dest: iframe
Sec-Fetch-Site: cross-site
With SameSite=None the Cookie: __Host-sid=... line is present and the request executes server-side; the browser then refuses to render, logging "Refused to display ... frame-ancestors 'none'". That is the key asymmetry: frame-ancestors blocks rendering, not the request. Any state-changing GET has already run.
Failure modes and detection. Header set at the CDN for static assets but not on app-rendered 500s: assert the header on every HTML route including 4xx/5xx. A permissive frame-ancestors shipped alongside X-Frame-Options: DENY: parse the effective policy, not the presence of a header. SameSite=Lax assumed to cover subdomain framing: it does not. Detection is a headless-browser test that loads each HTML route inside a cross-origin iframe and asserts the frame is blocked, plus a report-to endpoint for frame-ancestors violations.
When not to use 'none'. Partner portals and embedded checkout need frame-ancestors https://partner.example.com, which forces SameSite=None; Secure and means the cookie now flows cross-site: pair it with synchronizer CSRF tokens, Origin/Sec-Fetch-Site checks on state-changing endpoints, and step-up re-auth for password or payment changes. Sec-Fetch-* is a useful signal in Chromium and Firefox; treat it as defence in depth, never as the control.
Curated: · Written: · Reviewed:
Q&A-20The user logs out in one tab while two others stay open. What must the API destroy, and how do the other tabs find out?
What to destroy
Logout has to kill the server-side session, not just the cookie:
- Delete the session row (or revoke the refresh token) so the credential can't be replayed.
- Clear the session cookie on the logout response (
Set-Cookie: ...; Max-Age=0). - On stateless JWTs there is no row to delete. You need a revocation list keyed by
jti, or a per-usersession_versionbumped on logout and checked on every request, plus refresh-token revocation. A short access-token TTL alone leaves a window.
The cookie is shared across every tab in the browser profile, so clearing it in tab A clears it for B and C too. What is not shared is each tab's in-memory user object and rendered shell.
How the other tabs find out
Two mechanisms, and you want both:
- Broadcast.
localStorage.setItem('logout', Date.now())fires astorageevent in every other same-origin tab, not in the tab that wrote it.BroadcastChannelis the cleaner API (Safari 15.4+). On receipt, drop in-memory user state and render signed-out. - Revalidate. On
visibilitychange→ visible, and onfocus, callGET /api/me. A 401 means the session is gone regardless of what the broadcast did or didn't deliver.
The broadcast is a UX optimization; the server check is the correctness guarantee. A tab that missed the event (suspended, backgrounded, restored from bfcache) still converges the moment it is looked at.
Three tabs, one session row
| After logout in tab A | Server session | Tab B write | Tab B UI |
|---|---|---|---|
| cookie cleared only | still valid | 401 | still paints the name |
| session row deleted, no broadcast | gone | 401 | still paints the name |
| row deleted plus storage event | gone | 401 | signed-out |
Failure modes
- Broadcast never arrives: the tab was frozen (backgrounded Chrome, mobile Safari) or the event fired before the listener mounted. Detection: the tab still shows a name after
visibilitychange. Fix: the revalidate-on-visible path. - Write races the broadcast: tab B POSTs in the milliseconds before it processes the event. The server must reject on the session check: never trust the client's painted state. Detection: 401s in logs from sessions revoked moments earlier.
- Stateless JWT with no revocation: logout clears the cookie but the token stays valid until expiry. Detection: replay the captured token against a protected endpoint after logout; a 200 means you have no revocation.
- Multiple browser profiles or devices:
storageevents don't cross profiles. Only server-side revocation covers those.
Verification
Log out in tab A. From tab B, POST a write and confirm 401. Confirm tab B dropped its in-memory user and renders signed out. Then background tab B, log out again from A, foreground B, and confirm the visibilitychange revalidation catches it.
Curated: · Written: · Reviewed:
Q&A-58A user signs in, closes the tab, and comes back the next day still signed in, but a token copied off that device must not work anywhere else. How does the full stack hold both of those?
What separates a strong answer on session persistence versus stolen tokens is knowing which half can lie to the other.
Persistence and theft resistance pull against each other: the credential that survives a tab close is exactly the one an attacker wants to copy. Put the long life in a server-side session or a rotating refresh credential the page cannot read, and let the browser hold only a short-lived access token that expires before a leak becomes an incident.
Concretely, keep the session in a server store, a Redis key or a sessions table, with the cookie carrying only an opaque id marked HttpOnly, Secure and SameSite, and hand the page a 15-minute access token while the refresh credential rotates on every use. Treat a replayed refresh token as theft: revoke the whole token family and force both devices back to sign-in.
The reason for that specificity is a failure I have seen: A team issued a 90-day JWT into localStorage so users stayed signed in; a script injected through a third-party chat widget read it, and the same token drove 610 admin calls from another country over 12 days until the signing key was rotated and every session in the product died at once.
One family, one replay, both devices signed out.
| Time | Event | Server state |
|---|---|---|
| 09:00 | sign-in on laptop | family f1 created, refresh R1, access A1 (15 min) |
| 09:20 | tab reopened | A1 expired, R1 consumed, R2 issued |
| 09:22 | attacker replays stolen R1 | reuse detected on f1, family revoked |
| 09:22 | laptop calls API with A1 | 401, forced to sign in again |
I would not consider it settled without evidence: Sign in, close the tab, reopen after a day and confirm the page can read no long-lived secret, then replay yesterday's refresh token and confirm both the laptop and the attacker's device land on 401.
Long life belongs to the server store; the browser only borrows fifteen minutes at a time.
Curated: · Written: · Reviewed:
Q&A-102A crafted link and a third-party script both make a logged-in user's browser fire a state-changing request. Which defense stops each, and which layer (browser, page, or API) has to own it?
Two attacks, two owners. CSRF is the API's problem; XSS is the page's problem. A crafted link and a third-party script both end the same way: the browser sends a credentialed request to your API. The API can't tell that request from a real one unless it checks something the attacker's page can't produce, and that something is never the cookie, because the cookie is exactly what the browser attaches for the attacker.
Assume the session is an HttpOnly, Secure cookie, the SPA is on app.example.com and the API on api.example.com, so SameSite treats them as same-site.
CSRF. SameSite=Lax on the session cookie stops cross-site POST/PUT/DELETE, the cookie isn't sent, but it still allows top-level GET navigation. So never mutate on GET. Lax is the default in Chrome 80+ and most browsers now, but don't lean on it alone: it's a browser control, and "same-site" means registrable domain, so a sibling subdomain you don't control is same-site. The API has to require a token on every state-changing request: a synchronizer token stored in the session and echoed in a header such as X-CSRF-Token. A cross-origin form can't set a custom header, and a cross-origin fetch that tries triggers a CORS preflight the API can refuse. Check Origin and Sec-Fetch-Site too: Sec-Fetch-Site: same-origin is set by the browser and page JS can't forge it.
// API: state-changing routes only
app.use((req, res, next) => {
if (['GET', 'HEAD', 'OPTIONS'].includes(req.method)) return next();
const site = req.get('Sec-Fetch-Site');
if (site && site !== 'same-origin' && site !== 'same-site') return res.sendStatus(403);
if (req.get('X-CSRF-Token') !== req.session.csrf) return res.sendStatus(403);
next();
});
The page's entire CSRF job is to read the token from a non-HttpOnly cookie or a meta tag and echo it in that header.
XSS. If any script runs in your origin, it reads the CSRF token and calls the API with the user's cookies. Every defense above is bypassed. That's why XSS outranks CSRF. The page owns it: never put untrusted data into innerHTML, dangerouslySetInnerHTML or document.write. Use textContent or the framework's escaping. Ship a CSP with nonces and no unsafe-inline, and Trusted Types where supported. Third-party scripts are same-origin code; a compromised analytics vendor is XSS with a supply-chain wrapper. Use SRI on script tags, keep script-src to an allowlist, self-host where you can. The API owns its half: correct Content-Type and Content-Disposition, never reflect untrusted input into an HTML or JS context, and keep the session cookie HttpOnly so a script can't exfiltrate the token itself. It can still act as the user, but it can't walk away with the credential.
| Attack | What the browser does | Owner | Defense |
|---|---|---|---|
| Crafted link / cross-site form POST | Attaches the session cookie | API | SameSite=Lax + CSRF token header + Origin/Sec-Fetch-Site check |
| Compromised third-party script | Runs as same-origin code | Page | CSP nonce, SRI, no innerHTML, Trusted Types |
| Reflected input rendered as HTML | Renders attacker markup | API + Page | Output encoding at render, correct Content-Type |
Failure modes worth naming: SameSite=None without Secure is rejected outright by browsers; Lax still sends the cookie on a top-level GET, so a GET that mutates is exploitable; a double-submit cookie is bypassable if an attacker can set cookies on a sibling subdomain; a CSP with unsafe-inline or a wildcard script-src buys nothing; SRI hashes break silently when the vendor rotates the file. Detect by alerting on state-changing requests with a missing or foreign Origin/Sec-Fetch-Site, and by wiring CSP report-to.
Curated: · Written: · Reviewed: —
Q&A-65Fetches fail in the browser after a CSP deploy, and the API logs are empty. What did the policy miss?
The policy's connect-src is missing the API origin. The browser blocks the fetch before it opens a socket, so the request never reaches the API and the access log stays empty. Empty API logs plus a console violation is the signature: the block happened in the client, not on the server.
Confirm it in 30 seconds. The console names the directive:
Refused to connect to 'https://api.example.com/v1/orders'
because it violates the following Content Security Policy directive:
"connect-src 'self'"
Chrome's Network panel marks that row (blocked:csp) with no status code. If you instead see "blocked by CORS policy", the request did leave the browser: the API log would show the preflight OPTIONS, and the fix is server-side.
| Symptom | Where it broke |
|---|---|
| Console "Refused to connect", API log empty | CSP connect-src |
Console "blocked by CORS policy", API log shows OPTIONS | CORS |
| API log shows the request, 401/403 | Auth |
Why it's easy to miss: connect-src has a fallback. A policy that sets only default-src 'self' and never mentions the API still governs fetches, because connect-src inherits from default-src when absent. And connect-src covers more than fetch: XHR, WebSocket (wss://), EventSource and sendBeacon all fall under it, so a chat socket and a telemetry beacon break at the same moment as the API calls.
The fix:
Content-Security-Policy:
default-src 'self';
connect-src 'self' https://api.example.com;
report-to csp-endpoint
Details that bite:
https://*.example.commatches subdomains but nothttps://example.comitself.- Scheme, host and port must match exactly;
https://api.example.com:8443is a different origin from the default port. - Redirects are checked too, so an allowed origin that 302s to a host outside
connect-srcstill fails. - If the API is served under the same origin through a reverse proxy,
'self'covers it and no extra entry is needed: worth considering if the origin list keeps growing.
Roll it out under Content-Security-Policy-Report-Only with a report-to endpoint first. Report-only never blocks, so if fetches are failing, the deployed header is already enforcing. Violations land at the reporting endpoint, not in the API log: the second reason that log is empty.
Verify by loading the app under the enforcing policy and checking for zero connect-src violations in the console while a write returns 2xx in the Network panel.
Curated: · Written: · Reviewed:
Q&A-70The site is now on HTTPS. Checkout still fails in the console. What is left to fix?
Requiring TLS on the origin doesn't fix mixed content: the browser decides that per request, inside the page. Two behaviours matter:
- Active mixed content (scripts,
fetch/XHR, iframes, stylesheets, fonts) is blocked outright. The request never leaves the browser. - Passive mixed content (images, audio, video) is auto-upgraded to https since Chrome 86 and Firefox 127; if the https URL then fails, it's blocked.
So checkout failing after the cutover means at least one subresource or API call is still http://. Find it in the console:
| Browser | Console string |
|---|---|
| Chrome | Mixed Content: The page at 'https://shop.example/' was loaded over HTTPS, but requested an insecure resource 'http://api.example/checkout'. This request has been blocked. |
| Firefox | Blocked loading mixed active content 'http://api.example/checkout' |
To catch the ones you can't reproduce, ship a report-only policy first:
Content-Security-Policy-Report-Only: block-all-mixed-content; report-uri https://shop.example/csp-report
That reports every mixed-content violation without blocking, so you get the full production list before you enforce.
The decision that matters: rewrite vs upgrade
Content-Security-Policy: upgrade-insecure-requests rewrites every http subresource request to https at the browser, before it's sent. One header, and it covers URLs you can't edit: third-party SDKs, strings built in JS, and HTML already sitting in caches, provided the header is added at the edge so it rides along with the cached response. The cost: it hides the http URL instead of fixing it, so if the target has no https listener the request now fails as a TLS or connection error rather than a mixed-content block. You've moved the failure, not removed it. It also doesn't cover every scheme, so verify WebSockets and any non-HTTP endpoints yourself.
Rewriting the URLs is honest and permanent, but you own every occurrence: markup, JS, CSS url(), service-worker caches, and any cached HTML. That's the real reason a cutover "finishes" late.
Cached HTML: purge or wait
If the HTML that embeds http is in a CDN or browser cache, the fix isn't live until that copy is gone. Purge is immediate but you must hit every edge and every variant (Vary, gzip/br, locale). Waiting for TTL is safe but the window is the longest TTL on the HTML: fine at max-age=300, not at max-age=31536000, immutable, where you must purge or version the URL. Adding upgrade-insecure-requests at the edge sidesteps both, because it applies to the cached response too.
What an edge redirect does and doesn't do
A 301/308 from http to https at the CDN fixes navigations and passive content the browser still sends. It does nothing for active mixed content: the browser blocks the request before it is sent, so the redirect is never reached. Don't count it as the fix.
HSTS (Strict-Transport-Security: max-age=31536000; includeSubDomains; preload) is worth adding, but it governs navigations to your own origin: it doesn't rewrite http URLs in your markup.
Verify
Load checkout over https with the console open and the network tab filtered to http. Zero mixed-content warnings, zero http requests, and the report-only endpoint quiet. Then flip the policy to enforcing.
Curated: · Written: · Reviewed:
Q&A-1After a cart write, which store is a refresh allowed to trust, and how do you keep the page and the database from disagreeing?
Assumption: one user, one cart, writes go through an API that owns the cart rows; the page is a browser tab that can be reloaded at any moment.
A refresh may trust only the server. The client store is a working copy that exists to make the UI feel instant; it is never the thing a reload reads. So durable cart facts (line items, quantities, prices, ownership) live in the database, and the write response returns the authoritative cart so the client replaces its copy instead of guessing.
Mechanism, with TanStack Query v5 as the concrete client cache:
// TanStack Query v5
const cartKey = (userId: string) => ['cart', userId] as const;
useMutation({
mutationFn: (line: CartLine) => api.putCartLine(line), // returns the full cart
onMutate: async (line) => {
await queryClient.cancelQueries({ queryKey: cartKey(userId) });
const prev = queryClient.getQueryData<Cart>(cartKey(userId));
queryClient.setQueryData(cartKey(userId), (c) => applyLine(c, line));
return { prev };
},
onSuccess: (serverCart) => queryClient.setQueryData(cartKey(userId), serverCart),
onError: (_e, _line, ctx) => queryClient.setQueryData(cartKey(userId), ctx?.prev),
onSettled: () => queryClient.invalidateQueries({ queryKey: cartKey(userId) }),
});
Trace of one "+1 on line A":
onMutatecancels the in-flight['cart', userId]query, snapshots the previous cart, and writes the optimistic cart. UI shows quantity 2 immediately.PUT /cart/lines/Ahits the primary. The server recomputes price from the catalog and returns the full cart.onSuccessreplaces the cache with that response, not with the optimistic guess.onSettledinvalidates['cart', userId]; the next mount refetches. (v5 defaults arestaleTime: 0,gcTime: 5 min, so a refetch on mount is already the default: the invalidation is what forces it right after a write.)- A reload issues
GET /cartand reads the row. If that GET lands on a replica lagging behind the primary, it can return quantity 1 even though the write committed. That is read-after-write, not a client-cache bug.
Read-after-write: async replicas in the same region typically lag single-digit to low-tens of milliseconds, with p99 spikes into the hundreds under load. Two fixes: route the post-write GET to the primary for a short window, or have the write return a version/LSN and let the read wait until the replica has caught up. DynamoDB makes the choice explicit: eventually consistent reads are the default and cost half of strongly consistent ones.
Trade-offs:
| Approach | Latency | Correctness risk | Use when |
|---|---|---|---|
| Optimistic + rollback | instant | rollback flicker, lost update | conflicts rare, latency budget tight |
| Refetch after write | one RTT | stale replica read | correctness beats 100 ms |
| Server-driven (write returns cart) | one RTT | none extra | default for cart writes |
Failure modes and detection:
- Lost update: two tabs write the same line; last write wins silently. Detect with a version/ETag on the cart row and reject mismatches with 409.
- Double-add: a retry after a timeout adds the line twice. Use an idempotency key per mutation.
- Stale replica read: log client total vs server total on every cart response; a nonzero mismatch rate is your signal. A nightly reconciliation job comparing cart totals to line sums catches drift the logs miss.
What would change the decision: if the cart were purely ephemeral (guest, no checkout), client-only state with a server sync on login is fine. Once a price or an ownership claim is billed, the server owns it and the client only mirrors.
Curated: · Written: · Reviewed:
Q&A-30A write succeeds but a refresh still shows the old value. What should the page do?
What I'd say first
Assume the write committed on the primary and the refresh hit a lagging replica or a cache. The page should not treat the 200 as "everyone can see it now". It should carry a version token from the write and paint the value as confirmed only when a read returns that version or newer; until then it keeps the optimistic value, and if the wait exceeds a bound it reads the primary.
Clarify one thing before choosing: is the guarantee per session or per user across devices? Per-session is cheap. Cross-device needs a different mechanism, because the token lives in one browser.
Mechanism
The mutation returns a monotonic token: a per-row version column, or the commit LSN/GTID. The client stores it. Reads carry it. A replica that has not applied it yet either waits (bounded) or the read falls back to the primary. The token has to be comparable across nodes: a wall-clock timestamp from the app is not, because clocks skew; a per-row counter or the storage engine's commit position is.
# Python 3.12, psycopg 3
import time
def save_bio(conn, user_id, text):
return conn.execute(
"UPDATE profiles SET bio = %s, version = version + 1 "
"WHERE user_id = %s RETURNING version",
(text, user_id),
).fetchone()[0] # -> 12
def read_bio(replica, primary, user_id, min_version, deadline_s=2.0):
end = time.monotonic() + deadline_s
while time.monotonic() < end:
v, bio = replica.execute(
"SELECT version, bio FROM profiles WHERE user_id = %s",
(user_id,),
).fetchone()
if v >= min_version:
return bio, v
time.sleep(0.05)
return primary.execute(
"SELECT version, bio FROM profiles WHERE user_id = %s",
(user_id,),
).fetchone()
Trace: save_bio returns 12. The first replica read returns (11, old bio): 11 >= 12 is false, so it sleeps 50 ms and retries. The replica applies the write at t+1.6 s; the next poll returns (12, new bio) and the page paints it. If the replica is still behind at t+2.0 s, the primary read returns (12, new bio) and the page paints that. The user never sees the old value after a successful save.
Options
| Approach | Cost | When it breaks |
|---|---|---|
| Always read primary for the writer's session | primary read traffic, no cache benefit | multi-device: the laptop has no session token |
| Version token + bounded wait, fall back to primary | one extra column or LSN plumbing | token not monotonic across nodes (wall clock) |
| Optimistic UI, never reconcile | cheapest | user sees a value that was later rejected, and you cannot detect it |
| Cache delete-on-write | cheap | delete-then-read race repopulates the stale value |
Failure modes
- Token from the app's clock: skew between nodes makes
v >= min_versionmeaningless. Use the storage engine's commit position or a per-row counter. - Unbounded wait: a replica stuck for minutes pins the request. Bound it and fall back.
- Cache in front of the replica: deleting the key on write races with a concurrent read that repopulates the old value. Version the key or write through.
- Cross-device: the token lives in one browser. A second device needs lag-based routing or an explicit "your change may take a moment" state.
- Duplicate submissions: the user retries because the screen looked stale. An idempotency key on the mutation stops the second save from creating a second row.
Detection: count reads where the returned version is below the session's last written version. That is your read-your-writes violation rate. Alert on replica replay lag p99.
Worked example (observed): a profile save returned 200; a refresh 400 ms later hit a replica 1.6 s behind; 7% of editors re-submitted and created duplicate bios. The version token plus a 2 s bound removes both the stale paint and the retry.
Before calling it done
Delay a replica, save, refresh, and confirm the page holds the optimistic fields until the read version catches up. If it paints the old value at any point, the token is not being carried.
Curated: · Written: · Reviewed:
Q&A-26A product page is still slow after you added a cache. Which layer holds what, and what must the UI do after a write?
What a strong candidate says first
Assume a public product detail page (PDP) with a signed-in variant, and the write is a price or inventory change. Three caches sit in front of the same page and each holds a different thing:
| Layer | Holds | Keyed by | Typical lifetime |
|---|---|---|---|
| CDN / edge | public HTML, images, JS | URL (+ Vary) | 60–300 s |
| App cache (Redis) | session-keyed JSON fragments | user id + fragment | 10–60 s |
| DB | buffer pool, query plan cache | query + params | engine-managed |
A miss at one layer is not a miss at the others: that is why "we added a cache" does not fix a slow page. Find which layer is actually missing: X-Cache: HIT/MISS at the edge, Redis hit ratio, buffer-pool hit ratio. The slow one is usually the layer you did not add.
After a write: purge or version
Both stop the edge serving old bytes; they fail differently.
| Purge (invalidate key) | Version (change key) | |
|---|---|---|
| Mechanism | delete the object at every PoP | write /p/123?v=8, old key ages out |
| Correctness latency | seconds to minutes, per-PoP | immediate at origin; old key still served until TTL |
| Hit rate | preserved for untouched objects | one cold miss per write |
| Failure mode | purge API throttles or partially fails; silent staleness | stale versions accumulate; storage and origin load |
Deciding criteria: write rate against read rate, and whether TTL already bounds staleness.
- Few writes, many reads (catalog copy, images): purge. One API call, hit rate intact.
- High write rate (price or inventory feed): version. At 5,000 SKUs updated every 30 s you would issue ~167 purges/s sustained: that is where purge APIs throttle and where a failed purge is invisible. Versioning turns a correctness problem into a cache-efficiency problem.
- Many derived objects from one write (PDP + listing fragment + image variant): purge by surrogate key / cache tag so one call clears all of them instead of enumerating URLs.
The UI side: the write response returns the new version token, and the client drops its local copy and re-fetches with that token rather than reusing the pre-write URL. Optimistic render is fine if you reconcile on the response: the bug is rendering the new price while the edge still serves the old one to the next request.
When not to cache, and when TTL-only is fine
Do not cache anything where a stale read causes a wrong charge or an oversell: cart contents, checkout totals, the inventory number that gates "add to cart". Keep those at the app layer with a short TTL and revalidate on read, or leave them uncached.
TTL-only with no purge path is right when staleness is bounded and harmless: category navigation, marketing copy, image derivatives. Set TTL to the business tolerance (5 min for nav) and delete the invalidation code. It is the cheapest correct option and it removes a whole class of purge-failure bugs.
Failure modes and how you see them
- Purge API throttling or partial propagation: alert on non-2xx from the purge call, and probe the object from two regions after every deploy.
- Missing
Varyor cache-key collision: a personalized fragment cached under a public URL. Detect by fetching as anonymous and as signed-in and diffing. - Cold-key stampede after a version bump: origin QPS spike on the new key. Mitigate with request coalescing or
stale-while-revalidate.
A real one: a homepage fragment with a 30-minute edge TTL kept showing 49 after the price was written to 39. The purge fired but reached the PoP serving that region at minute 27; 18,000 shoppers saw the stale price in between. The fix was not a shorter TTL: it was versioning the fragment key so the write never depended on purge propagation.
How I would verify
After a price write, fetch the PDP as an anonymous client and as a signed-in client from two regions and confirm both show the new price within the stated bound. Then check the edge hit ratio did not collapse.
Curated: · Written: · Reviewed:
Q&A-2When can the page apply a click before the API answers, and what must both halves do if that guess is wrong?
The 60-second answer. Apply the click before the API answers when the client can compute the next state deterministically and the server has no rule that could reject it. Do not guess when the server assigns the value (IDs, prices, balances, inventory) or when the write is irreversible. When you do guess, the client owns the rollback and the server owns idempotency: the client keeps the last server-confirmed body and reverts to it on rejection, and the server dedupes the write on a client-supplied key so a retry after a timeout cannot apply it twice.
Assumptions I would state out loud: one user, one device, no offline queue; the write is a like on a post; the server is the only source of truth after a refresh.
What each half must do.
Client:
- Hold
confirmed, the last body the server returned, separately fromview, what is rendered. - Generate one idempotency key per user intent and reuse it on every retry of that intent.
- Tag each in-flight write with a monotonic sequence number and drop any response older than the newest one already applied.
- On 2xx, replace
viewwith the response body, not with the guess. - On a business rejection (409/422), set
view = confirmedand offer retry. On 5xx or timeout the outcome is unknown: retry with the same key instead of rolling back, because the write may have landed.
Server:
- Store the key with the result; a repeat of the same key returns the stored response and does not re-apply.
- Return the full authoritative body on success so the client reconciles without a second read.
- Separate "rejected" (409/422, do not retry blindly) from "unknown" (5xx, timeout, retry is safe).
Pseudocode (language-agnostic; the semantics are the point):
confirmed = { likes: 41 } // last server body
view = { likes: 41 } // rendered
seq = 0
appliedSeq = 0
onLikeClick():
key = newKey() // reused for every retry of this intent
mySeq = ++seq
view = { likes: confirmed.likes + 1, pending: true }
render(view)
send({ op: 'like', key }).then(res):
if mySeq < appliedSeq: return // stale response, drop it
appliedSeq = mySeq
if res.status == 200: confirmed = res.body; view = res.body
else if res.status in (409, 422): view = confirmed; showRetry(key)
else: showRetry(key) // unknown, keep the guess
render(view)
Trace, one click: confirmed.likes = 41, view renders 42 pending. Server returns 200 with {likes: 42}. confirmed = 42, view = 42. Guess and truth agree.
Trace, two clicks before the first resolves: click A sets view 42, key k1, mySeq 1. Click B sets view 43, key k2, mySeq 2. B's response arrives first with {likes: 43} → appliedSeq 2, confirmed 43, view 43. A's response then arrives with {likes: 42} → mySeq 1 < appliedSeq 2, dropped. Without that check the heart visibly un-fills after the user's second like.
Trace, timeout then retry: click sets view 42, key k1. The request times out but the write landed. Retry with k1. The server sees k1 already stored, returns the original 200 {likes: 42} without incrementing again. Client sets confirmed 42. A fresh key here would produce 43: that is the double-apply bug.
| Operation | Optimistic? | Why |
|---|---|---|
| like, follow, star | yes | client computes the next state; server accepts unless already in that state |
| create with server-assigned id | no | client cannot guess the id; render a placeholder row keyed by the idempotency key |
| payment, transfer, inventory decrement | no | server-assigned value plus a rejection rule the client cannot evaluate |
| edit a field the server validates | yes, with rollback | client computes the next state; server may reject on validation |
Failure modes and how to detect them.
- Double-apply on retry: count rows for one idempotency key; the answer must be 1.
- Stale response overwriting newer state: assert
appliedSeqis monotonic in client logs; the symptom is a control that flips back on its own. - Rollback clobbering a newer success: capture
confirmedat rollback time, not at click time, or a concurrent success gets erased. - Silent rejection: a client that ignores 409 keeps a state the server never stored. Hypothetically, in a feed with 2,600 concurrent likers, every one of them ends up with a like that vanishes on refresh and a permalink that 404s. Detection: after every settled write, assert the rendered state equals the last server body.
The click may guess; the row may not. Force the write to fail and confirm the rendered state equals the last server body: that test is what separates optimism from a lie.
Curated: · Written: · Reviewed:
Q&A-9The page wants an id before the create returns. Which identifier may the UI mint, and which identifier must later updates use?
What I'd say first. The UI mints an idempotency key, not an identity. Send it on the create; the server mints the canonical id and returns it; every later GET, PATCH and DELETE uses that server id. The client id is scoped to one intent and one owner, and it stops being useful the moment the create returns.
Assumptions: the create is a POST that can be retried (mobile network, timeout, double-tap), the resource is owned by a user, and the server is the authority on whether the row exists.
Mechanism. The UI generates a UUIDv4 at the moment of intent, before the request goes out, and sends it as Idempotency-Key or a body field. The server authorizes the caller, then inserts with a unique constraint on (owner_id, client_id):
-- Postgres 16
CREATE TABLE lists (
id uuid PRIMARY KEY, -- server id, app-minted UUIDv7
owner_id uuid NOT NULL REFERENCES users(id),
client_id uuid NOT NULL, -- UI-minted
title text NOT NULL,
UNIQUE (owner_id, client_id)
);
INSERT INTO lists (owner_id, client_id, title)
VALUES ($1, $2, $3)
ON CONFLICT (owner_id, client_id) DO NOTHING
RETURNING id;
A returned row is a fresh create (201). No row means this key already landed: SELECT id FROM lists WHERE owner_id=$1 AND client_id=$2 and return 200 with the same id. Either way the client ends up holding one server id.
Trace.
- Click. UI mints
client_id = 3f2a…(v4). POST /lists {client_id: 3f2a…, title}→ authorized → INSERT returnsid = 018f…(server, UUIDv7). 201.- Response lost. UI retries the same
3f2a…→ conflict → SELECT → 200 with018f…. One row. PATCH /lists/018f…→ 200.PATCH /lists/3f2a…→ 404.
Why not let the client mint the resource id. You can, some systems use a client-generated UUIDv7 as the primary key, but then the id is only as trustworthy as the client, and you still have to authorize and validate it. The idempotency-key design keeps the two jobs separate: the client id answers "have I already sent this?", the server id answers "which row is this?". A bigint sequence is the other server-id option; it is smaller and index-friendly but leaks row volume and needs a round trip before the client can address anything.
Failure modes.
- Client id used as the primary key with no owner check. A second account replays an intercepted id and writes to the first user's row. Detect with an audit assertion that every write's actor equals the row owner, and alert on the 409 rate.
- Key reused for a different payload. Two tabs, same key, different title. Store a hash of the request body alongside the key and return 422 on mismatch; otherwise the second intent silently returns the first row.
- Client keeps addressing by the minted id. PATCHes 404 once the server id differs. Detect with a 404 rate on PATCH and a client-side assertion that the path id came from a create response.
- Unique constraint on
client_idalone. Two users who coincidentally mint the same UUID collide, and a leaked id becomes a cross-tenant probe. Scope it to the owner.
How I'd settle it. Create twice with one key, assert one row and one id; PATCH with the server id, assert 200; PATCH with the client id, assert 404.
Curated: · Written: · Reviewed:
Q&A-8A user double-clicks checkout and a timeout retries. How do the form and the API make that still one order?
Two different events arrive as the same write. A double click sends the request twice from the browser. A timeout retry resends because the client never saw a response. Both must land on one order.
Assumptions: the endpoint is a POST that inserts, there is one primary database, and the client can generate a UUID before the first request. If any of those is false, the trade-offs below change.
What a strong candidate says first. Generate the key once per form instance (when the form renders, not when the submit handler fires) and send it on every attempt for that form. Disable the button while a request is in flight. On the server, persist the key with the first response and return that stored response for any repeat.
The client half is small:
// Browser, no framework
const key = crypto.randomUUID(); // once, at form render
form.addEventListener('submit', async (e) => {
e.preventDefault();
const btn = form.querySelector('button');
if (btn.disabled) return;
btn.disabled = true;
try {
await fetch('/checkout', {
method: 'POST',
headers: {'Content-Type': 'application/json', 'Idempotency-Key': key},
body: JSON.stringify(payload),
});
} finally {
btn.disabled = false;
}
});
The server half is where the guarantee lives. Insert the key first; a unique violation means this is a replay.
# Python 3.12, Flask 3.0, PostgreSQL 16
from hashlib import sha256
@app.post("/checkout")
def checkout():
key = request.headers["Idempotency-Key"]
body = request.get_json()
digest = sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
try:
with db.begin():
db.execute(
"INSERT INTO idempotency_keys (key, request_hash, state) "
"VALUES (%s, %s, 'in_progress')", (key, digest))
except UniqueViolation:
row = db.execute(
"SELECT request_hash, state, status_code, response_body "
"FROM idempotency_keys WHERE key = %s", (key,)).fetchone()
if row.request_hash != digest:
return Response('{"error":"key reused with different body"}', status=422)
if row.state == 'done':
return Response(row.response_body, status=row.status_code,
mimetype='application/json')
return Response('{"error":"in progress"}', status=409)
order = create_order(body)
resp = json.dumps({"order_id": order.id})
db.execute("UPDATE idempotency_keys SET state='done', status_code=201, "
"response_body=%s WHERE key=%s", (resp, key))
return Response(resp, status=201, mimetype='application/json')
Trace it: first submit with k-a inserts the key, create_order inserts one row, the update marks it done, 201. The timeout retry with k-a hits the unique violation, reads state='done', returns the stored 201 and body, inserts nothing. A second click that generated a new key k-b inserts a second key and a second order, which is why the key must be per form, not per submit.
| Attempt | Key sent | Orders inserted | Response |
|---|---|---|---|
| first submit | k-a | 1 | 201, order_id |
| timeout retry | k-a | 0 | 201, same order_id |
| second click, new key | k-b | 1 | 201, new order_id |
Alternatives, and when this is overkill.
- Unique constraint on a natural business key. If the client can produce a stable key the domain already owns,
(merchant_id, cart_id, attempt_seq), a unique index onordersgives you dedup with no extra table and no TTL. It fails when the write spans more than one row or touches a non-database side effect (a payment capture, an email), because the constraint only guards the row it is on. - Server-generated key returned in the first response. This does not solve the timeout case: the retry never received the first response, so it has no key to send. It only works if the client can re-fetch by some other correlation id it already holds.
- Dedicated key store (above). General, replays the exact response body, works for multi-step writes. Costs a table, a cleanup job, and the in-flight state you must handle.
- Retry semantics. Retry only on connection errors and timeouts, never on a 4xx, and use exponential backoff with jitter. A retry loop that fires on a 500 with a body will duplicate work the server already did.
Skip all of it for GETs (already idempotent) and for writes where a duplicate is harmless and cheap, such as an analytics ping.
Failure modes. Key generated per submit instead of per form: the double click sends two keys and inserts two orders. The exact bug the table's third row shows. Row written but response never stored because the process died between insert and update: every retry gets 409 forever; fix by giving in_progress a timeout after which a retry may take over. TTL shorter than the client's retry window: a retry after expiry inserts a second order. Detect all three by counting orders rows per idempotency key and alerting on any key with more than one.
Worked example (hypothetical figures): a checkout button stayed enabled during a 1.8 s wait. 1,655 orders were inserted twice, so finance matched 3,310 payment captures to 1,655 shipments: two captures per shipment.
To check it: submit the same form twice with one key and confirm one row and two identical responses; submit a second key and confirm a second row.
Curated: · Written: · Reviewed:
Q&A-23Two tabs edit the same record. How does the page send what it last read, and how does the API refuse a stale save without last-write-wins?
Two tabs, one row. The default answer: the GET returns an ETag, the write sends it back in If-Match, and the server compares before committing. Mismatch → 412 Precondition Failed, nothing written, client refetches and shows the conflict. A mutable resource that receives a write with no If-Match should answer 428 Precondition Required (RFC 6585) so a client that forgets the header can't silently fall back to last-write-wins.
Mechanism: the ETag is an opaque validator for the representation you read. If-Match uses strong comparison (RFC 9110), so a weak W/"..." tag will never satisfy it: that's deliberate, a weak tag means the bytes may differ semantically. The server does the compare-and-swap inside the same transaction as the write, not as a read-then-write, or two requests can both pass the check.
| Write | If-Match sent | Server | Result |
|---|---|---|---|
| tab A, fresh tag | matches current | commit | 200 + new ETag |
| tab B, stale tag | mismatch | abort | 412, no overwrite |
| tab B after refetch | new tag | commit | 200 |
| any client, header omitted | — | reject | 428 |
ETag is one option, not the only one. The trade-offs matter more than the header:
| Mechanism | Client sends | Server check | Where it breaks |
|---|---|---|---|
| ETag + If-Match | opaque tag from GET | byte-compare of current representation | tag changes for reasons unrelated to the edit (embedded timestamp, JSON key order) → spurious 412s; intermediaries can weaken or strip it |
| Version column | integer in the body | UPDATE t SET ... WHERE id=? AND version=?, affected-rows 0 → conflict | must be plumbed through every serializer; no HTTP caching benefit |
| If-Unmodified-Since | Last-Modified date | date compare | HTTP dates have 1-second resolution, so two writes in the same second both pass; it tracks modification time, not content |
| Merge-on-write | field-level patch or CRDT | per-field merge | needs a merge policy; wrong when an invariant spans fields (a transfer's two balances) |
A version column is usually the better pick when the write path is your own API and you want the conflict visible in logs and SQL, and when the resource is also written by jobs that don't speak HTTP. ETag wins when you already have caching and want the precondition enforced by the framework.
Optimistic locking is the wrong tool in three cases. Hot rows (a shared counter, a queue head) where retries thrash and a short lock or an atomic increment is cheaper. Long edit sessions, where holding a version for thirty minutes rejects every other editor; use a checkout/lease or field-level merge instead. And append-only data (comments, audit events, presence, cursor position) where the business rule genuinely is "last writer is authoritative" and a 412 just adds noise.
Failure modes worth naming. The retry loop: client gets 412, refetches, immediately re-PUTs with the new tag, and overwrites anyway. That's last-write-wins with extra steps. Detect it by counting 412s and retry-successes per resource; a high 412 rate with a high retry-success rate is this bug. The unstable tag: ETag computed over a body containing a server timestamp means every GET returns a new tag and every concurrent write conflicts. And the endpoint added later: a new PATCH route that never got the precondition check, which is invisible until two tabs hit it.
Verification: open the same record in two tabs, PUT from both, assert the second gets 412 and the UI refetches rather than clobbering the first write; then assert a PUT with no If-Match gets 428. Illustrative shape of the bug this prevents (hypothetical): two agents edited a ticket without the header, the second save overwrote the first's notes, and the first tab kept showing "saved" because its success state was never invalidated.
The tag you read is the lock you send.
Curated: · Written: · Reviewed:
Q&A-40A comment is posted and 40 people should see it. What does the writer do and what do the open pages do?
The short version
The writer makes one write and one publish. Fanning out to 40 people is the server's job, not the authoring browser's. Each open page holds a cursor and either receives a push or catches up from its inbox feed.
Assumptions I'd state up front: the 40 participants are already known (thread membership is stored), the comment is durable before anyone is told about it, and delivery is at-least-once, so clients dedupe.
Write path
POST /threads/42/commentsinserts intocomments(id 9001, thread_id 42, author_id 7) and commits.- The same transaction inserts into
outbox(id 5501, topicthread.42, payload{comment_id: 9001}). - A relay reads unsent outbox rows, publishes to broker topic
thread.42, then marks the row sent. - The notifier consumes
thread.42, readsthread_participants(42)→ 40 user ids, and inserts 40 rows intonotifications_inbox (user_id, comment_id, created_at)in one batched insert. - For each of the 40 with a live socket, it pushes
{comment_id: 9001}. The rest get nothing pushed.
The authoring browser sends exactly one request and never learns who the recipients are. That is the point: the write path's latency is independent of participant count.
Open page
Each open page holds a socket and a cursor: the highest notification id it has applied. On push it fetches comment 9001 (or the push carries the body), appends it, advances the cursor. On reconnect it calls GET /inbox?after=<cursor> and replays, deduping by comment_id. Push is the optimization; the cursor is the source of truth.
Fan-out on write vs read
| Fan-out on write | Fan-out on read | |
|---|---|---|
| Rows written per comment | 1 comment + 40 inbox rows | 1 comment |
| Cost when a page opens | one indexed lookup by user | join comments × participants |
| Late-joining participant | sees only new comments unless backfilled | sees full history |
| Thread with 10k participants | 10k rows per comment | 1 row per comment |
Write fan-out wins when threads are small and reads are frequent; read fan-out wins on huge threads or when most participants never open the page.
Failure modes
- Socket down at write time: covered by cursor catch-up. Detect by inbox rows whose
created_atis older than the client's last ack. - Notifier crashes after inserting some of the 40 rows: the outbox row stays unsent, the relay republishes, the notifier re-runs. Without a unique constraint on
(user_id, comment_id)you get duplicate inbox rows: add it. - Duplicate push: client dedupes by
comment_id. - Polling instead of push: 40 clients polling every 2 s is 20 req/s per thread; 10k active threads is 200k req/s. Push collapses that.
Arithmetic (hypothetical)
A 40-participant thread taking 100 comments/hour writes 4,000 inbox rows/hour, ~96k/day. Read fan-out writes none but pays a join over participants on every open.
How I'd verify
Post one comment with 40 recipients and assert: one comment row, one outbox row, 40 inbox rows, and 40 clients that either received the push or caught up from the feed.
Curated: · Written: · Reviewed:
Q&A-38A user clicks Export and the tab spins. How should that click reach the database?
The 90-second answer. The click should not touch the database. It POSTs an export request that writes a job row and returns 202 Accepted with a job id and a status URL. A worker consumes the job, streams the query in keyset pages, writes CSV to object storage, and marks the job done. The UI polls the job (or holds an SSE connection) and downloads through a short-lived signed URL. The database sees a sequence of bounded index range scans instead of one long read holding a connection and a snapshot open for the length of the export.
Why the click handler is the wrong place. A 20-connection pool with three concurrent exports loses 15% of its capacity for minutes, and the fourth user's checkout query queues behind them. On Postgres, a read transaction open for 94 seconds pins the xmin horizon, so autovacuum cannot reclaim dead tuples created during that window: the export leaves bloat behind as well as latency. On InnoDB the equivalent is a long consistent read holding an MVCC snapshot and growing the undo log.
Mechanism: keyset pages, no long transaction.
# Python 3.12, psycopg 3.2
BATCH = 5_000
last_id = 0
with open(tmp_path, "w", newline="") as f:
w = csv.writer(f)
w.writerow(HEADER)
while True:
rows = conn.execute(
"SELECT id, created_at, total_cents FROM orders "
"WHERE id > %s AND created_at <= %s ORDER BY id LIMIT %s",
(last_id, snapshot_ts, BATCH),
).fetchall()
if not rows:
break
w.writerows(rows)
last_id = rows[-1][0]
Trace it: each iteration is an index range scan on the primary key for at most 5,000 rows, ordered by id, and last_id advances to the last row returned. 620,000 rows means 124 iterations, each its own short transaction, so nothing is held open between pages. The created_at <= snapshot_ts bound is what makes the file consistent without a single long snapshot: rows inserted after the job started are excluded, so the export matches a point in time. OFFSET pagination would instead re-scan and discard everything before the offset: 124 pages of growing cost, quadratic in total rows.
Alternatives, and when each is right.
| Approach | Good for | Breaks when |
|---|---|---|
| Stream in the request | under ~10k rows, interactive | pool held for the duration; client disconnect cancels mid-write |
| Server-side cursor in one transaction | medium exports on a replica | transaction open for minutes → vacuum/undo pressure |
| Keyset pages, no long transaction | large exports | needs a monotonic indexed key; needs a snapshot bound for consistency |
| Nightly materialized dump | repeated identical exports | staleness; refresh cost |
| Read replica | offloading the primary | replica lag silently omits recent rows |
Failure modes and how they show up.
- Double click creates two jobs. Idempotency key on (user, filter hash) within a short window.
- Worker dies mid-export and the job sits
runningforever. Lease with a heartbeat; requeue after lease expiry and resume from the last checkpointedlast_id. - Queue grows unbounded. Per-tenant concurrency cap (2 is typical) plus a max row count that returns "narrow your filters" instead of enqueuing.
- Buffering the whole CSV in memory OOMs the worker. Write in chunks and stream to storage.
- CSV formula injection: a customer name of
=HYPERLINK(...)executes when opened in Excel. Prefix cells beginning with=,+,-or@with an apostrophe, per OWASP's CSV injection guidance. - Replica lag: an export from a replica 30 s behind omits the newest orders. Record
snapshot_tsin the job and put it in the filename.
Worked example (illustrative figures). 620,000 order rows, 20-connection pool:
| Path | Click returns | Checkout p95 during export |
|---|---|---|
| Query inside the request | 94 s | 2,400 ms for 11 min |
| Job, then object download | 180 ms | 180 ms |
What I would measure before calling it done. Run a production-size export and assert the POST returns in under 500 ms, checkout p95 stays inside its budget while the worker runs, pool wait time stays flat, and the row count in the file equals SELECT count(*) at snapshot_ts. The click starts a job; the file is a later download.
Curated: · Written: · Reviewed:
Q&A-15A user picks a large file. Should the bytes travel through your API, or should the API only mint permission while the browser talks to the store?
The 90-second answer
Mint permission in the API; send the bytes straight to the object store. The API authenticates the user, picks the key, signs a short-lived URL, and records metadata. The browser PUTs to the store. The API confirms the object exists before the UI says done.
State the assumptions out loud: the store speaks presigned URLs (S3, GCS, Azure Blob SAS); the browser can reach it directly, which means CORS is configured on the bucket; and you do not need to inspect bytes before they land. If any of those is false, proxy the upload and say which one forced it.
Mechanism
Three calls, and the middle one never touches your servers.
// Node 20, @aws-sdk/client-s3 v3, @aws-sdk/s3-request-presigner v3
import { S3Client, PutObjectCommand, HeadObjectCommand } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { randomUUID } from 'node:crypto';
const s3 = new S3Client({ region: 'us-east-1' });
const BUCKET = 'uploads-prod';
app.post('/uploads', async (req, res) => {
const key = `u/${req.user.id}/${randomUUID()}`; // unguessable, one per attempt
const url = await getSignedUrl(
s3,
new PutObjectCommand({ Bucket: BUCKET, Key: key, ContentType: req.body.contentType }),
{ expiresIn: 900 },
);
await db.insertUpload({ key, userId: req.user.id, state: 'pending' });
res.json({ url, key });
});
app.post('/uploads/:key/confirm', async (req, res) => {
const head = await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: req.params.key }));
if (head.ContentLength !== req.body.size) return res.status(409).end();
await db.markUploaded(req.params.key, head.ETag); // state: 'confirmed'
res.status(204).end();
});
The confirm step is the part people skip. HeadObject is the only proof the bytes arrived; the client's 200 from the PUT is not, because a closed tab between PUT and confirm leaves a row that never becomes real.
| Path | Bytes through app server | UI may show done | Failure it invites |
|---|---|---|---|
| POST file to API | whole body buffered | on 200 | OOM or 413 at the proxy; retry re-sends the body |
| Presigned PUT, no confirm | none | when PUT returns 200 | row missing, orphan object, UI lies |
| Presigned PUT + confirm | metadata only | after the row exists | one extra round trip; orphan if confirm never arrives |
Failure modes and how you see them
Orphans. Upload succeeds, confirm never arrives. Detect with a bucket lifecycle rule that deletes u/ keys with no matching confirmed row after 24 hours, and a gauge on pending rows older than an hour.
Proxy limits before your code runs. nginx's client_max_body_size defaults to 1 MB and answers 413 without invoking the handler. A worked example with hypothetical numbers: 20 concurrent 95 MB uploads buffered in a 512 MB container is 1.9 GB resident, so the process dies before any timeout fires. Direct-to-store removes the term entirely.
URL replay. The key is a UUID, so a leaked URL overwrites only its own object. Short expiry bounds the window: 900 s at 1 Mbit/s carries about 110 MB, so slow links need either a longer expiry or multipart with per-part URLs.
Spoofed content type. The client declares image/png and uploads HTML. Verify magic bytes on confirm or in a scan worker, and serve objects from a separate origin with Content-Disposition: attachment.
CORS. A missing PUT in the bucket's AllowedMethods shows up as an opaque network error in the browser console, not a 4xx.
When to proxy instead
Scan-before-land, EXIF stripping, transcoding, or a store the browser cannot reach. All four are real, and all four are reasons to accept the memory cost for that one endpoint rather than for every upload.
Verification
Upload at the product size cap and watch the app server's resident memory stay flat. Kill the tab between PUT and confirm and check the row stays pending and the UI never renders the file. Then confirm the lifecycle rule deletes the orphan.
Curated: · Written: · Reviewed:
Q&A-39The user starts a video upload that must be transcoded. How does the page wait without lying?
With background job the UI waits on, a green local UI is where I start checking rather than stop.
Work that takes longer than a request timeout cannot be hidden behind a spinner on the original POST. The UI needs a job identifier, a status it can poll or push, and a terminal state that matches the row the worker wrote.
Concretely, create the job row first and return its id. Show queued, running, failed, or ready from that row. Subscribe with a socket or poll every 3 seconds, and do not paint complete until the worker has written the output object.
The reason for that specificity is a failure I have seen: The upload POST waited 58 seconds for transcode, the load balancer cut it at 60, the client showed an error while the worker later succeeded, and 1,900 users uploaded the same video twice.
Transcode wait.
| Wait strategy | Client result | Duplicate uploads |
|---|---|---|
| block the POST for 58 s | error at 60 s, worker succeeds | 1,900 |
| poll job id every 3 s | ready when the object exists | 0 |
I would not consider it settled without evidence: Kill the tab after enqueue and reopen the page, then confirm the job status matches the worker and the object store.
Wait on the job row, not on the original request.
Curated: · Written: · Reviewed:
Q&A-24A payment provider posts a webhook. How does that become a badge on an open tab, and what must be stored before either the email or the toast fires?
The answer
The webhook never touches the tab. It hits your API, and the only thing it is allowed to do is write one row. Everything the user sees (badge, toast, email) is derived from that row. So: verify the signature, insert a notification row keyed by the provider's event id, commit, return 200, and only then fan out to email and to the open tab over SSE or WebSocket. Toast from the handler before the commit and a rolled-back transaction leaves a notification on screen that no query can find.
The path, step by step
Trace for a Stripe payment_intent.succeeded:
- Stripe POSTs the event with
Stripe-Signature: t=1712...,v1=.... The handler computes HMAC-SHA256 over"{t}.{raw_body}"with the endpoint secret and rejects if the timestamp is more than 300 s old (Stripe's default tolerance). Use the raw body, not parsed JSON: re-serializing breaks the HMAC. - Insert with the provider event id as a unique key:
INSERT INTO notifications (id, user_id, provider, event_id, type, entity_id, created_at, read_at)
VALUES ($1, $2, 'stripe', $3, 'payment_succeeded', $4, now(), NULL)
ON CONFLICT (provider, event_id) DO NOTHING
RETURNING id;
No row returned means this is a retry: return 200 and stop. Stripe retries failed deliveries with exponential backoff for up to three days in live mode, so duplicates are the normal case, not the exception.
- Commit, then enqueue the email job and publish to the user's channel. Return 200 within a couple of seconds; the provider's timeout is shorter than your email send.
- The open tab holds an
EventSourceon/notifications/stream. The server writesevent: notification/data: {"id":"01H...","unread":3}. The client sets the badge tounread, notunread + 1.
What has to be stored
| Column | Why it exists |
|---|---|
id (ULID) | Stable handle the client acks and the toast links to |
provider, event_id | Unique index; makes retries idempotent |
user_id | Routing, which stream, which inbox |
type, entity_id | Renders "Payment of $40 failed" without a join at read time |
created_at, read_at | Unread count is count(*) where read_at is null |
email_sent_at, push_sent_at | Per-channel delivery state, so a retry does not re-send |
Where it breaks
- Toast before commit: phantom badge, gone on refresh.
- No unique index on
(provider, event_id): a retry sends a second email. - Client-side increment: two open tabs each add one, so the badge reads 2 for one payment. Send the server-computed count.
- SSE behind a load balancer with a 60 s idle timeout: the connection dies silently. Send a
: heartbeatcomment every 15–30 s, and have the client refetchGET /notifications/unread_counton reconnect:Last-Event-IDreplay only works if you retained the events.
How I would check it
Post a signed test event, confirm one row, confirm the tab's badge moves, then replay the identical event and confirm the row count and the badge are unchanged.
Curated: · Written: · Reviewed:
Q&A-16A board should feel live. How do you choose poll versus a socket, and how do both halves stay authenticated when the transport changes?
The 90-second answer
Two questions decide the transport. First, what staleness can the board tolerate, if a two-second-old board is fine, poll; if a cursor or a comment has to land in under 200 ms, you need a push. Second, where does the session live, and the answer is the same for both, because a WebSocket handshake is an HTTP request. The browser attaches the same HttpOnly session cookie to the upgrade GET that it attaches to any other request, so the server authenticates the handshake exactly as it authenticates the API, then binds the connection to that principal. There is no second auth story to invent.
Choosing
Polling's cost scales with tabs × 1/interval, and it is cacheable, idempotent, and survives every proxy and deploy. A socket's cost scales with events, not tabs, but you pay a held connection per client: file descriptors, memory, sticky routing or a pub/sub fan-out, and a heartbeat under the load balancer's idle timeout.
| Poll | Socket | |
|---|---|---|
| Freshness | bounded by the interval | push latency |
| Cost model | tabs × 1/interval | events × subscribers + held connections |
| Auth | cookie per request | cookie on handshake, then bound principal |
| Reconnect | nothing to do | re-handshake, resubscribe, resume from cursor |
| Infra | any proxy, CDN-cacheable | sticky or pub/sub, heartbeat under LB idle timeout |
How the handshake carries the session
- Cookie on the upgrade. Works because the upgrade is an HTTP GET and the browser attaches cookies automatically. Caveat: the browser
WebSocketconstructor cannot set custom headers, so a bearer token in anAuthorizationheader is not available to browser clients. - Ticket pattern.
POST /api/ws-ticketwith the session cookie returns a single-use token with a 30-second TTL, consumed at handshake. This keeps the long-lived session out of URLs and access logs, and it gives you a real status code. - Why the status code matters. A failed upgrade surfaces to JS as a close with code 1006 and no reason. You cannot distinguish "your session expired" from "the network died". The ticket endpoint returns a real 401 you can act on.
- Authorize per room, not per message. Check once at subscribe time that the principal may see room 42; don't re-check every frame.
- Session expiry mid-connection. The socket outlives the cookie. Enforce an absolute connection lifetime (say 30 minutes) and force a re-handshake, or revalidate on the heartbeat: otherwise a revoked session keeps receiving pushes.
Reconnect
Reconnect is a fresh handshake; the cookie is still in the browser, so no second login. Two things must happen: resubscribe to rooms, and resume without assuming nothing was missed. Carry a cursor (last event id) and have the server replay from it, or refetch the snapshot over HTTP and resume the stream.
let lastEventId = 0;
let attempt = 0;
async function connect() {
// The session cookie authenticates this POST; the ticket authenticates the socket.
const res = await fetch('/api/ws-ticket', { method: 'POST', credentials: 'same-origin' });
if (res.status === 401) return redirectToLogin();
const { ticket } = await res.json();
const ws = new WebSocket(`wss://board.example.com/ws?ticket=${ticket}&since=${lastEventId}`);
ws.onopen = () => { attempt = 0; };
ws.onmessage = (e) => {
const msg = JSON.parse(e.data);
lastEventId = msg.id; // server replays everything after this on reconnect
apply(msg);
};
ws.onclose = () => {
const delay = Math.min(30000, 500 * 2 ** attempt++);
setTimeout(connect, delay);
};
}
connect();
Trace: the first call fetches a ticket and opens with since=0, so the server replays from the beginning. Messages advance lastEventId. When the socket drops, onclose computes min(30000, 500 * 2^0) = 500 ms and increments attempt to 1. The next drop gives 1000 ms, then 2000, 4000, capped at 30000. Each retry fetches a fresh ticket and passes the current lastEventId, so the gap is replayed rather than silently lost.
Failure modes
- ALB idle timeout defaults to 60 s and kills idle sockets: send ping/pong every 20–30 s.
- Sticky sessions break on deploy: fan out through Redis pub/sub or NATS so any node can push to a room.
- Poll thundering herd: 18,000 tabs at a 1.5 s interval is 12,000 req/s. Jitter the interval, serve 304s off an ETag, and back off on error.
Arithmetic (illustrative figures)
18,000 tabs polling every 1.5 s at 3 ms of origin CPU per request: 12,000 × 3 ms = 36 CPU-seconds per second, about 36 cores. The same board over a socket: 90 rooms of 200 subscribers, one event per room per second, is 18,000 deliveries/s at roughly 0.05 ms of serialization each. About 0.9 CPU-seconds per second, one core, plus 18,000 idle connections. The crossover is the fan-out ratio: when each event reaches only a handful of subscribers, polling wins on both cost and simplicity.
Curated: · Written: · Reviewed:
Q&A-10A list changes while the user scrolls or walks back. How do the page and the API resume without skipping or repeating rows?
Assumptions: the list is ordered by a column that doesn't change for a given row, and the client can hold one small token per list position. If either fails, say so before answering.
The API returns an opaque cursor, the sort-key values of the last row plus a unique tie-break, and the client stores that token, not a page index. Resume means re-issuing the same cursor, not recomputing an offset.
Why offset breaks. OFFSET n LIMIT k counts rows from the top of the current result set. An insert above the window shifts every row down by one, so the next page re-serves the last row of the previous page and drops one at the bottom. A delete shifts the other way and skips a row.
Keyset. Sort by (created_at DESC, id DESC) and page with:
SELECT id, created_at, body
FROM messages
WHERE (created_at, id) < (:c_created, :c_id)
ORDER BY created_at DESC, id DESC
LIMIT 3;
The row comparison is the whole trick: it means "everything strictly after this exact row in this exact order", so inserts and deletes elsewhere in the list can't move the boundary. id is in the tuple because created_at alone isn't unique: two rows sharing a timestamp can come back in either order between queries, which is the same skip/repeat bug in a smaller window.
Trace, rows as (id, created_at):
10 09:00 9 09:00 8 08:00 7 08:00 6 07:00 5 07:00
Page 1, LIMIT 3: 10, 9, 8. Cursor (08:00, 8).
Now insert 11 09:30 at the top.
Page 2 with the cursor: (created_at, id) < (08:00, 8) → 7, 6, 5. Nothing repeated, nothing skipped. Row 11 sits above the cursor and appears on refresh, which is correct: it wasn't in the list the user was reading.
Same insert with OFFSET 3 LIMIT 3 over the new ordering 11,10,9,8,7,6,5 returns 8,7,6. Delete row 9 instead and offset page 2 returns 6,5.
| Resume | Repeated | Skipped |
|---|---|---|
OFFSET 3 after one insert | row 8 | row 5 |
| keyset cursor after one insert | none | none |
Client side. Keep the cursor in component or route state keyed to the list position, so back navigation re-issues the token the server gave you. Don't derive it from items.length: that's an offset with extra steps. If the UI shows "page 7 of 20", it can't be resumed safely; use "load more" or infinite scroll.
Failure modes.
- The sort key mutates: an edit bumps
updated_at, a "mark unread" reorders. A row crosses the cursor boundary and is served twice or never. Sort on an immutable column, or dedupe by id on the client. - The cursor isn't validated. If a client can send a cursor for a different sort or filter, the server returns a window from another list. Sign or encode the cursor and reject a mismatch.
- The cursor is a raw
created_atwith no tie-break, so rows at a timestamp boundary duplicate.
When offset is fine: static or admin data that doesn't change under the reader, and when you need random access to page N or a total count. Keyset gives up both; it only walks forward.
Verification: a test that pages through while inserting and deleting between pages, collects every id, and asserts the set equals the table's set with no duplicates. In production, log the cursor per request and alert on duplicate ids within one session.
Curated: · Written: · Reviewed:
Q&A-22Search has debounce, filters, sort, and a URL. What does the client send, what does the API echo, and how do the box and the address bar stay one query?
The client sends one query document; the API echoes the normalized version of it back; the URL is the single source of truth both sides read from.
Concretely: the box debounces input, aborts the previous in-flight request, and serializes the whole query (text, filters, sort, cursor) into the URL. The API parses that URL, normalizes it, ranks against it, and returns the results plus the normalized query document. The client writes that echoed document back to the URL with history.replaceState, so the address bar and the box are always rendering the same object the server ranked.
What the client sends
A single GET whose query string is the state:
GET /api/search?q=redis+timeout&f=lang:go&f=lang:rust&sort=-created_at&cursor=eyJpZCI6MTIzfQ
q: raw text, URL-encoded viaURLSearchParams(spaces become+).f: repeated param for multi-value filters; order is not meaningful, so the server canonicalizes it.sort: signed field,-created_at= descending.cursor: opaque base64 keyset cursor, not an offset.
What the API echoes
{ "results": [...], "query": { "q": "redis timeout", "f": ["lang:go","lang:rust"], "sort": "-created_at", "cursor": "eyJpZCI6MTIzfQ" } }
The echo is the normalized document: trimmed, whitespace collapsed, filters sorted, defaults filled in. The client re-serializes query into the URL. Because replaceState does not fire popstate, this write does not re-trigger a fetch: that is what stops the echo from looping.
Mechanism: why the echo keeps them in sync
On load, the client parses location.search and hydrates the box and filter chips from it. On every subsequent change, the client is the only writer of the URL, and it writes exactly what the server returned. Back/forward navigation fires popstate, the client re-parses the URL, and re-fetches, so a shared link, a back button, and a fresh tab all reconstruct the same query.
Trade-offs
| Choice | Wins | Costs |
|---|---|---|
| URL as state | Shareable, back/forward works, no server session | Long URLs; sensitive filters land in history and Referer |
| Server-side session | Short URLs, hidden filters | Breaks sharing and back button; needs session store |
| Debounce 250 ms on typing | ~1 request per pause, not per keystroke | Feels laggy above ~400 ms |
| No debounce on filter/sort | Discrete actions, immediate feedback | None: these are clicks, not keystrokes |
| Echo normalization | Box and URL converge | Must not echo semantic rewrites (spell-correction) or it fights the user's typing |
Failure modes
- Overlapping fetches without abort. Responses arrive out of order and a stale one wins. Fix:
AbortController, abort the previous request before issuing the next. Detect by logging a request id per response and asserting monotonicity. pushStateon every keystroke. Back button walks through every character. UsereplaceStatefor typing,pushStateonly for discrete filter/sort changes.- Echo loop. If the URL write triggers a fetch, you get an infinite request cycle. Guard by comparing the serialized echo against the current URL before writing.
let controller, timer;
function onInput(value) {
clearTimeout(timer);
timer = setTimeout(() => runSearch(value), 250);
}
async function runSearch(q) {
controller?.abort();
controller = new AbortController();
const params = new URLSearchParams({ q, sort: '-created_at' });
filters.forEach(f => params.append('f', f));
const res = await fetch(`/api/search?${params}`, { signal: controller.signal });
const { results, query } = await res.json();
render(results);
const next = `?${new URLSearchParams(query)}`;
if (next !== location.search) history.replaceState(null, '', next);
}
Checked against Node 20 / evergreen browsers: AbortController, URLSearchParams, and replaceState are all standard.
Curated: · Written: · Reviewed:
Q&A-19The API caps a search at 120 per minute. What must it return, and what must the typeahead do instead of retrying in a loop?
The 60-second answer. Return 429 Too Many Requests with a Retry-After header giving the seconds until the window resets. The typeahead must not retry on 429 at all: it reads Retry-After, stops issuing searches until that moment passes, and paints a quota state. Retrying is what turns a per-user cap into a self-inflicted outage: every retry spends the same budget the real user needs.
Why a retry loop is worse than the 429. The cap is a fixed window of 120 requests per 60 s. A retry loop does not wait for the window; it re-enters it. A client retrying every 200 ms issues 300 requests a minute against a budget of 120, all rejected, and the origin still pays to parse, authenticate and rate-limit-check each one. The user's next real keystroke queues behind them.
What the client does.
// Browser, fetch + AbortController. Node 18+ exposes the same globals.
const LIMIT = 120, WINDOW_MS = 60_000;
let tokens = LIMIT, windowStart = Date.now();
let blockedUntil = 0; // ms epoch, set from Retry-After
let inFlight = null, seq = 0;
function takeToken() {
const now = Date.now();
if (now - windowStart >= WINDOW_MS) { windowStart = now; tokens = LIMIT; }
return tokens > 0 ? (tokens--, true) : false;
}
async function search(q) {
if (q.length < 2) return;
if (Date.now() < blockedUntil) return renderQuota(); // no fetch at all
if (!takeToken()) { blockedUntil = windowStart + WINDOW_MS; return renderQuota(); }
inFlight?.abort(); // drop the previous keystroke
const ctrl = new AbortController();
inFlight = ctrl;
const mine = ++seq;
const res = await fetch(`/api/search?q=${encodeURIComponent(q)}`, { signal: ctrl.signal });
if (res.status === 429) {
const secs = Number(res.headers.get('retry-after'));
blockedUntil = Date.now() + (Number.isFinite(secs) ? secs * 1000 : 5_000);
return renderQuota(); // return, do not retry
}
if (mine !== seq) return; // a newer keystroke won
render(await res.json());
}
Trace it. The user types redis with a 250 ms debounce; keystrokes land at 0, 90, 180, 270, 360 ms and the debounce fires once at 610 ms with q="redis". takeToken drops 120 to 119 and one fetch goes out. A fast typist pausing every 300 ms would otherwise generate 3 req/s, or 180/min, over the cap, so the local bucket is what keeps the 429 from happening in the first place. When the bucket hits zero, takeToken returns false, blockedUntil is set to the end of the window, and no fetch is issued. If the server 429s anyway (another tab, another user on the same key), Retry-After: 23 sets blockedUntil = now + 23000; every keystroke in those 23 s returns renderQuota() without touching the network.
What the API returns.
| Field | Example | Notes |
|---|---|---|
| Status | 429 Too Many Requests | RFC 9110 §15.5.29 |
Retry-After | 23 | delta-seconds, RFC 9110 §10.2.3; an HTTP-date is also legal |
Cache-Control | no-store | stops a shared cache replaying the 429 to other users |
RateLimit-Limit / -Remaining / -Reset | 120 / 0 / 23 | IETF draft, not an RFC; advisory only |
| Body | {"error":"rate_limited","retry_after":23} | for clients that cannot read headers |
That last row matters more than it looks: Retry-After is not a CORS-safelisted response header, so a cross-origin fetch cannot read it unless the server also sends Access-Control-Expose-Headers: Retry-After. Mirroring the value in the body is the cheap fix.
Failure modes.
Retry-Aftersent as an HTTP-date:Number("Wed, 21 Oct 2015 07:28:00 GMT")isNaN, so the 5 s fallback fires and the client waits the wrong amount. Send delta-seconds; it is also immune to client clock skew.- Header missing entirely: fall back to a fixed delay with jitter, never to zero.
- Limiting by IP: an office behind one NAT shares 120/min. Key the limit on the API key or user id.
- Window reset thundering herd: if every client's window starts on the same wall-clock boundary, they all resume together. Jitter the client's resume by 0–2 s.
- Blanking the result list on 429 makes the box look broken. Keep the last good results and label them stale.
Enforce the limit before the search work, not after: a 429 that arrives once the query has already run saves nothing. And keep 429 distinct from 503: 429 is this client's quota, 503 with Retry-After is the service being overloaded.
What I would check. Drive the box past 120 in a minute with a scripted typist and confirm zero requests go out between the 429 and the Retry-After deadline, that the quota state renders, and that the first keystroke after the deadline succeeds.
Curated: · Written: · Reviewed:
Q&A-49Typeahead hammers the API on every keystroke. How do you bound cost on both halves?
What I'd say first
Assume a read-heavy autocomplete over a corpus that fits a prefix index (say under 10M rows), a p99 budget around 100ms, and users typing in bursts. Bound both halves: the client stops sending requests it doesn't need, and the server refuses to do work it can't afford.
Client: debounce 150ms, require 2 characters, cancel the in-flight request when the next key arrives, render at most 8 rows. Server: serve from a prefix index or an edge cache keyed on the normalized query, and put a per-session token bucket in front so a stale or misbehaving client can't reach the primary.
Why those numbers
Debounce collapses a burst of keystrokes into one request per pause. 150ms sits below the ~200ms point where users notice lag, so it cuts requests without feeling sluggish. The 2-character floor kills the highest-volume, lowest-value prefixes ("a", "ab"). Cancelling in-flight matters as much as debouncing: without it you still have N requests on the wire, just spaced out.
let timer, ctrl;
input.addEventListener('input', e => {
clearTimeout(timer);
ctrl?.abort(); // drop the previous request
const q = e.target.value.trim().toLowerCase();
if (q.length < 2) return render([]);
timer = setTimeout(async () => {
ctrl = new AbortController();
const r = await fetch(`/suggest?q=${encodeURIComponent(q)}`, { signal: ctrl.signal });
render((await r.json()).slice(0, 8));
}, 150);
});
Trace: typing "london". Each keystroke clears the pending timer and aborts the previous fetch, so only the pause after the final "n" fires a request, and a slow earlier response can't paint over it.
Worked example (figures illustrative, from a search box I've tuned): a fast typist issuing ~11 requests/sec un-debounced, at a 2,400-user peak, is 26,400 QPS against the primary. Adding 150ms debounce, the 2-char floor, and a 30s prefix cache drops it to ~900 QPS: about 0.4 req/s per user.
| Client + index | QPS at peak | Failure it removes |
|---|---|---|
| every key, primary scan | 26,400 | stale in-flight responses |
| 150ms debounce, 2 chars, 30s prefix cache | 900 | latest prefix wins |
Alternatives and when the recipe is wrong
- Longer debounce (300ms) or fewer results: cheaper still, but adds perceptible lag. Use it only when the backend is genuinely saturated and you can't add capacity.
- Server-side rate limiting (token bucket, e.g. 5 req/s per session, 429 on excess): the backstop for old clients and bots. Too tight and legitimate typists get empty dropdowns: set it well above the debounced rate.
- Precomputed suggestions: materialize top-N completions per prefix offline into a KV store, serve at O(1). Best when the corpus is stable and query distribution is head-heavy. Wrong when content changes fast (fresh items never appear) or the tail is long (most prefixes miss).
- Edge cache vs prefix index: the cache is nearly free but hit rate tracks query skew; the index handles the tail at higher cost per query.
If the corpus is small enough to hold an in-process trie (roughly under 100k rows), skip the cache and rate limit entirely: the whole path is a memory lookup.
Failure modes to check
- Responses arriving out of order paint an older prefix over a newer one: cancel, or tag responses with a sequence number and drop stale ones.
- Cache key not normalized (case, whitespace, trailing spaces): hit rate collapses and you're back on the primary.
- Debounce without cancel: request count drops but concurrency doesn't.
- Rate limit below the debounced rate: 429s on normal typing.
How I'd verify
Throttle the CPU, type a 10-character query, and confirm three things: request count on the wire, server query rate, and that the painted suggestions match the latest prefix rather than a stale one.
Curated: · Written: · Reviewed:
Q&A-21Someone shares a project URL. What must be in the path, and what must the first paint fetch even if the original tab still has the entity in memory?
I would anchor deep links that require server data in a trace that starts at the click and ends at the query.
A URL the user can share must be enough for the server to load the record and for the client to render it. Client-only routers that hold the entity in memory make the shared link an empty shell.
Concretely, encode the server id in the path. Fetch that id on first paint even when the client has a cache. Return status 404 from the API when the record is gone so the page can show not found instead of a spinner.
The reason for that specificity is a failure I have seen: Shared project links depended on a client cache, 2,275 cold loads showed a blank canvas, and 310 of those users created duplicate projects with the same name.
Same URL, three first paints.
| First paint | Client cache | API |
|---|---|---|
| warm tab | hit | skipped, looks fine |
| cold shared link, no fetch | miss | never called, blank |
| cold shared link, GET by id | miss | 200 or 404 |
I would not consider it settled without evidence: Open the shared URL in a logged-in browser with an empty cache and confirm the client fetches the id and paints the record or a 404 from the API.
The link is an API read, not a souvenir of another tab.
Curated: · Written: · Reviewed:
Q&A-37Product images are huge on mobile. What does the client request and what does the pipeline produce?
The 90-second answer. The client asks for one variant, not the original. The image tag carries a srcset of candidate widths plus a sizes attribute describing the layout slot, and the browser picks the smallest candidate that covers slot width × devicePixelRatio. The request that leaves the device names that width, a format and a quality in the URL, and carries an Accept header listing the formats it can decode. The pipeline returns exactly that variant (encoded at the requested width, in the negotiated format) and caches it at the edge so the next viewer of the same variant never reaches the origin.
What the client sends.
GET /img/sku-4471/w360/q75.webp HTTP/1.1
Host: cdn.example.com
Accept: image/avif,image/webp,image/*;q=0.8
HTTP/1.1 200 OK
Content-Type: image/webp
Content-Length: 43008
Cache-Control: public, max-age=31536000, immutable
Vary: Accept
ETag: "sku-4471-w360-q75-webp-v3"
Three things make this correct. Vary: Accept keeps a WebP body from being served to a client that only asked for JPEG. The w360 in the path is the cache key, so each width is a separate object. immutable with a one-year TTL is only safe because the URL carries a content version (v3): re-encode the source and the path changes, so you never need to purge.
What the pipeline produces, and the decision that matters. Two ways to fill the cache: pre-generate every variant at upload time, or generate on first miss and let the edge hold it. The deciding factor is variant explosion against request distribution.
A ladder of 6 widths × 3 formats (AVIF, WebP, JPEG) is 18 objects per source image. For a 50,000-SKU catalog that is 900,000 objects; at an illustrative 120 KB average that is roughly 108 GB of storage and 900,000 encode jobs, most of which no one ever requests. Pre-generation pays that in full and buys you a warm cache after every deploy. On-demand pays only for variants actually fetched, but the first request for each one hits the origin and pays encode latency.
So: pre-generate when the catalog is small and hot, or when a cold cache after deploy is unacceptable. Generate on first miss when the catalog is large and the tail is long. The common production choice is the hybrid: pre-generate the two or three widths that cover most traffic, generate the rest on demand.
Failure modes.
- Missing
Vary: Accept. The CDN serves a WebP body to a JPEG-only client. Detect it by diffing responseContent-Typeagainst the requestAccepton a few URLs. - Cache key that ignores DPR. A 3× phone gets the 1× image and looks soft. Detect by comparing the requested width to
devicePixelRatio × CSS widthin the DOM. - Origin stampede. N concurrent requests for the same uncached variant trigger N encodes. Fix with single-flight coalescing at the edge.
- Upscaling. A 1080 request against a 600 px source wastes bytes. Clamp the ladder to the source width.
The cost of getting it wrong. A gallery shipped 12 originals of 3.2 MB each, CSS-scaled to 180 px. Switching to 360 px WebP variants:
| Request | Each | Gallery total | Displayed width |
|---|---|---|---|
| 4,000 px original | 3.2 MB | 38.4 MB | 180 px |
| 360 px WebP | 42 KB | 504 KB | 180 px |
38.4 MB → 504 KB on the same layout. Verify it on a real device: record the image URLs, confirm each response width matches the displayed size, and confirm the second view is served from the edge cache rather than the origin.
Curated: · Written: · Reviewed:
Q&A-48Faceted search is slow and some filters return empty while data exists. What must the query and the index share?
The short answer: the query and the index must agree on three things. The field name, the field's type and analyzer, and the value's normalization. Break any one and the filter matches nothing while the documents sit right there in the index.
Why a filter returns empty while data exists. The usual cause is a term-level filter pointed at an analyzed text field. Map color as text with the standard analyzer and ingest "Red": the analyzer lowercases it, so the index stores the token red. A term query does not run the analyzer on its input, so {"term": {"color": "Red"}} looks for the literal token Red and finds nothing. Same field, same value, zero hits. The fix is a keyword subfield that stores the original string verbatim:
PUT products
{ "mappings": { "properties": {
"color": { "type": "text", "fields": { "raw": { "type": "keyword" } } },
"price": { "type": "integer" }
}}}
Now {"term": {"color.raw": "Red"}} matches. The rule: filter and aggregate on keyword, full-text search on text.
Why facets show zero. Aggregations run over the query's result set. Put the color filter in the query clause and the colors aggregation only ever sees red documents, so every other color bucket comes back 0 or missing: the "empty while data exists" symptom. The standard faceted-search shape puts the filter in post_filter, which runs after aggregations:
GET products/_search
{ "size": 20,
"query": { "match": { "name": "shirt" } },
"post_filter": { "term": { "color.raw": "Red" } },
"aggs": { "colors": { "terms": { "field": "color.raw" } } } }
colors now counts all matching shirts; the 20 returned hits are red. If you need counts independent of the whole query, use a global aggregation.
Why it's slow. Aggregating or filtering on an analyzed text field forces fielddata: an uninverted index built on the heap, disabled by default in Elasticsearch 8 and slow when enabled. keyword fields use doc_values on disk, which is what filter context and terms aggregations are built for. Put filters in filter context so they skip scoring and get cached.
Trade-offs. Indexing every facet field is not free: each keyword field with doc_values costs disk per document and adds write latency. Don't index fields you never filter or aggregate on, and don't add a keyword subfield to a high-cardinality free-text field you only ever search. Client-side filtering is fine when the candidate set is already small and bounded: filtering 200 already-fetched rows in the browser beats a round trip. It breaks the moment the set exceeds what you fetched.
Failure modes and detection.
| Symptom | Likely cause |
|---|---|
| filter returns 0, docs exist | term filter on an analyzed text field |
| other facet buckets show 0 | filter in query instead of post_filter |
| slow aggregation | aggregating on text (fielddata) |
| counts differ from the database | normalization mismatch (case, trim, synonyms) |
Verify by comparing the hit count against a database count on the same predicates, and by running _analyze on the field to see the tokens the index actually stored.
Curated: · Written: · Reviewed:
Q&A-32Search results look wrong even though the documents are in the index. What do you check across the stack?
"Wrong" splits into three cases, and I pin down which one first: the document is missing from results, it's present but ranked below something irrelevant, or it's present but the returned fields are wrong. Each points at a different layer, so I check from the client inward.
1. What the client actually sent. Open the network tab and copy the raw request body. Typeaheads and front-end SDKs mutate queries: stripping punctuation, lowercasing, adding fuzziness, flipping default_operator to AND. If the UI sent A19 and the index holds A-19, nothing downstream recovers it.
2. Analyzer at query time vs index time. Run _analyze on both sides with the field's analyzer:
GET /products/_analyze
{ "field": "sku", "text": "A-19" }
If sku is keyword, you get one token, A-19. If it's text with the standard analyzer, you get a and 19: the hyphen is a token boundary. A term query for A-19 against the standard-analyzed field matches nothing. This is a frequent cause of "the doc is in the index but search can't find it."
3. Mapping. GET /products/_mapping. Confirm the field is index: true, that it's text where you want full-text and keyword where you want exact, and that dynamic mapping didn't guess. A SKU like A-19 routed into a long field silently fails to index the string.
4. Refresh and visibility. Elasticsearch is near-real-time: a doc indexed 200 ms ago isn't searchable until the next refresh, default refresh_interval 1s. Check GET /products/_count, if the count is right but _search misses, it's refresh or a filter, not indexing.
5. Shards, replicas, routing. GET _cat/shards/products?v and GET _cluster/health. An unassigned shard (yellow/red) drops a slice of results. Custom routing that doesn't match the write routing sends the query to the wrong shard. Replica lag shows as "found on one node, not another."
6. Scoring. If the doc is present but ordered wrong, add "explain": true and read the score breakdown: a function_score or a boost on the wrong field usually explains it.
7. Highlight and returned fields. If the row is there but the snippet is empty, the highlight field name or _source filtering is off, not the search.
The trade-off that decides the fix: index-time analysis is baked into the stored tokens, so changing the index analyzer means a reindex (alias swap, or accept downtime). Changing only search_analyzer is a live mapping update and fixes the query side without touching stored tokens. Mismatch on the index side → reindex; query side → don't.
| Observation | Layer | First check |
|---|---|---|
Findable by _id, not by term | analyzer / mapping | _analyze both sides, _mapping |
| Findable a second later | refresh | refresh_interval, _count |
| Findable on one node only | shard / routing / replica | _cat/shards, _cluster/health |
| Present but misordered | scoring | explain: true |
| Present but empty fields | highlight / source | highlight field name, _source |
I'd confirm with one live capture: same tokens, same filters, same top-10 ids from the index API as the UI shows.
Curated: · Written: · Reviewed:
Q&A-11The API returns 401, 404, 409, and 503. What should the screen do for each, and what must it never paint?
Assumptions first: a single-page app on one shared HTTP client, cookie or bearer session, and an API that returns a JSON body with a stable machine-readable code plus whatever the status needs to be resolvable. The status line picks which surface owns the screen; the body only fills it in.
- 401 means the session is gone, not the resource. One global handler clears auth state, purges rows cached for that session, and routes to sign-in with the intended path preserved. It fires once for a burst of parallel 401s, not once per request.
- 404 means this resource is not there (or the API is hiding a 403 behind it, which is common for existence-hiding). The region that asked for it paints not-found in place, with a route back. Not a full-screen crash, not an empty list.
- 409 means a write collided with current server state: optimistic concurrency, duplicate unique key, invalid transition. It is the only one of the four that carries a user decision. The body names the conflicting version or field; the UI shows the user's edit beside the server's value and offers reload-and-reapply or overwrite.
- 503 means the service is down or shedding load. Paint unavailable with a retry, honour Retry-After if present, and only auto-retry idempotent requests.
| Status | Who decides | Surface | Never paint |
|---|---|---|---|
| 401 | global interceptor | sign-in, intended path kept | another session's cached rows |
| 404 | call site | not-found in that region | crash overlay, empty collection |
| 409 | call site, write path | conflict state showing both versions | a generic error that discards the edit |
| 503 | call site or retry wrapper | unavailable + retry | empty collection, endless skeleton |
A trace, because 409 is where this usually goes wrong. User edits invoice 42, amount 100 to 120, and saves:
- Client sends PUT /invoices/42 with If-Match: "v7" and body {amount: 120}.
- Server's copy is v9, because a colleague edited the note. Server returns 409 with {code: "version_conflict", currentVersion: "v9", current: {amount: 100, note: "net 30"}}.
- Interceptor classifies 409 on a mutation as resolvable, does not retry, hands the parsed body to the caller. Retrying with If-Match "v9" would silently clobber the note change.
- The form stays mounted with the user's 120 still typed; a conflict banner shows server 100 versus yours 120 with Reload and Keep mine, where Keep mine re-sends with If-Match: "v9".
- No navigation, no toast-only acknowledgement, no list refetch that replaces the form with the server row and loses the typed 120.
Trade-offs worth stating out loud. Centralised interceptor versus per-call handling: 401 belongs globally because nothing about the resource changes the answer. 404, 409 and 503 are resource-scoped, so the interceptor classifies and the call site paints; a widget that 404s inside a dashboard must not blank the dashboard. Toast versus dedicated state: toasts suit actions whose result is already visible elsewhere, so a toast for a failed page load leaves an empty table sitting behind it; anything that replaces data needs its own state. Client map versus server-driven: HTTP status classes are stable, so classification lives in the client, while the server supplies resolution data (current version, field, retry seconds), not display strings. Unknown statuses fall back by class: unknown 4xx terminal, unknown 5xx retryable, neither empty.
Never paint: an empty collection for any non-2xx, a crash overlay for a per-resource 404, a success row for a write the server rejected, another user's cached content after a 401, or an endless skeleton where a timeout should have turned loading into unavailable.
Validation is a fixture at the HTTP boundary stubbing each status, asserting the rendered surface, plus a page test that no empty-state node appears when status >= 400. Instrument the counter for empty results following an error; when it moves, the map has a hole.
Curated: · Written: · Reviewed:
Q&A-12When the query returns no rows, when it is still in flight, and when it failed, how must the API and the page disagree on purpose?
What I'd say first. Four states, four contracts: loading, empty, success-with-rows, error. The API's job is to make "the query ran and matched nothing" distinguishable from "the query did not run". The page's job is to never render one as the other. They disagree on purpose: one API outcome can map to more than one surface, and no surface may ever be inferred from an absent body.
The contract. Successful query, zero matches → 200 with an explicit empty collection and the envelope intact: {"items": [], "total": 0, "next_cursor": null}. In flight → no server contract at all; that is client state, held only while the request is open. Failure → 5xx or a timeout, never a 200 with an empty body.
Where the alternatives sit.
| Choice | What it buys | What it costs |
|---|---|---|
200 + items: [] | one parse path; carries total and cursor | slightly larger body; client must check items.length |
| 204 No Content | smallest response | no body, so no total or cursor; res.json() throws on an empty body, so every client needs a special case |
| 404 for empty | — | conflates "collection exists, empty" with "collection missing"; breaks caching and error handling |
204 is right for DELETE or a genuinely representation-less resource, not for a list read that has metadata to carry. 404 for an empty collection is the one I would push back on hardest.
Retry. Retry only idempotent reads, only on timeout, connection reset, or 5xx: never on 4xx. GET is idempotent by HTTP semantics, but the retry is only safe if the read is side-effect free: a search endpoint that logs impressions or bills per query makes the retry non-idempotent in effect, and that is when you cap it server-side or drop the automatic retry. Jittered backoff, bounded attempts, and the manual "try again" button bounded too.
When not to use this. If the API cannot distinguish, a legacy endpoint that returns 200 with an empty body on error, the client must not guess. Surface an error, not an empty state. And never let the spinner be the empty state: a client-side timeout that flips to error is mandatory, otherwise "still in flight" and "nothing here" are the same pixels.
Failure modes and how I'd detect them. Spinner-as-empty: count empty renders preceded by more than 2 s in flight. Double-submit: duplicate requests per session. Retry storm: request rate against a degraded replica.
The incident. Search showed a spinner for 5.8 s, then an empty table on a gateway timeout. 428 users reran the query; the API logged 1,284 extra reads against a replica already saturated. The page had inferred "no results" from "no response".
The disagreement, stated. 200-empty on first load → "nothing here yet, create one". 200-empty after a filter change → "no matches, clear the filter". Same API outcome, two surfaces. Timeout → error with retry. In flight → spinner, and only in flight.
How I'd check it. Drive the client with a slow 200 empty body, a 200 with rows, and a timeout; confirm three distinct surfaces and that the empty state never appears before the response lands.
Zero rows is a successful 200. A timeout is not zero rows.
Curated: · Written: · Reviewed:
Q&A-79The API returns a useful error body. How do you make that error accessible in the page?
The 90-second answer
The API body is data, not UI. Three things have to happen before a screen reader user learns why the submit failed: the status and code are mapped to a sentence a human can hear, that sentence lands in a node that was already in the DOM, and focus moves to it once. Then the field that caused it carries the same sentence through aria-describedby and aria-invalid="true".
Never render the server string directly. 422 and {"errors":[{"field":"email","code":"invalid_format"}]} are not words a user can hear, they leak internals, and echoing the body through innerHTML is an XSS sink. Map code to copy in the client, and write it with textContent.
<form id="signup" novalidate>
<div id="error-summary" tabindex="-1" aria-labelledby="error-summary-title" hidden>
<h2 id="error-summary-title">There is a problem</h2>
<ul id="error-summary-list"></ul>
</div>
<label for="email">Email</label>
<input id="email" name="email" aria-describedby="email-error">
<p id="email-error"></p>
<button type="submit">Create account</button>
</form>
// Plain HTML + ES2020. replaceChildren/focus/setAttribute are baseline in current browsers.
const COPY = {
invalid_format: 'Enter an email address',
too_short: 'Password must be at least 12 characters',
};
form.addEventListener('submit', async (e) => {
e.preventDefault();
const res = await fetch('/api/signup', { method: 'POST', body: new FormData(form) });
if (res.ok) return;
const body = await res.json();
renderErrors(body.errors);
});
function renderErrors(errors) {
list.replaceChildren(...errors.map((err) => {
const li = document.createElement('li');
const a = document.createElement('a');
a.href = `#${err.field}`;
a.textContent = COPY[err.code] ?? 'Check this field'; // textContent, never innerHTML
li.append(a);
return li;
}));
summary.hidden = false; // must precede focus(): a hidden node is not focusable
summary.focus(); // tabindex="-1" makes the div a focus target
for (const err of errors) {
const input = document.getElementById(err.field);
input.setAttribute('aria-invalid', 'true');
document.getElementById(`${err.field}-error`).textContent = COPY[err.code] ?? 'Check this field';
}
}
Trace. A 422 returns {"errors":[{"field":"email","code":"invalid_format"},{"field":"password","code":"too_short"}]}.
| Step | DOM after the step | Exposed to assistive tech |
|---|---|---|
replaceChildren | two <li><a href="#email"> items | nothing yet: summary still hidden |
summary.hidden = false | summary visible | nothing yet: focus has not moved |
summary.focus() | focus on #error-summary | heading "There is a problem", list of 2 items, each a link |
| loop | #email aria-invalid="true", #email-error filled | on Tab to #email: "Email, edit, invalid entry, Enter an email address" |
Trade-offs. Focus and role="alert" on the same node double-announce: the live region fires on insertion, then focus re-reads it. Pick one. For submit-triggered errors, focus the summary and skip the live region: the user's focus was on the button, so moving it is expected. For errors that arrive with no user action (autosave, websocket), use a persistent empty <div role="alert"> and set its textContent; mounting <div role="alert">{msg}</div> in one render is missed by some screen readers because the region was not in the accessibility tree when the text arrived. Prefer aria-describedby over aria-errormessage: the latter has uneven screen reader support.
Failure modes. A red border with no text fails WCAG 2.2 1.4.1 (Use of Color) and 3.3.1 (Error Identification). role="alert" on inline validation interrupts on every keystroke; use aria-live="polite" or nothing there. A 500 or a dropped connection has no field to attach to: render a page-level message, keep the user's input, offer retry. Clear aria-invalid on input so a corrected field stops being announced as invalid.
Verify with NVDA + Firefox, JAWS + Chrome, and VoiceOver + Safari: submit an invalid payload, confirm each message is spoken once, focus lands on the summary, and Tab reaches the first invalid field with its message read. WCAG 2.2 3.3.3 (Error Suggestion) and 4.1.3 (Status Messages) are the criteria to cite.
Curated: · Written: · Reviewed:
Q&A-86Users see a 500 only after the page retries. How do you debug that as a full-stack failure?
The decisive question is whether the retry causes the 500 or merely arrives into a system the first request already broke. Those two branches need different evidence, so I prove which one I'm in before touching a fix.
1. Capture both requests with a shared fingerprint. Reproduce it deterministically: in devtools set the network to offline for ~250 ms on the first POST, or put a proxy in front that aborts the first response. Then log, on both client and server, a fingerprint of the request (method, path, user, and a hash of the body) alongside the request ID. The tell is two requests with the same body hash and different request IDs.
12:04:01.220 POST /checkout rid=7f2a body_sha=9c1e -> 201 charge=ch_881
12:04:01.470 client aborts after 250 ms (response never read)
12:04:01.480 POST /checkout rid=b31d body_sha=9c1e -> 500
duplicate key value violates unique constraint "charges_idem_key"
sqlstate 23505
Same body_sha, different rid, and the 500 is a unique-constraint violation: the retry re-ran a write that already committed. That is the duplicate-work branch. If instead the second request 500s with a pool-acquire timeout and you see dozens of distinct rids in the same second, you are in the retry-storm branch and the write path is a red herring.
2. Read the actual 500, don't infer it. The error class discriminates the cause faster than any hypothesis:
| Evidence in the logs | Cause | Next check |
|---|---|---|
| 2nd request, same body hash, unique violation (23505) | non-idempotent write retried | does the handler accept a client key? |
| 2nd request, pool-acquire timeout, many rids | retry storm saturating the pool | in-flight count, pool wait time |
| 2nd request lands on another instance, 500 = missing session/cache row | no shared state across instances | sticky routing or shared store |
| only POSTs 500, GETs fine | write path specifically | compare the two handlers |
3. Rule out the neighbours. A retry after a token refresh that 401s, a CDN re-fetching a GET, and a first request that is still in flight when the second arrives (a genuine race, not a lost response) all look similar from the browser. The fingerprint separates them: a race shows two requests overlapping in time, a lost response shows the first completing before the second starts.
4. Fix, with the trade-off stated.
- Idempotency key: client generates a UUID per logical operation, sends it on every attempt; server stores it under a unique index with the response and returns the stored response on replay. Correct for any write, but costs a table, a TTL, and stored response bodies, and the key must be scoped per user.
- Natural-key dedupe: a unique constraint on something the domain already has, e.g.
(user_id, cart_version). No client change, but only works when such a key exists. - Retry budget with exponential backoff and jitter, retrying only idempotent methods: cuts the frequency of the storm, does not make a duplicate write safe. Use it alongside a key, not instead of one.
Keys are not worth it for GETs, for naturally idempotent PUT/upsert writes, or when the client already supplies a natural ID.
Verify: abort the first write, allow exactly one retry, and assert the server returns the original result with one row written, not a 500 and not a second row.
Curated: · Written: · Reviewed:
Q&A-36A virtualized inbox paints 20 rows but the API returns 500. How do you size both halves?
Virtualization and page size are independent knobs, and the common mistake is treating one as the fix for the other. The window decides how many rows exist in the DOM; the page size decides how many rows cross the wire. A 20-row window over a 500-row response still parses 500 objects, keeps them in memory, and re-renders on every store mutation.
Default: page the server to roughly two to three painted windows. With 20 rows visible, request 40-60 rows per cursor page and fetch the next page when the scroll position is within one window of the loaded end. That keeps a fast flick ahead of the network without holding the mailbox in memory.
Three alternatives are worth weighing.
Fetch everything once. If a row is small (id, subject, timestamp, read flag at ~200 bytes) 500 rows is 100 KB, maybe 25 KB gzipped, one round trip. That beats 17 cursor requests at 50 ms RTT each (850 ms of pure latency) and gives instant client-side search and sort. The 500-row response is only wrong here because these rows are 8 KB.
Offset pagination. Wins when you need random access: jump to message #340, or a scrollbar that maps to absolute position. It also wins when the backend has no stable sort key to seek on. It loses on a prepend-heavy inbox: new mail arriving at the top shifts every subsequent row, so page 2 re-serves rows you already rendered. You see duplicates in the list and, if you dedupe by index, skipped messages.
Cursor/keyset. Wins on append- and prepend-heavy feeds with a stable key (received_at, id). Costs you random access and requires the sort key to be unique and immutable.
Bigger pages. A 200-row page cuts request count 4× versus 50 and is the right call on high-latency links (200 ms RTT) where each request is expensive, or when the user is scrolling fast and you would rather over-fetch than show blanks. It costs memory and parse time proportional to bytes, not rows.
| Rows/page | Bytes at 8 KB/row | Requests to cover 500 | Latency at 50 ms RTT |
|---|---|---|---|
| 500 | 4.0 MB | 1 | 50 ms |
| 100 | 800 KB | 5 | 250 ms |
| 50 | 400 KB | 10 | 500 ms |
| 30 | 240 KB | 17 | 850 ms |
The 500-row option is not slow because of request count; it is slow because 4.0 MB has to be transferred, parsed and retained. At 200 bytes/row the same table flips: 500 rows is 100 KB and one request, and 30-row pages are strictly worse.
Failure modes worth checking. A page smaller than the window gives blank rows on a fast flick and one request per wheel event. A page size that is not a multiple of the window leaves a partial window that triggers a fetch for three rows. Offset pagination on a mutating list produces duplicate keys in the rendered list: count distinct ids against rendered rows. A cursor over a non-unique sort key skips rows; assert the union of pages equals the full set.
Detection: count network requests per 1000 px of scroll, and measure time from scroll to first painted new row. Past ~100 ms you are under-fetching. If heap grows past a few MB for a list the user has scrolled ten screens, you are over-fetching.
Curated: · Written: · Reviewed:
Q&A-41A user saves a form and the next page is empty. How do you handle replica lag in the UI?
What I'd say first
Two things produce an empty page after a save: the write failed, or the write succeeded and the read missed it. Check which before touching the UI: a 2xx on the mutation with the persisted row in the body rules out the first. The rest is read-your-writes: the session's next read must come from a source that has the commit, or the UI must keep showing what it already has until one does.
Mechanism
Async replication means the primary commits and returns before the replica has replayed the WAL. Lag is bursty, not uniform: p50 is often sub-millisecond while p99 spikes to seconds under a long-running transaction or a bulk load. That is why the bug is intermittent and load-dependent, and why it survives staging.
The UI consequence is a probability, not a single event. If 2% of post-write reads land on a lagging replica (hypothetical) and a session does 10 of them, the chance of at least one empty page is 1 − 0.98¹⁰ ≈ 18%.
Three levers, cheapest first
- Render from the save response. The mutation already returned the persisted row. Put it in the client store and render the next view from it: no read at all. This covers most "save, then open detail" flows.
- Pin the session. After a write, route that session's reads to the primary for a bounded window, or until the replica's replay position passes the write's position.
- Write token. The mutation returns a position (PostgreSQL:
pg_current_wal_lsn()); the next read carries it, and the read layer either waits for a replica at or past that position or falls back to the primary.
// Node 20 / TypeScript 5.4 — read routing sketch, PostgreSQL 16
async function readAfterWrite(token: string | undefined, sql: string) {
if (!token) return replica.query(sql);
const { caught_up } = (await replica.query(
'SELECT pg_last_wal_replay_lsn() >= $1::pg_lsn AS caught_up', [token]
)).rows[0];
return caught_up ? replica.query(sql) : primary.query(sql);
}
Trace of the token path, hypothetical timings:
- t=0 ms: POST /form commits on the primary at LSN 0/16B3740; the response carries the row and the token.
- t=40 ms: client navigates; GET /next sends the token.
- t=40 ms: replica replay LSN is 0/16B2F10, behind, so the read goes to the primary.
- t=41 ms: primary returns the row; the page renders saved fields.
Without the token, step 3 returns zero rows and the page renders empty defaults.
Failure modes
- Token never satisfied. Replica down or lag unbounded → the read must time out and fall back to the primary. Otherwise you have swapped an empty page for a spinner.
- Pinning window too short. User thinks for 30 s, window was 5 s, still empty. Tie the window to the token, not the clock.
- Pinning everything. If every read after any write goes to the primary, a write-heavy burst doubles primary read load and the replica earns nothing.
- Optimistic render of a failed write. Only render client-side state after a 2xx; otherwise the user sees data that was never persisted.
- Multi-tab. Session pinning is per-session, so tab B's read is not covered. Share the token via cookie or localStorage, or accept the miss.
- Double submit. Empty looks like "not saved", so users re-submit. Make the save idempotent with a client-generated key so a retry cannot overwrite a later edit.
What I'd verify
Force replica delay in staging (pause WAL replay, or a proxy that delays replica responses) save, and confirm the next view shows the saved fields. Measure the p99 of the token wait: if it is dominated by lag rather than by the primary read, the fallback is doing the work and the replica is not.
Curated: · Written: · Reviewed:
Q&A-50A manager opens Insights and the request times out. What does the page load instead?
What the page loads
A precomputed snapshot, not the aggregate. The Insights page issues one indexed lookup against a summary table keyed by (tenant, metric, time bucket) and renders it with the snapshot's as-of timestamp. If the manager's requested range isn't covered by the snapshot, the page loads a job handle and a placeholder that polls: the request that paints the layout never touches the fact tables.
Why the timeout happens
The failing request is an aggregate over the fact tables: a multi-table join across months of rows, executed inside the page request. That work scales with the data, not with the page, so it collides with the gateway timeout (30 s is a common default) and runs on the primary that is also serving writes.
The read path
-- page load: one index range scan, no fact tables
SELECT bucket, value, computed_at
FROM insights_daily
WHERE tenant_id = $1
AND metric = $2
AND bucket >= $3 AND bucket < $4
ORDER BY bucket;
The snapshot is built by a scheduled job (nightly, or every N minutes for fresher metrics) that writes one row per bucket. The page returns computed_at so the UI can say "as of 02:00".
For an ad hoc range, the page enqueues a job with a dedupe key of (tenant, metric, range) and returns 202 with a job id; the tab polls /jobs/{id} and stays interactive. Same query, off the request path.
Trade-offs
| Snapshot read | Job + poll | |
|---|---|---|
| Freshness | as of last refresh | exact range |
| Latency | single-digit ms | seconds to minutes |
| Cost | one scheduled scan | one scan per distinct range |
| Failure mode | stale data shown | job queue backlog |
Alternatives: a materialized view refreshed on a schedule is the same idea with less code; an OLAP/columnar replica moves the scan off the primary but still spends the request's latency; a cache in front of the aggregate helps only if the key space is small and the TTL is acceptable. Don't reach for any of these when the data is small enough to aggregate inline: a few thousand rows over a bounded range is fine on the request path.
Failure modes
- Snapshot job fails: the page serves yesterday's numbers with an old as-of. Detect with an alert on refresh lag, not on page errors.
- Stale snapshot served silently: the as-of timestamp is the contract; alert when its age exceeds the refresh interval.
- Job enqueue storm: many managers request overlapping ranges. Dedupe on the range key and cap concurrent jobs.
- Snapshot table grows unbounded: partition by bucket and set retention.
Worked example (figures illustrative)
A 14-table join over 18 months of 90M rows ran past the 30 s gateway timeout and returned nothing. Replacing it with a nightly summary of ~12,000 rows made first paint ~400 ms; the ad hoc range became a job that never blocks the tab.
What I'd check
Open Insights during peak writes and confirm the page query's plan hits insights_daily, not the fact tables; request an ad hoc range and confirm the tab stays interactive and the job result matches a direct query.
Curated: · Written: · Reviewed:
Q&A-56A refund flow is hurting users. How do you turn it off without waiting on a deploy?
What I'd say first
Assume the flag service is already wired into both the web client and the API, and that "off" has to mean no new refunds are created, not just that the button is gone. The answer is one named flag, evaluated server-side on every mutating request, with the client reading that same flag to hide the control. The server check is the one that matters: it is the only one that stops a caller who never loaded your UI.
Mechanism
Resolve the flag inside the refund handler, before any state changes, and return a documented disabled status when it is off. Do not resolve it at startup or bake it into the build: a flag read once at boot needs a deploy to change, which is the thing we are avoiding. The client fetches the same flag and hides the refund control; that is a UX improvement, not a control.
Keep the flag default-off in the flag service so a missing or failed evaluation fails closed. If the SDK cannot reach the flag service, the handler should treat that as off for a mutating money path.
Failure modes
- Client-only flag. The endpoint stays live. Anything that can POST (a retry queue, an admin script, a mobile build that has not updated) keeps creating refunds. Detect by alerting on refund creation rate, not on UI errors.
- Propagation lag. Flag SDKs cache; a kill switch flip can take seconds to tens of seconds to reach every instance. During that window requests still succeed. Detect by checking the flag's evaluation timestamp in logs, and by knowing your SDK's polling interval before you promise an incident commander "it's off".
- Server-only flag. The button stays visible and every click errors. Users retry, which multiplies load on a path you just disabled. Detect by watching 4xx/5xx rate on the refund endpoint after the flip.
- Bypass paths. Service-to-service refund calls and internal tooling never see the client flag. The server check covers them; the client check does not.
The failure that motivates the server check
A team hid the refund button in the client and left the endpoint live. A leftover admin script kept calling it and issued 387 refunds in 8 minutes before anyone noticed. The UI looked correct the whole time.
| Switch location | Button visible | Endpoint accepts POST | Refunds in 8 min |
|---|---|---|---|
| client only | no | yes | 387 |
| server only | yes | no | 0, UI errors |
| both | no | no | 0 |
How I'd confirm it
Flip the flag in staging. Confirm the control disappears on a fresh page load and that a direct POST to the refund endpoint returns the documented disabled status. Then flip it back and confirm refunds work again: a kill switch you cannot un-kill is its own incident.
Curated: · Written: · Reviewed:
Q&A-44A cart shows a total that does not match the charge. Who computes the money?
The part of derived totals client versus server interviewers probe is the mismatch after deploy, not the happy render.
Display math in the browser is a preview. The charge, tax, and discount the customer pays are computed on the server from the same rules that write the ledger, and if the UI total is authoritative a patched client can pay the wrong amount.
Concretely, compute a preview in the client for snappy updates. On checkout, ignore that preview and persist the server total. Re-render the confirmation from the server figures, and block pay if the preview drifted, while still charging the server amount.
The reason for that specificity is a failure I have seen: The client applied a 15 percent coupon twice, showed 42.50, the server charged 85.00, and 2,200 orders that week generated support tickets because the confirmation then switched numbers.
Coupon applied twice in the browser.
| Total owner | Screen | Charged | Tickets in a week |
|---|---|---|---|
| client preview | 42.50 | 85.00 | 2,200 |
| server ledger, UI re-renders | 85.00 | 85.00 | 0 |
I would not consider it settled without evidence: Mutate the client total in a test and confirm the charged amount still matches the server ledger.
Preview in the client and charge what the server computed.
Curated: · Written: · Reviewed:
Q&A-46Tax of 8.875 percent on 3 line items disagrees between the UI and the invoice. How do you round?
What separates a strong answer on money rounding on both halves is knowing which half can lie to the other.
Rounding mid-stream in the client and again on the server produces pennies that do not match. Pick a currency minor unit, a rounding mode, and a single place that rounds, then have the UI display that already rounded amount.
Concretely, keep integer minor units in the API. Apply tax and discounts in a documented order on the server. Return the rounded line and grand totals, display those integers, and do not round each line in JavaScript and then again in SQL.
The reason for that specificity is a failure I have seen: The client rounded each of 3 lines to 2 decimals then summed, the server summed then rounded, and a 0.01 gap on 16,800 invoices that month failed a card settlement match.
Three 12.99 lines, tax 8.875 percent, half-up to cents on the grand total.
| Who rounds | Line cents | Tax cents | Grand cents |
|---|
| each line tax half-up then sum | 1299, 1299, 1299 | 345 | 4242 |
| sum 3897 then tax half-up on the server | 1299, 1299, 1299 | 346 | 4243 |
| server grand 4243, UI displays 4243 | 1299, 1299, 1299 | 346 | 4243 |
I would not consider it settled without evidence: Price a cart with repeating tax decimals and confirm the UI, the API response, and the ledger row share one integer total.
Round once, on the server, in minor units.
Curated: · Written: · Reviewed:
Q&A-45A meeting at nine in the morning in Paris looks like nine in the morning in New York after a refresh. What is stored and what is formatted?
Two different things get stored, and conflating them is the bug.
The instant, the point on the timeline, is stored as UTC. The zone (an IANA name like Europe/Paris, not an offset) is stored beside it whenever the event has a local wall-clock meaning. Formatting happens at render time, in the viewer's zone. Nothing is formatted at write time.
For a one-off meeting that is the whole answer. For a recurring or far-future event it isn't, and that is where the real trade-off sits.
Why the instant alone breaks. A weekly 09:00 Mexico City standup stored as one UTC instant per occurrence is a snapshot of today's DST rules. Mexico abolished DST in 2022; every stored instant for the second half of the year was suddenly an hour off, and the fix was rewriting rows. Store 09:00 + America/Mexico_City + FREQ=WEEKLY;BYDAY=MO instead and each occurrence is computed from the current tzdb at read time: the rule change is absorbed for free. tzdb ships exactly this kind of change, and releases are named like 2024a.
The reverse also holds: wall time plus zone is wrong for a one-off absolute instant, because if the zone's rules move, the instant moves with them. A flight departure, a signed contract timestamp, an audit-log entry: those are instants, store UTC.
So: instant for things pinned to a fixed point in time; wall time + zone for things that recur or sit far enough out that the rules may move.
The failure this prevents. A nine-in-the-morning Paris kickoff stored as a zoneless wall clock of 9, rendered by New York clients as 9 AM Eastern:
| Stored | New York screen | Outcome |
|---|---|---|
zoneless 09:00 | 9:00 AM ET | 480 attendees join 6 h late |
2024-01-15T08:00Z + Europe/Paris | 3:00 AM ET | correct instant; Paris still reads 09:00 |
480 attendees × 6 h is 2,880 person-hours from one missing field.
The shape. One-off events:
// Node 20, no libraries
const start = "2024-01-15T08:00:00Z"; // the instant, stored
const zone = "Europe/Paris"; // the wall meaning, stored
new Intl.DateTimeFormat("en-US", {
timeZone: "America/New_York", dateStyle: "short", timeStyle: "short"
}).format(new Date(start)); // "1/15/24, 3:00 AM"
Recurring events store the rule instead:
{ "wall": "09:00", "zone": "Europe/Paris", "rrule": "FREQ=WEEKLY;BYDAY=MO" }
Failure modes and how you catch them.
- Zoneless string parsed as browser-local on one path and UTC on another. Catch it with a round-trip test: create in
Asia/Tokyo, read inAmerica/New_York, assert the instant is byte-identical and the rendered clock matches the viewer's zone. - Offset stored instead of a zone name.
+01:00carries no DST rules, so a July occurrence renders an hour off. Catch it by asserting the stored zone resolves throughIntl.supportedValuesOf('timeZone')and rejecting bare offsets. - Ambiguous and nonexistent wall times. On fall-back day 02:30 occurs twice; on spring-forward day it doesn't exist. Catch it by rejecting a wall time that maps to zero or two instants unless the caller supplies an explicit offset.
- tzdb drift. Bump tzdb in CI, recompute every stored recurring event, diff the instants. Anything that moved is a row you would have had to rewrite under the instant-only design.
What I'd check before calling it done: create an event in one zone, open it in another, confirm the instant is identical and the displayed clock matches the viewer's zone, then repeat with a recurring event across a DST boundary.
Curated: · Written: · Reviewed:
Q&A-33The growth dashboard and the billing table disagree. Which number does the product UI show?
Short answer. The UI shows the ledger number. Anything a customer can be billed for, entitled to, or will dispute comes from the transactional store, read through the same API that generates the invoice. Analytics counts appear only where nothing is payable, and when they sit next to a ledger figure they carry a label saying they are approximate.
Why the two numbers diverge
Analytics events are a sampled, delayed, lossy trail of what the client fired. The dedupe key is usually a client-generated ID, so retries and offline flushes double-count; bot filtering runs in the warehouse, not at ingest; a client that fires before the entitlement check records an event the ledger never writes. The ledger row is written inside the same transaction that grants the entitlement, so its count and the invoice line are the same row. They are not two measurements of one quantity. They are two different quantities that happen to share a name on a dashboard.
What to do
- Fire analytics from the client for funnels, activation and feature adoption. Nothing payable rides on those numbers.
- Read billed totals from the API that reads the ledger, and point the in-app usage page at that same API.
- If a marketing tile needs both, label the analytics figure approximate and link the ledger figure as the one that can be invoiced.
Worked example
One tenant, one month. Numbers are illustrative.
| Source | Count | What it may be used for |
|---|---|---|
| Client events landed in the warehouse | 1,800,000 | Funnel and adoption charts only |
| Billing ledger rows | 1,410,000 | Invoices, entitlement, in-app usage page |
| Warehouse count after bot filtering | 1,520,000 | Nothing payable; still not the invoice |
1,800,000 − 1,410,000 = 390,000 events the customer can see but cannot be billed for. 1,520,000 − 1,410,000 = 110,000 that survive bot filtering and still are not billable, because they are retries, duplicate client IDs, or events fired before the entitlement check.
Failure mode and detection
A usage meter painted 1.8 million from the warehouse while invoices used 1.41 million, and 64 customers disputed the 390,000 gap shown on the in-app usage page. The detection that would have caught it earlier is a nightly per-tenant reconciliation comparing three values: the in-app total, the ledger sum, and the warehouse count. Alert when the in-app total and the ledger sum diverge by more than a rounding threshold. The fix is not to reconcile harder. It is to make the usage page read the ledger API so the two numbers cannot diverge on screen in the first place.
When analytics is the right number
Funnels, activation, feature adoption, and any trend line where the shape matters and the absolute value does not. Label it in those places too, because a reader who has seen the ledger figure elsewhere will assume this one is payable.
Trade-off
The ledger API is slower and lower-throughput than the warehouse. A dashboard that fans out per-tenant ledger queries will need caching. Cache the ledger read, not the analytics read, and key the cache by tenant and billing period so a late-arriving ledger row invalidates it rather than being masked by a stale total.
Show money from the ledger, not from the event pipe.
Curated: · Written: · Reviewed:
Q&A-29How do you take payment and decrement stock so the confirmation screen and the warehouse agree?
What I'd say first
The confirmation screen and the warehouse must read the same durable record: a server-side order row. Neither the wallet callback nor the client's local state is allowed to trigger the charge or the stock decrement. Concretely: reserve stock with a TTL, create an idempotent payment intent server-side, write the reservation and the intent request through an outbox in one local transaction, and have the confirmation page poll the order id until it reads PAID.
Why one transaction can't cover it
The card processor is an external system with its own commit point. You cannot hold a database transaction open across a network call to it: the lock lives for the duration of the call, and a timeout leaves you unable to tell whether the charge happened. So checkout is a state machine with compensation, not a single atomic write.
States: RESERVED → PAYMENT_PENDING → PAID, with RELEASED as the compensation branch.
Reservation is the atomic guard against overselling:
UPDATE inventory
SET reserved = reserved + 1
WHERE sku = :sku
AND on_hand - reserved >= 1;
Zero rows updated means out of stock: fail before touching the processor. on_hand is not decremented yet; the unit is held, not sold.
Then, in the same local transaction, insert the outbox row that will create the payment intent with idempotency_key = order_id. A relay publishes it. The processor's capture webhook moves the order to PAID and, in that same transaction, does on_hand = on_hand - 1, reserved = reserved - 1.
Worked example (hypothetical)
One day, 12,600 checkouts, two implementations:
| Success signal | Paid without stock | Warehouse shorts |
|---|---|---|
| Wallet callback, then a separate stock POST | 410 | 410 |
| Reserve → intent → outbox → UI polls order | 0 | 0 |
In the first row the client showed "paid" on the wallet callback and then POSTed the stock decrement; that POST timed out at 8 s on 410 orders. The customer saw a confirmation, the warehouse never got the decrement.
Failure modes to design for
- Capture times out after the reservation. The order stays
PAYMENT_PENDING; the reservation TTL (15 min) expires and releases the unit. The UI must keep showing pending, never paid. Detect by alerting on orders stuck inPAYMENT_PENDINGpast the TTL. - Duplicate capture webhook. Idempotency on
order_idmakes the second one a no-op; without it you decrement twice. - Outbox relay down. Reservations accumulate but no intent is created; alert on outbox lag, not on order volume.
- TTL expiry racing a late capture. Capture must be conditional on the order still being
PAYMENT_PENDINGand the reservation still held; if the reservation was released, refund rather than ship.
What I'd verify before calling it done
Kill the processor mid-capture, confirm the UI stays pending, the reservation releases at TTL, and no paid confirmation is rendered. Then replay the capture webhook twice and confirm on_hand moves once.
Curated: · Written: · Reviewed:
Q&A-47The product page says In stock. The user pays and it is not. What reservation does the click create?
The click on the product page creates no reservation. A stock badge is a cached read of a number that was true at some point in the past, and no amount of care at render time turns it into a promise. The reservation belongs to the transition into checkout: when the user commits to buying, you write a hold keyed to that checkout session, with an expiry, in the same transaction that decrements what other shoppers can see.
Why it has to be one transaction and not a check-then-write: two sessions reading "1 left" and then both inserting a hold is the classic lost update. Make the database enforce it.
CREATE TABLE stock (
sku text PRIMARY KEY,
on_hand int NOT NULL,
reserved int NOT NULL DEFAULT 0,
CHECK (on_hand - reserved >= 0)
);
CREATE TABLE reservation (
id uuid PRIMARY KEY,
sku text NOT NULL,
qty int NOT NULL,
session_id text NOT NULL,
expires_at timestamptz NOT NULL,
state text NOT NULL DEFAULT 'held', -- held | captured | released
UNIQUE (session_id, sku)
);
Reserving two units:
UPDATE stock SET reserved = reserved + 2
WHERE sku = 'SKU-1' AND on_hand - reserved >= 2;
-- rowcount 0 -> 409 sold out; rowcount 1 -> insert the reservation row, commit
Trace it under Postgres READ COMMITTED with on_hand = 1, reserved = 0, sessions A and B both POSTing /checkout for one unit:
- A's UPDATE takes the row lock, sets
reserved = 1, rowcount 1. - B's UPDATE blocks on the same row.
- A inserts its reservation row and commits.
- B's UPDATE unblocks; Postgres re-evaluates the WHERE against the committed row, where
on_hand - reserved = 0, so rowcount is 0. B returns 409 and the page shows unavailable before payment.
Exactly one hold exists. B's badge may still say In stock, and that is fine, because the badge is advisory and the checkout is authoritative.
Where you place the hold is a real decision, not a default:
| Hold point | Oversell risk | Stock locked by abandoners | Extra machinery |
|---|---|---|---|
| checkout start, TTL 10 min | low | up to 10 min per abandoned cart | sweeper to release expired holds |
| payment intent / capture | higher: two buyers can both reach the payment page | near zero | void or refund path when capture fails |
| no hold, oversell allowance | by design | none | backorder flow, support cost |
Reserving at capture is defensible when payment is fast and the item is restockable: you trade a small oversell rate for not locking inventory behind people who never pay. Reserving at checkout start is right when the item is scarce, non-restockable, or expensive to refund. A softer approach, allow a 1-2% oversell and email a backorder, wins on high-margin, long-lead goods where rejecting a buyer at the lock costs more than the apology. If stock lives in Redis, a Lua script doing DECR with a floor gives the same atomicity, but then the cache is the source of truth and needs its own durability story.
Hold duration matters as much as hold placement. The TTL must exceed the slowest realistic payment path, or you can capture money against a hold that already expired. Extend the TTL when the payment intent is created rather than setting it once, and make the read path filter expires_at > now() so an expired row is invisible even before the sweeper runs.
Failure modes worth designing for:
- Reservation leak: a committed hold whose checkout is abandoned. Detect it by counting
heldrows pastexpires_at; a sweeper releases them and decrementsreserved. - Retry double-hold: the
UNIQUE (session_id, sku)constraint plus an idempotency key on the checkout call. - Clock skew: compute
expires_atwith the database'snow(), not the app server's. - Replica lag on the badge: reads from a replica show stale availability, so never gate the sale on them.
Verification: fire two concurrent checkouts for the last unit and assert one 200 and one 409, then let the winner's hold expire and assert the unit returns to available.
Curated: · Written: · Reviewed:
Q&A-28A dashboard paints 9 tiles from live joins and is slow. What changes on the write path and on the page?
The 90-second answer. Two changes. On the write path, the transaction that changes the facts also upserts one summary row per tile, so the aggregate is computed once per write instead of once per page paint. On the page, the dashboard fetch reads one row per tile from that summary table, no joins, and renders a generated_at timestamp beside the tiles so staleness is visible instead of assumed.
Assumptions worth stating out loud: the nine tiles are aggregates (counts, sums, min/max) over the fact tables, not row-level drill-downs; a few seconds of staleness is acceptable; the store is a relational database with real transactions.
Write path. In the same transaction as the fact insert, run an idempotent upsert into a tile_summary table keyed by (tile_id, bucket). Guard it against out-of-order replays:
INSERT INTO tile_summary (tile_id, bucket, total, updated_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (tile_id, bucket) DO UPDATE
SET total = tile_summary.total + EXCLUDED.total,
updated_at = EXCLUDED.updated_at
WHERE EXCLUDED.updated_at > tile_summary.updated_at;
The WHERE clause is the part people skip: without it, a retried or replayed event silently double-counts. If the write path can't touch the summary table directly (separate service, separate store) use a transactional outbox: write the fact and an outbox row in one transaction, and let a consumer apply the upsert. Never dual-write to two stores without the outbox; a crash between the two writes is exactly how drift starts.
Page. The dashboard query becomes SELECT tile_id, total, updated_at FROM tile_summary WHERE ...: one row per tile, no joins, index-only scan. Render updated_at as "as of HH:MM:SS". Keep the HTTP cache short (a few seconds) so a refresh after a write shows the new number.
Alternatives and when not to use this. A materialized view with REFRESH MATERIALIZED VIEW CONCURRENTLY (needs a unique index) is fine when the tiles are global and a periodic refresh is acceptable: it's less code but the whole view recomputes. A cache in front of the live join helps only if the join is cheap enough to recompute on miss. Denormalization is the wrong call when tiles are per-user, per-filter, or high-cardinality drill-downs: you'd be maintaining a summary row per dimension combination, and the write amplification beats the read savings.
Failure modes. Drift from a missed or replayed event; out-of-order updates overwriting a newer total (the guard above); a backfill that rewrites history without recomputing summaries; a rolled-back transaction that already emitted an event. Detect all of them the same way: a nightly reconciliation job that recomputes each tile from the facts and compares, alerting on the count of mismatched tiles, plus a staleness alert on max(now() - updated_at).
Worked example (hypothetical figures). Nine tiles, each joining four fact tables on every paint:
| Read path | First paint | Drift over 14,000 days |
|---|---|---|
| 9 live joins of 4 tables | 6,800 ms | none, but slow |
| summary only in the API path | 90 ms | 220 days stale |
| summary in the transaction + nightly check | 90 ms | 0 days stale |
The middle row is the trap: moving the join out of the page but not into the write transaction just relocates the staleness.
Verification. Write a fact, refresh the dashboard, and confirm the tile equals a hand sum of the facts and that generated_at moved. If it didn't move, the write path isn't updating the summary.
Curated: · Written: · Reviewed:
Q&A-13A new checkout tender is flagged. Who evaluates the flag for the button, who evaluates it for the write, and what happens when they disagree?
Direct answer. The button is evaluated where it renders: in the client, against the flag payload the SDK or bundle already holds. The write is evaluated on the server, per request, in that request's own context. The server is authoritative. When the two disagree, the server's evaluation decides the write, and the client's stale value is corrected on the next bootstrap or stream. The disagreement itself is a bug to alert on, not a state to serve.
Why the split is forced. The client cannot be trusted with authorization: a user can edit the bundle, replay a request, or simply hold a tab open across a flag flip. So the write gate has to be server-side. The server cannot render the button, so the button gate has to be client-side. That gives two evaluations of one flag, and they only agree if they share three things: the flag key, the evaluation context (user id, tenant, environment), and the default. Percentage rollouts must bucket on the same attribute, hash the user id, or the same user lands in different buckets on each side.
The cleanest arrangement is one evaluation of the targeting rules, on the server, with the result shipped to the client at bootstrap or over a stream. The client renders the result; it does not re-run the rules. That removes rule drift and leaves staleness as the only failure mode. Most flag SDKs support both shapes: a client-side SDK that evaluates locally against a ruleset fetched at startup, and a server-side SDK that evaluates per request.
What disagreement looks like.
| Client (button) | Server (write) | What the user gets |
|---|---|---|
| on | on | button renders, POST succeeds |
| on | off | button lies; POST refused |
| off | on | control hidden; write still gated if called directly |
Row 2 generates support tickets. Row 3 generates security findings if the server gate is missing.
Failure modes and how to catch them.
- Stale bundle after a server-only flip. The client keeps rendering the button from its last bootstrap. Detect: alert on the refused-write rate for the tender endpoint, and log the flag key plus evaluation context on every refusal.
- Flag-gated route registration. If the flag controls whether the route is mounted, a refused write comes back 404, which reads as a routing bug and gets triaged as frontend. Prefer an explicit 403 with a machine-readable reason so the refusal is unambiguous.
- Client-only gating. The button hides, the API accepts. This is the security hole; the server gate is not optional.
- Server-only gating. The write is correct and the button is dead: a UX bug that looks like a backend outage.
Worked example (hypothetical). Tender pay_by_bank is at 25% rollout, bucketed on user id. A user in the bucket loads checkout; bootstrap says on; the button renders. Ops flips the flag off on the server. The user's tab still holds the old payload, so the button is still there. They tap it; the POST arrives, the server evaluates off, and returns 403. The client shows a generic error. On the next bootstrap the button disappears. The window between the flip and the next bootstrap is the exposure, and its length is the client's bootstrap interval: minutes, not seconds, unless you stream updates.
Verification. Bootstrap a flagged-off user and a flagged-on user; confirm button and write agree in both. Then flip the flag on the server only and confirm the stale bundle's POST is refused. That last step is the one that catches the split.
Curated: · Written: · Reviewed:
Q&A-42Two tenants use the same JavaScript app. How does a row from tenant A become visible to tenant B?
The row gets there through a cache key or a query predicate that never mentions the tenant. Almost nothing else.
Two halves have to hold. Server: every query is scoped by the tenant derived from the session, never from a client-supplied id or header. Client: every cache (React Query, Redux, IndexedDB, service worker, HTTP cache) is keyed by tenant and dropped on switch. Break either half and tenant A's row renders under tenant B.
The client half, traced. React Query v5, keyed by resource id only:
useQuery({ queryKey: ['invoice', invoiceId], queryFn: () => api.get(`/invoices/${invoiceId}`) })
- Tenant A opens invoice 42. Cache miss, fetch runs, entry
['invoice', 42]holds A's row. DefaultstaleTimeis 0 andgcTimeis 5 minutes, so the entry outlives the switch. - Session switches to tenant B. The component for invoice 42 remounts with the same key.
- Cache hit. React Query returns A's row synchronously and refetches in the background. B paints A's invoice, then it is replaced a few hundred ms later: long enough to screenshot.
Fix: queryKey: ['invoice', tenantId, invoiceId] plus queryClient.clear() on switch. The tenant id in the key must come from the session response, not a URL segment the user can edit.
The server half. SELECT * FROM invoices WHERE id = $1 returns A's row to B's session. WHERE id = $1 AND tenant_id = $2, with $2 taken from the session, does not. Postgres row-level security can enforce this so a forgotten predicate fails closed instead of leaking.
Where to enforce it, and when to go heavier.
| Model | Isolation | Cost | Use when |
|---|---|---|---|
Shared schema, tenant_id + RLS | Row-level | One deployment, one migration path | Hundreds of B2B tenants, no residency requirement |
| Schema per tenant | Table-level | Migration per schema, search_path per connection | Tens of tenants, per-tenant backup/restore needed |
| Database or deployment per tenant | Full | N deployments, N connection pools | Regulated data, residency, or one tenant's load would starve others |
Deciding criteria: tenant count, whether one tenant's traffic can degrade the rest, and whether a contract or regulator demands physical separation. Shared schema with RLS is the default for SaaS; reach for schema-per-tenant when you need per-tenant restore; separate deployments only when isolation is contractual, because the ops cost is real.
Other leak vectors, and how to catch them.
- Service worker caching API responses: the cache name isn't tenant-scoped, so a switch serves A's JSON. Detect in DevTools → Application → Cache Storage; fix by versioning the cache name per tenant and clearing on switch.
- CDN keyed on path only:
GET /api/invoicescached at the edge and served to every tenant. Detect by requesting the same path with two session cookies and diffing; fix withCache-Control: privateorVary: Cookie. - IndexedDB persisted store: survives reload, not just a route change. Clear it on switch.
- Trusting a tenant claim in a JWT that wasn't rotated on switch: re-derive tenant from the session server-side.
Verification. Switch tenants in one tab and watch the network panel: every request should carry the new session cookie, and no response should come from disk or memory cache. Then confirm Cache Storage and IndexedDB hold nothing for the old tenant. Tenant is a server-side predicate and a client cache key, not a URL segment.
Curated: · Written: · Reviewed:
Q&A-43The app works on a train. How do you queue writes and merge them when the network returns?
What I'd say first
Two questions decide the design: do writes depend on each other, and can two devices edit the same record? If writes are independent and the server assigns ids, this reduces to an idempotent outbox. If they aren't, you need ordering and conflict rules, and that's the expensive part.
Assume the common case: a field app, writes are per-record, the server owns ids, the same user may be on two devices. Then persist the intent locally with a client-generated key before touching the network, replay in order when connectivity returns, and have the server upsert on that key. A retry after a timeout then returns the row the first attempt created instead of creating a second one. Paint the pending row locally, swap it for the server row on success, and surface conflicts rather than resolving them silently.
The outbox
The record and the outbox entry go in one local transaction, IndexedDB or SQLite, so a crash between them can't lose the write or duplicate it.
type Outbox = {
clientKey: string; // crypto.randomUUID(), generated once, at intent time
seq: number; // monotonic per device; orders the replay
method: 'POST' | 'PATCH' | 'DELETE';
path: string;
body: unknown;
attempts: number;
nextAttemptAt: number;
};
async function flush() {
for (const item of await outbox.all()) { // ordered by seq
if (item.nextAttemptAt > Date.now()) continue;
const res = await fetch(item.path, {
method: item.method,
headers: { 'Idempotency-Key': item.clientKey, 'Content-Type': 'application/json' },
body: JSON.stringify(item.body),
});
if (res.ok) {
await local.replace(item.clientKey, await res.json()); // pending row -> server row
await outbox.delete(item.clientKey);
} else if (res.status === 409) {
await conflicts.add(item); // surface to the user, never drop
await outbox.delete(item.clientKey);
} else if (res.status >= 400 && res.status < 500) {
await deadLetter.add(item); // permanent: bad payload, forbidden
await outbox.delete(item.clientKey);
} else {
item.attempts += 1;
item.nextAttemptAt = Date.now() + Math.min(2 ** item.attempts * 500, 60_000);
await outbox.put(item);
break; // stop the flush; keep order for the rest
}
}
}
Trace: entries come back ordered by seq. A future nextAttemptAt is skipped. On 200 the local pending row is replaced by the server row and the entry is deleted. On 409 the entry moves to a conflict list and is deleted. On any other 4xx it moves to a dead-letter list: a permanently bad payload must not block the queue head forever. On 5xx or a network throw, attempts increments, backoff is set, the entry is written back, and the loop breaks so later writes don't overtake it.
Server side: a unique index on (user_id, idempotency_key). A replay hits the index and the handler returns the stored row with 200 rather than an error. Keep keys for a bounded window, 24 to 72 hours is typical, then let them expire so the table doesn't grow without limit.
Why the key matters
A worked example, numbers illustrative:
| Replay | Server rows | Duplicates |
|---|---|---|
| 73 writes, POST without a key, first batch times out at 30 s and retries | 92 | 19 |
| same 73 writes, upsert on client key | 73 | 0 |
The timeout is the whole problem. The first attempt reached the server and committed; the client only saw the connection drop. Without a key the server can't tell the retry from a new write, so 19 rows land twice.
Merging, not just replaying
Replay is the easy half. Merging is where the choices are:
- Server-authoritative upsert by key. Correct when writes are independent. Cheapest, and what I'd default to.
- Version check per record. The client sends the version it read; a stale version gets 409 and the UI shows both sides. Use when two people can edit the same record.
- Field-level merge or a CRDT. Only when concurrent edits to the same record are routine: collaborative text, shared counters. A lot of machinery for a form app.
Ordering: if write B needs the id from write A, don't invent a server id locally. Let B reference A's client key and have the server resolve it during replay, or block B until A lands. The break above is what enforces that.
Failure modes
- Double-create on retry after timeout. Detect with a duplicate query on
(user_id, natural_key), or by counting rows per client key. - Queue head blocks everything. One payload that always 400s stalls the flush. The dead-letter branch is the fix; alert on dead-letter depth.
- Unbounded queue. Cap the outbox by count and bytes, warn the user, and refuse new writes past the cap rather than silently dropping old ones.
- Token expiry mid-flush. A 401 should refresh and resume, not clear the queue. Treat it as transient.
- Clock skew.
nextAttemptAtis local; a device with a wrong clock can stall the queue. Compare against a monotonic clock where you can.
How I'd check it
Go offline, create a record, kill the tab, reopen, come back online. One server row, and the client view matches it. Then force the timeout case: point the client at a proxy that drops the response after the server commits, and confirm the row count stays at one.
Curated: · Written: · Reviewed:
Q&A-34A support screen needs to show a customer. What must never be in the bundle, the logs, or local storage?
Assumptions: the screen is read-only display for an authenticated support agent; the browser is a shared, replicated, long-lived store; the agent is trusted, the browser is not.
Answer: raw PII never enters the client process. The browser gets a mask or an opaque reference; the server keeps the identifier and re-authorizes every read.
Never in the bundle: static JS and HTML are served from a CDN with no auth and cached by intermediaries and by every laptop that loaded the page. A national id, a full card number, or an API key compiled into a bundle is public the moment it ships.
Never in client logs, error beacons, or session replay: replay records DOM text and input values; error beacons serialize the object that threw. Both land in a third-party store with its own retention and its own access list.
Never in localStorage, sessionStorage, IndexedDB, or the Cache API: localStorage survives tab close and logout, and any script on the origin, including an injected one, reads it. sessionStorage clears on tab close but is still readable by any script during the session.
Why masking works. The exposure chain is client-side sinks: replay, beacons, storage, caches. If the raw value never crosses into the client process, none of those sinks can capture it. Masking is not obfuscation; it is keeping the value on the server side of the boundary.
Alternatives and when they differ.
| Approach | Client sees | Use when | Cost |
|---|---|---|---|
| Masking (last 4) | display-only fragment | read-only display | client cannot act on the value |
| Tokenization | surrogate id | client must reference the value | mapping store + resolve endpoint |
| Gateway redaction | nothing | one policy across many services | raw still traverses the app tier and its logs |
| Short-lived signed ref | expiring handle | one-time reveal | re-auth and audit on resolve |
Deciding criteria: does the client need to act on the value or only display it, who must be able to read it, and how long must it live. Display-only → mask. Act-on-it → tokenize. One-time reveal → signed ref that expires and is audited.
Failure modes. XSS exfiltrates localStorage. A replay vendor retains DOM text you thought was transient. A JSON response with Cache-Control: public gets cached at the CDN. An error beacon ships the whole object because the handler logged err.context. A "temporary" debug log of the raw record outlives the ticket.
Server-side mask:
// Node 20, Express 4
const c = await db.getCustomer(id);
res.json({
id: c.id,
name: c.name,
ssnLast4: c.ssn.slice(-4), // display only
cardLast4: c.card.number.slice(-4),
// no ssn, no full PAN, no CVV, no session token
});
Verification: grep the built assets for the field names, inspect storage in devtools, and review a recorded session for the raw value. Only a masked form should appear.
Curated: · Written: · Reviewed:
Q&A-35A user asks to be forgotten. What do you delete besides the database row?
Assume a GDPR Art. 17 request with no exception in play. The row is the easy part. The answer is every copy a UI or a teammate can still read: derived stores, client-side state, and the append-only stores you cannot rewrite.
Delete vs anonymize vs crypto-shred
The store decides which one you get.
- Delete when the store is mutable and you own the write path: the users row, the profile document, the search doc, the CDN object. Cheapest to reason about, and the only option that satisfies "erasure" literally.
- Anonymize when the record has to survive for aggregate value: analytics events, model training rows. The bar is irreversibility. Replacing the user id with a random UUID is pseudonymization, not anonymization, because the mapping table still exists. Real anonymization means dropping the join key and the quasi-identifiers (email hash, device id, precise timestamps, IP) so the row cannot be re-linked. If you cannot get there, treat it as personal data and delete it.
- Crypto-shred when the store is append-only or you do not control the write path: backups, an event log, a data lake, a third-party warehouse. Encrypt each subject's data under a per-subject key held in a separate KMS; deleting the key makes the ciphertext unrecoverable without rewriting a byte. The key store must not live inside the backups it protects, and key deletion has to be durable across replicas: otherwise you have shredded nothing.
Legal hold overrides all three. Tax, financial-record and litigation obligations mean you keep the minimum set in a quarantined store with access logging, and you document why. That is an Art. 17(3) exception, not a failure to comply, but it has to be a named, auditable set, not "we kept everything just in case."
The copies people forget
Search index (delete-by-query, then verify: index lag means the document is still served), CDN (purge by URL and by cache tag so every object for that user goes), warehouse and event streams, ML feature store and embeddings, support desk and email/SMS provider, logs that captured PII, and the client: localStorage, IndexedDB, Cache Storage, cookies, and any persisted query cache.
Backups: do not rewrite them. Either crypto-shred, or keep a deletion ledger and replay it when a backup is restored.
A worked example (hypothetical figures)
Account deletion removed the users row. A service worker kept serving a cached /me payload, and the search index lagged. Neither was visible to the delete path.
| Store | Days until gone | Visible to teammates |
|---|---|---|
| users table | 0 | no |
service worker /me | 21 | yes, to the old session |
| search index | 14 | yes, 3 people |
Detection: query every store by every identifier the user ever had (user id, email, email hash, device id) and assert zero hits. Run it as a CI test after each erasure, and restore a backup into a scratch environment to confirm the person is absent there too.
When this changes
If the store is immutable, crypto-shred instead of delete. If a legal hold applies, retain the named minimum and record it. If you are a joint controller or a processor, propagate through the vendor's deletion API and keep the receipt.
Curated: · Written: · Reviewed:
Q&A-73Your client error service stores the full page URL and headers. Why is that a secret incident?
The 90-second answer
It is a secret incident because the error service turns a credential that was meant to exist for one request in one browser tab into a durable, replicated, broadly-readable record. The URL and headers on a failing request are not metadata about the request: they are the request's credentials. Authorization: Bearer …, Cookie: session=…, and ?access_token=… are the exact strings an attacker needs to impersonate the user, and you have just written them into a system with different retention, different access control, and different backup policy than the app that issued them.
Why the error service is the wrong place for them
- It is usually a vendor. Your tokens now sit in someone else's database, under their retention window and their support tooling.
- Access is broad. Everyone on the project can read events: contractors, support, whoever accepts an invite. The app's own authorization does not apply here.
- It replicates. Events get forwarded to Slack alerts, to a warehouse, into a ticket. Every copy is another place you have to revoke.
- Retention is long and often not configurable below a plan tier.
- The token outlives the session. A bearer token captured at 10:00 is still replayable at 10:00 tomorrow unless it was short-lived.
What actually leaks
| Source | Example | Why it is a credential |
|---|---|---|
| Request URL | /callback?code=…&access_token=… | OAuth code and token are directly replayable |
Authorization header | Bearer eyJ… | Full API access as the user |
Cookie header | session=… | Session hijack |
Referer | another page's URL carrying a token | Leaks a second request's secret |
| Error message / stack | fetch failed: https://…?token=… | Same URL, different field |
Trade-offs, because stripping is not free
Redaction costs debuggability: the header you stripped is often the one that explains the 401. A deny-list is easy to write and silently misses new keys; an allow-list is safer but you lose fields you did not anticipate. Sampling reduces volume, not exposure: a sampled event still contains the token. Less redaction is defensible when the error service is internal-only, retention is short, access is restricted to the owning team, and the tokens are short-lived and sender-bound. If any of those is false, scrub at the client before send.
Failure modes and how to detect them
- Client scrub misses
Refererand the error message string. Detect by replaying a real staging event through a regex foreyJ,Bearer,session=. - Redaction at ingest only: the raw payload already crossed the network and sits in the vendor's intake buffer. Detect by inspecting what the SDK sends on the wire, not what the dashboard shows.
- Over-scrubbing collapses grouping: every event becomes
[redacted]and you cannot tell two bugs apart. Detect by comparing event group counts before and after.
A scrub that handles the URL and headers
const SECRET_KEYS = /^(authorization|cookie|x-api-key)$/i;
const SECRET_PARAMS = /^(access_token|code|id_token|session|token)$/i;
function scrub(url, headers) {
const u = new URL(url);
for (const k of [...u.searchParams.keys()]) {
if (SECRET_PARAMS.test(k)) u.searchParams.set(k, '[redacted]');
}
const h = {};
for (const [k, v] of Object.entries(headers)) {
h[k] = SECRET_KEYS.test(k) ? '[redacted]' : v;
}
return { url: u.toString(), headers: h };
}
Trace with url = "https://app.example.com/callback?code=abc123&access_token=eyJhbGci&page=2" and headers = { Authorization: "Bearer eyJhbGci", Accept: "application/json" }. searchParams.keys() yields code, access_token, page; the first two match and are set to [redacted], page is untouched. URLSearchParams percent-encodes the brackets, so u.toString() is https://app.example.com/callback?code=%5Bredacted%5D&access_token=%5Bredacted%5D&page=2. In the header loop, Authorization matches and becomes [redacted]; Accept does not and is kept. Output: { url: "https://app.example.com/callback?code=%5Bredacted%5D&access_token=%5Bredacted%5D&page=2", headers: { Authorization: "[redacted]", Accept: "application/json" } }. Note this does not touch Referer, the error message, or the stack: those need the same treatment.
Verification
Trigger a handled error in staging, read the stored event, and grep it for eyJ, Bearer , session=, and access_token. If any hit, the incident is still open.
Curated: · Written: · Reviewed:
Q&A-80The API returns a snippet of HTML for a comment. How do you stop that becoming script in the page?
First 90 seconds
Ask one thing before answering: is the HTML authored by your own templates, or does it contain anything a user typed? That decides the fix.
If any part of the string is user-controlled, treat it as hostile, sanitize it on the server at write time with an allowlist library, and render it through a sink that cannot execute. If the markup is entirely yours, the bug is that user data reached the template unescaped: fix the template's contextual escaping and stop there.
The answer I'd give: sanitize once on write, render as text wherever the comment is plain text, and add CSP as a second layer for the places that genuinely need markup.
Sanitize at write
// Node 20, dompurify 3.x + jsdom
import createDOMPurify from 'dompurify';
import { JSDOM } from 'jsdom';
const purify = createDOMPurify(new JSDOM('').window);
const clean = purify.sanitize(rawComment, {
ALLOWED_TAGS: ['b', 'i', 'code', 'a', 'br'],
ALLOWED_ATTR: ['href'],
ALLOWED_URI_REGEXP: /^https?:/i,
});
Store clean. Every consumer (web, admin, mobile, the CSV export, the digest email) now reads a value that is already safe, instead of each one having to remember to sanitize.
Render at the sink
el.textContent = comment.body; // plain text: nothing parses
el.innerHTML = DOMPurify.sanitize(comment.body); // markup: sanitize again
textContent never creates elements. innerHTML with the raw string is the bug.
Trace one payload
Payload: <img src=x onerror="fetch('/steal?c='+document.cookie)">
textContent: the browser inserts a text node. No<img>element is created, no request to/steal. The string renders literally on screen.innerHTMLwith the raw string: the parser builds animgelement,src=xfails to load, the error event fires, the handler runs, and/stealreceives the cookie unless it isHttpOnly.innerHTMLafter the sanitizer above:imgis not inALLOWED_TAGS, so it is dropped and the output is an empty string. Nothing is created.innerHTMLwith the raw string plus CSPscript-src 'self' 'nonce-…'(nounsafe-inline): the element is still created, but the inline handler is refused and the console logs a violation. The script does not run; the injected layout does.
That last line is why CSP is a second layer, not the fix. It stops execution, not injection: a fake login form or a CSS-based exfiltration still lands.
Where teams get it wrong
Sanitizing only at render means every new consumer is a new chance to forget. Sanitizing only at write means you cannot tighten the allowlist later without the original. The common compromise: keep the raw value in one column for audit, and serve the sanitized value from the column the API reads.
Verification
Post <img src=x onerror=alert(1)> and <script>alert(1)</script> through the comment API, then check each consumer: the DOM should contain those characters, not an element. Confirm the CSP header is present on the response that serves the page, and check the console for violations.
Curated: · Written: · Reviewed:
Q&A-81A product search box talks to a SQL backend. Where does injection still hide in a full-stack app?
Injection survives wherever a request value decides the shape of the SQL instead of filling a slot in it. In a product search box that is almost never the WHERE text: everyone binds that. It is the sort dropdown, the page size, the filter operator, the facet field, the "search in" scope. Placeholders bind values; no mainstream driver (Postgres, MySQL, SQL Server) lets you bind an identifier. ORDER BY, LIMIT, table and column names have to be written into the SQL text, so that is where the box still reaches the parser.
Trace of one request, GET /search?q=shoes&sort=price_desc&size=20:
- Handler binds
qas a parameter: safe. sizegoes throughint()and a clamp: safe.sortis looked up in{"price_desc": "price_cents DESC", "newest": "created_at DESC"}: safe.- Replace step 3 with
f"ORDER BY {sort}". Nowsort=(CASE WHEN (SELECT substring(password_hash,1,1) FROM users LIMIT 1)='a' THEN price_cents ELSE name END)is valid SQL in Postgres and MySQL, and the result order flips depending on the first character of a password hash: a blind oracle that needs no error message. Stacked statements (; DROP TABLE) only land where the driver allows multi-statement queries (MySQLmulti_query, SQL Server); Postgres's extended protocol rejects them, which is why the subquery form is the one to test.
Three inputs leave that box: the search text (bind it), the sort column (map it), the page size (parse it as an int and clamp it).
Alternatives, and when each is wrong:
| Approach | What it protects | Where it fails |
|---|---|---|
| Bind parameters | values in WHERE/HAVING | cannot bind identifiers |
| Query builder / ORM typed API | identifiers, if the API validates against a metamodel | raw escape hatches (orderByRaw, text()) reopen it |
| Identifier quoting | injection, by making the payload one quoted name | not authorization: the caller can still sort by password_hash |
| Column map / allowlist | injection and column authorization | rots when the sortable set is user- or tenant-defined |
| Fixed query per sort option | everything | doesn't scale past a handful of options |
JPA's Criteria API is the good case: cb.asc(root.get(name)) throws on an attribute that isn't in the metamodel. SQLAlchemy's getattr(Model, name) is the trap: it restricts you to mapped attributes but also reaches metadata and dunders, so check membership in Model.__mapper__.columns first. Knex quotes identifiers, which stops injection but still lets a caller order by any column in the table.
An allowlist is the wrong tool when the sortable set is genuinely dynamic: tenant-defined columns, a report designer, an ad-hoc BI query box. A hardcoded list either blocks legitimate columns or rots. There you need a metadata layer: resolve the client token against a stored, pre-validated identifier (a saved-view table, or information_schema.columns read at startup), or parse and re-emit the SQL with a real parser and reject anything not in the catalog. Note too that "column" is sometimes an expression (relevance score, distance), which needs a named-expression registry, not a column list.
Failure modes worth testing for:
- Allowlist on the public API but not the CSV export or admin path that reuses the handler.
- Direction not validated:
sort=price_desc;--when direction is concatenated separately from the column. - Second-order: the value is stored safely, then concatenated by a nightly report job.
- LIKE wildcards:
%and_inqdon't inject but change the result set, and a leading%forces a full scan.
Detection: log the final SQL with parameters separated and assert the normalized statement fingerprint is constant across payloads. In staging, log_statement = 'all' or pg_stat_statements grouping shows it immediately: a payload that changes the fingerprint changed the structure.
Worked example, illustrative numbers: a search box concatenated q into ORDER BY; a crafted value returned 68,000 customer rows in 9 minutes before the endpoint was disabled. The row count is invented for the example; the shape (one unbound identifier, one endpoint, minutes to exfiltrate) is the part to remember.
Bind the search value. Never splice it into SQL. And treat the sort control as a query, not a string.
Curated: · Written: · Reviewed:
Q&A-66A third-party private key showed up in the main JavaScript chunk. What do you do, and how do you stop the next one?
First 60 seconds. Treat the key as public the moment it shipped. Revoke it at the provider, then issue a replacement that never touches client code. Rotation is the fix; purging the CDN cache is not, because copies are already in caches, proxies, and scrapers.
Then three things in parallel: notify the provider's abuse contact so they can flag anomalous calls on that key, pull the key's usage logs for its whole lifetime and diff them against expected traffic, and check the billing dashboard for charges you didn't make. If the key can move money or data, assume it was used until the logs say otherwise.
Why it happened. The usual cause is a build-time inline. Webpack's DefinePlugin replaces process.env.X with a string literal in the bundle. Next.js only inlines vars prefixed NEXT_PUBLIC_; Vite only VITE_. So a secret leaks when someone gives it a public prefix, or imports a server module into client code. Scanning source misses this: the literal only exists in the built output.
Stopping the next one. Two gates.
- Scan the built assets, not just source, in CI. gitleaks or trufflehog against
dist/:
# .github/workflows/secret-scan.yml
- uses: gitleaks/gitleaks-action@v2
env:
GITLEAKS_CONFIG: .gitleaks.toml
Point the config at your provider's key formats (e.g. sk_live_[0-9a-zA-Z]{24}) and fail the job on any hit. Run it on the artifact, after the build.
- Keep the credential server-side and give the browser something short-lived. The browser only needs to authenticate a session, not hold a long-lived credential, so the server mints a scoped, expiring token per user.
// server.js — Node 20, stripe-node v14
import Stripe from "stripe";
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY); // server-only env
app.post("/api/ephemeral-key", async (req, res) => {
const key = await stripe.ephemeralKeys.create(
{ customer: req.user.stripeCustomerId },
{ apiVersion: "2023-10-16" }
);
res.json({ ephemeralKey: key.secret });
});
The client gets a key bound to one customer that expires; the account-wide secret never leaves the server.
Trade-offs.
| Approach | Browser holds | Cost | Use when |
|---|---|---|---|
| Proxy calls through your server | nothing | +1 network hop, your server scales with traffic | default; provider has no browser SDK |
| Short-lived scoped token | expiring token | none extra | provider SDK must run in the browser (Stripe.js) |
| Per-user scoped key | long-lived but narrow key | none | provider has no ephemeral-key API |
Failure modes. A leaked ephemeral token is replayable until it expires: keep TTLs short and scope to one user. A CI scan misses secrets that are base64-encoded or split across strings; add entropy checks, not just regex. And a proxy that forwards the raw secret still leaks it if the proxy logs request bodies.
Curated: · Written: · Reviewed:
Q&A-83A full-stack app has a JavaScript manifest and a server manifest. How do you handle supply-chain risk in both?
Assumptions first: the JavaScript side is npm with package.json + package-lock.json, the server side is Python with requirements.txt (or poetry.lock). If the server is Go or Rust, the same three layers apply with go.sum / Cargo.lock; only the tool names change.
Core answer. Run the same three layers on both graphs (deterministic install, integrity, inventory) and add one thing the JS side needs and the Python side usually doesn't: install-time script control.
- Deterministic install.
package.jsonallows^1.2.3, so a freshnpm installcan resolve a new patch that was published after your last build.npm ciinstalls exactly the lockfile and fails ifpackage.jsonand the lockfile disagree: that failure is the drift gate. Python: pin==and install with--require-hashes. - Integrity. The lockfile carries a
sha512integrity hash per tarball;--require-hashesdoes the same for pip. This catches a tampered artifact on the registry, not a malicious-but-correctly-hashed publish. - Inventory. Emit an SBOM per release (CycloneDX or SPDX, via
syftorcdxgen) for both halves, and scan it withosv-scanneror Trivy. The point of the SBOM is answering "which release contains package X version Y" in minutes.
The asymmetry. npm runs preinstall/install/postinstall lifecycle scripts as your user at install time: that is arbitrary code execution before your app ever starts, and it can read ~/.npmrc or CI env tokens. npm ci --ignore-scripts disables it; pnpm 10 disables lifecycle scripts by default and requires an explicit onlyBuiltDependencies allowlist. Python wheels are prebuilt, so pip install --only-binary=:all: avoids running setup.py from an sdist.
npm ci --ignore-scripts
pip install --require-hashes --only-binary=:all: -r requirements.txt
Trade-off: pinning vs allowlists vs provenance. Pinning is cheap, always on, and stops drift, but it does not stop a bad version you pinned. A private registry proxy (Artifactory, Nexus) with an allowlist restricts what can enter at the cost of a curation step on every new dependency. Provenance (npm provenance via sigstore, SLSA build levels) proves an artifact came from the claimed repo and CI, but only for publishers who attest, and it says nothing about a compromised maintainer's own pipeline. Deciding criterion: if your transitive tree is small and you review lockfile diffs, pinning plus a scanner is enough; provenance earns its cost when you pull hundreds of transitive packages from publishers you can't audit.
| Control | Stops | Doesn't stop |
|---|---|---|
| Lockfile + frozen install | silent version drift on rebuild | a bad version you pinned |
--ignore-scripts / pnpm 10 default | install-time code execution | malicious runtime code |
Hashes / --require-hashes | tampered tarball on the registry | a correctly-hashed malicious publish |
| Provenance (sigstore, SLSA) | builds not from the claimed repo | a compromised maintainer CI |
| SBOM + osv-scanner | known CVEs | zero-days; it is noisy |
Failure modes and detection. Dependency confusion: an internal name like @acme/utils also exists on public npm, and npm resolves the public one. Detect by scoping every internal package and pinning its registry in .npmrc. Maintainer account takeover: a patch version appears in the lockfile diff with no package.json change. Review lockfile diffs in PRs and alert on new versions. Postinstall exfiltration: a script reads ~/.npmrc or CI secrets; --ignore-scripts plus CI egress rules catch it. Budget for scanner triage: osv-scanner on a large tree returns real CVEs mixed with unreachable ones, and reachability analysis is what keeps that queue short.
Curated: · Written: · Reviewed:
Q&A-52Which environment variables may go into the browser bundle, and which must stay on the server?
Short answer: only values you'd be comfortable publishing go in the bundle. Everything else stays in the API process's environment. The dividing line isn't "is it a secret": it's "did the bundler substitute it into a file the browser downloads."
Assumptions: a Next.js or Vite SPA served as static assets from a CDN, with secrets read from process env by the API. The same reasoning holds for CRA (REACT_APP_) and any bundler that uses a prefix allowlist.
Mechanism. The browser has no process environment: there is no process.env to read at runtime. So bundlers don't ship env vars; they do a literal, build-time string substitution. Next.js replaces every occurrence of process.env.NEXT_PUBLIC_FOO with the string value of NEXT_PUBLIC_FOO at build time. Vite does the same for import.meta.env.VITE_FOO. The prefix is the allowlist: only prefixed names are substituted, everything else is left alone.
That's why a wrong prefix leaks. The prefix isn't a naming convention, it's the switch that decides whether the value becomes a string literal in a public file.
Trace:
// src/config.js
export const apiBase = process.env.NEXT_PUBLIC_API_BASE;
export const dbUrl = process.env.DATABASE_URL;
Build with NEXT_PUBLIC_API_BASE=https://api.example.com and DATABASE_URL=postgres://user:pw@db:5432/app:
// emitted client chunk
const apiBase = "https://api.example.com";
const dbUrl = process.env.DATABASE_URL; // not substituted
apiBase is now a literal in a file anyone can fetch. dbUrl was never matched by the replace, so it stays a property lookup; in the browser the shim has no such key and it evaluates to undefined. On the server the same expression resolves from the real process env. grep -r "postgres://" .next/static returns nothing; grep -r "api.example.com" returns the chunk.
Trade-offs. Build-time inlining is the default and the cheapest (no extra request, no flash of unconfigured UI) but the value is frozen into the artifact, so one build can't be promoted from staging to prod, and rotating a public value means a rebuild.
| Approach | Keeps secret off client | Cost |
|---|---|---|
| Build-time inline (public prefix) | only for non-secrets | one artifact per environment |
Runtime config endpoint (/config.json, window.__ENV__) | yes, for non-secrets | extra request or blocking inline script before first render |
| Client-safe key (Stripe publishable, reCAPTCHA site key, Mapbox token) | yes, by design | must be domain/referrer restricted; still paired with a server secret |
| BFF: browser calls your API, API holds the key | yes | extra hop; your API becomes the abuse surface |
Use a runtime config endpoint when the same artifact must run in several environments. Use a BFF when the third party has no client-safe key at all: that's the only option that keeps the secret off the client.
Failure modes. Wrong prefix on a secret (the leak above). Dynamic access, process.env[name], is not substituted by Next.js, so it's undefined in the browser even for a public var; that's a silent misconfiguration, not a leak. Source maps shipped to prod expose the same literals. Build logs echoing env.
Detection. A CI step that greps the built client assets for known secret patterns (sk_live_, -----BEGIN, postgres://) and fails the build. That catches the wrong-prefix case before the CDN does.
Curated: · Written: · Reviewed:
Q&A-53A pull request preview is up. What must it be forbidden to call?
Forbidden: any production write path. Concretely, a preview must not be able to call:
- production API origins that mutate state (the live
api.example.com, not a preview or staging origin) - the production primary database, and any replica that isn't read-only
- production queues, topics, and cron triggers
- production object storage buckets
- third-party services with side effects: live payment keys, email/SMS senders, and any webhook that points back at production
- production secrets of any kind
Reads are a separate decision, not a default. A preview that can read production customer data is a data-exposure question; a preview that can write is an incident.
Why it happens. A preview is unreviewed code running with ambient configuration. Nothing in the branch says "production": the runtime's env vars and the SDK's default endpoint decide where the first request lands. The failure is almost never a deliberate call to prod; it's an unset API_BASE_URL falling back to a default, or a preview inheriting the production secret store because that's what the deploy template had.
What to give it instead, and when each is enough.
| Preview backend | What the preview calls | Cost | Enough when |
|---|---|---|---|
| Mock server in the preview runtime (MSW, WireMock) | nothing real | low | UI-only change; API contract already covered by tests |
| Preview API + seeded synthetic DB | preview API, preview DB | medium | query/schema changes, new endpoints |
| Preview API + read-only prod replica | preview API, prod read replica | medium-high | data-shape bugs that need prod-like volume |
| Preview API + full prod clone | preview API, clone | high | migration rehearsal, perf work |
Pick the cheapest row that can still fail the way the change can fail. A mock backend cannot catch a broken write path, so a PR that touches writes needs a real preview database. A migration needs a clone with production volume and constraints, because a seed of 100 rows won't surface a lock timeout on 40M.
The cost of the bottom row is not just storage: a clone per preview means personal data on every branch and every engineer's laptop, plus a refresh job to keep it from going stale. That's usually the argument for synthetic seeds with production-shaped distributions rather than a straight copy.
Failure modes and how you catch them.
- Default endpoint fallback. SDK falls back to a production URL when the env var is missing. Detect: make the variable required, no default, and assert at boot.
- Ambient production credentials. Preview inherits the prod secret store. Detect: check key prefixes in CI (
sk_live_vssk_test_) and fail the build. - Egress to production. Preview can still reach prod hostnames. Detect: deny at the network policy and alert on any attempt. The alert is the evidence, not the policy.
- Webhooks pointing home. Preview registers a callback against production. Detect: assert the registered callback host is the preview host.
// checked against Node 20
const PROD = /(^|\.)api\.example\.com$/;
const base = process.env.API_BASE_URL; // deliberately no default
if (!base) throw new Error('API_BASE_URL must be set for previews');
if (PROD.test(new URL(base).hostname)) {
throw new Error(`preview resolved a production origin: ${base}`);
}
Run that at process start, not at first request. A preview that boots against production and only fails on the first write has already written.
Curated: · Written: · Reviewed:
Q&A-64You want to reject anonymous HTML at the edge. What must the rule not block?
Rejecting anonymous HTML at the edge means the rule fires on session-gated document routes and nothing else. Everything a browser fetches without a session cookie has to pass through untouched: immutable hashed build assets, public HTML (login, signup, password reset, OAuth callback, error pages), health and readiness endpoints, webhook receivers, and API routes that authenticate with bearer tokens or signatures instead of cookies.
Why the rule is a path allowlist, not a content check
At request time the edge sees the method, the URL and the request headers. It does not see the response, so it cannot ask "is this HTML?" before the origin answers. Any rule phrased as "block HTML" has to be approximated with path patterns, and the approximation is where it breaks. Two consequences:
- Match on route prefixes you control (
/app,/settings,/admin), not on file extensions. Extension matching misses extensionless document routes and catches.htmlfixtures you serve publicly. - Cookie presence is not session validity. A stale or wrong-domain cookie passes a presence check and then fails at the origin, which is the blank-page failure below. If you need real validation at the edge, verify a signed token (JWT or a signed session ID) with a key you can rotate; if that requires a database round trip per request, the edge is the wrong layer.
Edge gate vs origin gate vs WAF rule
| Where | Sees | Good for | Wrong when |
|---|---|---|---|
| Edge middleware | URL, headers, cookies | Cheap rejection before origin; protecting cached HTML | Session validity needs a store lookup; per-object authorization |
| Origin / app | Full session store, DB, user record | Authorization decisions, per-resource checks | You are paying origin cost for anonymous traffic you could have dropped |
| WAF rule | IP, rate, headers, signatures | Bot and credential-stuffing mitigation, coarse blocks | Per-user auth: it has no session concept |
Use the edge gate to keep anonymous traffic off the origin and to stop private HTML from being cached and served to the wrong user. Do not use it as the authorization boundary: the origin still has to check that the session owns the resource it is asking for. A WAF rule complements both but never replaces either.
Worked example (hypothetical)
Rule: session required for /app/*; skip /static/*, /login, /healthz, /api/*.
| Request | Cookie | Edge decision | Result |
|---|---|---|---|
GET /app/dashboard | none | block | 302 to /login |
GET /app/dashboard | valid | allow | 200 from origin |
GET /static/main.8f2a.js | none | skip | 200 from CDN cache |
GET /login | none | skip | 200 |
POST /api/webhooks/stripe | none, signature header | skip | 200 |
GET /healthz | none | skip | 200 |
Now drop the /static/* skip. The document still loads, because /app/dashboard is gated correctly, but the browser's next request for main.8f2a.js gets a 302 to the login page, the script never executes, and the user sees a blank shell. If the session cookie is scoped to app.example.com while assets come from cdn.example.com, the cookie is not even sent, so the asset request looks anonymous every time. In a hypothetical rollout of that mistake, every logged-in user on the affected host saw the blank shell, not a fraction of them: the failure is deterministic, not load-dependent.
Failure modes and how to catch them
- Static prefix not skipped. Symptom: HTML 200, asset 302/401, blank page. Detect with a synthetic check that fetches a known hashed asset with no cookie and asserts 200.
- Redirect target itself gated.
/loginbehind the same rule gives an infinite redirect. Assert the login route returns 200 anonymously. - Cookie domain mismatch. Cookie set on the apex, assets on a subdomain. Check
Set-Cookiedomain against every host that serves gated HTML. - Webhook and health routes caught by a broad
/api/*-style rule. Alert on non-200 rates for those paths; they never carry a browser cookie.
Log every edge decision with path and outcome, and alert when the 302 rate for a path matching a static prefix goes above zero.
Guard the document, not the hashes.
Curated: · Written: · Reviewed:
Q&A-17The server HTML and the first client render disagree. What did each half read, and how do you make the first paint one tree?
What each half reads
The server runs your component tree through renderToString (or renderToPipeableStream) and emits an HTML string. The client runs the same tree again inside hydrateRoot (React 18+), but instead of creating DOM nodes, React walks the existing DOM and matches each fiber to a node, comparing element type, text content, and attributes. Hydration is a diff, not a render.
The contract: the tree the server serialized and the tree the client computes on its first pass must be identical. Anything nondeterministic (Date.now(), Math.random(), new Date().toLocaleTimeString(), window.innerWidth, localStorage, a cookie read only in the browser) makes the client compute a different tree than the server did.
What happens on mismatch
React 18 logs Hydration failed because the initial UI does not match what was rendered on the server (error #418) or Text content does not match server-rendered HTML (#425), then discards the server HTML for the nearest Suspense boundary and re-renders that subtree on the client. That is why a mismatch in one leaf can flash a whole section: the boundary, not the leaf, is thrown away.
React 19 reports the mismatch with a diff of expected vs actual and recovers by re-rendering the mismatched content rather than discarding the boundary. Same root cause, smaller blast radius.
Fixes and their costs
| Approach | First paint | Cost | Use when |
|---|---|---|---|
| server-computed props | correct, no flash | none | value is derivable server-side |
useEffect deferral | placeholder, then value | layout shift; absent from HTML (SEO) | genuinely client-only (window size, localStorage) |
| cookie-gated SSR | correct, personalized | uncacheable at CDN (Vary: Cookie) | theme, locale, auth-gated nav |
suppressHydrationWarning | server value kept | hides real bugs; one level deep | leaf timestamp only |
// React 18/19: server and client agree on first paint
function useOnline() {
return useSyncExternalStore(
(cb) => {
window.addEventListener('online', cb);
return () => window.removeEventListener('online', cb);
},
() => navigator.onLine, // client snapshot
() => true // getServerSnapshot: used for SSR and hydration
);
}
useSyncExternalStore's third argument is the server snapshot; React uses it for both the server render and the hydration pass, so the two agree, then switches to the live value after.
For a value you cannot compute on the server, render the server's value first and update in useEffect:
const [tz, setTz] = useState<string | null>(null); // matches server
useEffect(() => setTz(Intl.DateTimeFormat().resolvedOptions().timeZone), []);
useId gives stable IDs across both passes. suppressHydrationWarning goes only on a leaf whose text legitimately differs (a timestamp): it silences the warning one level deep and does not fix structure.
When not to
Do not cookie-gate markup that does not need it: you lose CDN caching for every user to personalize a header. Do not reach for suppressHydrationWarning on a structural mismatch: React still re-renders, you just stop seeing why. And do not server-render per-request data that changes on every load; the cache hit rate collapses.
Detecting it
Run the server render and hydrateRoot in a test with a fixed clock and a cookie the markup depends on; assert no console error. In production, watch for #418/#423 in client error reporting: they cluster on specific routes and correlate with the flash.
Curated: · Written: · Reviewed:
Q&A-31The homepage is personalized and you also want a high CDN hit ratio. How do you split the page?
Assumptions first. The homepage is mostly anonymous traffic, personalization is a rail (recently viewed, recommendations, cart count), and the shell must be in the initial HTML for SEO and first paint. If personalization has to be in the first byte (pricing, eligibility) this split is wrong; use edge assembly instead and accept a lower hit ratio.
The answer. Split by variance into three tiers and cache each where its variance allows.
- Public shell: HTML skeleton, CSS, JS, images, nav, footer. Identical for everyone. Cache at the CDN with a long TTL and versioned asset URLs.
- Segment-shared content: trending items, category rails, promo banners. Same for a coarse group (geo, logged-in vs anonymous, A/B bucket). Cache at the edge keyed on that segment, short TTL.
- Per-user fragment: recently viewed, recommendations, cart count, greeting. Never in a shared edge object. Fetch it client-side with credentials after first paint, or assemble it at the edge with a private cache.
The mechanism that decides everything is the CDN cache key. The default key is the URL plus whatever Vary names. Put Cookie in the key and every user is a distinct object: hit ratio collapses to near zero. Leave it out and any user-varying byte in the body leaks to the next visitor. So the fix is not to vary the key; it is to remove user-varying bytes from the shared object.
Worked example (hypothetical numbers). Homepage: 180 KB shell, 12 KB rail JSON, 1M views/day.
| Tier | Cache-Control | Cache key | Origin egress/day |
|---|---|---|---|
| Whole page personalized | private, no-store | n/a | 1M × 192 KB = 192 GB |
| Public shell | public, s-maxage=86400, stale-while-revalidate=60 | URL | ~10k × 180 KB = 1.8 GB |
| Personalized rail | private, max-age=20 | user | 1M × 12 KB = 12 GB |
Shell hit ratio lands around 99% at steady traffic; total origin egress is ~14 GB/day, a 93% cut. The rail is still a miss by construction: that is the price of per-user bytes.
The fetch.
<div id="rail" aria-busy="true"><!-- skeleton, reserved height --></div>
<script>
fetch('/api/rail', { credentials: 'include' })
.then(r => r.json())
.then(d => { /* render into #rail */ });
</script>
credentials: 'include' sends the session cookie to the origin. The rail response is Cache-Control: private, max-age=20, so the browser may reuse it for 20 seconds but no shared cache may store it. The shell response must not carry Vary: Cookie.
Failure modes.
- Cookie in the cache key, or
Vary: Cookieon the shell: hit ratio drops to near zero and origin CPU spikes. Detect on the CDN hit-ratio dashboard and origin request rate. - User data in a shared body without a key change: cross-user leak. Detect with a two-session synthetic check: fetch the homepage as two users and diff the shared object; it must be byte-identical.
- Rail fetch blocking first paint: LCP regression. Detect on RUM LCP p75; render a skeleton with reserved height to avoid layout shift.
- Stale shell after deploy: version asset URLs and purge by cache tag.
Alternatives. Edge-side includes or CDN compute (Cloudflare Workers, Lambda@Edge) assemble shell plus fragment at the edge with a private fragment cache: one round trip, more moving parts. Segment-level caching of the rail raises hit ratio at the cost of coarser personalization. If personalization must be in the initial HTML, none of the client-fetch options work; use edge assembly or accept the lower hit ratio.
Curated: · Written: · Reviewed:
Q&A-63A sale is coming. Do you scale HTML rendering the same way you scale JSON?
The 90-second answer
No, not the same pool, not the same concurrency limit, not the same autoscaling signal. JSON handlers are I/O-bound: they wait on a database or a downstream service and burn single-digit milliseconds of CPU. An SSR render is CPU-bound: it walks the component tree, serializes a string, and usually fans out to several downstream calls to fetch the data first. One render can cost as much CPU as dozens of JSON requests, and it holds a worker for far longer.
Two questions decide the rest. Is the sale page cacheable: identical HTML for everyone, or personalized per user? And is rendering on the request path or behind a queue? If the HTML is CDN-cached with a short TTL, the render fleet barely sees the spike and the question is mostly moot. If it is personalized per user, you cannot cache it, and the render fleet is the thing that breaks first.
Why the pools differ
Do the arithmetic with your own numbers. Assume 60 ms of CPU per render (component tree plus serialization) and 2 ms per JSON handler. On an 8 vCPU box held at 60% CPU:
- Render: 8 × 0.6 / 0.060 s ≈ 80 renders/s ≈ 4,800 rpm
- JSON: 8 × 0.6 / 0.002 s = 2,400 req/s ≈ 144,000 rpm
Two orders of magnitude. That gap is the whole argument for separate pools and separate limits.
Illustrative numbers, not a benchmark: the shape is what matters:
| Pool | HTML rpm | JSON rpm | Render queue delay | Checkout p95 |
|---|---|---|---|---|
| 8 shared boxes | 12,000 | 4,000 | 7 s | 6.2 s |
| 6 render + 8 JSON | 12,000 | 4,000 | 180 ms | 240 ms |
On a shared pool the render backlog consumes workers that checkout needs, so a page-rendering spike becomes a revenue outage. Split, and the render fleet degrades alone.
Bound the render queue, then shed
// Node 20 + Express 4 — bounded render concurrency, shed to a static shell
const MAX_RENDERS = 8; // ≈ vCPU count; render is CPU-bound
let inFlight = 0;
app.get('/p/:id', (req, res) => {
const started = Date.now();
if (inFlight >= MAX_RENDERS) {
return res.set('Cache-Control', 'no-store').send(SHELL); // client hydrates
}
inFlight++;
renderProduct(req.params.id)
.then(html => {
if (Date.now() - started > 250) {
return res.set('Cache-Control', 'no-store').send(SHELL);
}
res.send(html);
})
.catch(() => res.status(500).send(SHELL))
.finally(() => { inFlight--; });
});
Shedding to a shell keeps the page usable and the queue short. Queueing instead just moves the latency into the user's browser.
Failure modes and how you see them
- Event-loop starvation: a CPU-bound render blocks the loop, so JSON on the same process stalls even with idle cores. Watch p99 event-loop delay, not just CPU.
- Downstream amplification: each render makes N calls, so the render fleet multiplies load on the API's own dependencies. Watch downstream calls per render.
- Memory: concurrent renders each hold a tree and a string buffer. Watch RSS per instance against the render cap.
- Cache stampede at sale start: every instance misses the same key at once. Watch origin hit rate in the first 60 seconds.
- Shared connection pool: renders and JSON competing for the same DB pool. Watch pool wait time.
Alternatives, and when not to split
Pre-render the sale page at deploy time, or serve it with a short TTL and stale-while-revalidate: cheapest, and it removes most of the spike. A static shell plus client hydration is the fallback when personalization is light. A separate autoscaling group with its own concurrency limit is the general answer. If HTML is fully cached, or rendering happens off the request path in a worker, one pool is fine: the render cost never lands on the request thread. Validate with a mixed load test that records saturation of each pool separately before you raise either limit.
Curated: · Written: · Reviewed:
Q&A-74A product HTML route feels cheap per request. Why is it dominating the API bill?
Direct answer: the bill is metered per API call, not per HTML request. A route that feels cheap costs one HTML render but N downstream calls, so its real unit cost is N × (cost per call). If N is 20 and the render itself is a rounding error, the route's share of the API bill is set by fan-out, not by how light the page looks.
Assumptions that matter: server-side rendering or a BFF that fetches per component; the downstream APIs are metered per request (managed API, per-invocation function, per-query warehouse); no shared cache in front of those calls; the HTML itself is not cached at the edge because it is personalized or has a short TTL.
Why it dominates: three multipliers stack:
- Fan-out. Each page section fetches independently. A product page with price, inventory, reviews, recommendations, shipping and personalization is easily 10–25 calls.
- Waterfall / N+1. Per-item fetches inside a list turn one logical read into one call per row.
- Low cache hit rate. If the HTML is personalized the CDN cannot absorb views, so every view pays full fan-out. Retries against a slow dependency add calls on top.
Arithmetic (illustrative figures, $0.0004 per API call, 2.8M views/month):
| Route | Calls/view | Calls/month | API cost/month |
|---|---|---|---|
| unbatched | 22 | 61.6M | ~$24,600 |
| batched | 4 | 11.2M | ~$4,500 |
Same traffic, same page. The only thing that changed is calls per view.
Trace one request: HTML request → BFF → 6 parallel section fetches → the reviews section loops over 12 SKUs → 12 calls → recommendations calls 3 → 22 origin calls for one HTML response. That loop is the bill.
Alternatives, and when each is wrong:
- Batch or consolidate into one endpoint / BFF aggregation call. Cuts N directly, but a fat response raises payload and coupling; wrong when sections have very different cache lifetimes.
- Cache downstream reads (Redis, per-section TTL). Cheap win when data is shared across users; wrong for per-user data with no reuse.
- Cache the HTML at the edge. Removes the fan-out entirely, but only if the page is cacheable: personalized routes cannot use it.
- CI cap on fan-out. Catches regressions, but fails the build wrongly on long-tail personalized routes where high N is legitimate and traffic is tiny. Cap on cost-per-view, or calls × traffic, not raw N.
Failure modes to name: cache stampede when a popular key expires; stale price or inventory if TTL is too long; the BFF becoming a single bottleneck and a new SPOF.
How to confirm: trace one HTML request, count origin calls, multiply by views and per-call price. That number, not the render time, is the route's bill.
Curated: · Written: · Reviewed:
Q&A-72Which pages should a search engine be allowed to index in a logged-in product?
Before changing either half for indexing an authenticated app, I would write down how both sides will be checked.
Search engines should index marketing pages and must not index private app routes. robots.txt is not access control, and disallowing a path can stop a crawler from even seeing noindex.
Concretely, require a session for app HTML and return 401 or 404 to anonymous crawlers. Send noindex on any HTML that requires a session. Keep public marketing routes crawlable and linked, and do not rely on robots disallow as the only gate.
The reason for that specificity is a failure I have seen: The app shell was reachable without a session and was indexed for 18,000 private dashboard URLs, which leaked customer first names in titles, and the cleanup took 5 weeks to drop from results.
What was in the index.
| Route class | Anonymous fetch | Indexed URLs |
|---|---|---|
| marketing | 200, indexable | 42 |
| app dashboard, no auth | 200 public HTML | 18,000 |
| app dashboard, 401 plus noindex | 401 | 0 after 5 weeks |
I would not consider it settled without evidence: Fetch an app URL without cookies and confirm 401 plus noindex, then fetch marketing HTML and confirm it is indexable.
Index the brochure, not the account.
Curated: · Written: · Reviewed:
Q&A-69How do you send a known user to the new fullstack pair without rolling the whole site?
Open with this: set a signed cookie on the named accounts, and let the edge, not the app, pick the origin pool. The same cookie has to steer the HTML document and the API calls, or you ship a candidate frontend against a stable backend.
Assumptions: one apex domain (example.com) with app. and api. subdomains, an edge layer you control (CDN worker, Envoy, nginx), and a candidate pair already deployed and reachable. If the candidate needs a different API contract, cookie steering won't save you: the two pairs have to be wire-compatible first.
Mechanism. The edge reads canary=<user>.<hmac>, verifies the HMAC against a secret the app doesn't hold, and picks the pool. Signing matters: an unsigned cookie is a self-service opt-in, and someone will find it. Route on the cookie, never on IP or User-Agent: those change mid-session and you get a user whose document came from candidate and whose XHR came from stable.
// Cloudflare Worker, wrangler 3.x / workerd 2024-06
const POOLS = { stable: "https://stable.internal", candidate: "https://candidate.internal" };
export default {
async fetch(req, env) {
const raw = (req.headers.get("Cookie") || "")
.split(/;\s*/).find(c => c.startsWith("canary="))?.slice(7);
let pool = POOLS.stable;
if (raw) {
const [user, sigB64] = raw.split(".");
// env.CANARY_SECRET is a Uint8Array; b64ToBytes is atob -> Uint8Array
const key = await crypto.subtle.importKey(
"raw", env.CANARY_SECRET, { name: "HMAC", hash: "SHA-256" }, false, ["verify"]);
const ok = await crypto.subtle.verify(
"HMAC", key, b64ToBytes(sigB64), new TextEncoder().encode(user));
if (ok) pool = POOLS.candidate;
}
const url = new URL(req.url);
const res = await fetch(new Request(new URL(url.pathname + url.search, pool), req));
const out = new Response(res.body, { status: res.status, headers: res.headers });
out.headers.set("x-pool", pool === POOLS.candidate ? "candidate" : "stable");
return out;
}
};
Set the cookie with Domain=.example.com; Secure; SameSite=Lax; Path=/ and a short TTL: hours, not weeks. HttpOnly is optional; the app never needs to read it, so leave it on.
Verification trace. Four curls, run before you tell anyone it works:
curl -sI https://app.example.com/checkout -H 'Cookie: canary=alice.<sig>'→x-pool: candidate- same URL, no cookie →
x-pool: stable curl -sI https://api.example.com/cart -H 'Cookie: canary=alice.<sig>'→x-pool: candidate: this is the one that catches the subdomain bugcanary=alice.deadbeef→x-pool: stable
Alternatives.
| Method | Who lands on candidate | Blast radius | Pull-back |
|---|---|---|---|
| Signed cookie at edge | named accounts | exactly those accounts | delete cookie |
| Sticky % (hash of user id) | 1% of all traffic | ~1% of unconsenting users | lower the % |
| Header (staff only) | internal staff | staff only | drop the header |
| DNS weight | everyone behind that resolver | unpredictable | wait out the TTL |
At 800k DAU, 1% is 8,000 sessions; at a 3% checkout rate that's roughly 240 orders a day exposed to a bug you haven't found yet. A cookie on 80 staff is 80 sessions. Widen to the percent only after the named cohort has run clean for a full traffic cycle.
Failure modes.
- Cookie scoped to
app.example.comonly. The API never sees it, so candidate HTML calls the stable API. Detect: logx-poolon both document and XHR, alert on a mismatch inside one session. - Edge caches the HTML by URL. The first candidate request poisons the cache for everyone. Fix:
Cache-Control: privateorVary: Cookieon documents; hashed static assets can stay shared. - Long cookie TTL. If the candidate breaks, those users stay broken until expiry. Keep the TTL short and re-issue, and keep a server-side kill switch that overrides the cookie regardless of signature.
- Shared session store, changed schema. Candidate writes a session shape stable can't parse; users who bounce between pools get logged out. Detect: 401 rate on stable for canary users.
- "Known user" turns out to be a whole team or a customer's production account. Enroll staff accounts, not customer accounts, until you've watched a full day.
Curated: · Written: · Reviewed:
Q&A-57The new API is bad. Can you roll it back while leaving the new SPA in place?
Yes, but only if the new SPA's calls still resolve against the old API. That is the whole decision. Rolling back the API alone is safe exactly when the live bundle's request surface is a subset of the old API's routes. The moment the bundle calls an endpoint the old API no longer has, you have a one-sided rollback and a broken page.
Start by enumerating what the live bundle actually calls. Pull the deployed HTML, find the bundle hash, and grep the bundle for request paths, or read the API access log for the last hour and group by route. Diff that set against the old API's route table. Empty diff: roll the API back and leave the SPA. Non-empty diff: either roll both back together, or serve both API versions at once and route by path prefix (/v13/..., /v14/...) until the old bundle is the live HTML again.
Why the two deploys drift: the SPA is static assets behind a CDN with its own cache lifetime, and the API is a separate deploy with its own. Even a simultaneous rollback leaves a window where the CDN still serves the new bundle against the old API, or the reverse. Versioned paths or a version header on the API close that window; a single unversioned API does not.
The failure mode is silent, which is what makes it expensive. A removed endpoint returns 404, and a UI that treats a failed price fetch as "no price" renders $0.00 instead of an error. Nothing pages, because the request failed cleanly and the page still renders. In one incident, ops rolled the API back two versions and left the new SPA live; the price endpoint was gone and 14,600 sessions saw a zero price before anyone noticed. (Figures from that incident, not a benchmark.)
| API version | Live bundle | Price endpoint | Sessions showing $0 |
|---|---|---|---|
| v14 | v14 | present | 0 |
| v12 | v14 | removed | 14,600 |
| v12 | v12 | present | 0 |
The middle row is the one to avoid: the API looks healthy (200s on every route it still has), the SPA looks healthy (it renders), and the combination is wrong. Detect it by alerting on 404 rate per route and on a client-side "price missing" counter: status-code dashboards alone will not catch it.
Verification after either rollback: fetch the live HTML, confirm the bundle hash is the one you expect, then exercise the main journeys (checkout, price display, search) against the live API and inspect the response bodies, not just the status codes. A 200 with an empty body is the same bug wearing a different hat. Record the bundle hash and API version that were live together in the deploy log; the next rollback needs that pair.
One case where you do not roll the API back at all: if the new version carried a security fix, restoring the old one reopens the hole. Roll the SPA forward to a bundle that matches the fixed API instead.
Curated: · Written: · Reviewed:
Q&A-55You cut API traffic to green. Why can the site still break on static files?
The 90-second answer
Cutting API traffic to green changes which origin answers /api/*. It does nothing to the CDN edge, which caches by URL, and nothing to the browser cache. HTML and hashed assets are two separate cache domains with two separate lifetimes, and blue-green flips only one of them.
Assume: HTML is served with a short TTL or no-cache and names content-hashed assets; assets are served immutable with a long TTL; a CDN sits in front of both; the color flip is a DNS or load-balancer weight change.
The break: a user holds blue HTML from before the flip. That HTML names main.8f2a.js. If the edge has that object cached, fine. If it doesn't, the request goes to the origin, now green, and green never built 8f2a. 404, blank page. The API was healthy the entire time; the user never got far enough to call it.
Reverse case: green HTML is live but the green asset build hasn't propagated to every PoP, so the HTML names a hash that 404s at that edge for the propagation window.
| Step | Request | Edge state | Result |
|---|---|---|---|
| 1 | GET / | blue HTML cached | 200, names main.8f2a.js |
| 2 | GET /main.8f2a.js | miss, origin = green | 404 |
| 3 | page boot | no script | blank |
I have seen this exact shape: green HTML named a chunk that existed only on blue storage, and 31% of first loads after the cutover 404'd on the main script for 9 minutes.
Detection is a synthetic check, not a load test: after the flip, fetch the live HTML from each PoP, then fetch every asset URL it names, and require 200 on all of them.
What actually fixes it
The invariant is that HTML and the hashes it names must be resolvable at the same time, from the same edge. Four ways to get there:
- Dual-publish assets to both colors under a shared immutable prefix. HTML only ever names hashes present in both. Drain blue after HTML and assets agree. Cost: double storage and double upload per release, and it requires reproducible builds, if the same source yields different hashes per color, "both colors have it" is false.
- Color-agnostic asset origin. Assets live at one origin/CDN path that never flips; only HTML and the API move. Least work if you already run a static host separate from the app. Cost: one more origin to operate, and it becomes a shared dependency both colors must not break.
- Edge routing by cookie or header. Pin a session to one color for HTML and assets together, so a user never crosses. Cost: routing logic at the edge, and you must expire the pin or you strand users on blue.
- Short HTML TTL plus immutable assets. If the HTML TTL is under the flip window and assets are content-hashed, the window is small. Cost: it's a probability, not a guarantee. A user mid-load still crosses.
Deciding criteria: dual-publish is the wrong default when the asset set is large enough that doubling egress matters, when builds aren't reproducible, or when you already run a color-agnostic asset origin, then option 2 is strictly less work. Cookie pinning is wrong when you can't bound session lifetime.
Failure modes worth naming: 404 on the entry script (blank page, RUM script-error spike); mixed-version JS calling an API field green removed (500s that look like API bugs but are stale clients); stale HTML after blue drain (404s that persist for the HTML TTL, not the flip window).
Curated: · Written: · Reviewed:
Q&A-59The load balancer says the backend for frontend is healthy, but the first page is blank. What should the check have done?
I would treat health checks for the BFF as something that must survive a stale bundle talking to a new API.
A backend for frontend can return 200 while a required downstream is dead, which makes the web look healthy and the product unusable. The check that gates traffic must exercise the dependencies the first screen needs.
Concretely, probe the BFF with a request that talks to the session store and the primary API. Fail the check if either dependency misses its budget. Keep a shallow liveness probe separate so the process can be restarted without waiting on those dependencies.
The reason for that specificity is a failure I have seen: A load balancer used a process ping that stayed green while the session store was down, and 72 percent of first-page renders failed for 13 minutes with no pool marked unhealthy.
Ping versus a ready probe.
| Check | Session store down | Balancer state | First-page success |
|---|---|---|---|
| process ping | still 200 | in rotation | 28% |
| ready probe through store | 503 | drained | n/a, traffic shifted |
| after drain to healthy pool | store up | in rotation | 99.4% |
I would not consider it settled without evidence: Kill the session store in a drill and confirm the readiness check fails and the balancer stops sending users.
Ready means the first screen can still be built.
Curated: · Written: · Reviewed:
Q&A-67How should infrastructure as code describe the static host and the API so they cannot drift?
The 90-second answer. Drift between a static host and an API almost never comes from the resources themselves: it comes from the seams: the API origin the CDN points at, the certificate and DNS name, and the env map the static build bakes in. So the question isn't "one stack or two", it's "is every seam a reference or a copy". Two shapes are defensible:
- One workspace owning the bucket, distribution, service, DNS and env map. One plan, one apply, no cross-stack wiring.
- Separate stacks, where the static stack reads the API stack's origin through a remote-state output or a data source: never a hand-typed hostname.
Pick by blast radius and apply contention, not taste. A single workspace means one bad plan can replace the CDN distribution and the service in the same apply, and one state lock serializes every deploy; that's fine for a small team and a 30-second plan, and it becomes a liability once two teams share the state or plan time runs into minutes. Separate stacks shrink the blast radius and let the API ship on its own cadence, at the cost of a window between the two applies where they disagree: close it by ordering (API first, static second) and by having the static side resolve the API origin at plan time from the producer's output.
Mechanism. Terraform orders resources by dependency. If the distribution's origin is data.terraform_remote_state.api.outputs.api_origin, the graph forces the API stack to exist before the static plan can resolve, and a rename in the API stack shows up as a diff in the static plan. If the origin is a literal string in a tfvars file or an env map, nothing links them and the two drift silently.
# api stack
output "api_origin" { value = aws_lb.api.dns_name }
# static stack — reference, not a copy
data "terraform_remote_state" "api" {
backend = "s3"
config = { bucket = "tf-state", key = "api/prod.tfstate", region = "us-east-1" }
}
resource "aws_cloudfront_distribution" "site" {
origin {
domain_name = data.terraform_remote_state.api.outputs.api_origin
origin_id = "api"
}
}
Failure mode. The one that bites is a split apply where the env map is a literal. Worked example, two stacks, API applied first:
| Record | Intended | After split apply | Writes |
|---|---|---|---|
| www | new static | new static | pages 200 |
| api | new cluster | new cluster | none from the site |
| site API host env | new cluster | old cluster | 100% to the dead fleet |
The API stack moved the hostname; the static stack still baked the old one into the build. Detection: the static plan shows no diff while the API plan shows a hostname change. That asymmetry is the tell. Fix: make the env map an output reference, and gate the static apply on the API apply.
Verification. After apply, read the origin from the producer's state and compare it to what the site actually resolves and sends:
terraform -chdir=api output -raw api_origin # the API stack's load balancer DNS name
dig +short api.example.com
curl -s https://www.example.com/runtime-config.json | jq -r .apiOrigin # if the site reads its origin at runtime
All three must match. If the third differs, the build baked a stale value and the next deploy will drift again.
Curated: · Written: · Reviewed:
Q&A-68A pull request changes a query and a screen. What database should that preview use?
Give each pull request its own disposable database, and point the preview API at that. Not shared staging, not production, and not one long-lived "preview" instance that several open PRs share.
Assumptions I'd state before answering: previews are per-PR and torn down on merge or close; migrations run in CI as part of the preview build; and the migration in the PR is unreviewed, so it has to be allowed to be destructive somewhere harmless.
Why per-PR. The PR changes a query and a screen, but the migration that ships with it is the risk. On a shared preview database, one PR's ALTER TABLE ... DROP COLUMN is visible to every other open PR and to anyone treating that instance as a staging dry run. Isolation is the point, not the render.
The options, and when each is right.
| Option | Isolation | Fidelity | Cost |
|---|---|---|---|
| Shared staging | none | high | low |
| Schema-per-PR on one instance | partial: shared server, roles, extensions | high | low |
| Ephemeral clone per PR | full | high | high |
| Schema-only + seeded fixtures | full | medium | low |
Schema-per-PR is the usual compromise: cheap, and it catches most query and screen bugs. It leaks when a migration touches extensions, roles, or anything above the schema level, so it is the wrong default if PRs routinely alter those. A full clone per PR gives the highest fidelity and the highest cost. Schema-only plus a seeded subset is what I'd pick for most teams: the screen needs realistic shapes and a few hundred rows, not the whole table.
The failure this prevents. A worked example with hypothetical numbers: a preview migration drops a column on shared staging; a production deploy dry-run points at the same instance; the dry run deletes 1.1 million search rows. The drop ran once, on the wrong target.
| Target | Column dropped | Search rows left |
|---|---|---|
| dedicated preview copy | yes | copy only |
| shared staging | yes, by mistake | 0 of 1.1 million |
| production | no | unchanged |
How to build it. On PR open, create the database, load schema, run the PR's migrations, seed a subset:
# CI, on PR open
createdb "preview_pr_${PR_NUMBER}"
pg_dump --schema-only "$PROD_URL" | psql -d "preview_pr_${PR_NUMBER}"
psql -d "preview_pr_${PR_NUMBER}" -f pr/migrations/*.sql
Then a guard so a misconfigured pipeline cannot aim a migration at something shared:
case "$PREVIEW_DB_URL" in
*staging*|*prod*) echo "refusing to migrate a shared database"; exit 1;;
esac
Cost, so the choice is concrete. A 400 GB production clone at 150 MB/s takes roughly 45 minutes and doubles storage per open PR. A schema-only dump is a few megabytes and finishes in seconds; a 1% row sample is minutes. Ten open PRs on full clones is ten copies of production to keep in sync: that is the argument for schema-only plus fixtures unless the PR genuinely depends on production data distribution.
Teardown and detection. Drop the database when the PR closes, and assert in CI that the preview connection string's host and database differ from staging and production. If that assertion ever fails, the preview is not isolated and the migration should not run.
Curated: · Written: · Reviewed:
Q&A-76How do you split unit tests, contract tests, and end-to-end tests for a full-stack feature?
The rule
Test at the lowest layer that can fail for the reason you care about. Unit tests prove logic inside one half. Contract tests prove the two halves still agree on the payload. End-to-end tests prove what only exists once everything is assembled: auth propagation, routing, migrations, real database constraints.
Assumptions: two deployables (browser app + API), a shared repo or a broker both sides can read, and a CI budget of a few minutes for the fast layers.
Why the boundary is the payload
A UI unit test stubs the API client, so the stub encodes what its author believed the API returns. Rename amount to total in the handler and every UI unit test still passes: the stub still says amount. Contract tests close that gap: they pin the shape both sides must satisfy and fail on either side of the rename. E2E catches it too, but at 30-60 s per journey you cannot run enough of them per commit to cover every field of every payload. Contracts give breadth, e2e gives depth.
The split
| Layer | Covers | Example count | Wall clock |
|---|---|---|---|
| unit, per half | reducers, query builders, form validation, pure rendering | 360 | ~1.5 s (4 ms each) |
| contract | every payload the page consumes: status codes, field names, error bodies | 48 | ~4 s (90 ms each) |
| e2e | one write path per critical journey, browser → API → DB | 5 | ~3 min (35 s each) |
The fast layers gate every commit; e2e runs on the merge queue. If e2e exceeds ~10 minutes, you have unit or contract tests wearing a browser. Pact with a broker is the heavier alternative: worth it when the provider belongs to another team and you need versioned verification; a checked-in schema is enough inside one team.
Artefact: one schema, checked in, asserted from both sides
contracts/order.schema.json:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["id", "total", "currency", "status"],
"additionalProperties": false,
"properties": {
"id": { "type": "string" },
"total": { "type": "integer", "minimum": 0 },
"currency": { "type": "string", "enum": ["USD", "EUR"] },
"status": { "type": "string", "enum": ["pending", "paid", "refunded"] }
}
}
Consumer side (Jest + Ajv v8, draft-07):
const Ajv = require('ajv');
const schema = require('../../contracts/order.schema.json');
const { fetchOrder } = require('../order');
test('GET /orders/:id matches the shared contract', async () => {
const order = await fetchOrder('ord_123'); // recorded provider stub, no network
const validate = new Ajv({ allErrors: true }).compile(schema);
expect(validate(order)).toBe(true);
});
The provider side runs the same schema against the serialized handler response. additionalProperties: false plus required is what makes the rename fail: total missing is a required violation, amount present is an additional-property violation.
Failure modes
- A schema without
additionalProperties: falsepasses a rename. Verify once by mutation: delete a required field, confirm the contract test goes red while the unit suite stays green. - Contract tests pointed at a live provider become slow, flaky e2e. Watch p95 per test; above ~200 ms it is hitting the network.
- Hand-written stubs in unit tests re-encode the bug. Generate fixtures from the schema file.
- E2E asserting on CSS selectors breaks on every refactor. Assert on roles or API-visible state, and quarantine a journey after three flakes in 30 days.
What this looked like when it failed
A suite of 410 unit tests and 6 e2e journeys stayed green while the list endpoint renamed amount to total; checkout rendered blank totals for 14% of sessions for 2 hours. The e2e journeys ran only on merge, and the unit stubs still said amount. Adding 48 contract tests moved detection to the pull request that made the rename.
Curated: · Written: · Reviewed:
Q&A-77When should a full-stack team trust MSW, and when must they hit a real staging stack?
Short version: trust MSW when the claim is about the client's behaviour given a response; hit staging when the claim is about what the server will actually do. If the assertion names a DOM node, MSW. If it names a status code, header, or field the server produces, staging.
Why the line falls there
MSW intercepts at the network boundary, not inside your code. In the browser setupWorker registers a Service Worker that answers fetch/XHR; in Node setupServer patches the HTTP layer. Your components run unmodified, so you get real async timing, real fetch semantics, real abort handling, but the body is whatever your handler returns. MSW has no knowledge of the server's schema, so it cannot fail when the server changes. That is the whole trade: total control over responses, zero fidelity to the contract.
What that buys you: loading, empty, error, 409, 503, timeout, and pagination-edge states that are slow or impossible to trigger against a real stack. What it cannot buy you: real cookies and session refresh, DB constraints, serialization choices (ISO string vs epoch, nullability, enum values), p99 latency, chunked/streamed responses, CORS, rate limits, and third-party integrations.
The two layers in code
// handlers.ts — MSW 2.x (Node 18+)
import { http, HttpResponse } from 'msw'
export const handlers = [
http.get('/api/orders/:id', ({ params }) =>
HttpResponse.json({ id: params.id, status: 'shipped', customer: { id: 'c1' } })
),
]
// orders.test.ts — Vitest
import { setupServer } from 'msw/node'
import { http, HttpResponse } from 'msw'
import { handlers } from './handlers'
const server = setupServer(...handlers)
beforeAll(() => server.listen({ onUnhandledRequest: 'error' }))
afterEach(() => server.resetHandlers())
afterAll(() => server.close())
it('shows the conflict banner on 409', async () => {
server.use(http.post('/api/orders', () => new HttpResponse(null, { status: 409 })))
render(<Checkout />)
await userEvent.click(screen.getByRole('button', { name: /place order/i }))
expect(await screen.findByText(/already exists/i)).toBeVisible()
})
onUnhandledRequest: 'error' is the cheap drift alarm: any request your handlers don't cover fails the test instead of silently hitting the network.
What each layer can actually prove
| Claim | MSW | Staging |
|---|---|---|
| Renders loading / empty / error | yes, fast | slow, flaky |
| Retry and backoff on 503 | yes | hard to trigger |
| Session refresh on 401 | no: no real cookie | yes |
| Unique constraint returns 409 | only if you hand-write it | yes |
| Field renamed in the response | no | yes |
| p99 latency, streaming, CORS | no | yes |
Failure modes and how to catch them
Contract drift. Handlers keep returning a shape the server stopped sending. Catch it by validating fixtures against the OpenAPI/JSON Schema in CI, and by running the same journeys against staging nightly.
False green on auth. MSW bypasses the session, so a broken refresh flow passes every test. Catch it by running the login-and-refresh journey against staging pre-merge.
Fixture rot. Handlers diverge from reality over months. Catch it by recording staging responses and diffing them against fixtures on a schedule.
Illustrative arithmetic: say fixtures return 200 with a nested customer object across 22 routes, and staging returns 409 on 7 of them once a unique constraint ships. The MSW suite is green, the release is broken, and the diff that would have caught it is one status-code comparison per route.
When not to use MSW
Don't use it to test server logic, don't let it be the only gate before release, and don't hand-maintain hundreds of handlers: generate them from the OpenAPI spec where you can. The bridge between the two layers is contract testing: record staging, validate fixtures against the schema, run the same journey in both places.
Curated: · Written: · Reviewed:
Q&A-78A pixel diff is green and a contract test is red. What do you trust for a full-stack change?
Trust the contract test. A red contract means the response shape or status code changed; a green pixel diff only means the rendered pixels still match within tolerance. The contract is the narrower assertion (it names exact fields, types and status codes) so when the two disagree, it is the one that can see the change. Then, before paging anyone, confirm the contract test is running against the build you think it is.
Why the pixels can stay green while the payload is broken: a pixel diff compares a rasterized render with a tolerance and usually masks dynamic regions. A field that feeds a code path outside the snapshot (a modal, a second route, a below-the-fold section) never enters the comparison. A field whose absence renders the same as its presence, because the placeholder was already there or the value formats identically, also passes. The screenshot is a coarse, whole-render check; the contract is a field-level one.
The reverse happens too, and it is why neither check replaces the other: a button clipped by 12 px is a red pixel diff and a green contract, because the payload is fine and the layout is not. But for the conflict in the question, the contract wins.
Worked example (hypothetical). The contract asserts POST /orders returns 201 with orderId. The backend now returns 200 with id. The confirmation page renders "Order placed" from the status code, not the id, so the snapshot still matches.
| Change | Contract | Pixel diff | What the user sees |
|---|---|---|---|
orderId → id, 201 → 200 | red | green | confirmation page fine; the mobile client's order-tracking call breaks |
total "12.00" → 12.00 | red (type) | green | identical render |
| primary button clipped 12 px | green | red | missed taps |
Two failure modes to rule out before you act on the red.
Contract red for the wrong reason: a stale schema, a test pointed at a mock or an old deploy. Detect it by reading the failure, it prints the actual response, and checking the base URL and the schema version pinned in the test. If the response body is the one you expect, the test target is wrong, not the service.
Pixel green for the wrong reason: tolerance too loose, the changed region masked, or the snapshot captured against a mock server. Detect it by re-running the snapshot against the real staging response.
To settle it: remove one required field from the staging response and confirm the contract fails while the snapshot still matches within tolerance. Then point the contract test at the deployed URL to rule out a stale target.
A green screenshot is evidence about pixels. A red contract is evidence about the interface. Only one of them is the interface.
Curated: · Written: · Reviewed:
Q&A-84The network panel shows a fast API call and a slow TTFB. How do you read that waterfall as a full-stack engineer?
The waterfall's TTFB is the document request's "Waiting for server response" segment. The API call is a separate request, usually fired after the document is parsed, so it sits downstream of the bottleneck: a 90 ms API call cannot make a 1.1 s document TTFB fast. Read them as two independent paths and find which segment of which request owns the time.
Read the segments, not the total. Hover the document request → Timing. Chrome splits it into:
| Segment | Measures | Slow here means |
|---|---|---|
| Queueing / Stalled | browser connection limits, priority | too many parallel requests (HTTP/1.1 caps at 6 per origin) |
| DNS Lookup | name resolution | cold resolver, no cache |
| Initial connection + SSL | TCP + TLS handshake | new origin, no keep-alive |
| Request sent | upload | large request body |
| Waiting for server response | server think time + 1 RTT | gateway, SSR, DB, upstream call |
| Content Download | bytes over the wire | body size, no compression, slow client link |
TTFB is everything up to and including "Waiting". So a slow TTFB is either connection setup (cold) or server think time (warm). That distinction decides the fix.
Worked example. Navigation to /dashboard:
- DNS 20 ms, TLS 60 ms, Waiting 900 ms, Content Download 40 ms → TTFB ≈ 980 ms.
GET /api/userfires at 1.1 s, Waiting 90 ms.
The API is fast and irrelevant: first paint is gated by 900 ms of server think time on the document. Add Server-Timing: edge;dur=12, app;dur=780, db;dur=610 to the HTML response and the 780 ms app time is dominated by a 610 ms DB call inside SSR: that is the hop to fix.
When the API really is the problem. If the document is a 2 KB shell with TTFB 30 ms and the API is 800 ms, the API is on the critical path: a client-rendered SPA paints nothing until it returns. Same if the panel shows the API as fast because it was served from disk cache (check the Size column: 304 or (disk cache) is not a server round-trip). And if TTFB is slow only on the first load and fast on repeat, it is DNS/TLS/cold start, not the handler.
Failure modes. Confusing "Waiting" with "Content Download" sends you optimising payload when the server is slow. Measuring from localhost hides connection setup. And a fast API serialized behind a slow document, or 40 fast APIs fired in sequence, still blocks paint; check each request's start time against the first-paint marker, not just its duration.
Curated: · Written: · Reviewed:
Q&A-85Product blames the API for poor Core Web Vitals. How do you decide which half is actually guilty?
Start with the field data, not the ticket. Core Web Vitals are three metrics with three different phase breakdowns, and the API can only reach one of them directly.
Assumptions: you have RUM with a 75th percentile per vital, and a distributed trace that links the browser span to the server span. If all you have is Lighthouse, you have one machine's number and you cannot settle this.
What the API can and cannot touch
- LCP = TTFB + resource load delay + resource load time + render delay. The API owns TTFB and nothing else.
- INP = input delay + processing time + presentation delay. All client-side. The API reaches it only indirectly, when a large JSON payload or a response-triggered hydration blocks the main thread.
- CLS = layout shifts from late-loading content. The API is not in this metric at all.
So the first question is which vital moved. If CLS regressed, the API is innocent by construction.
The decision rule
Compare the API's share of the vital, not its absolute latency. TTFB p75 under 800 ms is the standard "good" threshold; LCP good is 2.5 s. If TTFB is 800 ms inside a 2.5 s LCP, the server is roughly a third of the budget and worth fixing. If TTFB is 70 ms inside a 4.1 s LCP, it is 1.7% of the budget: halving it buys 35 ms and cannot move the score.
Worked example (illustrative figures):
| Metric | Field 75th | API share | Actual owner |
|---|---|---|---|
| LCP | 4.1 s | 70 ms | 1.6 MB hero |
| CLS | 0.28 | 0 ms | late font swap |
| INP | 340 ms | 70 ms | 280 ms click handler |
Arithmetic: to bring that LCP under 2.5 s you must remove 1.6 s. The API's entire contribution is 70 ms. The hero image and the click handler are the budget.
When the API genuinely is guilty
- TTFB p75 above ~1 s and TTFB is the largest single term in LCP. Fix the server: query plans, N+1s, connection-pool saturation, missing cache.
- INP is bad and the trace shows the long task starting when the fetch resolves: a 2 MB JSON payload parsed and hydrated on the main thread. The API is guilty, but the fix is payload shape (pagination, server-rendering the first view, streaming), not query latency.
- LCP is bad on first navigation only, and TTFB is high only on cache misses. That is a caching/CDN problem, not a database problem.
RUM vs lab
RUM decides guilt: it is the same population the score is computed from, and it gives the phase breakdown per vital. Lab tooling (Lighthouse, WebPageTest) decides the fix: it reproduces the page on a controlled connection and lets you bisect the waterfall. A lab LCP of 1.8 s against a field LCP of 4.1 s means field users are on slower devices or cold caches: the API is not the difference.
Fix frontend vs fix API
Frontend fixes are usually cheaper and larger: preload the hero, font-display: swap with a matched fallback, defer non-critical JS. Do those first when the API share is small. Spend on the API when TTFB dominates and the server span in the trace shows the time inside your own code rather than on the network.
Failure mode to watch
The team scales the database because the API dashboard is the only one carrying a p75. Detect it by putting the API's p75 next to the vital's p75 on one chart, with the API's share of the vital as a percentage. Under 10%, capacity work is not the fix.
Curated: · Written: · Reviewed:
Q&A-60Checkout is slow. How do you prove whether the wait is in the browser, the API, or the query?
Bisect the path before you instrument it. Three measurements, in increasing cost, settle most cases:
curl -wthe endpoint from the same network the browser uses. If total is 200 ms, the API and the query are not the problem, and you have spent two minutes.- Open the HAR in DevTools and read the waterfall for the checkout request. If
requestStartlands 3 s after the click, the wait is in the browser before the request leaves. - If curl is slow,
EXPLAIN (ANALYZE, BUFFERS)the query the endpoint runs, and checkpg_stat_statementsfor that statement's mean and max execution time across all callers.
A joined trace is the fourth step, not the first. It is the only tool that answers "which hop owns the 4.8 s" when the hops belong to different teams, but it costs days of instrumentation and a sampling decision.
curl -w 'dns=%{time_namelookup} connect=%{time_connect} tls=%{time_appconnect} ttfb=%{time_starttransfer} total=%{time_total}\n' \
-o /dev/null -s https://api.example.com/checkout
Where each tool stops
| Tool | Answers | Cost | Blind spot |
|---|---|---|---|
| HAR / DevTools waterfall | browser-side wait, retries, third parties | zero | nothing server-side |
curl -w breakdown | DNS/TLS/TTFB/total for one endpoint | zero | no browser, no DB internals |
Server-Timing header | app vs DB split, inside the real browser request | one header, ~an hour | only what you instrument |
EXPLAIN ANALYZE / pg_stat_statements | plan, actual rows, time per query | zero | one statement at a time |
| Distributed tracing | click→SQL in one view, cross-service | days of setup, sampling config | head sampling misses p99 |
HAR is weaker than it looks for third-party calls: a cross-origin resource without a Timing-Allow-Origin response header exposes only startTime, duration, fetchStart and responseEnd. You can prove a payment SDK burned 800 ms, not why.
Server-Timing: db;dur=190, app;dur=230 is usually the best value per hour. It surfaces in DevTools and in PerformanceResourceTiming.serverTiming, so the server's own split appears in the real browser request rather than in a synthetic one.
Tracing has a sampling trap: head sampling at 1% will not contain your p99, so you read traces of fast requests and conclude nothing is wrong. You need tail-based sampling keyed on latency. And when the path is one service and one database, tracing is not worth the setup: HAR plus curl plus EXPLAIN gets there in ten minutes.
Failure modes and how they show up
- Client retry loop. A fetch wrapper retries a timeout or 5xx with no backoff. HAR shows the same URL three or four times; a trace shows sibling spans with the same name. This is the most common cause of "the API is slow" that isn't.
- Connection pool saturation. p99 climbs once concurrent requests exceed pool size while the query itself stays fast. Detect with pool wait time, or
pg_stat_activityshowing sessionsidle in transaction. - N+1. Forty queries per checkout at 5 ms each. Detect by query count per request, not query duration.
- Main-thread blocking. The response arrives, then 1 s of paint is blocked by a 300 ms long task. Detect with the Long Tasks API or Total Blocking Time.
- Third-party tag. A payment or analytics script on the critical path. Detect in the HAR waterfall as a domain you do not own.
Worked example
A checkout p99 sat at 4.8 s. Web blamed the API, the API blamed the database, and three teams spent 90 minutes before a joined trace split it:
| Span | Time | Owner |
|---|---|---|
| click to first fetch | 40 ms | client |
| client retry loop | 3,100 ms | client |
| BFF plus API | 420 ms | server |
| SQL | 190 ms | database |
| remaining paint | 1,050 ms | client |
4,100 of the 4,800 ms were client-side, and the API was never slow. A HAR would have shown the repeated request in under a minute; the trace was needed only to prove the retries were the same logical call.
The decision rule: curl fast and HAR showing delay before requestStart means the browser; curl slow with a sequential scan in EXPLAIN means the query; both fast with repeated requests in the HAR means the client. Do not stand up tracing until two of the three cheap tools disagree.
Curated: · Written: · Reviewed:
Q&A-61API graphs look fine. Users still say the page is slow. What else do you measure?
What I'd add: field data from the device, and a way to attribute main-thread time.
API APM measures one hop. A page is a waterfall: DNS/TLS, TTFB, HTML, render-blocking CSS, JS download and parse, hydration, then the API call. If the API is 180 ms of a 3.9 s LCP, the API graph is describing about 5% of the user's wait.
Four signals, in the order I'd add them:
| Signal | Answers | Blind spot |
|---|---|---|
| RUM field data (LCP, INP, CLS, TTFB) segmented by device class, connection type, geo, route | What real users felt | Sampled, no stack traces, thin on rare segments |
Browser tracing (OTel browser SDK, PerformanceObserver) | Which long task, which resource, which fetch | Adds overhead; needs sampling; cross-origin gaps |
| Synthetic checks from real devices and geos on a schedule | Deterministic regression detection, cold cache | Clean network, no real user mix |
Resource Timing plus Server-Timing | Splits TTFB into server, network and queue | Cross-origin entries are zeroed without Timing-Allow-Origin |
A worked trace. Mid-range Android, Chrome, throttled 4G at roughly 200 KB/s, 4x CPU slowdown, cold cache. RUM p75 for the route was 3.9 s.
| Phase | Bytes | Time |
|---|---|---|
| DNS + TCP + TLS | — | 240 ms |
| TTFB | — | 310 ms |
| HTML | 48 KB | 240 ms |
| Render-blocking CSS | 96 KB | 480 ms |
| Third-party tag manager JS | 320 KB | 1600 ms |
| LCP hero image | 210 KB | 1050 ms |
| LCP | 3920 ms |
At 200 KB/s those byte counts are 240, 480, 1600 and 1050 ms, and the column sums to 3920 ms. Hydration then ran a 1180 ms long task, which is where the INP p75 of 1.4 s came from. API APM p95 for the same route was 180 ms. Nothing in the API graph could have shown the tag manager or the hydration cost.
// Chromium 120+. report() is your beacon, tagged with the route id.
new PerformanceObserver((list) => {
for (const e of list.getEntries()) {
if (e.duration >= 50) report('longtask', { start: e.startTime, duration: e.duration });
}
}).observe({ type: 'longtask', buffered: true });
new PerformanceObserver((list) => {
const lcp = list.getEntries().at(-1);
report('lcp', { start: lcp.startTime, size: lcp.size, url: lcp.url });
}).observe({ type: 'largest-contentful-paint', buffered: true });
When RUM-plus-APM correlation is not enough.
- Third-party origins. Without
Timing-Allow-Originyou getdurationbut the phase breakdown is zeroed, and there is no server span at all. Bound the cost by canary: block the tag for 1% of sessions and diff LCP p75. - Cross-origin iframes (consent, ads, payment) are opaque end to end. Measure by difference, not by attribution.
- Client-side rendering. The API is fast and the main thread is busy; hydration has no server span. The Long Tasks API is the only direct measure, and it is Chromium-only: Safari and Firefox need a synthetic proxy.
- Low-volume segments. A p95 over 40 samples a day is noise. Require n ≥ 100 per segment per day before alerting, otherwise fall back to synthetic.
- Perceived slowness with good metrics. SPA soft navigations and route transitions are not LCP. Measure per-interaction INP and route-change timing separately.
Failure modes and how I'd catch them.
Beacon loss: RUM beacons dropped on unload. Send with navigator.sendBeacon on visibilitychange, and detect the loss by comparing beacon count against CDN access logs for the same route. Synthetic/RUM divergence: synthetic runs warm-cache from a datacenter, so a synthetic p75 of 1.2 s against a RUM p75 of 3.9 s is device and network, not code. Check the gap before filing a bug. Alerting on the wrong side: alert on the user-side budget (LCP p75 above 2.5 s for three consecutive hourly windows) rather than on API latency, which stayed green through all of this.
The criterion I'd hold: for the same route and the same hour, put RUM p95 next to API p95. If the gap exceeds 500 ms and API payload transfer does not explain it, the missing time is on the main thread or in a third-party origin, and that is what you go measure next.
Curated: · Written: · Reviewed:
Q&A-62API availability is above the SLO. Signups still fail. What should the budget have counted?
The budget should have counted completed signup journeys, not API availability. A journey is a named sequence of client events and server outcomes, and it counts as successful only when the last server write and the last client confirmation both happen. Anything short of that burns budget.
Per-service availability misses this because availability is computed over requests that reach the service. A client that gives up at 2 seconds either never sends the request or abandons the response, so the server records a fast, successful request while the user sees a failure. Both halves are healthy; the seam between them is not. The same gap shows up when a retry succeeds server-side after the user has navigated away, and when a write commits but its response is lost to a dropped connection.
Worked example, a signup week with a 2-second client timeout:
| Metric | Value |
|---|---|
| API availability | 99.95% |
| signup journeys attempted | 50,000 |
| confirm screen reached | 45,700 |
| journey success | 91.4% |
4,300 journeys, 8.6%, never reached the confirm screen while the API reported 99.95%. If the SLO is 99.5% journey success, the weekly budget is 0.5% of 50,000 = 250 failed journeys, and this week burned 4,300, about 17x the weekly budget. That is the number that should have paged.
Instrumenting it: emit a journey ID at the first client event, carry it through every server call, and emit a terminal client event on success or abandonment. Success means the terminal success event and the server-side completion record for the same journey ID. Track the ratio over a rolling window.
Trade-offs and when not to use it:
- Client telemetry is lossy. Ad blockers, offline clients and crashes drop the terminal event, which looks identical to failure. Pair the client counter with a server-side completion counter and treat the difference as its own signal rather than folding it into the SLI.
- Journey SLIs need a stable journey definition. If the flow changes weekly, the number is not comparable week to week, so version the definition alongside the SLI.
- For a single-endpoint service with no client, per-endpoint availability is the right SLI. Journey accounting pays off when the failure mode lives at the seam.
Failure modes to watch:
- Retries double-counting. Dedupe on journey ID, not on request.
- Success defined too loosely. "Confirm screen rendered" is not "account usable"; if the screen can render before the write commits, you will count failures as successes.
- Sampling. At 1% client-event sampling, a 250-journey budget is 2.5 sampled events, too few to alert on. Sample the terminal event at 100% or alert on the server-side counter.
Detection: alert when the server-side completion count and the client-side success count diverge by more than the budget, and page on remaining budget at 25%.
Curated: · Written: · Reviewed:
Q&A-75Web dashboards are green and API dashboards are green, yet checkout fails. How do you run that incident?
I treat it as one incident for the checkout journey, not two component incidents. First 60 seconds: confirm the failing journey, the affected user slice, and the last deploy on both sides. Assume a checkout success SLO of 99.9% over 28 days; page on a 14.4x burn rate over 1h (2% of the 28-day budget). Freeze deploys. Declare one incident commander for the journey. If web and API are separate teams, that commander gets a liaison from each; I do not run two parallel incidents.
The reason both dashboards are green is that they measure component health, not the join. The web dashboard watches JS errors, CDN status, and page load. The API dashboard watches endpoint latency and 5xx. Checkout is a distributed transaction across client, API, payment, and inventory. An API that returns HTTP 200 with {error:"invalid_cart"} is green on the API dashboard and red on the journey. A stale service worker serving old JS is green on the web dashboard and red on the journey. So the first diagnostic move is to get a journey-level signal: a synthetic checkout every minute from a real browser, plus RUM success rate.
Correlate one failing session. With OpenTelemetry browser instrumentation, the browser injects the W3C Trace Context traceparent header on fetch/XHR, and the server continues the same trace. Search that trace ID in Jaeger, Tempo, or Datadog. If you do not have RUM, capture a HAR in Chrome DevTools, find the failing POST /api/checkout, note the x-request-id or traceparent, and grep server logs for it. Then compare the client asset version to the API version and check the service worker cache.
Hypothetical trace: at 10:00, web v412 and API v88 ship. Checkout success drops to 89%. Web dashboard: JS error rate 0.1%, CDN 200s. API dashboard: p99 120 ms, 5xx 0.02%. Browser sends POST /api/checkout with traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. The server span with that trace ID shows cart.items[].price missing and returns 200 {error:"invalid_cart"}. The client service worker v411 expects price. Fix: bump the service worker cache version and make the API accept both shapes. Success returns to 99.6%.
Trade-offs. Joint commander versus single commander: joint when two teams own the halves and the contract is unclear; single when one team owns both or the failure is clearly one side. Rollback versus fix-forward: roll back if the last deploy is under 30 minutes old and no schema migration is involved; fix forward if rollback would lose data or break a migration. Do not create a joint incident for a single-user failure; that is a support ticket.
The incident closes when the journey SLI is back inside SLO and the timeline contains both a client artefact and a server artefact. If both halves look green and the user is still stuck, it is one incident.
Curated: · Written: · Reviewed:
Q&A-89You are on call for a service that is both a web app and an API. How do you page and mitigate?
What I'd say in the first two minutes. Assume one on-call rotation owns the whole user journey (the bundle, the CDN, and the API) and that the bundle and API ship from separate pipelines. Page on the user-facing outcome, not on 5xx. A 5xx-only pager is blind to the worst case here: the API returns 200 with an empty result set because the new bundle calls a field the API no longer returns. Nothing 5xx, nothing in the API logs, white screen for the customer. So the paging signal is an SLO burn rate on the critical journey (checkout, login, search), fed by synthetic checks that assert data, plus RUM error-free session rate as a second source. Mitigation is: confirm blast radius, correlate with the two deploy timelines, then roll back the half that moved, bundle or API, independently, and roll forward the contract if it's skew.
Why burn rate, not raw error rate. Raw thresholds page on noise and sleep through slow burns. Multi-window multi-burn-rate alerts (SRE Workbook) page at 14.4x burn over a 1h window with a 5m short window, and open a ticket at 6x over 6h. That pages fast on a hard outage and still catches a 2%-an-hour bleed.
The synthetic check has to assert rows, not status.
// Playwright, Chromium 126, every 60s from 3 regions
const res = await page.goto('/checkout');
const rows = await page.locator('[data-test="order-row"]').count();
if (res.status() !== 200 || rows === 0)
throw new Error(`checkout empty: status=${res.status()} rows=${rows}`);
A status-only check passes this incident. The row count does not.
A worked incident (figures hypothetical). A bundle deploy removed a field the API had already stopped returning.
| Signal | Value | Who watched it |
|---|---|---|
| RUM error-free sessions | 82% (from 99.6%) | nobody paged |
| API 5xx | 0.2% | API pager, quiet |
| Synthetic checkout rows | 0 rows at t+2m | added after |
| Time to mitigate, joint runbook | 8 min | one on-call |
Before the fix this ran 74 minutes because the frontend team waited on the API team and vice versa. One pager owning the journey collapses that.
Runbook, in order.
- 0–2 min: confirm from two regions; check RUM and synthetic together so a single flaky region doesn't page.
- 2–5 min: pull both deploy timelines. Bundle deploy in the last 30 min → suspect skew; API deploy → suspect contract.
- 5–10 min: if the bundle moved, roll it back and purge the CDN, then verify the served asset hash matches the rolled-back build. If the API moved, roll back or flip the flag. If it's skew, roll the API forward to accept both field names: rolling the bundle back is faster but leaves the skew for the next deploy.
Failure modes and how they show up.
- Version skew: 200s with empty rows, flat 5xx. Detected only by a data assertion.
- Stale CDN after rollback: rollback looks done but users still get the bad bundle. Verify by asset hash, not by deploy status.
- Synthetic flakiness: page on two consecutive failures across regions, or the check becomes noise and gets muted.
- Split ownership: two pagers, each waiting on the other. One rotation owns the journey.
Trade-offs. Synthetic checks cost money and can flake; RUM is sampled and lags by minutes, so it's a corroborating signal, not the pager. Rolling back the bundle is seconds; rolling back an API with a schema change may need a migration-safe flag, which is why the contract should accept both shapes during a deploy window. I'd validate the whole thing with a game day that breaks the bundle and the API separately and confirms one on-call can mitigate each from the same runbook.
Curated: · Written: · Reviewed:
Q&A-88A ticket needs a UI change and an API change. How do you pair so the work actually lands together?
Assume one ticket, one deployable, and a UI change that only makes sense once the API returns the new field. Pair on a single branch, fix the contract before either of you writes code, and gate the merge on one end-to-end journey rather than on two diffs.
That is the whole answer. The rest is how to run it.
First 30 minutes: the contract
Write the example before the code. The exact JSON the page sends, the exact shape it gets back, including the new field's name, its type, and what it holds when the value is absent. Both of you read it against the ticket. Most UI/API tickets that go wrong go wrong here, and it costs half an hour to find out.
Then: one branch, two seats
One of you implements the handler, the other the page. Switch seats at the write and the render: whoever wrote the handler writes the fetch call, whoever wrote the page writes the response mapping. That switch is what catches the mismatch, because the person who knows the API's edge cases is now the one consuming them.
The gate
Do not merge until both of you have run the same click in the same preview environment: create the thing, confirm the row in the store, reload, confirm the page renders the new value. Not "the tests pass": the journey.
Why not split by file type
Splitting by folder is the default and it fails in a specific way: the contract lives only in chat. The UI ships against a guessed field name, the API ships against a guessed consumer, and the mismatch surfaces in staging. A representative shape of that, with illustrative numbers:
| Approach | Calendar days | Handoffs | Enum mismatches |
|---|---|---|---|
| split by folder | 11 | 4 | 3 |
| one branch, contract first | 3 | 0 | 0 |
| contract first, then split anyway | 8 | 2 | 1 |
The third row is the interesting one. Agreeing the contract and then splitting still costs you the handoffs, because the journey never runs end to end until merge day.
When not to pair this way
If the API change breaks an existing contract with other consumers, pairing on one branch does not help: version the endpoint, keep the old shape live, and pair only on the additive part. If the field already exists and the UI just reads it, pairing is waste; a contract test is enough. If the two of you cannot share a preview environment, the journey gate degrades to a contract test plus a recorded trace, which is weaker: say so out loud rather than pretending it is equivalent.
Failure modes and how you would catch them
- Case drift: the page sends
"refunded", the handler expects"REFUNDED". Catch it with a contract test that runs the real serializer against the real parser, not a hand-written fixture. - Merge-day skew: the halves land days apart, so no environment ever exercises both. Catch it by making the journey a required check on the PR.
- Silent null: the field is optional in the API and the page renders
undefined. Catch it by asserting the field is present and non-null inside the journey.
Done looks like
One click in a shared preview writes the new status, the same session reads it back, both halves on the same commit.
Curated: · Written: · Reviewed:
Q&A-96What goes wrong when a full-stack review only comments on the half the reviewer knows?
The failure in one line: when a review only reads the half the reviewer knows, the boundary between the halves gets approved by default. The reviewer's silence on the other side is read as a pass.
Assumptions that make this the answer: the change crosses the boundary (a new field, status code, or retry), the reviewer is the only approver, and nothing else pins the contract. No typed schema, no contract test.
Three things go wrong, roughly in order of cost:
- The payload contract becomes unowned. The UI author assumes the server validates; the server author assumes the UI sends a clean value. Neither is wrong locally, so both files pass review and the gap ships.
- The reviewer's confidence is miscalibrated. "I read the diff" quietly becomes "I reviewed the change." The skipped half is exactly where authz, idempotency and empty/error states live.
- Failure modes never get named, so they never get tested. Nobody writes the test for the 403, the double-submit, or the empty list, because nobody said them out loud.
Worked trace: a tenant leak that passed a UI-only review:
- UI adds an Export button. It calls
GET /export?org=Acme Inc: the org's display name, because that is what the page already holds in state. - The handler reads
orgfrom the query string and runsSELECT * FROM rows WHERE org_name = $1. It never checks that the caller is a member of that org. - The reviewer reads only the React diff: button, loading state, disabled while pending. Approves.
- Any authenticated user can now type another org's display name and stream its rows. The demo tenant returns 900 rows.
The UI-only review found nothing because nothing was wrong in the UI. A handler-only review would have found nothing either: the query is indexed and correct for the value it is handed. Only reading both, and asking who is allowed to send that value, finds it.
| Review scope | What it sees | What it misses |
|---|---|---|
| UI diff only | button, loading, disabled state | handler trusts a client-supplied org |
| handler diff only | indexed query, correct for its input | caller is never checked against the org |
| both, plus "who may send this value" | missing membership check | — |
Trade-offs. "Refuse approval unless both halves are exercised" is the safe default, but it costs something: it blocks small, genuinely one-sided changes and turns the reviewer into a bottleneck on work they cannot judge. Cheaper options that cover the same gap:
- Pair a frontend reviewer with a backend reviewer on the same PR, each signing the half they own and explicitly flagging the boundary.
- Pin the boundary with a contract test or a typed schema, so the review is not the only control.
- Split the PR: land the contract change first, reviewed by both, then land each half against a fixed interface.
Partial review is acceptable when the change does not cross the boundary, or when something else already pins it: a schema, a contract test, a generated client. Then a one-sided review is a real review rather than a coin flip.
Detection: a PR that touches both halves with a comment from only one reviewer is the signal. Ask for the example request and response in the PR description; if the author cannot produce them, the boundary was never designed.
Curated: · Written: · Reviewed:
Q&A-95What belongs in a design doc for a vertical slice that cuts through UI, API, and store?
The 90-second answer
A vertical slice doc pins the contract at each seam so one request can be built, shipped and rolled back on its own. It contains, in order: the slice statement, the wire contract, the store change, the UI states, the failure and rollback plan, and the one metric that says it shipped. Anything that doesn't cross a seam in this slice goes in an out-of-scope list.
Slice statement. One user-visible action, one request, one row, one check the user performs. "Customer places an order and sees it in their order list." If you can't write the check as a sentence a QA engineer can run, the slice is too big.
Wire contract. The exact request and response, with field names, types, nullability, and, the part that gets skipped, which layer mints the id. Also the error envelope: status codes, whether the body carries a machine-readable code, and whether a retry is safe.
Store. Tables and columns touched, the migration, whether it is reversible, and the transaction boundary. If the migration adds a NOT NULL column, say how existing rows get backfilled before the code that reads it ships.
UI states. Loading, empty, error, partial. Which fields the view reads, and what it does with fields it doesn't know: ignore or fail.
Failure and rollback. Feature flag name, what happens when the flag is off, and how to undo the migration. Expand-contract when the store change and the code can't ship together.
Metric. One number. "Orders created per day with a rendered confirmation" beats "API latency".
Worked trace: who owns the id
Request:
POST /v1/orders
{"customer_id":"cus_8f21","items":[{"sku":"SKU-114","qty":2}],"idempotency_key":"9c1f4a"}
Handler mints ord_01H8Z... (ULID), inserts one row, returns 201:
{"id":"ord_01H8Z...","status":"pending","total_cents":4998,"created_at":"2025-06-02T14:03:11Z"}
The list endpoint returns the same id under the same name, and the UI keys its row on it. One identifier, minted once, echoed everywhere. The doc states this explicitly because the alternative, UI keying on customer_id + created_at, is what produces duplicate rows when two orders land in the same second.
What each omission costs
| Omitted | Symptom | Detected by |
|---|---|---|
| id ownership | UI invents a second key; duplicate rows | two orders in the same second |
| error envelope | UI shows "something went wrong" for a 409 it could retry | retry storm or silent drop |
| migration reversibility | rollback leaves a NOT NULL column | rollback drill |
| out-of-scope list | reviewer argues about the auth model for a week | review latency |
When not to use a slice doc. Cross-cutting changes (auth model, tenancy, a new event bus) don't fit; they touch every slice and need their own doc. A slice doc is also wrong if the slice can't be rolled back alone, because then the rollback section is fiction.
The doc is done when a reviewer can walk one payload from request to rendered row without inventing a second identifier.
Curated: · Written: · Reviewed:
Q&A-94How do you estimate a feature that needs a schema change and a new screen?
Estimate three things, not two: the schema change and its backfill, the screen with all its states, and the contract that couples them. Then give a range whose width is set by the single largest unknown, and name that unknown out loud.
The contract is the coupling point because it is the only place where a change on one side invalidates the other's estimate. The API shape decides which screen states exist (a 409 on a stale write, an empty list, a partial page) and the schema decides what the API can return without a join or a backfill. Add a column and the screen gains a state; add a screen state and the API needs a field the schema may not have. Estimate the halves separately and you price each one against a contract that hasn't been agreed.
Sizing the unknown: pick the one question that moves the estimate most. Usually "does the backfill fit in the maintenance window?", and timebox a half-day spike to answer it. If the spike answers it, the estimate tightens; if it can't, the unknown stays as its own line at the top of the range. Don't pad every line to cover it; that hides where the risk is.
Turning the breakdown into a range: sum the parts you can size, then add the unknown as a separate low/high. The range width comes from that line, not from a blanket multiplier. A 9-day point estimate with a 5-day unknown is "9 to 14", and the reviewer can see why.
Three approaches, and when each wins:
- T-shirt sizing plus a named unknown: enough when the feature is small, reversible, and the team has shipped something similar. Fast, low precision.
- Bottom-up breakdown: worth it when the schema change is irreversible (a migration on a large table, a column drop) or the screen has many states. This is where the contract tests earn their line.
- Spike then re-estimate: when the largest unknown is a technical question you can answer by building a throwaway. Costs a day, removes the widest part of the range.
Failure modes: estimating from the mockup alone; treating the migration as free; no dual rollback; contract drift found at integration. Each shows up as a line that was zero in the estimate.
Worked example (hypothetical): a billing screen.
| Part | Estimate | Actual |
|---|---|---|
| happy path | 3 d | 3 d |
| 7 states incl. 409, empty | 2 d | 5 d |
| migration + backfill | 2 d | 4 d |
| contract tests + dual rollback | 2 d | 5 d |
Estimate 9 d, range 9–14 d with the backfill window as the unknown; actual 17 d. The range was right to be wide: the miss was that the unknown line was the one that grew, and the states were priced at 2 d against a contract that changed twice.
Afterwards, compare estimate to actual and record which half and which contract item moved. That is the only way the next estimate of this shape gets tighter.
Curated: · Written: · Reviewed:
Q&A-87A bug only happens for one customer. How do you reproduce it with a HAR file and SQL?
Core answer. Assume two things before you start: the customer can export a HAR from the failing session, and your stack propagates a request id from the HTTP layer into the database session. If the second is false, fix that first: it is the whole method.
Reproduce the bug by replaying the exact request the browser sent and reading the SQL that request caused, joined on the request id. A green local UI proves nothing; the customer's request is the input, the query log is the output.
Mechanism. A HAR is JSON: log.entries[], each with request (method, url, headers, postData) and response (status, content). Replay the failing entry with a script, tagging it so the server's SQL log can be attributed to it:
import json, requests
har = json.load(open("failing.har"))["log"]["entries"]
e = next(x for x in har if "/api/orders" in x["request"]["url"])
req = e["request"]
hdrs = {h["name"]: h["value"] for h in req["headers"] if h["name"].lower() != "cookie"}
hdrs["X-Request-Id"] = "replay-1"
r = requests.request(req["method"], req["url"], headers=hdrs,
data=req.get("postData", {}).get("text"))
print(r.status_code, len(r.content))
On the database side, make the request id visible in the log. In Postgres 16, set log_min_duration_statement = 0 and log_line_prefix = '%m [%p] %q%a '; %a is application_name, which your app sets per request (SET LOCAL application_name = 'replay-1' inside the transaction, so it survives connection pooling). Then grep replay-1 postgresql.log gives every statement that request ran, in order.
Why the request id and not the timestamp: under concurrency two requests land in the same millisecond, and the same URL is hit repeatedly, so neither is unique. The request id is the only field both the HAR and the query log carry that identifies one request.
What it looks like. A list page that took 1.8 s for one tenant:
| Source | Evidence |
|---|---|
| HAR | one GET /api/orders, 200, 1.8 s, 60 rows in the body |
Query log, replay-1 | 1 SELECT ... FROM orders (12 ms) + 9 SELECT ... FROM line_items per order × 60 = 541 statements, 1.71 s DB time |
| Joined | the N+1 is server-side; the browser never issued those 540 requests |
That is the reproduction: same input, same 541 statements, same 1.8 s. Now you can bisect.
Alternatives and when this fails. Timestamp correlation is fine for a single-user repro and useless under load. Browser devtools gives you the same HAR live, but you cannot get it from the customer. A proxy (mitmproxy, Charles) captures non-browser clients and websockets, at the cost of TLS interception. For replay, requests is fine for one request; Playwright's routeFromHAR replays a whole session including XHR ordering.
It fails when the HAR cannot see the cause: async work (queues, cron, webhooks) never appears in a HAR; SSE and websockets show as one entry with no frames; a 304 or disk-cached response has no body; and if the request id is not propagated to the DB, you are back to timestamps. Scrub PII from HAR bodies and bind values before you share either artifact.
Curated: · Written: · Reviewed:
Q&A-93How do you mentor a strong frontend or backend specialist so they can work across the full-stack boundary?
The 90-second answer
Mentoring a specialist across the boundary is not teaching them the other language or framework. It is teaching them the other half's failure mode, and the fastest way to do that is to make them own one end-to-end journey until they can debug it without a handoff.
State the assumptions first. The specialist is already strong on their own side, so the gap is context, not skill: they don't know what the other half does with what they produce. And you have a real production journey to use, not a tutorial repo.
So pick one journey, "user opens the order page", and trace it together from the click to the row. Then hand them the unfamiliar half of the next real ticket, and review the contract between the halves: request shape, response shape, error semantics. Not the style of the code.
The trace is the teaching tool
Worked example, hypothetical, to show the shape of the exercise:
Backend returns 200 with {"order_total_cents": 4200}. Frontend reads order.total. Renders undefined. No exception, no network error, blank price. The backend specialist sees a green test suite and a 200 in the logs. The frontend specialist sees a white screen and a clean network tab. Neither is wrong on their own side.
Trace it: click → fetch → 200 → parse → field lookup → undefined → render. The break is at the field lookup, and the fix could legitimately land on either side. That one trace teaches the backend specialist why a renamed field is a breaking change even though the status code is 200, and teaches the frontend specialist why a missing field should fail loudly instead of rendering nothing.
What to review, and what to measure
| Stage | What the mentor reviews | Signal it's working |
|---|---|---|
| First ticket | the contract: request/response shape, error cases | they ask about the other half's assumptions unprompted |
| Second ticket | contract only, nothing on their home half | they catch a breaking change before review |
| Third ticket | nothing | they diagnose a cross-boundary incident without paging the other specialist |
Failure modes
Mentoring as a tutorial repo teaches syntax, not the failure mode, and it doesn't transfer to production. Reviewing style on the unfamiliar half makes the specialist defensive and teaches them nothing about the boundary. Skipping the production journey leaves the exercise abstract, so they can't connect it to the ticket in front of them.
How you know it worked
Have them reproduce one incident from a HAR and a query log with the other specialist out of the room, then ship the fix on both sides. If they can do that, the boundary has stopped being a handoff.
Curated: · Written: · Reviewed:
Q&A-90Tell me about a conflict over whether the frontend or the backend owned a piece of behaviour.
Situation. At a retailer's checkout, the React/TypeScript cart computed tax per line item for display, and the Java/Spring invoicing service computed tax on the order total for the invoice. Both shipped, both passed their own tests, and finance reconciliation started flagging orders where the cart total and the invoice total disagreed by a cent or two.
Task. I owned the checkout frontend; invoicing was another team's. We had to name a single owner of the tax figure before the next billing cycle, without losing the per-line tax the cart showed.
Action. I traced both implementations on the same basket. Worked example, rate 8.875%, three items at $19.99:
| Step | Client (round per line) | Server (round total) |
|---|---|---|
| line tax | 1999 × 0.08875 = 177.41125 → 177 | — |
| three lines | 177 + 177 + 177 = 531 | — |
| total tax | — | 5997 × 0.08875 = 532.23375 → 532 |
One cent apart, every order, deterministically. That killed the "it's a race condition" theory and pointed at the rule itself.
The mechanism is that rounding is not associative: round(a) + round(b) ≠ round(a + b). Two implementations of "the tax rule" diverge whenever any of three things differ: the rounding mode (half-up vs banker's), the unit being rounded (per line vs total), or the rate source (client config vs server tax table). The client had a hardcoded rate that lagged the server table by a deploy, so the gap widened after every rate change.
Result. The server became the only owner of persisted tax. The quote endpoint returns line-level tax in integer cents; the client formats those integers and never computes tax. Reconciliation went clean, and the per-line display survived because the server now returns per-line values.
When the client should own it instead. Client-side ownership is right when the number is presentational and never persisted or billed: a shipping estimate before the address is confirmed, a "you'll pay about" preview, and the server recomputes authoritatively at commit. It's also right for offline-first flows where you can't round-trip. Server ownership is right for anything persisted, billed, audited, or shown as a legal total. The deciding question is whether the number ever leaves the session as a fact. If it does, one owner, and it's the side that persists.
Failure modes and how they surface.
- One-sided rate change: the client's rate drifts from the server's. Detected by a nightly reconciliation job diffing quote totals against invoice totals, alerting on any nonzero diff.
- Float accumulation: if either side stores tax as a double, error compounds across lines. Use integer cents end to end; a double tax column diverges at scale.
- Silent divergence: the client is often the only side with UI tests, so the server's rule is untested against real baskets. A contract test that runs the same basket through both implementations catches it before reconciliation does.
The conflict resolved because the trace made the disagreement a one-cent arithmetic fact rather than a matter of opinion, and the contract (server returns integer cents per line, client formats) gave both teams a single place to look.
Curated: · Written: · Reviewed:
Q&A-91Tell me about a time you missed an SLA that was defined on the whole user journey, not on one service.
Last year I owned the checkout journey SLO: 99.9% of checkout attempts complete end to end over a rolling 30 days. We missed it. Journey success sat at 99.2% for 11 hours while both half-SLAs underneath stayed green: the API reported 99.95% availability and the web client's error rate never moved. Nobody paged, because no single service was failing.
Here's the shape of it.
| Situation | Journey SLO 99.9%/30d; observed 99.2% for 11 hours; API 99.95%, client error rate flat |
| Task | Find which hop burned the budget while both half-dashboards stayed green |
| Action | Joined client attempts to server outcomes on a request id; found a 3-retry, no-jitter storm on 502s; capped retries at 1 with jitter and a 10% retry budget |
| Result | Journey back to 99.94%; pool wait p99 800 ms → 40 ms |
The cost: 2M checkouts a day is 60M a month, so the 0.1% budget is 60,000 failed journeys. Eleven hours is about 917,000 journeys; at 0.8% failure that's roughly 7,300 failed checkouts, about 12% of the month's budget gone in half a day.
The mechanism was retry amplification across the boundary. A downstream dependency got slow, the API started returning 502s after its 2-second timeout, and the client retried every 502 three times with no jitter. Attempts per request went from 1.0 to 2.4. The API's connection pool was 20 connections, each held for about 2 seconds while the downstream timed out, so the pool could serve roughly 10 requests a second against an offered load of about 24. The queue grew, p99 pool wait hit 800 ms, more requests timed out, and more retries followed. Both halves looked healthy by their own definitions: the API counted a 502 as a completed request, and the client counted each retry as a fresh attempt, so the retries never appeared as errors on either dashboard.
Capping retries at 1 is one fix, not the fix. The alternatives I weighed: a retry budget, retries capped at 10% of successful requests, which keeps retries for genuine blips without letting them amplify; jittered exponential backoff, which spreads the herd; a circuit breaker, which stops calling a dependency that's already failing; and fixing the downstream slowness, which was the actual root cause. I chose the cap plus jitter because it was the smallest change that broke the feedback loop, and I filed the downstream fix separately.
Capping is the wrong call when the failure is a short transient (a single-AZ blip, a leader election) and the request is idempotent. There, retries recover the user with no visible error, and capping at 1 turns a recoverable blip into a failed checkout. And if the write isn't idempotent, the fix isn't a retry cap at all: it's idempotency keys, because retrying a POST without one can double-charge.
What I'd watch for now: attempts-per-request above 1.2 means a retry storm is forming; a spike in duplicate order ids means non-idempotent retries; and journey success diverging from either half by more than 0.2 points means the halves are lying. That last one is the alert I added: join client attempts to server outcomes on a request id and page on the divergence, not on either half's own rate.
The lesson I'd carry into the next role: an end-to-end SLA is successful journeys over attempted journeys. It is not two half-rates added together, and it will not be protected by two dashboards that are each green.
Curated: · Written: · Reviewed:
Q&A-92Tell me about a time you refused to ship a frontend-only fix because the store would still be wrong.
Last year I owned the profile settings page. Product wanted a client-side cache so the page felt instant: write to local state, flip the button to "Saved", sync in the background. Three days of work. I refused it, because the store enforces a unique constraint on email that the client can't evaluate, and the cache would have turned every rejected write into a lie on screen.
The situation. The email column has a unique index: two accounts can't share an email. The client has no way to know whether an email is taken; that check lives in the database, and the only honest answer comes back as a 409 from the write endpoint. The proposed cache wrote to local state first, showed "Saved", and fired the request afterwards. When the server returned 409, nothing in the UI listened for it. The page said saved; the row was unchanged. A refresh, the one action a user takes when they doubt the page, would show the old email, and now the user trusts the screen less than before.
The task was narrow: stop the page from claiming a write succeeded when the store rejected it.
The action was to decline the cache and bind the constraint to the form instead. The write endpoint already returned 409 with the offending field. I made the form await the response, map the 409 onto the email input, and only then show success. No local "saved" state until a 2xx. That was nine days, not three: the extra time was the error contract and the tests around it.
The result: after launch, refresh matched the row every time, and the 409 surfaced on the field the user could actually fix.
Why the cache hides the failure: optimistic UI assumes the write will succeed. That assumption holds when the operation is idempotent and conflict-free: a like counter, a draft autosave. It breaks the moment the store enforces a constraint the client can't check. The client is guessing at a decision only the database can make.
POST /profile {"email":"a@b.com"}
-> 409 {"field":"email","error":"already_taken"}
shortcut branch: local state = saved, 409 ignored
fix branch: email field = "already_taken", no saved state
GET /profile after refresh
shortcut: old email, screen claimed success
fix: old email, field still flagged
Failure modes worth naming: silent divergence between screen and row (detect with a refresh-after-save test), a 409 that never reaches a field (detect with a contract test asserting the error shape), and a retry loop that re-sends a write the store will always reject.
The trade-off: optimistic UI earns its keep when the round trip is the bottleneck and conflicts are rare. Here the round trip was about 120 ms and the conflict was a hard constraint, so the cache bought latency at the cost of correctness.
The evidence I'd want before calling it fixed: refresh after save on both branches, showing the shortcut lying while the row is unchanged. In the staging run before launch, the shortcut branch produced 4,200 writes that never reached the store while the page showed saved: a hypothetical figure from our load test, not production, but it's the shape of the failure.
Curated: · Written: · Reviewed:
Q&A-99When does a backend-for-frontend belong on the team tech radar, and when is it extra hop theatre?
What I'd say first
Assume "radar" means the team's assess/trial/adopt rings, and that we're talking about a BFF as a shared, team-owned deployable, not a library. A BFF belongs on the radar when it does a boundary job the client cannot do for itself: terminate the session, hold a credential the browser must never see, or collapse several backends into one screen-shaped response. It's theatre when it's a 1:1 proxy in front of a single API the client already calls directly.
Why the hop is cheap or expensive, depending on where it sits
A BFF is a server process between the client and the domain services. At runtime it does four things: terminates the client connection and its auth, fans out to N backends, merges the responses into one payload shaped for one screen, and returns. The fan-out happens inside the datacenter, where a round trip is sub-millisecond. The client hop happens across whatever the user is on: 40-80 ms RTT on LTE, worse on a bad link.
That asymmetry is the whole argument. Aggregation pays when it replaces N client round trips with one client round trip plus N datacenter round trips. It pays most when the calls are sequential and dependent, because sequential client calls multiply the RTT.
Worked example: a screen that needs user → org → feature flags → flag config, four dependent calls:
| Path | Client RTTs | Network wait at 60 ms RTT | Datacenter hops |
|---|---|---|---|
| Client calls each backend | 4 sequential | 240 ms | 0 |
| BFF aggregates | 1 | 60 ms | 4 × ~0.5 ms |
Roughly 180 ms saved, before counting the four TLS handshakes and four auth checks the client would otherwise do. Now put the same BFF in front of a single backend: one client RTT either way, plus 5-15 ms of extra serialization and a process to deploy. That's the theatre case: you've added a deploy target and a failure domain and bought nothing.
When it isn't worth it
- One backend, one client. The BFF is a proxy with a deploy pipeline.
- The client is a first-party app on a fast link that can already call the API in parallel over HTTP/2. Parallel calls don't multiply RTT, so aggregation saves little.
- The "aggregation" is just concatenating JSON the client could fetch itself, with no credential or session boundary crossed.
Failure modes and how you'd see them
- Hop tax with no payoff. p75 per click rises and the response body is byte-identical to the direct API. Detect by diffing the BFF response against the backend response and running p75 with and without the hop.
- Bypass. Later consumers skip the BFF because it only serves one screen shape. Detect by counting distinct callers in the API's access logs; if the BFF has one caller and the API has four, the BFF is a detour.
- Cold process. A BFF that scales to zero adds a cold start to the first request after idle. Detect in p99, not p75.
What I'd bring to the radar vote
A table of consumers, the fields each actually uses, and p75 with and without the hop. Put it in Assess/Trial when at least one boundary job is real: credential hiding, cookie-to-token session translation, or aggregation of two or more backends for one screen. Move it to Adopt when two or more consumers share it and the hop is measured, not assumed. Leave it off when it's a 1:1 proxy.
Curated: · Written: · Reviewed:
Q&A-100When should a full-stack team split a monorepo, and when should they keep UI and API together?
The 60-second answer
Assume one product, one shared HTTP/JSON contract between UI and API, and a CI pipeline that builds both. Keep them in one repo until the two halves deploy on different clocks and can roll back independently. The signal is not repo size: it is whether a UI-only change and an API-only change can ship, fail, and revert without touching each other. Until that is true, a split buys you nothing and costs you a version-skew window.
Measure three things over a month: CI wall time per half, deploys per half per week, and co-change rate (fraction of merges touching both trees). Keep one repo while co-change rate is high and contract tests need a single version. Split when each half has its own rollback and you can version the payload independently.
Why the coupling is the cost
In a monorepo, CI is the coupling. Without path filters, every API-only commit rebuilds and retests the UI bundle, and vice versa. Path filters fix the CI tax cheaply: try that before any split. What path filters do not fix is release coupling: one pipeline, one version, one rollback. That is fine while the contract changes atomically, and it is the thing that breaks when you split.
A worked month (hypothetical figures from one team)
| Shape | CI wall time | Deploys desynced in a week | Contract breaks |
|---|---|---|---|
| one repo, no path filters | 40 min | 0 | 0 |
| split too early | 9 min each | 3 | 1 missing field |
| one repo with path filters | 11 min | 0 | 0 |
Read the middle row as the failure mode, not the goal. The split cut CI from 40 to 9 minutes per half, but the UI shipped 9 minutes after the API on 3 of that week's deploys, and in one of those windows a required field was dropped from the handler while the deployed bundle still sent it. The 31 minutes saved per run were paid back as a production incident.
Arithmetic on the CI tax: 40 min × ~30 runs/day ≈ 20 CPU-hours/day spent rebuilding the untouched half. Path filters take that to ~11 min, recovering most of it without touching the contract. That is why "the folder feels large" is the wrong trigger: the fix for size is filters, not a split.
When to actually split
Split when all three hold: each half deploys on its own cadence, each half can roll back alone, and the contract is versioned so a UI at vN can talk to an API at vN-1 for the length of a deploy window. If you cannot state that compatibility window, you are not ready.
Failure modes and detection
- Premature split: contract skew. Detect with contract tests run against the deployed pair, plus a schema check in CI that fails when the payload shape drifts.
- Late split: CI tax and merge contention. Detect with CI minutes per merged PR and co-change rate; when co-change drops below ~10% and CI is dominated by the other half, the split is overdue.
- Split with a shared release train: you get two repos and one deploy. Worst of both. Detect by checking whether a UI revert forces an API redeploy.
Do not decide from a feeling. Track CI minutes, co-change rate, and contract breaks for a month before and after, and let the numbers pick.
Curated: · Written: · Reviewed:
Q&A-101A list screen takes 3 seconds to paint 200 rows. How do you prove whether the API ran one query or 201, and what do you change?
The 60-second answer
Assume GET /orders?limit=200 returns each order with its customer name, p95 is 3.2 s, API CPU is flat and DB CPU is pinned. Before touching a cache, count the queries the request runs. If it is 1 + N, the endpoint is doing an N+1 and the fix belongs in the query, not in front of it.
Confirm it with the ORM's query log or an APM trace, not by guessing. Then replace the per-row load with one of: a JOIN (many-to-one), a batched IN/ANY load (one-to-many), or a denormalized read model if the join is genuinely expensive. Caching the response is the wrong fix: it hides the cost until the first miss after TTL, and the miss still runs 401 queries.
What N+1 looks like from the API side
The trace is the artefact. A 200-row list with a lazy-loaded customer and address:
GET /orders?limit=200 3204 ms
SELECT * FROM orders LIMIT 200 18 ms
SELECT * FROM customers WHERE id = $1 ×200 2810 ms
SELECT * FROM addresses WHERE customer_id = $1 ×200 340 ms
401 round trips. Each is ~1.2 ms of network plus parse, but they run through a 20-connection pool and serialize behind the request, so wall time is the sum, not the max. The API process is idle; the DB is doing 401 plans.
How you confirm it
Three checks, cheapest first:
- ORM query log: Rails
ActiveRecord::Base.logger = Logger.new($stdout), Djangoconnection.queriesor the debug toolbar, Prismalog: ['query'], Hibernateshow_sql. Count lines per request. - APM trace: one span per query. Datadog, Honeycomb, or OTel with the DB instrumentation. This also catches N+1 in the serializer (a computed field calling a service per row), which the ORM log misses.
- DB side:
pg_stat_statementscallsdelta per request, orSELECT count(*) FROM pg_stat_activity WHERE state = 'active'.
Then make it permanent: emit db.queries_per_request as a histogram and alert when p95 crosses ~20. That is the metric that catches the next one before a user does.
The fix, and which one
| Shape | Fix | Queries | Watch out |
|---|---|---|---|
| many-to-one (order → customer) | JOIN | 1 | fine, no row multiplication |
| one-to-many (order → items) | batched IN / ANY | 2 | chunk ids at ~1000; Postgres caps at 65535 params |
| expensive join, read-heavy | denormalized read model | 1 | write path must maintain it |
Batched load in the ORM, Prisma example:
const orders = await prisma.order.findMany({ take: 200 });
const customers = await prisma.customer.findMany({
where: { id: { in: orders.map(o => o.customerId) } },
});
Two queries, no row multiplication, and the FK index does the work. Check EXPLAIN that the FK is indexed: an unindexed IN over 200 ids is 200 sequential scans.
The wrong fix
Cache-Control or Redis in front of the endpoint. It converts a 3.2 s p95 into a 3.2 s p99 plus a cache-hit p50, and the DB still runs 401 queries per miss. If the cache key includes the user, hit rate collapses and you have added a dependency for nothing. Fix the query first; cache after, when the query is already one round trip and you are buying latency, not hiding cost.
Failure modes to name
- JOIN on a one-to-many multiplies rows (200 orders × 5 items = 1000 rows) and the serializer must dedupe.
INwith 10k ids degrades the planner; chunk it.- N+1 in the serializer, not the ORM: only the trace shows it.
- Replica lag: after the fix, the list may read a replica; that is a separate problem, not this one.
Curated: · Written: · Reviewed: —
Q&A-103A marketing page must hit LCP under one second and the logged-in dashboard must feel native. Walk me through which rendering mode you pick per route, what hydration costs you, and where the data fetch lands in each.
Two route classes, two budgets. Marketing pages are identical HTML for every visitor, so they should never touch an origin at request time: build them static, serve from the CDN edge, ship almost no JavaScript. The dashboard is per-user and interactive, so it gets a server-rendered shell that flushes immediately plus client components for the tiles that actually change. Picking one mode for the whole app is the mistake.
| Route | Mode | HTML from | Data fetched | JS shipped |
|---|---|---|---|---|
| /pricing, /about | SSG, rebuilt on deploy | CDN edge | build time | ~15 KB gz |
| /blog/[slug] | ISR, revalidate 300s | CDN edge, stale-while-revalidate | build + CMS webhook | ~15 KB gz |
| /app shell | SSR + streaming RSC | origin, Cache-Control: private | server, per request, parallel | ~80 KB gz |
| /app live tile | CSR inside the shell | client | client fetch after paint | in shell |
| /settings | SSR shell + CSR form | origin | client, after session check | ~120 KB gz |
Arithmetic for the marketing page on Lighthouse Slow 4G (~1.6 Mbps): CDN TTFB 40 ms + 15 KB gzipped HTML (75 ms) + 14 KB inline critical CSS (70 ms) + 60 KB AVIF hero (300 ms) + 100 ms render ≈ 585 ms LCP. Serve it from origin instead: TTFB 250 ms → ~800 ms, and you pay origin CPU per request. Render it client-side: 200 KB gzipped bundle (~1 s download plus parse), then fetch, then render → LCP 1.5–2 s. Three numbers, whole argument.
Hydration is the tax on SSR. The server HTML is inert until the client parses the bundle and re-renders the same tree. On a Moto G4-class device, budget 0.1 ms per DOM node as a planning figure: a 4,000-node dashboard shell is ~400 ms of main-thread block before it answers a tap, and that shows up as INP, not LCP. Three ways down: React Server Components for the chrome so it ships zero JS, islands/partial hydration for the marketing page, and next/dynamic({ ssr: false }) for below-fold widgets so they hydrate after the fold is interactive.
Where the fetch lands:
- SSG/ISR: at build, or on the revalidate tick. No user request waits on your database.
- SSR: on the server before the HTML flushes, unless you stream. Nine tiles at 200 ms sequential is 1.8 s TTFB;
Promise.allor Suspense boundaries flush the shell at ~50 ms and fill tiles as they resolve. - CSR: after paint, in
useEffector React Query. Fast shell, but a waterfall: bundle → auth → fetch → render.
Failure modes worth naming. Hydration mismatch: server rendered new Date().toLocaleString() or Math.random(), client renders a different string, React 18 throws away the server tree and re-renders. You pay hydration twice and the page flashes. Detect it with the "Hydration failed" console error and a mismatch counter in RUM; fix it by rendering time- and locale-sensitive values in an effect or passing a stable ISO string. ISR staleness: a price changed four minutes ago and revalidate: 300 hasn't fired; detect with a cache-age metric and fix with a CMS webhook calling revalidatePath on publish. Caching authenticated HTML at a shared edge: Cache-Control: public on /app serves user A's dashboard to user B. Authenticated HTML is private, no-store, or the page is split so only the anonymous shell is cacheable.
When not to: a page behind auth that nobody shares and no crawler sees doesn't need SSR; CSR is cheaper and simpler. And don't ISR data that changes per second: that's a client subscription, not a cache.
Curated: · Written: · Reviewed: —
Q&A-104Two users open the same invoice and both hit Save. Walk me through how you stop one edit from silently overwriting the other: where the check lives, and what the loser sees.
First I'd pin down what's being edited. If it's a user-authored record (an invoice, a ticket, a profile) a lost update is data loss, and I wouldn't ship last-write-wins. If it's a presence flag or a counter, LWW is fine and I'd say so out loud.
For the record case: optimistic concurrency control with a version the database checks atomically.
The client loads the record and gets version: 7. It keeps that number. On save it sends it back: as If-Match: "7" or as a field in the body. The API does not read-then-compare-then-write; that's a TOCTOU race where two requests both read 7, both pass, both write. The check has to be in the same statement as the write:
UPDATE invoices
SET total_cents = $2,
note = $3,
version = version + 1,
updated_at = now()
WHERE id = $1
AND version = $4;
-- rowcount 0 → someone else won
rowcount = 0 means the version moved. The API returns a conflict with the current row (or at least its version and the fields that changed), and the client shows a merge. rowcount = 1 returns the new version, 8, which the client stores for the next save.
The check lives in the DB because that's the only place it's atomic with the write. The API translates: If-Match header → $4, rowcount 0 → conflict response. The client carries the version and never invents one. Strictly, a failed If-Match is 412 Precondition Failed per RFC 9110; a version field in the body is conventionally 409. Pick one and document it: clients that branch on the status code need consistency.
Trace of two saves:
t0 A GET /invoices/42 → 200, ETag "7"
t1 B GET /invoices/42 → 200, ETag "7"
t2 A PUT If-Match "7" → 200, ETag "8"
t3 B PUT If-Match "7" → 409, {version: 8, changed: ["note"]}
B's write never touched the row. That's the point: the conflict is detected, not resolved by whoever typed last.
What B sees matters as much as the 409. A silent retry with the fresh version is last-write-wins wearing a hat. The UI keeps B's typed text in local state, shows what changed on the server since B loaded, and offers: keep mine (re-apply on top of version 8), take theirs, or merge field by field. On a long form, a field-level diff beats a whole-record one.
| Strategy | Check | Cost | Use when |
|---|---|---|---|
| Last-write-wins | none | silent data loss | presence, last-seen, single-writer rows |
| OCC (version / If-Match) | DB conditional UPDATE | conflict response + merge UI | user-authored records, low contention |
| Pessimistic lock | advisory lock + TTL | blocked writers, lock expiry bugs | high contention, short edits |
| CRDT / field merge | per-field clocks | complexity, storage | offline, collaborative text |
A DB transaction with SELECT ... FOR UPDATE is not the pessimistic option here: you can't hold a transaction open while a human thinks for four minutes. If you need a lock, it's an advisory lock with a TTL and a heartbeat, and you still want OCC underneath for when the TTL expires.
When is LWW acceptable? When the newest value is by definition correct, or the write is commutative. SET view_count = view_count + 1 is not a lost update: it's an atomic increment. A "mark as read" flag is LWW and nobody cares. The test is the cost of the dialog versus the cost of the lost edit.
Contention decides the design. If a record is edited 3 times a day and a session runs 4 minutes, the chance another edit lands inside your window is roughly 3 × (4/1440) ≈ 0.8%: a conflict dialog is cheap. At 200 edits a day it's ~55%, and you want presence indicators or a lock, not a dialog on every other save.
Failure modes I've hit: a fetch wrapper that retries the conflict with the new ETag, reintroducing the lost update. You see it as a 409 immediately followed by a 200 from the same session. A background job that updates the row without bumping version, so clients hold versions that never move and conflicts go undetected: catch it by comparing updated_at against the last version bump in an audit query. And an ETag computed as a hash of the response body, which changes on whitespace and serialization order, so clients conflict against themselves; make the ETag the version number.
Curated: · Written: · Reviewed: —