Skip to content
Tech Interview Prep

Top 100 Technical Product Manager Interview Questions and Answers

The questions most likely to come up in your Technical Product Manager interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 1

101 questions

Q&A-1Sales wrote "as a user I want a Salesforce sync so that I can see accounts". What is wrong with that story, and what do you write instead?

The story names a solution in the want-clause and hides the job. "Salesforce sync" is a mechanism; "see accounts" is the want restated, not a benefit: no user, no measure, no failure condition. You can't test it, and you can't tell whether a cheaper design satisfies it. Three checks decide whether a story is a story or a procurement ticket:

  1. Does the want-clause name a mechanism? If so, the design is frozen before the need is testable.
  2. Is the benefit measurable? "See accounts": how fast, how correct, compared to what?
  3. Does the story survive a vendor swap? If replacing Salesforce with HubSpot changes the story, the vendor was the requirement.

What you write instead

Name the job, the evidence, and the measure; demote the vendor to a design option.

As a seller, I need to open the canonical account record for the company I'm on a call with in under 10 seconds, so that I quote the right legal entity instead of pasting stale fields from email.

Acceptance: 9 of 12 observed sellers open the correct account in under 10s without leaving the call view. Salesforce is one candidate implementation; the story doesn't name it.

Trade-offs: when the vendor does belong in the story

Three forms, and the question that picks between them:

FormWhen it's rightDeciding questionExample
Job story (vendor in design)The need is user-visible and the vendor is one of several optionsCan you name a user-visible outcome and measure it?the rewrite above
Constraint story (vendor named)The vendor is imposed from outside: contract, data-residency, compliance deadlineIs the vendor a constraint you didn't choose?"As sales ops, we need all account records resident in Salesforce by 30 June so the data-residency clause is met"
SpikeFeasibility or cost is unknownIs the unknown technical rather than user-facing?"Two days to determine whether the Salesforce API can reconcile accounts at our volume"

The original picks the first form's shape (a user story) and fills it with the second form's content (a constraint nobody stated). If the vendor really is a constraint, say so and put it in the benefit clause. If it isn't, it belongs in the design doc, not the story.

Failure mode

Worked example, hypothetical figures: a team shipped a two-way sync in 11 weeks. 64% of sellers still copied fields by hand, because the job was "trust the number on the call", not "sync every object". Support took 180 tickets/week on duplicate accounts. The story was satisfied; the job wasn't. You detect this by instrumenting the job's measure, correct account opened in under 10s, before the sync ships, not the sync's uptime.

How I'd settle it

Five observed seller sessions plus the completion metric before any CRM vendor enters the story. If the sessions show sellers already trust the number, the sync is the wrong fix and the story is about the data, not the pipe.

Curated: · Written: · Reviewed:

Q&A-27The epic split is "frontend, backend, data model". Why is that a bad split for a first release, and what is a vertical slice here?

Assumptions: a first release, a small team, and a product whose value is a user completing a task. If the work is a platform migration with no user-visible outcome, the calculus changes: I'll come back to that.

The answer: "frontend, backend, data model" splits by architectural layer, and no layer epic is independently shippable or independently verifiable. Each is only "done" against a spec no real user has exercised, and the riskiest part, the contract where the layers meet, belongs to no epic. So integration risk, the thing most likely to be wrong, gets deferred to the end and discovered all at once.

Mechanism, concretely:

  • No testable increment. A finished frontend epic with no backend is inventory. You can't demo it, can't put it in front of a user, can't get feedback.
  • Feedback latency. With a slice you learn in week 2 that the data model can't answer the query the UI needs. With layers you learn in week 8, when the rework is expensive.
  • Unowned seams. Who owns the API contract, pagination semantics, the error shape? Each layer epic assumes the other side will match.
  • False progress. Three epics at 80% is zero value; the burndown looks healthy while nothing ships.

What a vertical slice is here: the thinnest path through every layer that one named user segment can complete end-to-end and that produces a usable outcome. For onboarding: one segment (self-serve signup), one happy path (create an org, invite one teammate, land in the dashboard) with auth, the data model, and the UI all inside that one slice, plus a manual fallback for anything not built (provisioning done by hand, no billing).

SplitEnd of week 4A user canIntegration risk surfaces
FE / BE / data3 epics ~80%nothingweek 8+
one-segment slice1 path shippedcreate an org, invite a teammateweek 2
all segments at oncestill buildingnothinglate

(Week numbers are illustrative; the ordering is the point.)

When the layer split is right: a cross-cutting migration with no user-visible outcome (moving a datastore, upgrading a framework) genuinely is horizontal, and slicing it by user outcome is artificial. The test: can a user tell the difference when it ships? If no, track it as a platform epic with its own risk plan.

Done means: one named segment can complete the path in staging, end to end, with no engineer in the loop. Not "frontend merged."

Curated: · Written: · Reviewed:

Q&A-2The story is "as a buyer I want two-factor authentication so that the account is secure". Engineering asks whether SMS or a hardware key is in scope. How does the benefit clause decide?

Start from the decision the benefit clause is supposed to drive, not from the feature request that arrived first.

The benefit clause states the outcome the buyer wants. Engineering's question, SMS or hardware key, is a control-selection question, and a benefit clause can only answer it if it names the asset being protected and the threat being defended against. "Secure" names neither, so it cannot choose between two controls that differ precisely in which attacks they stop.

So I would clarify three things before answering. What action is being protected: login, step-up on a payment, or account recovery. Who the realistic attacker is: credential stuffing, SIM-swap, a phishing proxy, or device theft. And whether a named control from a compliance regime is actually in scope. Once those are on the table, the benefit clause becomes a filter rather than a slogan.

Assumptions I would state out loud: the account holds something worth stealing, the buyer has not named a specific compliance regime, and the team can ship either control.

On the controls themselves. SMS one-time codes stop password-only takeover but are exposed to SIM-swap and to real-time phishing proxies that relay the code. TOTP apps remove the SIM-swap path, but a code can still be relayed by a phishing proxy. WebAuthn security keys bind the credential to the origin, so a relayed or replayed credential fails: that is the phishing resistance the other two lack. None of these is "more secure" in the abstract; each closes a different attack path.

The table below is how the benefit clause maps onto a control.

Benefit clauseStored assetControl that fits
"be secure"unspecifiedanything labelled 2FA
stop takeover of stored cardsPAN + walletapp TOTP or hardware key
satisfy a named controlpolicy-defined scopecontrol verified against that policy

A worked example with hypothetical numbers. A clause reading "stop takeover of stored payment cards" names the asset (stored PANs) and the threat (account takeover). That rules SMS out as the sole control, because SIM-swap reaches the account without the cardholder. A clause reading "satisfy a named control" names a policy, not an asset, so the control is whatever that policy specifies and the team verifies against the standard instead of guessing.

The failure mode to name: a benefit clause that says only "secure" passes review, ships SMS, and the gap surfaces in an incident review rather than in refinement. Detection is cheap, if two engineers read the clause and pick different controls, the clause is underspecified.

Before the story is Ready I would want a threat row: stored asset, likely attacker, acceptable residual risk, and a named risk owner. That is the evidence that the benefit clause actually decided something.

Curated: · Written: · Reviewed:

Q&A-3The criterion is "the checkout feels fast". What do you replace it with so QA and the browser can disagree with the PRD in the same language?

Replace the adjective with a criterion that names five fields: population, action, percentile, measurement point, failure behaviour. The measurement point is the load-bearing one: without it, "fast" is measured wherever the measurer happens to be standing, which is why QA on office Wi-Fi and the PRD author on a laptop can both be right and both be useless.

The replacement, as it would go in the ticket:

Criterion: checkout latency
Population: authenticated buyer, non-empty cart, supported browser
Action: click Pay -> committed order-id rendered on confirmation screen
Percentile: p95 and p99
Target: p95 < 800 ms, p99 < 2.5 s
Measurement point: client RUM at confirmation render, segmented by network class
Failure behaviour: at 8 s show retry reusing the same idempotency key;
                  a retry must never produce a second capture
Named cases: empty cart, 3DS challenge, declined card, connection drop mid-redirect

QA's lab run and the browser's field data are now the same quantity measured in the same place. When they disagree, the argument is about the sample, not about what "fast" means.

Which measurement point

Client RUM is the right default here because the criterion is about what the buyer experiences, and only the client sees network, device and third-party redirects. It is not the only style, and the choice is a real trade-off:

StyleMeasuresGood forBlind to
Client RUM p95buyer-perceived latency, includes network + devicethe criterion that decides "feels fast"server regressions masked by fast clients; needs traffic volume to be stable
Server-side SLO (Pay handler p95)your own code, always on, cheapalerting, capacity planning, catching regressions before users donetwork, device, 3DS and PSP latency
Synthetic on throttled 4G (Playwright/Lighthouse)reproducible, runs per PRgating merges in CIreal device and network mix

Use all three, but only one is the acceptance criterion. The others are supporting signals: the server SLO pages you at 3am, the synthetic run blocks the merge, the RUM panel decides whether the ticket is done.

When a looser criterion is acceptable

A p95 is meaningless over a handful of samples a day. For a low-traffic internal admin tool, a B2B back-office, or a pre-PMF product without the volume to make a percentile stable, the honest criterion is a synthetic budget on a named throttled profile: "Pay to confirmation under 1.5 s on a simulated 4G profile in CI", and you say out loud that you are not measuring real users. What you never do is keep the adjective and call the synthetic run a substitute for it.

Failure modes

  • Timer ends at the wrong place. If the measurement stops at the HTTP 200 from the Pay handler, 3DS redirects and PSP latency are invisible and the number is a lie. It must end at the order-id render.
  • Retry without an idempotency key. The 8 s timeout is only safe if the retry reuses the same key. Detect it with a replay test: two concurrent requests with the same intent key produce one capture and the same response; the same key with a different payload is rejected (409).
  • p95 hides the tail. A p95 of 700 ms with a p99 of 9 s still loses buyers. Report both.
  • Unsegmented samples. A Wi-Fi-heavy sample makes 4G look fine. Segment by network class or the panel lies in the same way the adjective did.

Illustrative arithmetic for why the argument is worth having: at 100k checkouts/day, a p95 budget of 800 ms means 5,000 checkouts/day above budget. If those convert at 70% against a 92% baseline, that is roughly 1,100 lost orders a day: the figure that turns "feels fast" into something a PM will trade scope for.

If a stopwatch cannot fail the criterion, it is not acceptance.

Curated: · Written: · Reviewed:

Q&A-28A story sits in Ready with 14 unchecked template boxes but the team cannot explain the empty-state. Is it Ready?

No, but the 14 unchecked boxes are not the reason.

Ready is a shared understanding of behaviour, not a checklist state. The boxes are a proxy for that understanding. When the proxy is empty and the thing it proxies is missing, you have your answer. The deciding question is narrower than the template: can the team say what a user sees when there is nothing to show? If not, the story is not Ready. A fully ticked ticket with the empty state marked "TBD" is not Ready either: same gap, better camouflage.

Assumptions that matter here: the story is user-facing and renders a list, table, or feed; the empty state is reachable in normal use (new account, filter matching nothing, everything deleted); "Ready" is your pre-sprint gate, not a release gate.

The test I would run is a walkthrough, not a box count. Put two engineers and a designer in a room, hand nobody the ticket, and ask each to describe what a brand-new user with zero records sees. Trace it:

  • Engineer A: "empty table, just the column headers."
  • Designer: "illustration, headline, and a 'Create your first project' button."
  • Engineer B: "the onboarding modal fires first, so they never reach the table."

Three answers, one story. That divergence is the finding. It is not Ready. Fifteen minutes of argument now picks one behaviour, names the trigger condition, and writes the copy, and the story is Ready with the boxes still unchecked, because the boxes were tracking the same question.

SignalWhat it tells youReady?
14 unchecked, empty state "TBD"neither behaviour nor templateNo
14 unchecked, empty/error/success agreed aloudbehaviour known, template laggingYes
14 checked, empty state "TBD"template satisfied, behaviour unknownNo
14 checked, empty state agreedbothYes

Two failure modes to name. The first is checklist theatre: the template gets ticked because ticking is cheap, the behaviour stays unagreed, and the gap ships. The second is the opposite: treating the template as the gate and blocking a one-line copy change behind a 14-item ceremony. Both come from reading the boxes as the thing rather than the signal.

Detection is the walkthrough above. If the three descriptions match without anyone opening the ticket, you have shared understanding. If they diverge, you have a conversation to have, not a story to pull.

What it costs, illustratively: say 500 accounts sign up a week and 20% hit the empty state on day one. If half of those bounce when they see a blank page, that is 50 lost activations a week, and a fix two weeks later has already cost you 100. The 15-minute conversation is the cheaper artefact. (Those numbers are a worked example, not a measurement: swap in your own funnel.)

So: not Ready. Not because 14 boxes are unchecked, but because nobody can finish the sentence "when there is nothing here, the user sees ___." Fill that in and the boxes stop mattering.

Curated: · Written: · Reviewed:

Q&A-46Accessibility is a Should for "after GA". The form is the only way to apply. What is wrong, and what is the Must test?

Core answer

The classification is wrong, and it's wrong for a reason that has nothing to do with how much the team cares about accessibility. "Should, after GA" is a statement about priority. Whether accessibility is a Must is a statement about whether the user has another way to get the thing. When the form is the only way to apply, a user who can't operate the form can't apply at all: no degraded path, no workaround, no partial credit. That makes the accessible apply journey an acceptance criterion of the release, not a follow-up ticket. The ticket framing also hides the real cost: it schedules the fix after the users you excluded have already been excluded.

So the Must test is not "run an accessibility scan." It's this: a person using only a keyboard, and a person using a screen reader, each complete the apply journey end to end on the release candidate, unaided, and reach the confirmation state. That's the gate. Everything else supports it.

Assumptions I'd state out loud: the form is genuinely the sole channel. No staffed phone intake, no paper equivalent; the target is WCAG 2.2 AA, which is the level most teams are held to (EN 301 549 points at WCAG 2.1 AA and the Revised Section 508 standards at WCAG 2.0 AA, so 2.2 AA is the stricter internal bar); and "apply" means submit, not just fill in.

The Must test, concretely

CriterionWhat it means on this formHow you detect it
2.1.1 Keyboard (A)every control, including the date picker and file upload, operable without a mousekeyboard-only run, mouse unplugged
2.4.3 Focus Order / 2.4.7 Focus Visible (A/AA)tab order follows visual order; focus ring visible at 3:1 against adjacent colors (1.4.11)tab through, watch the ring
1.3.1 / 3.3.2 Labels or Instructions (A)every input has a programmatically associated label, not a placeholderscreen reader announces the field name
3.3.1 / 3.3.3 Error Identification and Suggestion (A/AA)on submit failure, focus moves to an error summary; each error names the field and says how to fix itsubmit an empty form with a screen reader on
1.4.3 Contrast (AA)body text 4.5:1, large text 3:1contrast checker on the real palette
3.3.4 Error Prevention (AA)for a legal or financial commitment, submission is reversible, checked, or confirmedreview step before final submit
2.5.8 Target Size (AA, 2.2)interactive targets at least 24×24 CSS pxmeasure the real controls

Worked trace: keyboard-only, screen reader on, empty form:

  1. Tab from the page header. First stop should be the form's first field, not a skip link that goes nowhere. Screen reader reads "Full name, edit, required."
  2. Tab to the date-of-birth field. If it's a custom picker, arrow keys must move within it and Escape must close it without trapping focus. A trap here is a hard fail: the user cannot reach submit.
  3. Tab to submit, press Enter with the form empty. Focus should land on an error summary at the top and the screen reader should read the count and each message. If focus stays on the button, the user hears nothing and assumes the form worked.
  4. Fix the fields, submit, reach confirmation. Focus should move to the confirmation heading, not stay at the bottom of a page that no longer exists.

If any step fails, the release doesn't ship. That's what "Must" means.

Trade-offs

Should-after-GA is defensible in exactly one shape: a genuinely equivalent alternative channel that is staffed and reachable. A phone line with the same service level, or a paper form processed the same way. "We'll fix it later" is not that. The cost-versus-delay trade-off also doesn't run the way teams assume: label association, focus order and error messaging are cheap before GA because they're markup decisions, and expensive after because they touch shipped code, need re-testing, and arrive under complaint or legal pressure. The delay you buy by deferring is days; the work you defer is the same work plus the incident.

Failure modes and how you catch them

  • Keyboard trap in a custom widget: caught by the keyboard-only run, invisible to a scan.
  • Errors conveyed by color alone (1.4.1): red border, no text; caught by submitting empty with a screen reader.
  • Focus lost on validation: caught in step 3 above.
  • Session timeout mid-form with no warning (2.2.1): caught by leaving the form idle past the timeout.
  • CAPTCHA with no non-visual alternative (1.1.1, and 3.3.8 in 2.2): caught by attempting it with a screen reader.

Automated tooling is worth running: Deque, which maintains axe-core, puts automated coverage at roughly 30–40% of accessibility defects, but it cannot see any of the five above. The scan is a smoke test; the manual pass is the gate.

Curated: · Written: · Reviewed:

Q&A-47The spec says "authenticated users can GET /documents/{id}". A user guesses another id and sees it. What requirement was missing?

The missing requirement is object-level authorization. The spec says "authenticated users can GET /documents/{id}": that is an authentication rule wearing an authorization costume. It answers who is calling, not whether this caller may read this document. A valid session is a precondition, never a grant.

What a strong candidate says first: the spec needs a per-object predicate, the document's owner or tenant must match the principal derived from the session, plus a defined response when it doesn't. Without that predicate, every id in the system is readable by every logged-in user, and the id space is the only thing standing between a user and someone else's data.

Mechanism. The check runs after the object is resolved and before the body is serialized, against the principal the session produced, not a client-supplied user id, header, or query param. Three outcomes, not two:

# Python 3.12
def get_document(doc_id, principal):
    doc = db.documents.find_one({"_id": doc_id})
    if doc is None:
        return Response(404)
    if doc.tenant_id != principal.tenant_id:
        return Response(404)   # no relationship: don't confirm the id exists
    if not acl.can_read(doc, principal):
        return Response(403)   # caller can see it exists, cannot read it
    return Response(200, body=doc)

Trace it: a guessed id from another tenant fails the tenant comparison and returns 404, so the attacker learns nothing. A colleague in the same tenant without a read grant gets 403. The owner gets 200. The original spec collapses all three into "authenticated → 200".

404 vs 403 is a real trade-off, and the deciding criterion is whether the caller already knows the object exists.

SituationResponseWhy
Object in another tenant, or no relationship404Denies an enumeration oracle: 403 confirms the id is real
Object in caller's tenant, caller lacks the grant403Caller can already list it; 403 is debuggable and honest
Object doesn't exist404Same as the first row, deliberately indistinguishable

Pick 404 for cross-tenant and unknown, 403 for known-but-forbidden. Return 403 everywhere and an attacker walks the id space to map your document count; return 404 everywhere and your own support engineers can't tell "deleted" from "not yours" without a log dive.

Failure modes. The check drifts: it lives in the controller for GET /documents/{id} but not in the export job, the batch endpoint, the search index, or the signed-URL generator. Each of which takes an id and returns bytes. Tenant leakage also arrives through caches keyed on URL alone, so a response cached for tenant A is served to tenant B. Detect it with a CI test that authenticates as tenant A, requests a known tenant-B id, and asserts 404, not 200, not 403. That test is the regression guard; the incident is what happens without it.

Illustrative example, not a measured figure: an IDOR of this shape exposed 18k documents, two of which were KYC images, and ran 31 hours from first report to containment.

What the spec should say. "GET /documents/{id} returns the document when the authenticated principal's tenant matches the document's tenant and the principal holds read on it; otherwise 404 (no relationship) or 403 (known, forbidden)." That sentence is the requirement that was missing.

Curated: · Written: · Reviewed:

Q&A-49You change the unit of weight from lb to kg in the same field. Is that a version or a silent fix?

Short answer: it's a breaking change, and shipping it under the same field name makes it a silent one. A version bump on its own doesn't fix that.

The field name is the contract; the unit is part of the field's meaning. weight: 10 isn't a number, it's a quantity: lb versus kg is the difference between 4.54 kg and 10 kg. Nothing in the wire format changes: same key, same JSON number, same bytes. No schema validator, type checker, or round-trip test will flag it. Only a consumer that knows the unit will.

So the real options:

  1. Add weight_kg, keep weight meaning lb, deprecate weight with a date. Additive, safe, consumers migrate on their own schedule.
  2. Version the resource (/v2/...) and freeze weight as lb in v1 permanently. Safe, but only if the old field's meaning is genuinely frozen. A v2 that reuses the name weight with kg is the same trap one URL segment later.
  3. Change in place. Not a version, not a fix. A silent breaking change.
ChangeField nameWire bytesSafe?
lb→kg in placeweightidenticalno
add weight_kg, keep weightweight, weight_kgadditiveyes
bump version, same name, new unitweightidenticalno: silent for anyone who upgrades
// v1, frozen
{"id": "p-91", "weight": 10, "weight_unit": "lb"}
// v1 after the migration window — additive
{"id": "p-91", "weight": 10, "weight_unit": "lb", "weight_kg": 4.5359}

Arithmetic: 10 lb = 4.5359 kg. A consumer that reads 10 and treats it as kg now bills 10 kg = 22.046 lb, a 2.2046× overstatement. On a 1,100-parcel batch that's 2.2× the intended weight for as long as it takes someone to notice: in the case I've seen, 9 days, because nothing in the pipeline errored.

Failure modes and how they surface:

  • Silent over- or under-billing. No error, no 4xx, no schema violation. Caught only by a reconciliation job comparing declared weight against a re-derived one, or by a canary consumer pinned to the old semantics.
  • Partial migration. Some consumers read kg, some lb, same field. Caught by a per-consumer contract test that asserts the unit, not just the type.
  • Round-trip corruption. A service reads lb, writes kg back into the same field, doubling on every hop. Caught by an idempotency test: write the same record twice, assert the value is stable.

What actually prevents it: put the unit in the name (weight_kg, weight_lb) or in a companion field (weight: {value: 10, unit: "lb"}). JSON Schema and OpenAPI can't express units: a description and an enum on a unit field are documentation, not enforcement. The name is the only thing a consumer sees without reading your docs.

When in-place is acceptable: you own every consumer, you deploy them together, and you treat it as a flag day with a rollback. That's a coordinated migration, not a silent fix: the difference is whether anyone can be surprised.

Curated: · Written: · Reviewed:

Q&A-53The feature has no trace, no business metric, and a log line "handled". How do you know it worked for the users you claimed?

The 90-second answer

From the feature alone, I don't know, and I'd say that plainly rather than dress up the log line. "handled" fires on a code path; it tells me the handler ran, not that the user got what we promised. So I'd split the question into two: what can I reconstruct from evidence that already exists, and what do I change so the next launch isn't blind.

Assumptions I'd state out loud: I have the deploy timestamp, read access to the warehouse, and at least one adjacent system the outcome touches. An orders table, a payments ledger, a downstream queue.

Reconstructing it retroactively

First I pin the claim. "It worked" is only meaningful against the outcome we promised in the PRD: say, "users complete checkout." That becomes a measurable event: a row in the orders table with a paid status.

Then I look for a signal that already exists rather than instrumenting anything new. In priority order: a durable side effect in a system we own (if completing the flow writes an order row, that's ground truth and I can query it directly. No new instrumentation, no sampling error); a client-side event already firing on the page (page view, button click) as a proxy; support tickets, refunds or ad spend as a coarse bound.

The query is boring, which is the point (Postgres; FILTER is standard since 9.4):

SELECT date_trunc('day', created_at) AS day,
       count(*) FILTER (WHERE status = 'paid') AS completions,
       count(*) AS attempts
FROM orders
WHERE created_at BETWEEN '2024-05-01' AND '2024-05-20'
GROUP BY 1 ORDER BY 1;

If completions drop the day after deploy and recover when we roll back, that's the answer. If they don't move, the feature didn't change the outcome, which is also an answer.

Trade-offs: proxy vs sampling vs full instrumentation

A durable side effect exists, so query it: cheapest and exact. No side effect, high volume, low stakes, so sample, but be honest about power. A 1% sample of 10k daily completions is 100 events/day, and the 95% CI on a 50% rate is about ±10 percentage points. A 6% relative drop is 3 points, inside the noise. Sampling is fine for "did it break completely," useless for "did it move 6%." Outcome visible only in the client and the decision is expensive, so instrument fully, and that is a launch requirement, not a follow-up ticket.

Failure modes

A proxy diverges from the outcome: clicks aren't completions. Survivorship bites hardest: downstream systems only contain users who succeeded, so the dropouts leave no row; you need the denominator (attempts), not just the numerator. And "handled" often fires on both the success and the error branch, so read the code before trusting it.

What I'd change

Define the golden event and its dimensions before GA, put it on the launch scorecard, and verify it fires in the warehouse against the PRD's success definition. Redact PII at the source.

Curated: · Written: · Reviewed:

Q&A-30The story's acceptance tests are green in a mock. Production observability and a rollback path are missing. May it be Done?

No, not Done. If your Definition of Done includes observability and a rollback path, and for anything that writes to shared state or serves live traffic it should, then a story with green mock tests and neither of those is code complete, not Done. Those are different states, and treating them as the same is how a sprint's "done" becomes next week's incident.

Assumptions worth stating out loud: the story ships to production traffic rather than a dark launch; the DoD is the team's shared bar for a releasable Increment while acceptance criteria are this story's specific behaviours; and "mock" means the tests stub the network, the datastore, or both.

Mocks verify that your code calls the right things with the right arguments. They cannot verify that the real dependency answers, that a failure is visible when it happens, or that you can undo the change. Observability and rollback are properties of the deployed system, so no amount of stubbing tests them.

When a lighter DoD is defensible. Not every story needs a drilled rollback.

  • Feature-flagged, off by default, writes nothing: the flag is the rollback, and the flag's own exposure metric is the observability. Done is reasonable.
  • Internal tool, no customer data, blast radius of one team: a documented manual revert can be enough.
  • Additive, backward-compatible migration: rollback is "stop writing the new column," not a restore.

What doesn't qualify: anything that mutates shared state, anything a customer can reach, or any migration that isn't reversible.

Failure modes and how you'd catch them.

  • No trace id on the write path: you can't correlate a 5xx to the request that caused it, so the first signal is a support ticket rather than a dashboard.
  • Untested rollback: you don't know your MTTR until you need it. The drill is the detection.
  • Partial write: the row commits and the downstream event never publishes. Detection is a reconciliation job or a consumer-lag alert, not an error rate.
GateMock-green storyReleasable Increment
acceptance testspasspass
integration test against the real dependencynot runrequired
trace id on the write pathabsentpresent
rollback executed in stagingneverdrilled, time recorded

Worked example (hypothetical). An order service does 200 writes/sec at peak. New code writes the order row, then publishes to a topic. Mock tests stub the publisher and pass. In production the publish fails for 3% of writes because the topic's ACL was never granted to the new service account. There's no trace id on the publish path, so the first signal is a consumer-lag alert ten minutes later. 200 × 0.03 × 600 s = 3,600 orders written but never published. Rollback is a redeploy of the previous build, which does not undo those rows, so the actual fix is a backfill. The two DoD gates that would have caught it: an integration test against the real topic in staging, and a trace id asserted on the publish span.

So: not Done. Move it back, add the two gates, re-run. If the team genuinely can't afford observability and rollback for this change, that's a decision to make explicitly, flag it off or scope it down, not a box to tick.

Curated: · Written: · Reviewed:

Q&A-4Two items score the same RICE except confidence: one is 100% because the VP is sure, the other is 50% from a 40-person diary study. Which number is wrong, and what do you put in the confidence cell?

The 100% is wrong. The 50% is defensible. RICE confidence is a statement about the quality of the evidence behind reach, impact and effort, not about how strongly anyone believes the estimate. A VP being sure is an opinion, and an opinion is the weakest evidence tier there is, so the VP-sure item drops to the bottom of your rubric and the diary study keeps its 50%.

Mechanically, RICE = (Reach * Impact * Confidence) / Effort. Confidence multiplies the other three cells, so 100% asserts that the reach, impact and effort numbers are exactly right. Nobody knows a future reach. 100% is only available for something already measured: an instrumented counter, an A/B test run on the exact population. It is not a claim a sponsor can make on a whiteboard.

Here is what that costs. Take two items, both with reach 200,000 eligible users, impact 2, effort 8 person-weeks (hypothetical numbers).

ItemConfidenceScore
VP-sure100%200,000 * 2 * 1.0 / 8 = 50,000
diary study50%200,000 * 2 * 0.5 / 8 = 25,000
VP-sure rescored20%200,000 * 2 * 0.2 / 8 = 10,000

Same reach, same impact, same effort, and the roadmap order flips twice on the strength of one cell. That is the whole decision.

A rubric that produces a usable number:

Evidence behind reach / impact / effortConfidence
Measured on the exact population: instrumented counter, shipped A/B test100%
Direct research on a matched sample: diary study, usability sessions, survey, n >= 30 recruited from the segment80%
Adjacent or historical data: analogous feature, previous launch, small or skewed sample50%
Stakeholder judgment with no data behind it20%

Intercom's published RICE scale (Sean McBride, 2016) has three tiers (100% high, 80% medium, 50% low) and bottoms out at 50%. That floor is exactly the bug in the question: it forces a VP's opinion and a 40-person diary study to the same number, so the VP's item looks like research. Add the fourth tier.

Whether the diary study is 80% or 50% depends on how the 40 people were recruited and what you are extrapolating from them. Forty diarists drawn from the target segment, whose own behaviour produces the reach figure, can carry 80%. Forty self-selecting, articulate users whose frequency you are scaling to 200,000 is 50%, which matches what the team already put in the cell. The number that must change is the 100%.

One caveat before you stop: confidence is rarely the real error. If the VP item's reach also came from the VP ("everyone will want this"), that cell is fiction, and rescoring confidence from 100% to 20% still leaves 10,000 sitting on a made-up number. Fix reach first, then confidence.

Three ways this goes wrong in practice:

  • Confidence inflation. Plot confidence against the sponsor's level across a quarter of backlog items. If they correlate, the score is laundering authority rather than ranking evidence.
  • Re-scoring after ranking. If cells get adjusted until the preferred item wins, the rubric is decoration. Fix the rubric before scoring, write the evidence source and date next to each confidence cell, and require a written reason for any mid-cycle change.
  • Permanent 100%s. Track the share of items scored at the top tier and how many then missed their reach estimate. If top-tier items routinely miss, the top tier means "confident", not "measured".

Curated: · Written: · Reviewed:

Q&A-5Engineering estimated 5 days of coding for a new export. Support already spends 12 hours/week on the CSV the export would replace. What belongs in the effort cell?

The effort cell gets the total cross-functional person-time to ship and support the export over the horizon, expressed in person-weeks or person-months. It does not get 5 days of coding alone, and it does not get 5 days minus the 12 hours/week support savings. Those savings belong in Impact.

Assume one engineer, 5 days coding, support 12h/week, horizon 1 year. If I had to put a number: 5 days engineering = 1 person-week. Add PM, design, QA, docs, rollout, and expected maintenance. A defensible floor is 1 person-week; a realistic estimate is 2-3 person-weeks. The 12h/week is impact: 12 × 52 = 624 hours/year ≈ 15.6 person-weeks ≈ 0.3 FTE.

Why: RICE is impact per unit effort. If you put 5 days, you overstate ROI. If you subtract support savings from effort, you double-count: the savings already raise impact, and you make effort artificially small or negative. Effort is the denominator; it must be positive. If you net savings, you can get zero or negative, which makes the score infinite or undefined. The 12h/week is not a cost of the export; it's the cost of the status quo. The export's impact is the delta.

Alternatives: Some teams use engineering-only effort for speed. That's fine if consistent, but then impact must include support savings. The failure mode is mixing: coding-only effort with support savings in impact, which inflates score.

Failure mode: The 5-day export shipped; format wrong for 3 ERPs; support rose from 12h to 31h/week. That's +19h/week = 988h/year ≈ 24.7 person-weeks ≈ 0.47 FTE. That's negative impact, not effort. Detection: track support hours before/after a 4-week canary; if they rise, the impact estimate was wrong.

ItemCalculationPerson-time
Engineering coding5 days1 person-week
PM/design/QA/docs/rolloutexample: 5 days total1 person-week
Expected maintenanceexample: 2h/week × 52104h ≈ 2.6 person-weeks
Effort cellsum~4.6 person-weeks
Support savings (Impact)12h/week × 52624h ≈ 15.6 person-weeks ≈ 0.3 FTE
If support rises to 31h/week+19h/week × 52988h ≈ 24.7 person-weeks ≈ 0.47 FTE (negative impact)

Read the table as: effort cell is the sum of the first three rows; support savings are a separate impact line.

I would not consider it settled without evidence: a cost table with build days, weekly support hours before/after a 4-week canary, and who eats the hours. Coding days without support and rollout is a fiction that always wins RICE.

Curated: · Written: · Reviewed:

Q&A-7Legal, sales, and the CEO each labelled their item Must Have for the same release. What test do you apply before the word Must is allowed?

The test is a counterfactual with a named horizon: if this item is missing on the release date, does the release fail a law, a safety bar, or the viable outcome we defined for that horizon? If the answer is no, the word Must is not allowed. Whoever said it.

Three things do not mint a Must: preference, revenue, and seniority. Legal, sales and the CEO each brought one of those to the table, so each item goes through the same gate and the sponsor's title is not an input to it.

Applied to the three:

  • Legal's consent receipt. Absent, the release processes personal data with no lawful basis. That is a legal failure on day one, not a risk. Must.
  • Sales' logo on the invoice. Absent, invoices still go out and get paid; the sponsor is annoyed. Could.
  • CEO's colour tweak. Absent, nothing in the release's outcome changes. Won't, for this horizon.
ItemSponsorAbsent meansCategory
consent receiptlegalunlawful processingMust
logo on invoicesalesawkwardCould
colour tweakCEOnothing changesWon't (this horizon)

The horizon is doing real work in that table. "Must eventually" and "Must by Friday" are different claims, and only the second one is allowed to hold a release date hostage. A CEO colour tweak may well be a Must for the next horizon; it is not one for this one.

Then cap it. A Must list that eats all the contingency is a schedule with no slack, and the first slip takes the date with it. If more than about three items are Must, or the Musts fill the critical path with no buffer, the categorisation has failed: you have a list of everything, which is a list of nothing. Recategorise the rest as Should/Could/Won't and name the person who accepts the consequence of each slipping.

The failure mode I have seen: 14 Musts went into a 6-week release and 9 slipped. The one genuine Must, the consent receipt, shipped on day 41 as a hotfix after a regulator query. The other 13 were preferences wearing the label, and each one had displaced work that would have made the date.

How you detect a fake Must: it has no named failure mode. Ask the sponsor to finish "if this is absent, we fail ___" and watch whether the blank is a law, a safety bar, or the viable outcome, or a feeling.

What settles it: a Must list of three or fewer, each with the failure it prevents written next to it, and a Should list that can slip without killing the release. If everything is Must, MoSCoW has made no decision.

Curated: · Written: · Reviewed:

Q&A-9A 2-week checkout bug loses $40k/week. A 10-week personalisation bet might add $15k/week later. Which do you sequence first, and why is that WSJF rather than "the bigger vision"?

Where candidates lose the interview on cost of delay versus job size is usually a solution they named before they could name the need.

WSJF sequences by Cost of Delay ÷ job size so a short, expensive leak outranks a long bet with later value.

Concretely, cost of Delay here is $40k/week versus a speculative $15k/week. Job size is 2 versus 10 weeks. 40/2 = 20; 15/10 = 1.5. Fix the leak first unless a hard date (law, contract) overrides.

The reason for that specificity is a failure I have seen: The team protected the 10-week bet; the bug ran 8 more weeks ($320k) and the bet slipped anyway because checkout was on fire.

WSJF on two jobs.

JobCoD / weekSizeWSJF
checkout leak$40k measured2w20
personalisation$15k hoped10w1.5
sequenceleak first—then bet

I would not consider it settled without evidence: A two-row WSJF table with the leak's measured weekly loss, not a vision slide.

Vision does not earn interest while a leak is compounding.

Curated: · Written: · Reviewed:

Q&A-8A strategic customer asks for on-prem in the same quarter as the cloud GA. You cannot do both. What does a usable Won't look like?

The 90-second version

A usable Won't is a decision record, not a status. It names the need you are declining, the outcome that outranks it, the cost you are accepting, the horizon it holds for, and a falsifiable trigger that reopens it. It lives where the roadmap lives, is copied into the account plan, has a named owner, and can be read back to the customer verbatim without embarrassing you.

The specificity is the point. A Won't is the only artefact that lets a strategic customer plan around you. "Next half" is a date, and dates are commitments you will be held to. "Never" is a verdict on the customer's problem, which they hear as a verdict on them. A Won't with a trigger is a claim about the world (when X is true, we revisit) and the customer can hold you to that instead.

What goes in the record

Five fields, in this order:

  1. Need heard: in the customer's words. "EU data residency for PII," not "compliance."
  2. Governing outcome: the metric or contract this protects, and why it outranks the request.
  3. Opportunity cost: what shipping the alternative this quarter displaces, with a number.
  4. Horizon: the period the exclusion holds.
  5. Revisit trigger: a falsifiable condition with a threshold and a review date.

Plus the workaround, if one exists, and the owner.

Three ways to say no

ReplyCustomer hearsRevisit
"next half"a datenone
"never"an insultnone
"Won't this GA; reopen at 20 EU cloud logos"an exclusionnamed

What the record looks like (illustrative figures)

Won't Have, H1: self-hosted deployment for [Customer]. Need heard: EU data residency; their security team will not approve a US-region control plane. Governing outcome: cloud GA on [date] with three design partners, two of them regulated. Opportunity cost: on-prem packaging is roughly two engineer-quarters; it moves GA by a quarter and pushes the SOC 2 Type II window. Horizon: through GA plus one quarter. Revisit trigger: 20 paying cloud customers in regulated EU, or a signed contract with residency as a condition. Review [date]. Workaround: BYOK plus EU region pinning covers most of the stated requirement today; we will run their security review against it. Owner: [name].

Failure modes

  • The soft no. "Next half" gets staffed against. Illustrative: the AE said next half, the customer stood up a nine-person on-prem project, four months later there was no binary, and the account paused $1.2M ARR. Detect it by asking the AE what the customer is hiring and staffing for, if headcount is moving, the Won't failed.
  • The hard no. The customer stops bringing you problems. You lose roadmap input, not just the deal.
  • The secret yes. Engineering keeps a branch alive. You pay the cost without the commitment. Detect it by checking whether anyone is assigned to it in the next two sprints.
  • The orphan Won't. No owner, so nobody notices when the trigger fires.

Designing the trigger

It has to be falsifiable and cheap to check. "When the market demands it" is not a trigger; "20 paying cloud customers in regulated EU" is. Put the review date in the record so the trigger gets checked even if nobody complains.

A vague later is a commitment you will be quoted on.

Curated: · Written: · Reviewed:

Q&A-6The board wants all three of: new onboarding, EU data residency, and a partner API, each a quarter of capacity. How do you make the yes visible as a no?

What I say in the room: "Three asks at a quarter of capacity each is 75% of the quarter. On the sum, it fits: I'll show you the arithmetic. It doesn't fit on the people: all three draw on the same two engineers who own the entitlements and account model, and their combined draw is about 20% over what those two have. So the real choice is which one gets those two, and the other two become Won't for this horizon. Tell me which one, and I'll write down what we're not doing." A yes that doesn't name a no isn't a priority, it's a wish list.

Assumptions. 6 engineers, 13-week quarter. Overhead (on-call, support, interviews, PTO, holidays) 25%. So 6 × 13 = 78 engineer-weeks gross, 78 × 0.75 ≈ 58 engineer-weeks of build capacity. Each ask at a quarter of capacity = 0.25 × 58 ≈ 14.5 engineer-weeks. Three asks = 43.5 ≈ 44. Remaining: 58 − 44 = 14 engineer-weeks, about 25%.

So the sum fits with a quarter to spare. That is the number the board will quote back at you, and it is correct. The no is not in the sum.

The no is in the intersection. Capacity is not fungible. Two of the six engineers are the only ones who can change the entitlements service and the account model. Their build capacity: 2 × 13 × 0.75 ≈ 20 engineer-weeks. Now allocate the three asks to who can actually do them:

  • Partner API: 10 of its 14.5 engineer-weeks land on those two (entitlements, auth, rate limits); the rest is SDK, docs, frontend.
  • EU data residency: 8 engineer-weeks on the schema owner, one of the same two, for the data model and the residency-aware storage path.
  • Onboarding: 6 engineer-weeks on the same pair for the account-model changes.

Demand on the pair: 10 + 8 + 6 = 24 engineer-weeks against 20 available. Over by 4, about 20%. And that is before estimate error: each "quarter of capacity" was scoped before the work was understood, so treat it as a point estimate with a wide band, not a commitment.

That is the argument. Not "three quarters doesn't fit in one quarter": it does. "Three quarters of work all land on the same two people, and those two are already 20% over."

Show the menu, not the ranking.

MixOnboardingResidencyPartnerShips in 90d
25/25/25startedstartedstarted0
100/0/0doneWon'tWon't1
50/50/0slicesliceWon't2 partial

(Illustrative outcomes, not measured.) The 25/25/25 row is the board's ask, and it is the row that produces the cure letter: residency carries the external date, so it is the item that fails loudly when it is starved by the other two.

Write the no down. The yes is only visible as a no if the no is a line item someone signs. One page: the envelope (people, weeks, dates), the funded item, and the displaced item named explicitly ("Partner API: Won't this horizon, revisit when residency is certified") signed by the decision owner, not by me. A verbal "we'll revisit" is a deferral, and it reappears next quarter as the same three asks.

Failure modes and how I'd detect them.

  • The sum-fits trap. The board quotes 75% < 100% and declares it funded. Detect by asking which named people each ask consumes; if two asks name the same person, the sum is not the constraint.
  • The soft no. "We'll revisit" with no owner and no date. Detect by checking whether the Won't line has a name and a revisit trigger on it. No name, no no.
  • Constraint treated as a choice. Residency has an external date, so it is scheduled first, not ranked third. Detect by asking which of the three has a date attached to a contract or a regulator.

When this doesn't apply. If the board can add headcount or extend the horizon, the answer changes: it becomes a hiring or sequencing question, not a no. Say that out loud too: "two more backend engineers or one more quarter" is a legitimate answer, and often the one they actually want.

Curated: · Written: · Reviewed:

Q&A-20The team is at 95% utilisation on committed delivery. Leadership wants "more discovery". Where does the time come from?

The time comes out of committed scope. At 95% utilisation there is no slack to reallocate: discovery is work, and if it is not in the plan it either does not happen or it happens as unpaid overtime. So the answer to leadership is a scope conversation with three options (cut committed scope, move the date, or add capacity) plus a recommendation about which one to take.

My recommendation, when the date is a commitment and hiring cannot land inside the quarter, is to cut scope explicitly and protect a fixed discovery allocation. If the date is a forecast rather than a commitment, move the date instead; it is cheaper than the rework that skipped discovery causes.

What discovery actually consumes

Discovery is not a meeting. On a typical team it is spikes, prototypes, user interviews, instrumentation and data pulls, and each has a real cost:

  • A technical spike: 2-5 engineer-days, and the output is a decision, not code.
  • A throwaway prototype to test a UX assumption: 1-2 weeks for one engineer.
  • User research: 8-12 sessions plus synthesis, roughly 5-8 engineer-days of researcher and PM time.
  • Instrumentation to answer "do users actually do X": 3-5 engineer-days plus a week of data collection.

So "30% discovery" is a number, not a vibe. On a 10-engineer team with 2-week sprints, gross capacity is 100 engineer-days. At 95% committed utilisation, 95 days are already promised and 5 are free. A 30% discovery allocation is 30 days, so you must remove 25 days of committed scope to fund it. That is the conversation to have with leadership, and it is arithmetic rather than negotiation.

The three options, weighed

OptionCostWhen it is the right callWhat it breaks
Cut scopeFree in cash, costs stakeholder trustDate is a commitment, hiring cannot land in timeNamed features slip; unnamed quality work slips silently
Move the dateCosts schedule, preserves scopeDate is a forecast, not a contractDownstream commitments cascade
Add capacityCosts money and ramp timeDiscovery is a durable need, not a one-quarter fixA new engineer is net-negative for 4-8 weeks

The deciding question is whether the date is a commitment or a forecast. If it is a commitment, cutting scope is the only option that lands this quarter. If it is a forecast, moving the date is cheaper than the rework.

Failure modes and how to detect them

Discovery debt. Skipping discovery does not remove uncertainty; it defers it to build time, where rework costs 5-10x more than a spike. Detect it with a rising rework ratio and stories reopened after "done".

Invisible scope cuts. If you cut scope without naming what is dropped, teams cut tests, docs and refactoring first. Detect it with defect escape rate and coverage trend.

Evening discovery. Unscheduled discovery is unpaid overtime; it is the first thing dropped when a delivery crunch hits, and it drives attrition. Detect it with after-hours commit timestamps and PTO usage.

Discovery without a decision. A spike that produces a document nobody acts on has produced no decision. Detect it by checking whether each spike closed with a decision-log entry.

What I would show leadership

A capacity view where discovery hours are a first-class slice rather than leftover, and a named list of what was cut to fund them. Without both, leadership cannot tell whether the allocation was real.

Curated: · Written: · Reviewed:

Q&A-18Research wants a 6-week study "to learn about onboarding". What do you require before that study occupies a discovery slot?

Before I hand over a discovery slot, I want a one-page experiment brief. Five fields, and if any is blank the study waits.

  1. The decision it changes. Name the decision, the owner, and the date it's due. "Learn about onboarding" isn't a decision; "do we ship the guided checklist or the empty-state redesign in Q3" is.
  2. The hypothesis, falsifiable. "New users who see the checklist reach first value in one session more often than those who don't": direction and metric stated.
  3. The evidence threshold. Sample size and the number that flips the call. Not "we'll talk to users"; "8 sessions, kill if fewer than 3 of 8 complete the flow unaided."
  4. The budget. Weeks, people, and what those people aren't doing instead.
  5. The stop/pivot rule and a calendar kill date. What happens at week 2 if the signal is already clear.

Assumptions I'd say out loud: the slot is scarce. One per cycle, and taking it means something else doesn't run. Research is a stakeholder, not the decision owner; the PM or eng lead owns the call. And permission to learn is not permission to scale: a study can greenlight a prototype, never a rollout.

Why the brief matters. I've watched a 6-week study produce a 42-page deck and zero backlog change. The same question answered by a 5-day prototype: 8 users, none found the CTA. The rule of three puts the 95% upper bound on that failure rate at 3/8 ≈ 37%, which is enough to kill the flow. Cost: 2 person-weeks against 18 for the study (3 people × 6 weeks).

WorkCostDecision it enablesResult
6-week study18 person-weeksnone named42-page deck, no backlog change
5-day prototype2 person-weeksship or kill the CTA0/8 found it → kill
A/B test2 weeksscale?runs only if the prototype survives

Failure modes I'd watch for. The study that grows a second question mid-flight, with no kill date to stop it. The "we learned a lot" readout with no decision attached. The study run before the decision owner is named, so nobody can act on the result. All three are visible at the brief review, if the decision owner can't be named, the slot isn't ready.

What I'd accept as settled: a signed one-page brief with the decision, n, threshold, budget, and a kill date on the calendar. Then the study can have the slot.

Curated: · Written: · Reviewed:

Q&A-19A partner integration is in Now. What kill criterion do you write on day 0 so it cannot become a zombie for three quarters?

A kill criterion is a stop you write before the build starts, not after the first bad review: one metric, one window, one threshold, one named owner with the authority to sunset. For a partner integration sitting in Now, mine reads:

If 8 weeks after GA fewer than 200 accounts have completed a successful sync, or support load exceeds 15 hours/week, the integration owner sunsets it (flag off, contract wind-down, partner comms) and reports the sunset in the same forum that approved the build.

Assumptions I'd state out loud: "successful sync" means a full bidirectional sync with no manual repair, read from the integration's own telemetry rather than partner-reported numbers; support hours come from tickets tagged to the integration; the owner is the engineer who built it, not the PM who wants it to succeed. The figures below are hypothetical.

A hard kill line is not always the right tool, and the alternative you pick changes what you write on day 0.

Hard kill line. Right when the integration is additive, the metric is instrumented and stable, and sunsetting is cheap: a flag and an email. Wrong when any of those fail: if the partner is a contractual dependency, if the integration is a prerequisite for other roadmap work, or if the metric can't be read reliably at week 8. A hard line on a noisy metric kills things that were about to work.

Staged investment gate. Instead of one cliff, gate at week 4 (say 50 accounts, ≤25 support h/w) and again at week 8, each gate releasing the next slice of engineering. Use it when the metric is noisy early or the integration is genuinely strategic: you keep the option to stop without betting on a single date.

Timeboxed spike. When you don't yet know whether the partner's API supports the use case, the kill criterion belongs on the spike, not the integration: two weeks, and if there's no working sync against the sandbox, the integration never enters Now.

Failure mode I've watched: no kill line at all. The integration sat at 37 accounts and 22 support hours/week for 11 months, occupying 1.5 engineers, because every review asked how to grow it and none asked what would make us stop.

Week 8AccountsSupport h/wAction
bar≥200≤15keep
actual3722sunset
no bar3722zombie, 11 months

Detection matters as much as the threshold: if support hours aren't tagged to the integration, you can't see the line approaching, so both numbers go on the dashboard the review already reads. What makes it settled is that the Now card carries the metric, the date and the named owner, and the approving forum sees the sunset path before it sees the build.

Curated: · Written: · Reviewed:

Q&A-22A regional VP wants a custom workflow that would delay the shared GA by 6 weeks. How do you say no without promising a fake quarter?

What I'd say in the room

Assumptions I'd state before answering: the 6-week delay is real (I'd confirm with the platform lead first, if it's actually two weeks, this is a different conversation), the shared GA date is already committed to other regions, and I either own the shared roadmap or have a PM who does.

Then, directly: I can't move the shared GA for one region's workflow. Here's what I can do, and here's when we revisit.

The script:

"I hear the need: your team can't run the approval chain the way it's configured today, and that's real. What I can't do is push the shared GA six weeks; three other regions are sequenced behind it and the date is in their commitments. Two paths. If the workflow can be expressed as configuration on the shared flow, I'll get an engineer to scope that this week and it ships with GA. If it genuinely needs a fork, it goes on the roadmap after GA, and we revisit when three regions are live on the shared flow. I'll write that down today, with the trigger and who owns the next review."

Everything below is why that shape works.

Why not "Q4 maybe"

A maybe is a yes with plausible deniability, and the VP will staff against the yes. I watched this happen: a PM told a regional VP "Q4 maybe." The VP read it as a commitment and stood up a shadow team of four to build the fork themselves. It shipped behind a flag nobody owned, the flag stayed on, and the region's incident count doubled over the next quarter as the fork drifted from the shared flow's auth changes. The four engineers weren't the cost: the permanent second code path was.

ReplyGA delayShadow buildWhat the VP hears
"Q4 maybe"0 now, unknown lateryes: 4 peopleyes
"Not now; revisit after GA + 3 regions live"0nono, with a date
Silence0yesyes

The trigger has to be falsifiable

"After GA" alone is a maybe wearing a date. "After GA and three regions live on the shared flow" is checkable: anyone can count regions. Name the owner of the next review, not just the condition, or the trigger never fires.

If the request is actually a blocker

If the VP's workflow is a compliance requirement (data residency, a regulator's approval chain) the answer changes. That's a constraint, not a feature request, and it should have been in GA scope. I'd ask that question before saying no. If it is compliance, the right move is to re-scope GA, not to promise a quarter.

How I'd know the no landed

A written "Not now" in the decision log, copied to the VP, with the trigger and the named owner of the next review. Then watch two signals over the next sprint: no new headcount appearing against the region, and no new flag in that region's config. If either shows up, the no didn't land and I'd escalate it myself rather than let the fork grow.

Curated: · Written: · Reviewed:

Q&A-21The launch invite has 47 names. How do you rebuild it so a decision can actually be made?

Assumptions. The 47 names are on the invite by accretion: each person added the next. The decision on the table is a scope-and-date call: ship the full feature past the event window, or cut scope and hit it. If the decision is irreversible, regulated, or the 47 are affected customers rather than colleagues, the rebuild changes; I flag where below.

The answer. Rebuild the invite as a decision-rights list with four tiers, one named decider, and an expiry on every veto.

TierWhoCount (illustrative)What they can do
Deciderone named person who owns the date1makes the call, can cut scope without asking
Contributorsmust be heard before the call4input by deadline; silence is assent
Approversbounded veto, named domain2veto only in-domain, expires in 48h
Notifyeveryone else40read the decision, no meeting

The rebuild in one pass. Take the 47 names and ask each one question: if you disagree with the call, what happens?

  • "The launch stops" → approver, and only for a named domain. Two survive: legal (a data-residency claim in the copy) and security (a new third-party SDK).
  • "I hold information the decider doesn't" → contributor. Four survive: the engineer who owns the migration, support with the top three ticket drivers, the PM who owns the event contract, finance on the penalty clause.
  • "I'd want to know" → notify. The remaining 40.
  • One decider: the launch owner, who can cut scope without escalating.

The call then runs on a clock. The decider posts the cut list; contributors have 48 hours to object with evidence; approvers have 48 hours to veto in-domain or the veto lapses. Decision lands in two days, not three weeks.

Alternatives, and when each wins.

  • RACI fits ongoing process ownership, not a single dated call: C is unbounded and A is often a committee.
  • DACI (Intuit) splits the Driver from the Approver, which is better when the person doing the work isn't the person who decides.
  • Single-threaded owner (Amazon) drops the approver tier entirely. Best when the blast radius is internal and reversible; wrong when a regulator or a contract is in the loop.
  • Direct escalation skips the map: take the two contested options to the exec who owns the date. Fastest, but it burns the escalation path and teaches the org that maps don't work.

Pick by reversibility and blast radius. Reversible and internal → single-threaded owner. Irreversible or an external commitment → DACI with bounded approvers. Regulated → RACI with named approvers and documented sign-off.

When a wide consult list is warranted. When the 47 are the affected population, not the deciders: a pricing change touching 47 accounts, a migration touching 47 services. Then consult is the point, and you run it as a structured review with a deadline, not a meeting.

Failure modes.

  • Veto without expiry. One approver's silence reads as objection and the date slips. Fix: the veto lapses at the deadline; silence is assent.
  • Decider who can't cut scope. The "owner" has to escalate to cut, so nothing gets decided. Fix: put the scope lever in writing.
  • C-list that grows. Contributors add their own contributors. Fix: contributors attend once, by name, and don't delegate.
  • Escalation to a committee. The path ends in a forum: the 47-name problem again. Fix: escalation ends at one named exec.

Evidence it's settled. A one-page map (one D, ≤5 C, approvers with named domains and expiry, one escalation name) plus the decision logged with the date it was made.

Curated: · Written: · Reviewed:

Q&A-23Three people left the same meeting with three different "decisions". What do you write before anyone stands up?

Core answer, the first thing out of my mouth: before anyone stands up, I write one decision record and read it back to the room. Six fields: the decision in a single sentence, the options we considered, the one we chose, the owner and date, the dissent, and the revisit trigger. If the room doesn't agree with what I read back, we haven't decided yet: we've had a meeting.

Assumptions I'd state up front: this is a consequential, hard-to-reverse decision, not a status sync, and there's no existing ADR process to slot into. If there is one, use it and skip the rest.

Here's the record I'd actually write:

DR-014: Session store for new checkout
Date: 2026-03-11   Owner: Priya (payments)   Status: accepted
Decision: Redis-backed sessions, not sticky sessions on the ALB.
Options: (A) sticky sessions  (B) Redis session store  (C) JWT in cookie
Chosen: B
Dissent: Marco prefers C — avoids a new stateful dependency.
Revisit if: Redis p99 > 5 ms, or session count > 2M.
Supersedes: none

That's the whole artefact. Fifteen minutes to write, and it's the difference between one decision and three.

Why the disagreement happens. People leave a meeting with different memories because the meeting produced a discussion, not a record. Each person encodes what they argued for, not what was chosen. The dissent field is what fixes this: it forces the losing position into writing, so Marco can't later build C believing the room agreed with him. The revisit trigger fixes the other half: a decision with no expiry quietly becomes permanent, and nobody reopens it until it breaks in production.

The trade-off is format, not whether to record. Three credible options:

FormatCostGood forFails when
ADR in the repo~15 min, versionedarchitecture, cross-teamdaily small choices: too heavy
Append-only decision log~2 minmany small decisionsno context, just a line
Ticket comment~0reversible choicesburied, no revisit trigger

When a full record is not worth it: reversible, single-owner, low-blast-radius decisions. Picking a JSON library doesn't need an ADR: a one-line log entry is enough. The test is blast radius and reversibility, not importance in the moment.

Failure mode I've watched. Engineering built option B, sales sold option A, legal believed it was deferred. Nobody was lying; there was no record. Unwinding it took 19 days and a public status page. Detection is cheap: after reading the decision aloud, ask each person to state their position in one sentence. If two sentences disagree, you're not done.

Version, don't edit. When DR-014 changes, write DR-021 that supersedes it. Editing history destroys the audit trail that made the record worth writing.

If it isn't written, it wasn't decided.

Curated: · Written: · Reviewed:

Q&A-24The CEO pings "can we just add a chatbot on the homepage this sprint". What is the first translation, and what do you refuse to do?

The first translation is from a solution to a job. "Add a chatbot" is a proposed implementation; what I can actually plan against is the job it's meant to do, the evidence it will do it, why it has to be this sprint, and what it displaces. So I go back with one question: what outcome are we buying? Deflecting support contacts, converting homepage visitors, or something else? Those lead to different builds and different success measures, and the CEO usually has an answer in one sentence.

The refusal is the other half. I won't add it to the sprint silently. A sprint is a commitment to a set of outcomes; adding work without removing work doesn't create capacity, it moves the slip somewhere the CEO can't see it. So the request either fits the current Sprint Goal or it forces a trade-off, and the person who asked owns that trade-off, not me.

Concretely, I write a short intake note: the job, the evidence, the urgency, and the item it displaces. Then one of three things happens. It doesn't preempt the goal, so it goes in the backlog with a date. It does preempt, so I name the item coming out and ask the CEO to confirm. Or the CEO says ship both, and then I say the goal slips and get that in writing.

Worked example, hypothetical but the shape is what matters: the sprint goal is a checkout tax fix. A chatbot gets built in four days by pulling two engineers off the tax work. It deflects zero support tickets in the first two weeks, because nobody trained it on the top contact reasons and it launched with no entry point above the fold. The tax fix slips a sprint, the miscalculation reaches production, and refunds cost $91k. Nothing about the chatbot was wrong in principle. The failure was that the trade-off was never named, so nobody weighed $91k against a deflection number that turned out to be zero.

What happensTax fixChatbotCost
Silent addslipsships$91k refunds
Intake, trade-off namedholdsnext sprint$0
CEO confirms the slipslips on purposeships$91k, but chosen

The difference between the last two rows isn't the outcome, it's who decided. That's the whole point of the translation.

Two failure modes to watch. If I refuse flatly, "that's not in the sprint", the CEO routes around me and I lose the ability to name the trade-off at all. If I accept silently, I've taught the org that the Sprint Goal is advisory, and every future request arrives as a ping. The middle path is a two-line note and a yes/no.

I'd also want the measurement defined before the build, because "did the chatbot work" is unanswerable after the fact. Pick the number now: support contacts per 1,000 homepage sessions, or homepage conversion rate, measured two weeks before and two weeks after. If it's deflection, you need the baseline contact rate and the top five contact reasons, or you're building blind.

What I need to consider it settled: an intake note naming the job, the displaced item, and a yes/no from the person who asked, on the record rather than in a thread.

Curated: · Written: · Reviewed:

Q&A-10The CEO wants June 12 on a Later theme. What does Now/Next/Later actually communicate, and what do you put on the slide instead of a date?

Now/Next/Later communicates one thing: how much evidence sits behind each item. Now is funded and in build. Next is ordered and being refined. Later is a problem you intend to solve, with no committed sequence and no date. A CEO asking for June 12 on a Later theme is asking you to convert a confidence gradient into a schedule, which is the one thing the framework exists to prevent.

What goes on the slide instead of a date: an outcome, the blocker that keeps it out of Next, and the trigger that promotes it. Three fields, no calendar.

Worked example (figures hypothetical):

Later card: Guest checkout

  • Outcome: guest checkout completion 61% → 75%
  • Why not Next: blocked on the payments service migration (Next, owner: payments team)
  • Promotion trigger: migration in production 2 weeks with error rate < 0.5%
  • Date: none

If a real external date exists, it does not belong on this slide. It belongs on a separate commitments list with a different shape:

Commitment: June 12

  • Obligation: PCI-DSS 4.0 SAQ-D attestation to acquirer
  • Owner: payments compliance lead
  • Depends on: tokenization work in Now
  • Change control: slip goes to exec review, not to the roadmap

The distinction matters because the two lists fail differently. A Later item that slips costs you a reorder. A commitment that slips costs you a contract. Putting June 12 on the Later theme makes the roadmap carry commitment risk without commitment controls.

Failure mode I've seen: Later said "Q3 marketplace". Sales quoted June 12 to two prospects. Engineering had not started discovery. The date moved three times; both logos churned and cited the slide. Detection is mechanical: count the dates on the roadmap slide. If it isn't zero, or if a date appears there that isn't on the commitments list, the slide is doing commitment work.

LabelEvidence barDate on cardCard contents
Nowin build, fundedsprint range onlyoutcome, owner, sprint
Nextordered, scoped, not startednoneoutcome, owner, promotion order
Laterdirectional, unscopednoneoutcome, blocker, promotion trigger

The test for whether you've got it right: a reader can tell from the slide alone what would have to be true for a Later item to move, and no reader can mistake any of it for a promise.

Curated: · Written: · Reviewed:

Q&A-25Sales has a customer roadmap, product has a Now/Next/Later, and the board has a Gantt. They disagree on whether marketplace is this year. What do you do?

Core answer (first 90 seconds)

I'd treat this as a decision problem, not a documentation problem. Three artifacts disagree because they're answering three different questions, and nobody labeled which one is a commitment.

Assumptions I'd state up front: marketplace is a real candidate for this year, not a fantasy; the disagreement is about timing, not whether to build it; and I don't yet know who owns the commit.

My first move is to separate the decision from the representation. The decision, "is marketplace committed for this year", has one owner: whoever owns the capacity-and-revenue tradeoff, usually the CPO with the CEO. The board's Gantt is a financial plan; sales' PDF is a customer promise. Neither is the source of truth for delivery intent.

Then I'd run one test with that owner: what would have to be true for marketplace to ship this year? Named dependencies, named capacity, named revenue at stake. If those don't hold, the answer is Later, and the job becomes making that legible to both audiences without deleting their artifacts.

The three artifacts

ArtifactOwnerWhat it's actually forStatus it should carry
Board GanttCEO/CFOHiring and revenue timingPlan, not promise
Sales roadmapSales/CSDeal conversationsCustomer-safe; dated only when committed
Now/Next/LaterProductIntent and sequencingDirectional until committed

The board Gantt isn't wrong: a dated bar is legitimate for headcount and revenue planning. What's wrong is that it's unlabeled, so sales reads it as a delivery promise. I'd fix the label, not kill the artifact.

Bringing people along

Sales: give them a customer-safe view and a rule. No dates on uncommitted items, and a named person to check with before a customer meeting. Sales doesn't want to lie to customers; they want to close. Give them something true to say.

Board: show the dependency chain and the cost of committing. "Marketplace in H1 requires the payments migration to land by March; if it slips, we either move marketplace or take the migration risk." That's a decision the board can make, not a roadmap they can overrule.

Trade-offs

Committing early can pull a deal forward, but a public commitment you miss costs more than the deal: the customer re-plans around you and the board discounts your next forecast. Holding Later costs you some deals now and preserves the ability to say yes credibly later. The board Gantt's dates are worth keeping for finance; the fix is scope, not deletion.

Failure mode and detection

The failure is a customer hearing a date product never committed to. Detection: diff the sales deck against the canonical status before any customer meeting; a date on an uncommitted item is the signal. Second failure: the board treats the Gantt as a delivery commitment and plans revenue on it. Detect by checking whether the revenue plan assumes marketplace.

What I'd leave with

One owner for the commit, one status per item, three views that agree on the status and differ only in framing.

Curated: · Written: · Reviewed:

Q&A-11A customer attaches last quarter's roadmap PDF to a contract as Appendix B. What was the roadmap for, and what do you put in the contract instead?

A roadmap is a statement of direction under uncertainty. A contract is a set of obligations. Attaching the first as an appendix to the second converts every "we're exploring" into "we will deliver," and that is the whole problem.

Assume the setup: sales attached last quarter's roadmap PDF as Appendix B to an MSA or SOW, probably because the customer asked for a commitment and the roadmap was the closest thing on hand. That roadmap was for internal prioritisation: sequencing bets, aligning engineering and go-to-market on what we'd work on next, and giving leadership a way to re-plan when evidence changed. None of that is a promise, and none of it survives contact with a contract's remedies clause.

The mechanism is that the two artifacts encode different things. A roadmap item is a probability: "we think this is the highest-value next bet, and we'll drop it if the data says otherwise." A contract line item is a binary: delivered or breached. Once the roadmap is an exhibit, the customer's lawyer reads "Later" as "committed, unscheduled," and your own change-control process, the thing that lets you re-plan, now has to run through the customer's procurement.

What goes in the contract instead:

  • Named outcomes, not features. Three or fewer, each with an acceptance test the customer can run.
  • SLOs for anything already in production, with the measurement window and the remedy.
  • A change-control clause: how scope changes, who signs, what happens to dates.
  • An explicit statement that roadmaps, planning documents, and presentations are not exhibits and create no obligation.

The failure mode is concrete. I've seen Appendix B list 11 items from the "Later" column as deliverables. Legal spent six weeks unwinding it. One of the 11 was already marked Won't, we'd killed it before the contract was signed, and we still paid. Credits totalled $180k.

ArtifactWhat it containsBinding?
Q2 roadmap PDF11 "Later" itemsaccidentally yes
MSA exhibit3 outcomes with acceptance testsyes, by design
Roadmap labelled non-bindingthemesno

Detection is early and cheap: any time a roadmap, deck, or planning doc is named as an appendix, exhibit, or schedule in a contract, that's the trigger. Sales pasting slides into an MSA is the leading indicator; legal review flagging roadmap language is the lagging one.

The trade-off: you give up the ability to show the customer a rich forward-looking picture inside the contract itself. You keep it in the relationship: share the roadmap in business reviews, labelled non-binding, and keep the contract to what you'll be held to. The alternative, signing the roadmap as-is, is only defensible if you're willing to staff every item and accept penalties for the ones you drop; almost no roadmap survives that, which is why the exhibit route is the one that gets litigated.

I wouldn't call this settled without evidence: an exhibit with three or fewer commitments, each with an acceptance test, plus a clause stating roadmaps are not exhibits.

Curated: · Written: · Reviewed:

Q&A-12The board pack says "we will hit 18% activation by 30 September". Engineering's current evidence says 12–15%. Which word is missing, and who is allowed to turn a forecast into a commitment?

The missing word is target. The pack says "we will hit 18% activation by 30 September" with no label on the number, so every reader supplies their own, and a board supplies "commitment". Engineering's 12–15% is a forecast. Nobody turns a forecast into a commitment by editing the sentence; only the accountable owner named against that metric, normally the product owner, can commit, and only with capacity, dependencies and a change-control path attached.

LabelWhat 18% would meanWho owns itWhat it obliges
ForecastWrong number: the evidence-based range is 12–15%Whoever ran the analysisNothing. It is revised when evidence changes.
TargetStretch ambition above the forecastBoard or exec sponsorA plan and a review date, not a promise.
CommitmentA number capacity and dependencies can deliverNamed accountable owner (product owner)Change control: scope, date or number moves, in writing.

Same number, three statuses, three different owners. The word on the page decides which one applies.

Why the label is load-bearing. A forecast is a statement about the world and is falsified by new evidence: that is its job. A target is a statement about intent and is falsified by missing it, which is allowed. A commitment is a statement about delivery and is falsified by not delivering, which is not. "Will" is a commitment verb; "target" is a planning noun. Put "will" on an unlabelled number and you have minted a promise nobody agreed to make.

The arithmetic that decides it. Hypothetical but representative: 40,000 signups a month, activation 13% = 5,200. 18% = 7,200. The gap is 2,000 activations a month. In that funnel the only lever that size is the last step, invite a teammate, moving from 38% to 53%, a 15-point move on a step that has never moved more than 3 points in a quarter. That is a target, not a commitment. A commitment would be "ship the invite flow by 15 August", which engineering can own, because it is a change in the product rather than a change in the market.

Who can commit. The accountable owner named against the metric, usually the product owner, and only with the capacity and dependencies written down. Engineering can commit to shipping a change; it cannot commit to an outcome that also depends on traffic mix, seasonality and a partner's API. The board can set a target; it cannot commit on the team's behalf. If the pack has no status column, the number defaults to commitment in the reader's head, which is exactly how 18% reached the minutes, closed at 13.4%, and got read by two directors as a broken promise rather than a missed target.

How to detect it before it happens. Check the pack for a status column and a named owner next to each number. If either is missing, the number is unlabelled and will be read as a commitment. Then check the verb: "will" with no date-bound plan behind it is the tell.

What to write instead. "Target 18% activation by 30 September. Forecast 12–15% on current evidence. Committed: invite flow shipped by 15 August, re-forecast at the September review." Three numbers, three labels, one owner each. When the board wants stretch and will fund the work, target is the right label. When a contract or a dependency is already closed, commitment is. When the number depends on something you do not control, refuse to commit and say which dependency has to close first.

Curated: · Written: · Reviewed:

Q&A-74Two quarters of roadmap are committed. Platform work (a migration off a deprecated dependency) and a revenue feature both need the same two engineers; the migration has no visible customer impact, and the feature has a named prospect waiting. How do you decide, what do you write down, and what do you tell the prospect?

I would start deprecation migration versus revenue feature from the decision it informs, not from the first feature request that arrived.

A deprecation is a dated obligation with a cost curve, not invisible debt: price it against the deadline and blast radius it retires, date the prospect's revenue the same way, and break one date on purpose instead of both by accident.

Concretely, write the migration as a dated obligation (vendor EOL date, blast radius if it slips, engineer-weeks) and the feature as dated revenue with the commitment date and the cost of moving it. Put both on one timeline, then sequence or split, and record the deferral with an owner and a review date.

The reason for that specificity is a failure I have seen: We shipped a named $240k prospect's feature and pushed the migration off a deprecated auth SDK to the next quarter. The vendor's EOL landed in week 11; the pinned fork needed 3 engineer-weeks of emergency patching and auth p99 went from 180 ms to 900 ms for 6 days. The prospect's security review slipped and it closed two months later anyway: the revenue was never as dated as the EOL was.

Same two engineers, one quarter, both items dated.

ItemDate it must beatEngineer-weeksCost of missing it
SDK migrationvendor EOL, week 912unsupported fork, ~3 ewk emergency patching
Prospect featurecontract date, week 1310$240k deal slips a quarter
Capacity—16 (2 eng × 8 weeks after KTLO/on-call)—

12 + 10 > 16, so one date breaks this quarter. The record says which one, why, and when it is re-examined, and the prospect hears the dated alternative, not a hopeful one.

I would not consider it settled without evidence: Name the one dated constraint that decides, the vendor EOL or support cutoff, and show both items' engineer-weeks and deadlines on a single timeline, with the deferral's owner and review date written down.

Every deferral carries a price and a review date, or it is just a hope.

Curated: · Written: · Reviewed:

Q&A-90Engineers want a quarter to rewrite the billing module. How do you put that on a product roadmap without making it a blank cheque?

What I'd say first

"Rewrite billing" is not a roadmap item: it's a scope statement with no stopping condition. The codebase defines the work, and the codebase is unbounded, so the estimate is unbounded too. That's the blank cheque. My move is to re-express the same engineering work as a bounded user outcome, put that on the roadmap, and let the outcome define when we stop.

Assumptions I'd state out loud: billing is revenue-critical, so a defect has a dollar cost; the team's complaint is structural (the module can't represent something the business now needs), not cosmetic; and there's a defect or incident count we can actually query.

The mechanism

Three things turn it into a bounded item:

  1. A user-visible metric with a current baseline. Not "rewrite billing" but "invoice defect rate from X% to under Y%." The baseline comes from the incident tracker or support tickets, not from memory.
  2. A time box shorter than the full rewrite. The box is the maximum you'll spend before re-deciding, not the estimate.
  3. A kill criterion. If the metric hits the bar, you stop: even if the rewrite is half done. If the box expires without the metric moving, you stop and re-plan.

The reason this works: the rewrite is a means, the defect rate is the end. Roadmaps fund ends. When you fund the means, the means has no natural end.

Trade-offs, when a full rewrite is actually right

Incremental hardening is usually cheaper and lower-risk. A full rewrite wins when the defects share one structural cause that patching can't reach: e.g., the schema can't represent partial refunds, so every fix is a new special case. Test: if the last several fixes each added a branch rather than removing one, the structure is the problem. If fixes are converging, harden instead.

Failure modes

  • The box expires, the metric hasn't moved, and the team asks for another quarter. Detection: the metric is flat at the box boundary. Response: re-plan, don't auto-renew.
  • The metric is met by patching symptoms, so the structural work never happens and defects return in six months. Detection: defect count drops but the branch count in the module keeps rising.
  • The metric is gamed: defects reclassified as "expected behaviour." Detection: audit a sample of closed tickets.

Artefacts

Illustrative figures, not measured:

Roadmap frameTime boxUser metricStop condition
"Rewrite billing"opennonenever
"Cut invoice defects to <0.3%"6 weeks2.1% → target 0.3%metric met or box expires
"Harden billing"3 weekssame metricmetric met

Arithmetic on the box: if the full rewrite is estimated at 26 weeks and the defect bar is reachable in 6, the roadmap carries 6 weeks of committed capacity and a 20-week option you exercise only if the 6-week slice proves the structure is the blocker.

The card itself: "Now: reduce invoice defect rate from 2.1% to <0.3% in 6 weeks. Owner: billing. Kill: metric met → stop; box expires → re-plan."

That's how it goes on the roadmap without a blank cheque: the roadmap funds a bounded outcome, and the rewrite is what you do inside the box if it's the cheapest way to move the metric.

Curated: · Written: · Reviewed:

Q&A-100Keep-the-lights-on is "whatever is left after features". On-call overtime is 18 hours/week. Where does KTLO sit on a TPM roadmap?

What I'd say in the first two minutes

KTLO is a capacity line on the roadmap, not the remainder after features. I'd put it there as an explicit percentage with a trigger to raise it and a kill criterion for work that stops earning its place. The default I'd bring to planning is 20% of engineering capacity, reviewed monthly against on-call hours and error-budget burn. If on-call overtime is already running at 18 hours a week, that 20% is wrong and the slice goes up before the next feature commitment is made.

Assumptions I'd state out loud: the team owns its own services (no separate SRE org absorbing the toil), the roadmap covers one quarter, and capacity means engineer-weeks, not story points.

Why leftover fails

Unplanned work preempts planned work rather than sharing it. A team allocated 100% to features has no slack, so every page, certificate renewal and dependency CVE lands as a schedule slip on something already promised. The problem isn't that KTLO is unglamorous; it's that it's invisible until it's an incident, and by then the roadmap is being rewritten under pressure.

What the roadmap line looks like

  • A named KTLO bucket with a percentage and a single owner.
  • A trigger to raise it: on-call overtime above a threshold, or error-budget burn rate above target for two consecutive weeks.
  • A kill criterion: work that has run two quarters without reducing page volume or toil hours gets cut, and the hours go back to features.

Illustrative numbers from a team I worked with: the shape matters more than the exact figures:

AllocationOn-call overtimeNext-quarter delivery
100% features18 h/week−40% vs prior baseline
80/20 KTLO4 h/weekheld
no trackingunknownattrition surfaced at exit interviews

Failure mode and how I'd detect it

I've watched a quarter where features took the whole envelope. Overtime reached 18 hours a week, two engineers left within a month of each other, and the following quarter's delivery came in roughly 40% below the prior baseline. None of that was sudden. Overtime hours per engineer per week, error-budget burn rate, and the ratio of unplanned to planned work all move first. If the only dashboard in the room is feature burndown, none of those signals are visible.

Alternatives, and when not to use the slice

A dedicated reliability rotation, one or two engineers per sprint on KTLO only, works when toil is concentrated and interrupt-driven, and fails when the work needs deep feature context to fix. A separate reliability roadmap owned by an SRE team works at org scale, but only if the handoff cost is budgeted; otherwise the product team has just exported its toil and still pays for it. The percentage slice is the cheapest to run and the one I'd default to for a team of five to ten. I'd drop it in favour of a rotation if the same two services generate most of the pages, because then the fix is ownership, not allocation.

What I'd bring as evidence

Last month's timesheet hours split by KTLO and feature work, error-budget burn, and page volume per on-call shift, reviewed with the engineering manager. Without those numbers the percentage is a guess, and a guessed percentage is the first thing cut when a deadline moves.

Curated: · Written: · Reviewed:

Q&A-91A platform team wants all product squads on a new CI by Friday. Product has a launch Friday. Who has the right to say not now?

Direct answer, first 90 seconds. The product owner of the launching journey has the right to say "not now" for their squad. A launch freeze is a risk-control decision on a change that carries a customer commitment; a CI migration is an internal change that can slip a week without a customer noticing. So the platform team proposes the date, the product owner can defer their squad's migration past the launch, and the platform team's Friday deadline is a coordination goal, not a commitment, unless it is backed by a hard constraint like a vendor end-of-life date or an open CVE. In that case the decision escalates to the engineering leader who owns both teams, and the outcome is recorded.

Why the risk concentrates at cutover. CI migration is not a flag flip. At cutover you change the runner image, toolchain versions, cache keys, secret injection, artifact registry path, and the deploy step's credentials. The failure mode is a green build that produces a different artifact than the one the launch rehearsed. Hypothetical trace: same commit, old pipeline builds sha256:abc123, new pipeline builds sha256:def456 because the base image moved from ubuntu-22.04 to ubuntu-24.04. CI is green in both. The launch deploys the wrong digest. Detection is cheap: compare artifact digests between old and new pipelines for the same commit, and dry-run the deploy to a canary environment from the new pipeline, not just the build.

Trade-offs. Delaying platform migration is not free. You keep dual maintenance: two pipeline definitions, two sets of secrets, two cache backends. If the old CI is under a decommission date, delaying past it means no security patches. So the deadline wins when the old system has a hard EOL with no extension, there is an active CVE or compliance finding, or the launching journey is not revenue-critical and can absorb a pipeline change mid-week. Otherwise the freeze wins.

How the right is enforced. Not by a Slack argument. A freeze calendar that the platform team cannot override without the journey owner's written approval, plus a named escalation path.

DecisionOwnerBound
Freeze a squad's migration during launch weekProduct owner of the launching journeyDeferral, not permanent veto; one sprint max
Set the migration deadlinePlatform team, with engineering leadershipMust name the constraint (EOL date, CVE)
Override the freezeEngineering leader owning both teamsRecorded, with reason
Security or CVE fixSecurity ownerOverrides freeze

Failure mode I would name. Migrating during launch week and finding the new pipeline's deploy step uses a different service account with narrower IAM, so the deploy fails at the last step. Detection: run the deploy, not just the build, against a canary before the freeze.

Arithmetic on the window. If the freeze runs T-3 to T+1 business days and a squad's migration takes two days including verification, the last safe start is T-5. A Friday deadline with a Friday launch puts the migration inside the freeze. That is the collision, and it is why the product owner's "not now" is the right call.

Curated: · Written: · Reviewed:

Q&A-88A new CSV importer saves sales demos but creates 40 tickets/week. How does that enter prioritisation?

The 60-second answer

Support load is a recurring cost, so it enters prioritisation twice: as a subtraction from the feature's value, and as a kill criterion with a number and a date. I'd tag the tickets to the importer, measure median handle time, convert to hours per week, and compare that against the hours the importer saves.

Using your 40 tickets/week and an assumed 15 min median handle time, plus an assumed 20 demos/week at 30 min saved each:

  • Value: 20 × 0.5 h = 10 h/week saved.
  • Cost: 40 × 0.25 h = 10 h/week.
  • Net: zero.

The importer is currently a wash. It earns its place only if support drops, so the real decision is about the error states, not about whether the importer was a good idea.

How ticket volume becomes an effort estimate

Three steps, each of which can be wrong:

  1. Tag every ticket to the feature. Untagged tickets make the cost invisible and the feature look free.
  2. Take the median handle time, not the mean. One data-corruption ticket that takes 6 hours drags the mean and makes a healthy feature look expensive; the median tracks the routine case, and you track the tail separately.
  3. Multiply and convert. 40 tickets/week × 15 min = 600 min = 10 h/week. At a loaded $80/h that's $800/week, roughly $41.6k/year: before the escalation tickets, which are the ones that actually hurt.
StateTickets/wkSupport h/wkValue h/wkNet h/wk
Launch4010.010.00.0
After error-state fix51.2510.0+8.75
Ignored for a year4010.010.00.0, 520 h of support

Where it lands in the framework

RICE's Effort term is one-time build cost in person-months, so recurring support doesn't fit cleanly. Two honest options: amortise support over the payback window and add it to Effort, or subtract it from Impact. I prefer a plain payback calculation, because it forces the question the framework hides: how many weeks until the saved hours exceed the build hours plus accumulated support? If the importer took 60 h to build and nets zero per week, it never pays back. If the error-state fix costs 20 h and moves the net to +8.75 h/week, payback is under three weeks.

The kill criterion

Write it as a testable sentence with a threshold, a window, an owner and a date: "If tagged importer tickets stay above 10/week for four consecutive weeks after the error-state fix ships, we remove the importer." Ten per week is 2.5 h/week. The point where support stops being noise. The evidence is a ticket tag plus a weekly hours chart on the Now card, reviewed at the date, not whenever someone remembers.

Failure modes

  • Untagged tickets. The cost is real but unattributable, so it never enters the estimate. Detect it by reconciling total support hours against the sum of tagged features; the gap is your blind spot.
  • Mean instead of median. A single escalation hides a healthy routine case, or a healthy mean hides a brutal tail. Track both.
  • Counting support once. Treating 10 h/week as a launch cost rather than a recurring one understates a year by 52×.
  • A kill criterion with no date or owner. This is the one I've watched happen: the importer stayed, the tickets stayed, and the fix never got scheduled because nothing forced the review. A criterion nobody evaluates is a wish.

A feature that hires a support person is a staffing decision, and staffing decisions get a number, a date and a name.

Curated: · Written: · Reviewed:

Q&A-96A legacy export is used by 0.4% of accounts but 40% of support time. How do you retire it without a surprise?

What I'd say first

Treat this as a deprecation with a measurable exit criterion, not a code deletion. The 0.4% is the wrong number to decide on: I need the revenue and the request volume behind it. So the first move is instrumentation, not a plan: find the authoritative usage source, count distinct accounts per week, and join that to ARR. Then pick one of three paths and commit to a date.

Assumptions I'd state

  • The export is reachable through one endpoint or job I can gate behind a flag.
  • There's request-level telemetry (gateway/API logs) or an export-job table I can query by account.
  • I can reach the account owners: a named contact, not a broadcast email.
  • Any contractual notice period is known before I announce.

The decision

Three credible paths, and what decides between them:

PathWhen it's rightCost
Hard cutoverusage is ~0.4% of ARR too, no contractual noticecheapest, highest surprise risk
Sunset with a shimreplacement exists but callers move slowlyyou run two paths for a quarter
Keep and fixthe 40% support load is a bug, not the featureyou pay support forever

The deciding number is concentration. 0.4% of accounts is safe to cut if it's also ~0.4% of ARR. If those accounts are 12% of ARR, hard cutover is off the table and you fund the shim.

Mechanism: why the sequence prevents surprises

Surprises come from deleting before you can see who's still calling. So the order is instrument → announce → degrade → disable → delete, and each step is gated on a measurement, not a calendar date.

Arithmetic, hypothetical: 50,000 accounts, 0.4% = 200 accounts. If support is 4 FTE and this export is 40% of that, you're spending 1.6 FTE on 200 accounts. That's the number that funds the project.

Staged rollout, with the flag:

  • T-0: counter on the export path, weekly distinct-account table. Reconcile against the support queue, if the table says 200 and support sees 260, your telemetry is missing a caller.
  • T-30: announce to the named list, with the replacement and a date.
  • T-60: flag on. Export still returns 200, adds a Deprecation header (RFC 8594) and a link. Now you see who ignores it.
  • T-90: disable for accounts that migrated; holdouts stay on.
  • T-120: delete the code, keep the data for the retention window.

Failure modes

  • A caller you can't see: a partner's nightly job hitting an internal route that bypasses the counter. Detect by reconciling telemetry against support tickets, and by alerting on 4xx/5xx at the export path after disable.
  • The holdout is the revenue: one account in the 0.4% is large. Detect by joining usage to ARR before you announce, not after.
  • Silent breakage: the job fails at 02:00 and nobody notices for a week. Detect with a synthetic call to the export path in staging and a post-disable alert on the endpoint.

I'd call it done when the named list is empty or explicitly signed off, and the endpoint has had zero calls for two weeks.

Curated: · Written: · Reviewed:

Q&A-92Sales wants to discount 40% instead of removing a feature. What is the difference, and which should you prefer when the cost is COGS?

The 60-second answer. A 40% discount is a price change on the whole contract; removing the feature is a scope change. When the feature carries real variable COGS, prefer removing it, or moving it to a paid add-on, because the discount cuts revenue on every line the customer buys while leaving the cost driver untouched, and it hands the same cut to accounts that never touch the expensive feature. Discount only when the COGS is fixed or negligible, or when the concession is time-boxed and does not renew at the discounted rate.

Why the discount leaks and removal does not. A discount is priced against the contract, not against the feature. If SMS delivery costs you $25 a month for this account, a 40% discount does not reduce that $25 by 40%: it reduces the $100 list price by 40% and leaves the $25 where it was. The customer who never sends a message gets the same 40% off as the customer who sends a million. Removal is the opposite: it targets the cost line directly, and it self-selects, because only the buyers who need SMS pay for it.

Worked example (hypothetical figures, illustrative only). List price $100/mo per account; total COGS $70/mo, of which SMS delivery is $25/mo at this account's usage.

OptionRevenueCOGSContribution
Do nothing$100$70$30 (30%)
40% discount, SMS kept$60$70−$10 (−17%)
SMS removed from base$100$45$55 (55%)
SMS sold back as $30 add-on$130$70$60 (46%)

The discount is the only row that loses money, and it loses more the bigger the account: a customer with 4× the SMS volume gets 4× the discount in absolute dollars and 4× the COGS bill.

When the discount is the right call. When marginal COGS is near zero, most seat-based software, the discount costs you revenue but not cash cost, and it can be the cheapest way to close a deal. It is also defensible as a one-time, time-boxed concession with a written end date and no auto-renewal at the discounted rate, or as the price of a multi-year commitment that lowers your own per-unit cost. The test is whether the discount buys something you can point to in the contract.

Failure modes. The discount becomes the street price: sales templates it, renewals inherit it, and the next negotiation opens at $60. A discount applied to a usage-based line item scales with the very usage that drives COGS, so your heaviest users get your deepest discount. And a discount is reversible in principle only: you can raise list price, but you cannot easily raise a live customer's price, whereas a packaging change you can undo next quarter.

How to detect it. Track contribution margin by account and cohort, not by discount list. Watch discount depth against usage of the costly feature, and check renewal uplift, if renewals come in at the discounted rate, the discount is now list price.

What to say to sales. Offer the customer a choice between "same price, no SMS" and "same price, SMS at $30" rather than "40% off everything". That keeps the anchor, covers the COGS, and leaves the feature available to the buyers who will pay for it.

Curated: · Written: · Reviewed:

Q&A-86A noisy neighbour on shared compute made a customer's p95 8× for 2 hours. They want dedicated "for free because we are large". How do you package it?

Clarify before quoting. "Dedicated" means at least three different things and the price differs by an order of magnitude: a separate node pool with reserved capacity, a separate namespace on shared nodes with hard cgroup limits, or just a scheduler priority class. I'd ask which one they mean, and confirm the 8× was contention. If their own query got 8× slower because they shipped a bad plan, isolation relocates the problem instead of fixing it.

Assumption I'd state out loud: the pool is multi-tenant, the 8× came from another tenant's burst, and the customer's own peak is well below what they're asking to reserve.

The answer: dedicated is a paid SKU priced at the capacity it reserves, not the capacity the customer uses. Shared stays the default and gets real isolation controls, shipped to every tenant. If you want to reward a large customer, discount the SKU: don't zero it.

Why price it at reserved capacity. Isolation means headroom nobody else can touch. If their peak is 40 vCPU and their average is 12, you buy and hold 40. Illustrative at $0.03/vCPU-hour:

  • Dedicated: 40 × 730 × $0.03 = $876/month
  • Their shared allocation at average use: 12 × 730 × $0.03 = $263/month

Dedicated is ~3.3× their shared bill. That gap is the negotiation. Quote the number and the conversation becomes "is the p95 guarantee worth $613/month to you": a business question with an owner. Gift it and the cost lands on every other tenant's margin with nobody accountable.

Why free exceptions rot. Each one is a permanent reservation with no budget line. Four of them and the segment's margin is gone while shared customers still eat the 8×: you paid for isolation and shipped none of it.

How shared actually improves. The 8× tail is a scheduling problem, fixable without dedicated capacity:

  • Per-tenant cgroup v2 cpu.max caps sustained CPU at a hard ceiling; cpu.weight decides who wins when the pool is oversubscribed.
  • Fair-share scheduling (weighted fair queueing, DRF) gives each tenant a floor, so one tenant's queue depth can't starve another's.
  • Per-tenant concurrency quotas plus burst credits: a steady-state limit with short bursts allowed, so a legitimate spike isn't throttled into a p99 cliff.
PathMarginShared p95Customer p95
4 free dedicatedgonestill 8×fixed for 4
Paid dedicatedheldunchangedfixed, paid
Shared + quotasheld~1.2×~1.2×

Failure modes to name. Over-tight cpu.max converts contention into throttling stalls: p99 gets worse, not better, so watch nr_throttled and throttled time in cpu.stat per cgroup. Burst credits without a refill cap become a second free-dedicated program. And a dedicated pool doesn't protect a customer from their own bad plan; if their p95 is self-inflicted, you've sold them a $876/month non-fix.

Before committing I'd want: the contention trace showing the other tenant's burst, the customer's peak-to-average ratio, and an exception log with an expiry date on anything given away.

Curated: · Written: · Reviewed:

Q&A-13The dashboard's headline is total signups, up 44% after a contest. Weekly active collaborators is flat. Which is the North Star candidate, and what test disqualifies signups?

Weekly active collaborators, with a quality bar attached, is the North Star candidate. Signups is disqualified by the counterfactual test: if the metric can jump without any user getting recurring value, it isn't measuring the product.

State the test precisely, because "vanity metric" is a label people apply by vibe. A metric passes if a large, sustained rise in it, holding quality and guardrails constant, would mean the product is working. Signups fail on the second clause. A contest, a giveaway, a paid-install campaign, or a bot can move signups without a single person returning. The 44% headline is the evidence, not the objection: signups rose 44% while weekly collaborators moved 0.5%. Two metrics that are supposed to be causally linked moved independently, so the link is broken and the upstream one is the vanity count.

Why signups behave this way: it sits at the top of a leaky funnel, and the leak is where the value is. Signup → first doc → second editor → return next week. Each step drops most of the cohort, and the drop rate is exactly what a contest changes. A contest buys the first step and nothing downstream. For a docs product, the value moment is the second editor: a doc with one author is a private note, a doc with two is a shared artifact someone will come back to.

Illustrative numbers (hypothetical, for the shape of the failure):

MetricBeforeAfter contestMoves with value?
Signups500k720k (+44%)no
Weekly active collaborators41.0k41.2k (+0.5%)yes
Docs with ≥2 editors18.0k18.1kyes

Same contest, 220k new signups: 91% never created a doc, and the extra storage and compute ran about $28k/month. The board celebrated the 44%.

Detection, not vibes: split the contest cohort from organic and compare 30-day retention, docs created per signup, and cost per activated collaborator. If contest signups retain at a fraction of organic, the headline is buying noise.

The definition that makes collaborators/week usable: one page, five fields:

  • Event: a collaborator edits or comments on a shared doc.
  • Eligible actor: a human account, not a service account or bot.
  • Window: 7 days, rolling.
  • Quality bar: the doc has ≥2 distinct editors, so a solo draft doesn't count.
  • Guardrail: contest and paid-acquisition cohorts reported separately, never blended into the headline.

When collaborators/week is the wrong pick: a single-player product has no collaborators, and a metric that counts any edit can be gamed by one person opening the same doc daily. That's what the quality bar is for. If neither holds, take the last funnel step that still correlates with retention (docs shared, or teams active in a week) and apply the same counterfactual test.

Curated: · Written: · Reviewed:

Q&A-14Activation is 62% or 19% depending on who you ask. What did each person put in the denominator, and which one should the team use?

Both numbers are real. They are the same numerator over two different denominators, so they answer two different questions, and the team should adopt the all-signups rate, 19%, as the activation KPI, keeping 62% as a step diagnostic inside the funnel. The deciding criterion is whether the denominator is the population the team can actually move. If the team owns the signup-to-value journey, the denominator has to be signups; conditioning on clicking Start hides the largest leak, the roughly 69% of signups who never click Start at all.

62% is activated ÷ clicked Start, a conditional rate: P(activate | clicked Start). 19% is activated ÷ all signups, the unconditional rate: P(activate | signup).

DenominatornActivatedRate
clicked Start10,0006,20062%
all signups32,0006,20019%

The two reconcile exactly. 10,000 ÷ 32,000 = 31% of signups click Start, and 0.62 × 0.31 ≈ 19%. The 43-point gap is the signup-to-Start drop-off, not a measurement disagreement, and it is the part of the funnel the onboarding team owns.

62% is the right KPI in one case: when the team owns only the post-Start experience, a setup or checkout flow that begins at Start, so Start is the eligible population and 62% is the number their changes move. The test is not which rate is larger. It is which population the team's work can shift.

I have seen the cost of getting this wrong. Growth reported 62% using clicked-Start; finance used all signups at 19%. The board funded a channel that looked 3× better than it was, because the channel brought signups that never clicked Start, so the 62% never applied to them.

To stop that recurring, write a metric contract and version it: numerator, denominator, exclusions, window, owner, and the date the definition last changed. For this KPI that means activated users ÷ signups, excluding bots, employees and duplicate accounts, measured over a 28-day window from signup, owned by the onboarding team. Freeze the definition before you observe the cohort, and never condition eligibility on the step you are trying to improve: that is the rule that separates the two numbers here. If you later want the post-Start rate, publish it as a separate step metric with its own contract, not as the activation number.

Curated: · Written: · Reviewed:

Q&A-57Week-1 retention is 40% or 22% depending on whether "week 1" is 7 days from signup or calendar week. Which do you pick, and why must it be versioned?

The 60-second answer

Before quoting either number I'd pin down what "week 1" means, because 40% and 22% are not two estimates of the same thing: they're two different metrics sharing a name.

I'd pick tenure: day 7 from signup, or more precisely the day-7-to-day-13 window, counted once per user. Calendar weeks are fine for ops dashboards and capacity planning, but they are not a retention cohort.

The reason is exposure. A tenure-d7 cohort gives every user exactly seven days to come back. A calendar-week cohort doesn't: a user who signs up Monday has seven days of that week left, a user who signs up Sunday has one. And if the calendar-week number is computed the way it usually is, active users in week W who are still active in week W+1, the denominator isn't new users at all, it's everyone active that week, most of them long-tenured and slow to churn. That's why it reads higher.

Why the numbers differ

Illustrative, 100,000 weekly actives:

SegmentUsersWeek-over-week returnRetained
New (tenure < 7d)5,00022% (d7)1,100
Established95,00040%38,000
Blended "calendar week"100,000—39,100 → 39%

Same week, same product. The 40% is the established base carrying the number; the 22% is what actually happens to a new user. Report the 40% to the board and you've described your existing users, not your acquisition.

Why it must be versioned

Because the number is a contract, and contracts change. If you switch definitions, every historical point on the chart is now measured differently from the new ones, and the line jumps for reasons that have nothing to do with the product. That's the failure I'd flag: a team switched to calendar weeks, retention "rose" 18 points, and a channel got funded on a clock change.

Versioning means the metric has a name and a version (retention_d7_v2), the definition is written down (cohort = users with a signup event; retained = any session in days 7–13; timezone = UTC; signup day = day 0), and the backfill policy is frozen: you do not recompute history under the new definition and splice it onto the old line. If you must restate, you restate the whole series and label the break.

Failure modes to name

  • Silent definition switch mid-series: the +18-point fake win.
  • Backfill that rewrites past cohorts, so last quarter's number changes after the fact.
  • Timezone and DST: a "day" that's 23 or 25 hours shifts the day-7 boundary for part of the cohort.
  • Signup-day boundary: users who sign up at 23:50 land in a different cohort than users at 00:10, and cohort size swings with traffic shape.
  • Comparing a tenure cohort to a calendar-week period ratio on the same chart.

What I'd ship

A metric contract: definition, version, owner, timezone, cohort rule, retention rule, backfill policy. Dashboards cite the version. Experiments pin the version for the duration of the test. Then the 22% and the 40% can both live on the page, labelled, and nobody funds a channel on a clock change.

Curated: · Written: · Reviewed:

Q&A-61Average order value is $54 or $41 depending on whether NULL discounts are treated as 0. Which is the product number?

Neither number is the product number on its own. The product number is whichever one matches what the discount column actually means, and that meaning is a product decision, not an analyst's. If the column is optional and a NULL means "no discount was applied", the product number is $54. If a NULL means "we don't know whether a discount applied", the product number is $41, and the chart has to say "excludes orders with unknown discount" next to it. The two numbers differ because AVG skips NULLs and COALESCE(discount, 0) does not: those are two different questions wearing the same label.

Here is the arithmetic on a five-order day. Four orders rang up $100 with a $59 discount, so each nets $41. One order rang up $106 with no discount recorded: the discount column is NULL.

ordergrossdiscountnet = gross - discount
A1005941
B1005941
C1005941
D1005941
E106NULL106

AVG(net) over the rows where discount is not NULL is 164 / 4 = $41. AVG(gross - COALESCE(discount, 0)) over all five rows is (164 + 106) / 5 = $54. Same data, two policies, $13 apart.

Which one is right turns on the field's contract. If NULL means "no discount", the $106 order is a full-price order and belongs in the average, so $54 is the product number. If NULL means "unknown", you are averaging four orders and dropping one, and $41 is the product number only if the dropped order is missing at random. That is the part people skip. If the orders with NULL discounts are systematically different, say only guest checkouts skip the discount step, then excluding them biases the number, and $41 is wrong in a way no footnote repairs.

The failure I would watch for is a silent policy switch. Someone swaps COALESCE(discount, 0) for WHERE discount IS NOT NULL, AOV moves $13, and it gets read as a pricing win. I have seen exactly that $13 jump attributed to a promotion when it was a NULL-handling change in the ETL. The number and the policy have to move together, or the metric is not trustworthy.

To settle it: read the column definition in the schema or the ETL doc, confirm with whoever owns the discount field, and write a test that inserts a row with a NULL discount and asserts the documented behaviour. Until the field's meaning is written down, neither $54 nor $41 is the product number: you have two candidate numbers and an undefined column.

Curated: · Written: · Reviewed:

Q&A-62DAU in UTC drops 8% on Sunday evening in the US. Product thinks usage fell. What clock should DAU use?

Answer

The 8% Sunday drop is a bucketing artifact, not a usage decline. Pick one reporting clock, name it in the metric definition, and never splice clocks into a single series.

For globally additive DAU, the number that goes in a board deck, use one fixed reporting timezone. UTC is the default because it has no DST. If the product and audience are US-centric, America/Los_Angeles or America/New_York is defensible, but then you own the DST artifact below. For behavioral work (next-day return, cohort retention, notification timing) use the user's local date, because that is the day the user actually experienced.

Why the Sunday number moves

Take a US user in America/Los_Angeles, which is PDT (UTC−7) in summer. Their Sunday-evening session at 21:00 PDT is Monday 04:00 UTC. Under UTC bucketing that session counts toward Monday's DAU, not Sunday's. Every US session from 17:00 PDT onward lands on the next UTC date.

Illustrative arithmetic: 10M global DAU, 40% US (4M), and 20% of US actives are evening-only (18:00–24:00 local). That is 800k users. Move them off Sunday's UTC bucket and Sunday's UTC DAU falls ~8%, the reported drop, while Monday's UTC DAU rises by the same 800k. A real usage decline does not produce an equal-and-opposite bump on the adjacent day; a boundary artifact does.

ClockSunday 21:00 PDT session lands onSunday US evening counted?Additive across regions?
UTCMondaynoyes
America/Los_AngelesSundayyesyes, for that one region
user-local dateSundayyesno: sum ≠ global DAU

Failure modes

  • Non-additive local dates. A user active at 23:30 and again at 00:30 local is two local dates but one UTC date. Summing per-user local DAU double-counts; the global total will not reconcile.
  • DST. America/Los_Angeles has 23-hour and 25-hour days twice a year. A fixed-offset series (UTC) does not. If you use a named zone, expect two anomalous days per year and annotate them.
  • Half-hour offsets. Asia/Kolkata is UTC+5:30, so its local midnight is 18:30 UTC. The boundary is not on a UTC hour. Do not assume whole-hour offsets when you build the rollup.
  • Notification timing. A campaign scheduled at UTC midnight hits Europe/Berlin users at 01:00 local (CEST, UTC+2). That is the 01:00 case. US users get the same send at 17:00 PDT. Schedule sends on the user's local clock, not the reporting clock.

How to confirm

Plot the same DAU under UTC and under America/Los_Angeles. If the Sunday dip vanishes and Monday gains the same users, it is bucketing. Ship the metric definition with the clock named, and keep the two series separate.

Curated: · Written: · Reviewed:

Q&A-58Revenue by customer is $4.2M. Revenue by invoice rolled to customer is $5.1M. What grain bug is this, and which number is the KPI?

Core answer. This is a join fan-out bug: a one-to-many grain mismatch between the fact table and the dimension it is joined to. The $4.2M is the KPI; $5.1M is inflated by duplicated fact rows.

Said in the first minute: the fact table is at invoice grain, one row per invoice. The customer dimension carries more than one row per customer: customer × month, customer × segment, customer × ship-to. Joining invoice to that dimension duplicates each invoice once per matching dimension row, and the SUM then counts the same invoice several times. Grouping straight to customer, $4.2M, counts each invoice once and is the number that ties to the ledger.

The mechanism, traced. Take three customers. A has two invoices of $100; B has one of $150; C has one of $50. True revenue is $400. Now the customer dimension has two rows for A (two segments). Join invoice to dimension:

  • A: 2 invoices × 2 dimension rows = 4 rows × $100 = $400
  • B: 1 × 1 = $150
  • C: 1 × 1 = $50
  • Sum = $600, overstated by 50%

A's two invoices were each counted twice. The ratio you see is the average fan-out: $5.1M / $4.2M = 1.21, so customers with multiple dimension rows are inflating the total by about 21% on average.

Which number is the KPI. The one that reconciles to the ledger at the declared grain: here, $4.2M. But the grain itself is a business decision, not a technical default. If the KPI is revenue per customer per month, the grain is customer-month: pre-aggregate invoices to that grain first, then join attributes. If the KPI is recognized revenue, it is a different number again: invoice totals are not recognized revenue under ASC 606, so neither figure is the finance KPI. What decides is the question the KPI answers and what the ledger reconciles to.

GrainSumTies to ledger?
customer (deduplicated)$4.2Myes
invoice joined to customer dim, then summed$5.1Mno

Detection and failure mode. Compare COUNT(*) after the join against COUNT(DISTINCT invoice_id); if they differ, you have fan-out. The safe pattern is to aggregate at the fact grain before joining any attribute table. The failure mode is real: a region leader paid on the inflated $5.1M, finance's $4.2M was the books, and the clawback ran six weeks. Reconcile to the ledger within a stated tolerance before the number leaves the warehouse.

Curated: · Written: · Reviewed:

Q&A-59Last-touch attribution uses MAX(ts) without a tie-breaker. Two events share a timestamp. What happens to 4% of conversions?

What happens: For those 4% of conversions, MAX(ts) returns a timestamp that two touchpoints share, so it no longer identifies a single row. Joining that timestamp back to the touchpoint table matches both rows. The conversion is counted twice, once per channel, and whichever channel your downstream code reads is decided by physical row order, not by the data. Re-run the job after a shuffle or a table rewrite and the channel can flip.

Why it behaves this way: MAX(ts) is a scalar aggregate. It collapses the group to a value, not to a row. WHERE ts = (SELECT MAX(ts) ...) is an equality join on a non-unique key, so every tied row survives. If you then do SELECT channel ... LIMIT 1 with no ORDER BY, the engine may return any of them, and the plan (partition pruning, shuffle order, cluster layout) decides which.

Fix: make the ordering total.

-- standard SQL; verified against PostgreSQL 16 and BigQuery
SELECT conversion_id, channel
FROM (
  SELECT conversion_id, channel,
         ROW_NUMBER() OVER (
           PARTITION BY conversion_id
           ORDER BY ts DESC, event_id DESC   -- event_id breaks the tie
         ) AS rn
  FROM touchpoints
) t
WHERE rn = 1;

ROW_NUMBER emits exactly one row per partition, and the ORDER BY is a total order as long as event_id is unique within a conversion. BigQuery equivalent: ARRAY_AGG(channel ORDER BY ts DESC, event_id DESC LIMIT 1)[OFFSET(0)].

Trace. Conversion C1 has two touchpoints:

event_idtschannel
71000paid_search
91000email

MAX(ts) = 1000. Join back → 2 rows → C1 counted twice. ROW_NUMBER orders by (1000, 9) then (1000, 7); rn = 1 is event 9 → email, on every run.

Choosing the tie-breaker is a policy call, not a technical one. event_id is stable but arbitrary: it picks whichever event was ingested last. A channel-priority list (ORDER BY ts DESC, channel_priority ASC) encodes a business rule you can defend to finance. Pick one and write it down; the failure is not the tie, it is the undocumented tie.

When not to use last-touch at all: if ties are this frequent (4%), the model is already fragile. A position-based or linear credit model removes the single-winner question entirely.

Detection. After the join, SELECT conversion_id, COUNT(*) FROM ... GROUP BY 1 HAVING COUNT(*) > 1: any row is fan-out. For nondeterminism, run the attribution twice and hash the output; differing hashes mean an unstable ORDER BY.

Arithmetic (hypothetical): 100k conversions, 4% tied = 4,000 conversions. At a $22.50 average order value that is roughly $90k of attributed revenue that can land on a different channel between two runs of the same job.

Curated: · Written: · Reviewed:

Q&A-60The KPI SQL is SELECT COUNT(DISTINCT user_id) FROM events WHERE type IN (...12 types). Why is this a product smell?

The smell is that the numerator is undefined. COUNT(DISTINCT user_id) over twelve OR'd event types counts a user if they did any of twelve different things, so the KPI is a union of twelve intents and its value is set by whichever type has the most users. That is almost always the cheapest, highest-volume one: a view, not a completion. The number moves, but it cannot tell you which of the twelve moved, so it cannot drive a decision.

Teams write this query for a reason worth naming in the interview: nobody agreed on the success event. Each stakeholder wanted their event in the list, unioning them makes the number go up, and adding a thirteenth type is cheaper than settling the argument. The query is the artifact of an unresolved product definition.

Worked example (hypothetical week, twelve types collapsed into three stages):

typedistinct users
cta_view500,000
signup_start120,000
activation_complete41,000

500,000 users viewed the CTA; 41,000 of them activated; 20,000 activated through a deep link without ever seeing the CTA. The union is 520,000, so the KPI reports 520,000. The outcome it is supposed to track is 41,000: the KPI is 12.7× the thing it stands for.

The sensitivity is the real problem. CTA viewers are 500,000/520,000 ≈ 96% of the union. A 10% lift in CTA views (a marketing spend change, not a product change) moves the KPI to 570,000, +9.6%, with activation flat. You now have a marketing-traffic proxy wearing a product KPI's name, with a headcount decision attached to it.

What DISTINCT conceals, step by step: it collapses each user's forty events into one row, so "viewed once" and "completed" are indistinguishable; it drops rows where user_id IS NULL, so anonymous traffic silently disappears and the metric drifts with login rate; and it hides double-firing, so an instrumentation bug that doubles events is invisible in this number.

Alternatives, and when each wins:

  • One success event. COUNT(DISTINCT user_id) WHERE type = 'activation_complete'. Use when the KPI must be attributable to one team's action. This is the default fix.
  • Per-stage funnel. Report each stage's distinct users plus the conversion between them. Use when you need to know where the drop is, not just that there is one.
  • Pre-aggregated user-state table. (user_id, first_activation_date), counted with COUNT(*). Use when the query scans raw events daily; it also moves the definition out of a query string and into a table you can version.
  • Approximate distinct (APPROX_COUNT_DISTINCT, HyperLogLog) when the count is large and latency matters. Error is roughly 0.5–2% at typical precision: fine for a dashboard, not for a payout.

Failure modes to name: silent taxonomy drift (someone adds type 13, the KPI jumps, nobody knows why. Detect with an alert on the metric definition's hash); NULL user_id (detect with COUNT(*) - COUNT(user_id) per type); bot or internal traffic inside one of the twelve types inflating the union (detect by segmenting on a known-internal flag).

DISTINCT enforces user grain. It cannot define success.

Curated: · Written: · Reviewed:

Q&A-55Activation is 71% in the warehouse and 44% in product analytics. The SQL INNER JOINs to a properties table that 38% of users lack. What is the product mistake?

I would settle INNER JOIN dropping users from a funnel against a counter-example first, so the roadmap date has to survive it.

INNER JOIN keeps matches only; using it as the funnel denominator silently drops people and invents a better rate.

Concretely, funnel denominators are LEFT JOIN or a base population table. Properties are attributes, not eligibility.

The reason for that specificity is a failure I have seen: The 71% was reported to the board. The missing 38% were mostly mobile. A mobile rewrite was delayed a quarter.

Inner join as a silent filter.

StepUsersRate
signups10k—
INNER properties6.2k71% fake
all signups activated4.4k44% true

I would not consider it settled without evidence: A count of distinct users before and after each join, committed as a test.

A join is a product filter whether you meant it or not.

Curated: · Written: · Reviewed:

Q&A-56The funnel is View → Click → Pay. 12% of Pays have no Click because of a cached button. How do you report conversion?

Answer

Report two funnels side by side and never let the 12% disappear: a strict funnel that requires Click, and a stitched funnel that attributes each Pay to the session's last Click. Label the orphan pays as their own segment, and reconcile both against the payment provider's settled GMV.

Assumptions I'd state up front: Pay is server-authoritative (a Stripe payment_intent.succeeded webhook), Click is client-side and therefore lossy. Sessions carry a session_id that survives the pay redirect. The 12% is measured, not assumed.

Why the cached button breaks attribution: if the pay button's markup is served from a CDN or the browser cache, the inline onclick beacon that fires click may not be attached. The button still posts to the payment endpoint, so Pay lands, but Click never does. You get a Pay with no preceding Click in the event stream even though the user did click. That is an instrumentation gap, not a user behaviour.

Worked example (hypothetical, one day):

StageCountRate
Views (sessions)100,000—
Clicks10,00010% view→click
Pays, total2270.227% view→pay
Pays with a Click2002.0% click→pay (strict)
Pays without a Click2711.9% of pays

Strict click→pay = 200 / 10,000 = 2.0%. Stitched click→pay = 227 / 10,000 = 2.27%. The strict number understates by 13.5% relative, not a 4x error, but enough to flip a marginal experiment. Note the denominators differ: view→pay uses sessions, click→pay uses clicks. Put the denominator in the label so nobody compares 0.227% to 2.0% as if they were the same metric.

How to derive the rule: pick the denominator that matches the decision. Judging checkout UX, use click→pay and stitch. Judging top-of-funnel acquisition, use view→pay. Never mix them in one chart.

Failure mode I've seen: a dashboard showed 2.1% click→pay while finance saw 8.4% of sessions paying. Different denominators, unlabeled, and growth killed a checkout experiment that was working. The fix is not a colour change; it is naming the denominator and the eligibility rule.

Reconciliation: compare funnel-attributed GMV to the payment provider's settled GMV. A gap under 2% means the stitched funnel is trustworthy; a larger gap means you are still losing events and should not ship decisions on it.

Instrumentation fix: attach the click listener via event delegation on document, or send the click beacon server-side from the pay endpoint, so cached markup still tracks. Until that ships, keep the orphan segment visible in every report.

Curated: · Written: · Reviewed:

Q&A-64The experiment randomises page views. The metric is conversion per user. Why is the test lying, and what unit do you randomise?

Core answer (first 90 seconds)

The test is lying because you randomise at a finer grain than you measure. Assignment is per page view; the metric is per user. Two things break.

First, mixed exposure. A user with ten page views gets roughly five in control and five in treatment. Their single outcome, did this user convert, can't be attributed to either arm. When you compare "users assigned to A" against "users assigned to B," both groups contain mostly the same mix of treated and untreated views, so the true effect is diluted toward zero. You under-report the winner.

Second, clustering. A user's views aren't independent draws; they share the same person, intent and session. Treating views as independent units inflates the effective sample size, so variance is understated and the p-value is too small. You get significance you didn't earn.

So: randomise at the user. The unit you randomise should be the unit the metric is defined on and the unit the experience persists across.

Why the variance is wrong: the arithmetic

Design effect = 1 + (m − 1) × ICC, where m is views per user and ICC is the intra-user correlation of the outcome. Say a user averages 10 views and ICC = 0.05 (hypothetical, but typical for engagement metrics): 1 + 9 × 0.05 = 1.45. Variance is 45% higher than the naive view-level calculation, so you need roughly 45% more users for the same power, and your confidence interval is about √1.45 ≈ 1.20: 20% too narrow. That is the overconfidence.

Unit choices

Assignment unitMetric unitWhat happens
useruser conversioncorrect; variance matches
page viewuser conversionmixed exposure dilutes the effect; clustered variance understated
sessionsession metricfine if the metric is genuinely per session
accountuser conversionfine if users can't span accounts; otherwise cluster at account

Failure modes and how to detect them

  • Mixed exposure: log the arm of every exposure per user and check what fraction of users saw both arms. If it isn't ~0, the per-user contrast is diluted.
  • Sample-ratio mismatch: a check on the counts of randomised units. If you assigned 50/50 but observe 48/52, assignment or logging is broken. It is a count check, not a randomisation unit.
  • Understated variance: recompute with a cluster-robust or delta-method variance at the user level. If the interval widens materially, the view-level test was overconfident.

When a non-user unit is right

If the metric is genuinely per session, randomise sessions. If treatment spills across users (marketplace, social) randomise clusters (city, market) or use a switchback in time, and cluster the variance at that level. The rule: assignment unit = metric unit = the level at which interference is contained.

Curated: · Written: · Reviewed:

Q&A-65A nominal 50/50 test shows 53,000/47,000 exposures after 10 days. What must you calculate before reading the +2% lift?

Core answer

Before you read the +2%, calculate the sample ratio mismatch (SRM) test on the exposure split: the expected counts under the nominal 50/50 allocation, the chi-square statistic, its degrees of freedom, and the p-value. If that test fails, the +2% is not a lift: it is a measurement of a broken split.

Here you have 100,000 total exposures, so the expected count is 50,000 per arm:

chi-square = (53,000 − 50,000)² / 50,000 + (47,000 − 50,000)² / 50,000 = 180 + 180 = 360, with df = 1.

That gives p ≈ 3 × 10⁻⁸⁰. The usual SRM alarm threshold is p < 0.001 (some teams use 0.0005); you are roughly 77 orders of magnitude past it. The split is not sampling noise, so the +2% is uninterpretable until you find what moved 3,000 exposures out of control.

The calculation

from scipy.stats import chisquare
obs = [53_000, 47_000]
chi2, p = chisquare(obs, f_exp=[50_000, 50_000])
# chi2 = 360.0, p ≈ 2.8e-80
QuantityValue
Total exposures100,000
Expected per arm50,000
Observed53,000 / 47,000
chi-square, df = 1360
p-value≈ 3 × 10⁻⁸⁰
SRM thresholdp < 0.001
Verdictfail: do not read the lift

Why these numbers

With two arms, df = k − 1 = 1. Chi-square is the right test because under the null each exposure is an independent Bernoulli draw into an arm, and the sum of squared standardized deviations is asymptotically χ². At n = 100,000 the normal approximation is tight enough that the p-value is trustworthy.

Failure modes and how you detect them

  • Assignment or redirect bug: one arm loses users on a platform. Detect by running the SRM test per segment (device, browser, app version, country). SRM that appears in one segment and not others points at the assignment path, not the metric.
  • Eligibility filter applied after assignment: one arm is dropped by a post-assignment gate.
  • Bot or prefetch traffic: non-human exposures land unevenly across arms.
  • Logging loss: exposures dropped downstream of assignment. Detect by reconciling assignment logs against exposure logs.

A concrete shape of the first one: a redirect that drops control on mobile. The split looks like 53/47 overall, and the metric you would have read as +2% is really a mobile-only artifact.

What to do

If assignment was corrupted, restart. If only downstream logging was defective and recoverable, repair it and re-run the SRM test before reading any metric. Only once the split passes do you compute the metric's confidence interval and read the +2%.

Curated: · Written: · Reviewed:

Q&A-66Logged-out mobile and desktop are two users. After login they are one. How do you stop DAU from double-counting the morning commute?

The 90-second answer

DAU should count distinct canonical user ids, not distinct device or session ids. Every logged-out session gets an anonymous id scoped to that device. When the user logs in, you write one edge into an identity graph: anon_id -> user_id. From then on the anonymous id and the user id resolve to the same canonical id, and DAU is count(distinct canonical_id) over the day.

The commute case is the one that trips people up, because the login happens after the anonymous activity. Forward-only merging, bind the anon id at login and leave earlier events alone, does not fix it. The merge has to be retroactive within the day.

Trace

One person, one morning:

timesurfaceeventid
07:40mobilecold start, logged outanon A_m
07:42desktoppage view, logged outanon A_d
08:05mobileloginuser U
08:30desktoploginuser U

Raw distinct ids: {A_m, A_d, U} = 3. Forward-only merge: mobile emits A_m then U, desktop emits A_d then U. Still {A_m, A_d, U} = 3. Same-day retroactive merge: A_m -> U, A_d -> U. Distinct = 1.

So when the login edge is written, re-attribute that day's earlier events from the anon id to the canonical id, then aggregate.

How the identifiers work

  • Mobile: iOS identifierForVendor (per-vendor, resets when the last vendor app is uninstalled). Android has had no stable hardware id since Android 10, so use App Set ID or a UUID you persist in app storage.
  • Web: a first-party cookie or localStorage UUID minted on your own domain. Safari's ITP caps script-writable storage at 7 days, so treat the web anon id as short-lived.
  • Server: the login request carries the anon id and the server writes the edge. This is deterministic: you have the exact anon id, no fingerprinting needed.

Probabilistic matching (IP + user agent + screen size) is for linking before login and is not good enough to drive DAU. It produces false merges that undercount and is unexplainable when someone asks why the number moved.

Policy you must pick and document

  • Retroactive restatement: recompute history under the current graph. Yesterday's DAU changes when someone logs in today. Accurate, but dashboards move.
  • As-of / point-in-time: freeze the graph at end of day. Stable, but the commute double-counts on the day it happens.

Most teams want as-of for the headline number plus a restated "true" number, and a merge-rate chart beside DAU so a login-flow change can't be read as growth.

Failure modes

  • Shared device (family iPad): one anon id, several people. Logout should break the edge, or you undercount.
  • Resold or handed-down device: a new login on an anon id already bound to someone else. Break the old edge on logout or on a conflicting login.
  • Streaming aggregation: you can't retract an already-emitted count. Either recompute the day in batch or emit a correction event the aggregator consumes.
  • Logout then anonymous browsing: the anon id should revert to unbound, not stay glued to the last user.

Detect all of these with a merge-rate metric: edges written per day, and the share of anon ids that map to more than one user id.

Curated: · Written: · Reviewed:

Q&A-6720% of mobile events arrive 36 hours late. Yesterday's activation "fell" 6 points this morning. What do you show the board?

The 90-second answer

"Yesterday's activation isn't down 6 points: it's incomplete. Roughly 20% of our mobile events arrive up to 36 hours late, so a number read at T+6h is missing about a fifth of its events. The frozen number at T+48h is flat. I'd hold the roadmap and roll nothing back."

Why events are late

Mobile SDKs don't stream. They append events to a local disk queue and flush on a timer or on app lifecycle transitions: foreground, background, session end. When the device is online those flushes are quick, seconds to minutes. The tail comes from devices that were offline: airplane mode, a tunnel, a dead zone. The queue holds, and failed uploads retry with backoff. A user who activates at 09:00 on the subway can have that event land at 21:00 or the next morning. "20% arrive 36 hours late" means 36h is the p80 of the delay distribution: 80% land sooner, 20% take longer.

Why that looks like a morning dip

Yesterday's number is read at T+6h. The 20% still in flight are missing from the numerator, so the observed value is roughly 80% of the eventual value. If true activation is 40%, the T+6h read is ~32%: an 8-point gap. A 6-point drop sits inside that band. It's the measurement, not the product.

ReadActivationWhat it means
T+6h (preliminary)34%~15% of events still missing
T+48h (frozen)40%settled, flat vs prior days

The three ways to handle it

  • Restatement: recompute past days as late data lands. Simple, but numbers move under people's feet: every chart needs an as-of label.
  • Event-time attribution: bucket by when the event happened, not when it landed. Recent days are visibly incomplete; older days are stable. This is what most warehouses do.
  • Event-time watermark: a stream processor (Flink, Beam) holds state and emits when the watermark passes the allowed lateness. Correct for streaming, but a 36h watermark means 36h of state and 36h-stale output.
-- event-time attribution: bucket by when it happened, not when it landed
SELECT date(event_time) AS day,
       count(*) FILTER (WHERE name = 'activation') AS activations
FROM events
GROUP BY 1;

When a freeze window is the wrong tool

Operational alerting (fraud, outage detection) can't wait 48h. Those need a separate low-latency path that tolerates noise. And if the late tail is short (server-side events land in seconds), a freeze window just adds staleness for nothing.

Failure modes and how to catch them

  • Comparing a preliminary day to a frozen day → the fake regression above. Detect by tracking the late-arrival rate: the share of events landing after the freeze window. Alert when it exceeds the assumed tail.
  • Backfilling without versioning → "why did last week's number change?" Keep restatements logged with the as-of timestamp.
  • Watermark too short → late events dropped silently. Too long → dashboards stale. Both show up as a restatement rate that doesn't match the tail.

What I'd show the board

One chart, two lines: preliminary and frozen, with the as-of time and the % late printed on it. The headline is "yesterday is incomplete, not down." The decision is no rollback.

Curated: · Written: · Reviewed:

Q&A-71A deck says "we improved conversion 10% to 11%, a 10% win". Another slide says "+1 percentage point". Which phrase do you standardise on?

The answer, up front

Standardise on percentage points as the unit for any change in a rate, with the relative figure in parentheses and the baseline spelled out. The slide reads: "Conversion 10% → 11%, +1 pp (+10% relative)." The phrase "+10% win" gets deleted, not reworded.

Assumption I'd state before answering: the metric is a rate with a meaningful 0–1 range. Conversion, CTR, error rate, churn. If it's a count or a dollar figure, pp doesn't apply and plain percent is correct.

Why pp leads

pp is additive and comparable across segments. Relative change isn't: the same "+10%" is 0.1 pp on a 1% baseline and 5 pp on a 50% baseline. Rank segments by relative lift and you systematically promote the smallest baselines, because a 1-in-1,000 event moving to 2-in-1,000 is a "+100% win" no capacity plan can use.

pp also keeps the interval honest. Relative confidence intervals are asymmetric and blow up as the baseline approaches zero; a pp interval stays bounded by [0, 1].

When relative is the correct framing

Three cases, and only these:

  1. Baseline below the reporting precision. 0.02% → 0.04% is +0.02 pp, which rounds to nothing on a slide; "+100% relative" is the only readable version. Show the absolute pair anyway.
  2. Comparing effect sizes across arms whose baselines sit within roughly 2× of each other: a meta-analysis across markets, say. Outside that band, use pp or an odds ratio.
  3. Metrics where the base is arbitrary, like a ratio of two counts with no natural denominator.

Anything feeding a revenue, headcount or capacity model uses pp.

Failure modes

The one I've watched happen: a deck said "+10%" for a 10% → 11% move, and the plan was built as if conversion had reached 20%. Incremental conversions were overstated 10×, and the GMV forecast inherited it. Detection is cheap: a template rule that rejects a bare % on a rate, plus a dashboard column computing both and flagging any row where relative change exceeds pp change by more than 5×. That flag catches exactly the low-baseline segments where the two diverge hardest.

Second failure: relative change on a rate near 0 or 1. 98% → 99% is +1 pp and +1.02% relative; the relative number makes a large operational win look like noise.

Worked check (hypothetical figures)

20,000 sessions per arm, baseline 10%, observed 11%.

QuantityValue
pp change+1.0 pp
relative change+10%
SE of difference (pooled p = 0.105)≈ 0.31 pp
lift in standard errors≈ 3.2

The pp framing tells you the effect is ~3.2 standard errors and worth shipping. The relative framing tells you nothing about whether 1 pp is real.

The standard, one line

"+X pp (Y% relative, from A to B)": pp first, baseline always, relative only when the baseline is too small to read in pp.

Curated: · Written: · Reviewed:

Q&A-72Week-over-week signups are +20% every early September. A new campaign claims the credit. How do you check?

What I'd say first

The +20% is a seasonal pattern until proven otherwise. Early September signups rise because of back-to-school and the end of summer, and that lift shows up whether or not you run a campaign. So the question isn't "did signups go up", they did, it's "did signups go up more than the seasonal baseline." The campaign only gets credit for the increment above baseline.

Assumptions I'd state: signups is the metric; the campaign ran through early September; I have at least three prior Septembers of data; nothing else shipped in the same window (pricing change, launch, PR).

Step 1: build the seasonal baseline

Pull WoW signup growth for early September in prior years. Illustrative figures:

PeriodWoW signup growthCampaign live?
Early Sep 2022+18%no
Early Sep 2023+21%no
Early Sep 2024+19%no
Early Sep 2025+20%yes: claimed

Baseline is the mean of the three clean years: (18+21+19)/3 ≈ +19.3%. The campaign period is +20%. Incremental lift ≈ +0.7pp: inside normal week-to-week noise. On this evidence the campaign gets no credit.

Step 2: control for the pre-period slope

Seasonality isn't the only thing that moves signups. Fit the WoW growth trend over the 4–6 weeks before September (August). If signups were already running +15% WoW, the September bump is only +5pp of new movement, not +20%. Compare the campaign period's growth to the extrapolated pre-period trend, not to zero.

Step 3: holdout

Baseline comparison is observational; a holdout makes it causal. Two options:

  • Geo holdout: withhold the campaign from a set of comparable markets. Cheap, no contamination, works for broadcast campaigns. Bias risk: geos differ in market maturity and channel mix, so match on pre-period signup level and trend.
  • User holdout: randomly withhold the campaign from a slice of users. Cleaner randomization, tighter CI. Bias risk: spillover if the campaign is brand/TV, and it's politically harder to defend.

Decision rule: incremental lift = treated lift − control lift. If the 95% CI on that difference includes zero, the campaign doesn't get credit.

Failure mode to watch

The tell is replication outside the seasonal window. A campaign that genuinely drove the September lift should show some lift in a non-seasonal month. If you scale it in November at 8× spend and get +2%, the September number was the calendar, not the campaign. That's the check that settles it.

Curated: · Written: · Reviewed:

Q&A-73Overall conversion rose 1 pp but every region fell. What happened, and which number do you act on?

Core answer

This is Simpson's paradox, and it happens because the overall rate is a traffic-weighted average of the segment rates, not a fifth region. If traffic shifts toward the segment with the higher rate, the weighted average can rise while every segment's own rate falls.

Which number you act on turns on one question: is the mix a lever you control?

  • If the mix shift is exogenous (a partner changed their traffic, a market moved, a seasonal effect) you cannot reverse it, so the aggregate is your actual outcome. Act on it, and treat the segment declines as context.
  • If the mix is something you set (ad spend, targeting, a rollout) the aggregate is a blend of your own decisions. Act on the segment rates and the mix separately, because "overall up" can be entirely your own reallocation masking a regression in every segment.

Worked example

Two regions, total sessions held at 100,000 so the only thing moving is mix.

RegionPeriod 1 ratePeriod 1 sessionsPeriod 2 ratePeriod 2 sessions
US6.0%10,0005.0%36,000
EU0.5%90,0000.4%64,000
Overall1.05%100,0002.06%100,000

US conversions: 600 → 1,800. EU: 450 → 256. Overall: 1,050 → 2,056, so 1.05% → 2.06%, about +1 pp. Both regions fell (US −1.0 pp, EU −0.1 pp).

Decompose the +1.0 pp into mix and rate, using period-1 rates as the base:

  • Mix effect: (0.36 − 0.10)×6.0% + (0.64 − 0.90)×0.5% = 1.43 pp
  • Rate effect: 0.36×(5.0 − 6.0)% + 0.64×(0.4 − 0.5)% = −0.42 pp
  • Total: +1.01 pp

The mix shift added 1.43 pp; the rate declines took back 0.42 pp. The headline is entirely mix.

What would change the decision

  • Whether the mix shift is caused by your own action. If you moved spend to US, the aggregate is your decision reflected back at you, not a result.
  • Whether the segment deltas are outside noise. A −0.1 pp move on 64,000 sessions is about 64 conversions; check the interval before calling it a regression.
  • Whether the mix shift is real. Region reclassification, a bot-traffic source, or an attribution change can move "share" without any user behaving differently. Verify the mix moved in the raw session counts, not just the reporting layer.
  • Whether the high-converting segment's rate is itself decaying. If US is falling 1 pp per period, the mix tailwind runs out and the aggregate follows.

Failure mode

The one I've seen: the aggregate "win" hid a product regression in every region, and the mix shift was a one-off enterprise campaign that didn't repeat. The next period, mix reverted and the total dropped below where it started. Acting on the total would have shipped nothing and lost a quarter.

Curated: · Written: · Reviewed:

Q&A-63Paid signups are up 35%. Completed onboarding is flat. 28% of "users" fail a cheap bot heuristic. What do you do to CAC?

What I'd say first

Before touching CAC, I'd pin down three assumptions: the 35% signup lift came from roughly a 35% spend increase on the same channel; "users" on the dashboard means signups, not activated accounts; and onboarding completion is the activation event we actually pay for. If those hold, the headline CAC is wrong and the real problem is worse than a bot-filtering bug.

Two moves. First, recompute CAC on a denominator that excludes the 28% flagged as bots. Second, and this is the one that decides the budget, look at CAC per completed onboarding, not per signup. Paid signups up 35% with onboarding flat means the incremental spend produced zero incremental activated users. The marginal CAC on that spend is effectively infinite, so the answer to "what do you do to CAC" is: stop scaling this channel now, then fix the measurement.

The arithmetic

Baseline: $18,000 spend, 1,000 signups, 300 onboarded.

MetricBaselineAfter +35% spend
Spend$18,000$24,300
Signups1,0001,350
Onboarded300300
CAC / signup (raw)$18$18
CAC / signup (human, 72%)$25$25
CAC / onboarded$60$81

Raw CAC per signup is flat at $18: that's the number that looks fine. Human CAC per signup is $18 / 0.72 = $25. CAC per onboarded user went from $60 to $81, up 35%, because the denominator didn't move. The incremental $6,300 bought 350 signups and 0 onboardings.

Why the heuristic isn't the whole answer

A cheap heuristic that flags 28% of traffic is a triage signal, not a verdict. If its false-positive rate on real humans is even 5%, you're wrongly excluding ~49 of 972 humans, and if those skew toward the users who would have onboarded, you've hidden the problem instead of fixing it. So I'd route flagged signups to a verification step (email or SMS OTP, device attestation) rather than deleting them, and keep a raw count for fraud. When the heuristic is the wrong tool: bot share under ~5%, or an unmeasured precision, then a vendor fraud score or a challenge at signup is the better spend.

Failure modes to name

  • Scaling on raw CAC: you double spend, human CAC per onboarded goes to $162, and nobody notices for a quarter.
  • Hard-filtering on the heuristic: you understate signups, kill a channel that was actually working, and lose the real users it misclassified.
  • Trusting flat onboarding as "bots": if the new traffic is human but low-intent (wrong geo, incentivized offer), filtering bots won't move onboarding at all.

What I'd actually do

Pause the incremental spend. Recompute CAC per onboarded human and put it on the dashboard next to raw CAC. Add verification at signup and measure its false-positive rate against a labeled sample. Then decide the channel on payback: at $81 CAC per onboarded user, if gross margin is $9/month, payback is 9 months. Renegotiate the channel or shift budget to one with a better onboarded rate.

Curated: · Written: · Reviewed:

Q&A-15Time-on-site is the goal and it is up 22%. Refunds and accessibility complaints are also up. What did the goal miss, and how do you instrument the next quarter?

Time-on-site is a proxy, and a proxy can be won by making the product worse. The goal moved 22% because the bet kept people on the page; refunds and accessibility complaints moved because it also made the page harder to leave and harder to use. Instrument the next quarter with the goal plus guardrails that can veto it, and a decision rule written down before launch.

What the goal missed. A single outcome metric has no way to say no. Once time-on-site is the target, infinite scroll, autoplay and hard-to-dismiss interstitials all raise it. What's missing is the guardrail set: refund rate, WCAG defects, p95 latency, support contacts per 1k sessions, privacy incidents, and the authority to stop on one.

Worked example, figures hypothetical:

SignalWeek 0Week 2ThresholdVeto?
time-on-site4.1 min5.0 min—goal (+22%)
refund rate1.1%2.0%>1.5%yes (1.8×)
a11y tickets/wk12140>40yes
p95 latency380 ms410 ms>500 msno

The day-10 read called this a win because refunds hadn't settled. Refund rate is lagging: in this example the refund window closes 30 days after purchase, so a 10-day read sees only the fastest cohort and understates the final number. Accessibility tickets are leading but noisy: 12 to 140 is a real signal, yet ticket counts also move with how easy the reporting path is, so a redesign of the feedback form can look like a regression.

Hard veto vs weighted scorecard. A hard veto, any breach reverts the launch, is simple, fast, and defensible in a postmortem; nobody argues about weights at 2am. Its cost is that it treats a 0.1pp refund blip like a 2× jump and can kill a bet that is net-positive. Use it when the harm is irreversible or regulatory: a privacy incident, a WCAG failure on a core flow, a payment error.

A weighted scorecard folds goal and guardrails into one number. It lets you trade a small guardrail cost for a large goal gain, and it is the right tool when guardrails are noisy and the harm is recoverable. Its cost is that weights are arbitrary, get retuned after the fact, and "we lost 0.3 points" is not a decision anyone can defend.

The middle ground usually wins: hard vetoes on 2–4 non-negotiables, weighted review for the rest. Guardrail count is itself a trade-off: every guardrail is another alert, and past five or six the on-call starts rubber-stamping them, which converts a veto into a dashboard. Pick guardrails that are tied to a user harm, measurable inside the experiment window, and owned by a named person.

Hard vetoes are the wrong tool when traffic is low enough that a breach is one user, when the guardrail's noise exceeds its signal (1.1% to 1.3% refunds on 400 orders is inside the confidence interval), and when you cannot measure the guardrail until after the decision point: a veto you can't enforce is not a veto.

Instrumenting the quarter.

  • Pre-register goal, guardrails, thresholds, minimum detectable effect, sample size, abort rule and owner before launch.
  • Abort rule: any non-negotiable breached on two consecutive daily reads, or one breach above 2× threshold, reverts within 24 hours.
  • Hold back 5% of traffic on the old experience for 30 days so lagging guardrails (refunds, retention) can be compared after the window closes.
  • Segment guardrails: a11y complaints by assistive tech and flow, refunds by reason code. Aggregate rates hide harm concentrated in one segment.
  • Weekly read with the guardrail owner present; a breach stops the line rather than opening a discussion.

If harm can't veto the metric, you aren't measuring success: you're measuring what you're willing to trade it for.

Curated: · Written: · Reviewed:

Q&A-17A tax-filing product team is told to maximise Engagement from HEART. Why might that be the wrong category, and what do you put in its place?

HEART is a menu, not a mandate. Engagement counts how much of the product a user consumes: sessions per user, minutes per session, actions per session. That is a decent proxy for value where consumption is the value: a feed, a game, a marketplace. A tax return is the inverse. The user's goal is to be finished. Extra minutes are friction, confusion, or a form that failed to save, so maximising engagement optimises for exactly what the user is trying to minimise.

Put Task Success in its place as the primary category, Happiness as the near-term secondary, and Retention as the annual guardrail. Engagement stays on the dashboard as a diagnostic, never as a target.

Goals → signals → metrics

CategorySignalMetricDirection
Task Success (primary)Return filed and accepted without correctionCompletion rate; median time-to-complete; reject/correction ratecompletion ↑, time ↓, rejects ↓
Happiness (secondary)Perceived effort after submitPost-submit CES ("how easy was it?")↑
Retention (guardrail, annual)Same user files here next yearYear-over-year return rate↑
Engagement (diagnostic)Depth of useSessions per filing; minutes per sessionwatch, don't target

Why Task Success leads: it is the only category that is true when the user's job is done. Happiness is a leading indicator for next year's Retention but is cheap to move with copy changes, so it can't carry the goal alone. Retention is the real business outcome, but it's measured once a year and is confounded by price, refund size, and life events, so it can't steer a sprint.

Failure mode, with numbers. A team shipped a "keep going" streak badge on the filing flow. Illustrative figures: 40,000 filing attempts in the quarter, completion 71% → 64%. That's 2,800 more abandoned returns; at a $22 filing fee refunded on abandonment, $61,600 in refunds. Sessions per user rose 30% and the engagement chart looked great. The tell was on the same dashboard: completion rate and time-to-complete moved the wrong way in the same cohort. Detection rule: if engagement rises while completion is flat or falling, treat it as a regression, not a win. Segment by cohort, users who complete in one session versus several, because a rising multi-session share is the signature of a broken flow, not a stickier product.

When engagement is the right category. If the roadmap moves into year-round tax planning, advice, or a document vault, consumption becomes the value and Engagement earns its place. Until then it's a cost line. If leadership still wants it on the scorecard, keep it as a diagnostic and gate every launch on completion rate and time-to-complete, so the engagement number can't be improved at the expense of the user's actual job.

Curated: · Written: · Reviewed:

Q&A-26The quarter recap lists 47 stories shipped. Activation is unchanged. What do you put on the recap instead?

What I'd put on the recap

First, clarify what the recap is for. A quarter recap informs one of three decisions: where next quarter's investment goes, how much capacity the team gets, or whether a commitment was met. The 47-story number only answers the third. If the board goal was activation, the recap has to lead with activation.

So the lead line is the goal metric and its movement, then the bets that were supposed to move it, then the stories as evidence of what was tried.

The recap I'd write (numbers illustrative)

Lead: "Activation flat at 41% (was 41%). Nine bets run: three killed, two scaled, four still running."

Then a bet table:

BetHypothesisResultDecision
Onboarding checklist+3 pts activation+0.4 pts, CI includes 0killed
SSO for teams+2 pts activation+1.8 pts, held 6 weeksscaled
In-app templates+1 pt activation+0.1 ptskilled

Then the arithmetic: 47 stories across 9 bets is roughly 5 stories per bet. Three bets consumed about 15 stories and moved nothing measurable. That is the finding: shipping capacity was not the constraint, so adding stories next quarter is not the fix.

When the story count is the right lead

Not every quarter is a bet. If the work was a commitment: a compliance deadline, a contractual migration, a platform change with no user-visible metric: output is the honest lead, because the outcome measure either doesn't exist or isn't attributable to this team. Same if the audience is doing capacity planning: they need throughput, not direction.

The deciding criteria: (1) what decision the recap informs; (2) whether the outcome metric is measurable and attributable within the quarter; (3) whether the work was a bet or a commitment.

Alternatives and failure modes

Outcome-only recaps fail too. A metric can move for seasonal or market reasons; without a holdout or a control segment you can't attribute it. And an outcome-only recap hides that the team shipped nothing while the metric drifted up.

Story-count recaps fail by padding: splitting tickets, counting chores. Detection is the same in both cases: check whether the metric moved in the segment the work targeted, and whether the story count changed the metric at all.

The recap I'd ship: goal metric first, bet table second, story count as the denominator.

Curated: · Written: · Reviewed:

Q&A-16Revenue is down 6% this quarter. Feature adoption of the new editor is up 18%. Which number can you still act on this month, and what must you not claim?

Act on adoption. Do not claim it caused anything.

Revenue is a lagging number: the 6% is already booked, and nothing you do this month changes it. Adoption is the only one of the two you can still move before the quarter closes, so it is the one to act on, but "act on" means run a decision, not declare a win.

What you must not claim: that the 18% adoption increase caused, offset, or will reverse the 6% revenue drop. You have two numbers moving in opposite directions and no tested link between them. Adoption up while revenue falls is equally consistent with adoption being irrelevant, with the drop being driven by pricing or churn in a different segment, or with the editor attracting users who were never going to pay.

The trade-off cuts both ways.

Act now, on adoption. A 6% revenue drop with a month left is a forcing function. If the action is cheap and reversible (instrument the funnel, ship an onboarding nudge, read a cohort) the cost of waiting for a clean causal test is a quarter you do not have. You learn the link by acting, not by waiting.

Wait for evidence. If acting means committing roadmap, headcount, or a pricing change, you are betting the quarter on an untested hypothesis. The 18% is a usage number; it says nothing about willingness to pay.

The criterion that decides it: reversibility and cost of being wrong. Cheap and reversible, act now and treat it as an experiment. Expensive and hard to unwind, get the link tested first.

Failure mode I have seen: the 18% was celebrated as a turnaround. Finished-document rate stayed flat, and revenue fell another 4% the next quarter. Nobody had written down what would have to be true for adoption to matter.

MetricRoleThis quarterWhat it tells you
Editor adoptioncandidate leading+18%usage changed
Finished documentshypothesized linkflatthe link is not firing
Revenuelagging−6%already booked

Check the base before you trust the 18%. If adoption went from 4% to 4.7% of monthly actives, that is 0.7 points: a rounding error next to a 6% revenue drop. (Illustrative numbers.) A relative move on a small base is not a signal.

What I would do this month: write the hypothesis as a chain, adoption → finished documents → retained paid seats, pick the one link I can test in four weeks, and set the kill criterion before I start. If finished-document rate does not move inside that window, adoption is not leading anything and I stop spending on it.

Curated: · Written: · Reviewed:

Q&A-50A vendor is $40k/year. A build estimate is 6 engineer-weeks. What costs are still missing before you choose?

Direct answer: The $40k and the 6 engineer-weeks are the two smallest numbers in the decision. What's missing is the run cost on the build side, the integration/migration/training cost on both sides, and the vendor's price escalation. On a $200k fully loaded engineer, 6 engineer-weeks is $24k of coding, but the realistic build is 1.5–2× that once review, QA, docs and integration are counted, and the service then costs 0.5–1.2 FTE a year to run. That run line is what decides it.

Missing line items

Line itemBuildVendor
Coding (6 eng-weeks @ ~$100/hr)$24k—
Review, QA, docs, integration (1.75×)~$42k—
Integration with auth/billing/CI (2 wks)$8k$8k
Data migration + dual-run (1 wk)$4k$4k
Support training (0.5 wk)$2k$2k
Licence, year 1—$40k
Run, year 10.5–1.2 FTEvendor's
Price escalation, 5%/yr—$40k → $42k → $44.1k
Exit (export, notice, re-implementation)—30–90 day notice

The arithmetic that matters. A fully loaded engineer at $200k/yr is ~$100/hr. Build one-time: 6 weeks coding = $24k; at a 1.75× multiplier for review, QA, docs and integration ≈ $42k; plus $8k integration, $4k migration, $2k training ≈ $56k one-time. Run: 0.5 FTE maintenance (the vendor ships features, you now build them) + 0.4 FTE on-call and incident response + 0.3 FTE dependency patching = 1.2 FTE = $240k/year. Three-year build TCO ≈ $56k + 3 × $240k = $776k. Vendor three-year: $56k one-time + $126k licence = $182k. The gap is the run FTE, not the build.

When building still wins. If the capability is core IP; if the vendor prices per call or per seat and your volume makes that scale worse than a fixed team; if you need in-process latency the vendor's API can't hit; or if the vendor is a small company with acquisition risk. The test: is run FTE below ~0.2? Then build can be cheaper.

Failure modes and how you'd see them.

  • Maintenance tail: the service keeps generating tickets after launch. Detect: tickets touching it per quarter. Anything above zero means you own it forever.
  • Key-person risk: bus factor 1. Detect: time until a second engineer merges a non-trivial PR.
  • On-call load: detect pager volume; >2 pages/week ≈ 0.2 FTE of interrupt cost.
  • Vendor escalation: detect at renewal; model 5–7%/yr, and check per-seat/per-call overage tiers.

What flips the decision: a 3-year TCO table with run FTE and escalation modelled, not a one-line build estimate.

Curated: · Written: · Reviewed:

Q&A-68A self-serve cohort tool scans 14 TB/query. The bill is $18k this month from 3 PMs. What do you change besides yelling?

First, the assumption I'd state out loud: this is BigQuery-style on-demand pricing. Billed per byte scanned, $6.25 per TB, and the 14 TB is a 30-day cohort window over the raw event table. At that rate each query costs about $87, and $18k is roughly 205 queries a month across three PMs. The tool isn't broken; the pricing model is doing exactly what it says.

What I'd change, in order: pre-aggregate the cohort into a daily table, cap interactive scan per user per day, and show the estimated dollar cost before the query runs.

The mechanism matters more than the list. Columnar engines bill bytes read from storage, not rows returned or CPU. A cohort query filters on cohort membership, which isn't the partition key, so the engine reads every partition in the window and every column the query references. 14 TB is days-in-window × bytes-per-day × columns-touched. The query can return 200 rows and still scan 14 TB.

A daily cohort table moves the join and the aggregation into a scheduled job that runs once. It stores one row per user per day: cohort_id, the day, and the metric columns. The interactive query then reads a narrow, pre-joined table partitioned by date and clustered by cohort_id, so it touches only the partitions for that cohort's window and only the metric columns. Same answer, different bytes.

Numbers, assuming 2M users and ~400 bytes per row in the pre-aggregate:

PathScan per queryCost per query205 queries/mo
raw events14 TB$87.50$17,900
daily cohort table24 GB$0.15$31
build job (daily)467 GB/day$2.92/day$88

So ~$120/month all in, and the build is the only thing that touches raw events.

A quota is the backstop, not the fix. A per-user daily byte cap rejects the query before it runs, so the worst case is bounded: one PM can't spend the month in an afternoon. It doesn't make anything cheaper. Pair it with a dry-run estimate in the UI so the cap is visible before someone hits it.

Where this breaks: pre-aggregation freezes the metric set. A new question needs a new column or a raw query, so keep a raw path with a hard cap for exploration. Freshness drops to daily, if PMs need intraday, add an hourly rollup for the last two days only. And if users belong to many cohorts, the pre-agg explodes; keep cohort membership in a separate small table and join it.

Two failure modes I'd watch. Backfill: someone re-runs the build over two years. 730 days × 467 GB is 341 TB, about $2,100 in one job. Cap backfills and require approval. Quota evasion: if the three PMs share a service account, a per-user cap never binds; cap per service account and per query.

What I'd measure: p99 bytes scanned per interactive query, and the daily build's scan. If p99 is under 50 GB and the build is under 500 GB, the bill is a rounding error.

Curated: · Written: · Reviewed:

Q&A-69You compute "currently premium" from a snapshot captured Monday. A user who churned Tuesday is still premium in Friday's board pack. Why?

The snapshot answers "was this user premium at the moment we captured it," and the board pack is reading that answer on Friday. Nothing in between re-evaluates the predicate. Monday's capture wrote status = premium for user 8812; Tuesday's cancel landed in the events stream, not in the Monday partition. The pack's query says "currently premium," but the row it reads was written Monday, so "currently" is doing work the pipeline never did.

Assume the usual shape: an append-only subscription_snapshot partitioned by snapshot_date, and the pack reading the newest partition.

-- what the board pack runs
SELECT status FROM subscription_snapshot
WHERE snapshot_date = (SELECT MAX(snapshot_date) FROM subscription_snapshot)
  AND user_id = 8812;
-- returns 'premium' — MAX(snapshot_date) is Monday

The predicate is evaluated at read time, but the row it lands on was written Monday. A cancel on Tuesday is a new row in subscription_events, and no query in the pack touches that table.

Take a hypothetical 1,000-subscriber base with 40 cancels between Tuesday and Friday.

Sourceuser 8812 on Fridaychurn rate shown
Monday snapshot, read Fridaypremium0%
events with effective_at <= Fridaychurned4%

The pack shows 0% churn while the revenue ledger is down 4 points. That gap is the tell.

The correct computation is an as-of reconstruction: fold events with effective_at <= as_of to get state at any timestamp.

SELECT e.user_id, e.status
FROM subscription_events e
WHERE e.effective_at <= TIMESTAMP '2025-03-07 00:00:00'   -- Friday
  AND e.effective_at = (SELECT MAX(e2.effective_at)
                        FROM subscription_events e2
                        WHERE e2.user_id = e.user_id
                          AND e2.effective_at <= TIMESTAMP '2025-03-07 00:00:00');

Alternatives: capture a fresh snapshot at period end (cheap, but only correct if the capture job actually runs and you label the as-of on the slide), or maintain a CDC-updated current-state table (fast reads, but you now own ordering and idempotency for every event).

Three failure modes matter here. Silent staleness is the dangerous one: no null, no error, just a plausible wrong number. Detect it with a freshness assertion: max(snapshot_date) must equal the report date, or the pack fails to build. Second, late-arriving events: a cancel with effective_at Tuesday that lands in the stream Friday is invisible to the snapshot forever, but an as-of reconstruction picks it up when you re-run it. Third, timezone boundaries: a snapshot taken Monday 00:00 UTC misses a Monday 23:00 local cancel that a "Monday" snapshot in local time would include.

The trade-off: snapshots are one partition scan and are fast; reconstruction scans the event table and needs a rule for corrections and out-of-order arrivals. Most teams keep both: daily snapshots for dashboards, events as the source of truth for anything that goes to the board.

Whatever you pick, put the as-of timestamp on the slide and reconcile the premium count to billed revenue. If they disagree, the snapshot is stale.

Curated: · Written: · Reviewed:

Q&A-70A prospect's security reviewer asks "is our data encrypted?" Sales wants to paste "yes" into the RFP. What analogy do you use in the meeting, and what business consequence changes if you give the one-word answer?

The 90-second version

"Encrypted" is three claims wearing one word: in transit, at rest, and who holds the key. I'd say that out loud, then hand the reviewer one analogy that carries all three.

Think of a records-storage company holding your files.

  • In transit: the files ride in a sealed, armoured truck. Anyone on the road sees a truck; nobody can read what's inside. That's TLS 1.3 on the wire.
  • At rest: the files sit in a locked warehouse. The lock is strong, but the storage company holds the master key. That's AES-256 on disk with keys in our KMS: a stolen disk is useless, and our staff can still open the cabinet.
  • End-to-end: a safe inside the warehouse that only you can open and the storage company genuinely cannot. We don't offer that.

So the honest answer is: yes in transit, yes at rest, no end-to-end. Here's the mapping I'd put on the screen:

ClaimAnalogyWhat we implementWhat the buyer may write
In transitarmoured truckTLS 1.3"unreadable on the wire"
At restlocked warehouse, we hold the keyAES-256, keys in KMS"useless if the disk is stolen"
End-to-endsafe only they can opennot implementednothing: never claim it

The business consequence

"Encrypted" in an RFP is a representation, not a description. The moment Sales pastes "yes", the buyer's security reviewer writes it into the contract, and their auditor reads it as the strongest of the three claims: usually zero-knowledge, because that's the one that sounds like encryption to a non-engineer.

What changes: the buyer may write "data is encrypted in transit with TLS 1.3 and at rest with AES-256; the provider holds the keys." They must never write "zero-knowledge", "provider cannot access customer data", or "end-to-end encrypted." Those three phrases are what turn a true statement into a breach.

An illustrative example with made-up figures: Sales answered "yes, fully encrypted" on a hospital group's RFP. The buyer assumed zero-knowledge. A court-ordered audit pulled 62,000 message bodies in readable form from our side. The $1.9M renewal closed at $1.2M, with a right-to-audit clause and an eight-week remediation plan attached.

How I'd catch it before signing

Before any RFP goes out, grep the security section for "zero-knowledge", "end-to-end", and "cannot access". If any appear, the claim has outrun the architecture. Then pin the approved sentence: "TLS 1.3 in transit, AES-256 at rest, keys held in our KMS, staff can read content." An auditor can check each clause against the config, and it's the only wording Sales is allowed to reuse.

Encrypted isn't a yes or no. It's a claim about who holds the key.

Curated: · Written: · Reviewed:

Q&A-31The spec lints clean. A partner still cannot complete "create order" because 200 is returned for a stock failure. What did lint not prove?

Lint proves the document is well-formed. It does not prove the document is true.

A spec that answers POST /orders with 200 and a body of {"error": "out of stock"} passes every structural rule a linter enforces: the path is declared, the response code is listed, the schema resolves, the media type is legal, the $refs point somewhere real. OpenAPI describes the shape of a response, never the meaning of its status code. Nothing in the schema asserts that 200 means "order created": that meaning lives in the product contract, and the linter has no access to it.

So the precise answer to "what did lint not prove" is: it did not prove a partner can branch correctly on the response. Branching is the only thing partners do with your API.

What lint actually checks. $ref resolution, required fields, parameter and path syntax, that declared response codes are syntactically valid, that security schemes are defined, that examples match their schemas, that operationIds are unique. All of it is internal consistency. A spec can be 100% lint-clean and still tell every consumer the wrong story about what happened.

The failure, traced. A partner client written against the spec above:

r = post("/orders", json=payload, headers={"Idempotency-Key": key})
if r.status_code == 200:
    fulfil(r.json()["order_id"])

Server returns 200, body {"error": "out of stock"}. r.status_code == 200 is true, so the branch is taken. r.json()["order_id"] raises KeyError: the fulfilment job dies mid-run. A client that instead reads the message field and retries loops forever on a request that can never succeed, and with no idempotency key each retry is a fresh order attempt. Both behaviours are the spec's fault, not the client's: the status code said success.

The fix is a mapping, not a schema change. Every domain outcome gets a status and a stable machine-readable problem type (RFC 9457 application/problem+json), so partners branch on the code and never parse prose:

OutcomeStatusProblem typePartner branch
Order created201—proceed to fulfilment
Insufficient stock409urn:problem:out-of-stockstop, surface to user
Payload semantically invalid422urn:problem:validationfix request, do not retry
Replay of an idempotency keyoriginal status + body—treat as the original result

409 for stock is a state conflict against current inventory; 422 is for a request that is well-formed but semantically wrong. The exact choice matters less than the invariant: no failure is ever a 2xx, and every failure carries a type a client can switch on.

What would settle it. Consumer-driven contract tests (Pact or equivalent) run against the real service, one per row of that table, asserting status, problem type, and retryability. Plus a replay test: same idempotency key twice, assert one order exists and both responses are identical. Until those exist, the spec is a claim, not a contract.

Lint cannot see a lie told with a 200.

Curated: · Written: · Reviewed:

Q&A-32Mobile drops after Pay; 8% of users tap again. The API is POST /charges with no key. What do you put in the PRD?

What goes in the PRD

One hard requirement: POST /charges requires an Idempotency-Key header, scoped to a single payment intent, and the server guarantees at most one capture per key.

Assumptions I'd state before writing it: the client can generate a UUID per checkout attempt and persist it across the retry; we control the charge service, not the processor's own dedup; the gateway sits in front of the charge store.

Then the semantics, because "add an idempotency key" is not a spec:

  • The client generates one key when the user first taps Pay and reuses that exact key for every retry of that intent: including the automatic retry after a dropped connection.
  • The server claims the key atomically before doing any work: a unique index insert, or INSERT ... ON CONFLICT DO NOTHING returning the row. A second request with the same key either waits on the in-flight claim or gets the stored response.
  • It stores the request fingerprint (method, path, body hash) and the terminal response. An identical replay returns the stored response verbatim, same status code. Reuse with a different body is rejected: pick 409 or 422 and document it.
  • TTL: 24 hours, after which the key is evicted and a replay is treated as new. State it, so nobody assumes forever.

Why the atomic claim is the whole point: two taps 40 ms apart both read "key not seen" if the check and the insert aren't one operation, and you capture twice anyway. The key is only as good as the claim.

Alternatives I'd reject in the same PRD: disabling the button after the first tap (doesn't cover the retry after a dropped response, which is the same double-charge), and leaning on the processor's own dedup (not every processor has it, and you can't exercise it in your own test suite).

The failure this prevents, from a real incident: double-tap charged 8% of users twice. Chargebacks plus goodwill ran $63k over 21 days.

DesignTapsCaptures
POST, no key22
Key per intent, atomic claim21
8% of 50k pays (assumed volume)—4,000 extra captures

Acceptance criteria, written as tests rather than prose: replay and concurrency runs over 100,000 payment intents produce exactly one capture per intent, identical responses on identical replay, and an explicit error on key reuse with a changed payload. Sequential and concurrent, both. Detection in production: alert when captures per intent exceeds 1.

If the user can tap twice and get charged twice, POST without a key is a product defect, not a nice-to-have.

Curated: · Written: · Reviewed:

Q&A-33The export API returns 202 and a message "we're on it". Support asks when the file exists. What was missing from the product contract?

What a strong candidate says first

The contract stopped at acceptance. 202 Accepted (RFC 9110 §15.3.3) means the server took the request for processing; it makes no promise about when, or whether, the file will exist. So support has nothing to answer with, and that is the gap: the API specified the start of the job and never specified the end.

Assumptions I'd state out loud: the export is asynchronous, the file is not ready when the 202 is returned, and the client is either a UI or an API consumer that can make a second request.

What was missing

A completion contract, in six parts:

  1. Job identity. The 202 should carry Location: /exports/{id} and the id in the body. Without it there is no handle to ask about.
  2. A status resource. GET /exports/{id} returning state, progress, timestamps, and on failure a machine-readable error.
  3. Terminal states. queued → running → succeeded | failed | cancelled | expired. "On it" is not a state; it is the absence of one.
  4. A time bound. A published p95 (say 60s for ≤10k rows) plus Retry-After, so clients back off instead of hammering.
  5. Result delivery. A signed download URL with an explicit TTL, not a bare path.
  6. Idempotency and cancellation. Idempotency-Key on the POST so a retry returns the same job; DELETE /exports/{id} so a wrong export can be stopped.

The wire format

POST /exports
Idempotency-Key: 7f3c9a...
→ 202 Accepted
Location: /exports/9f2a
Retry-After: 5
{"job_id":"9f2a","state":"queued"}

GET /exports/9f2a
→ 200
{"state":"running","progress":0.4}

GET /exports/9f2a
→ 200
{"state":"succeeded","rows":8421,
 "download_url":"https://cdn/exports/9f2a?sig=...",
 "expires_at":"2025-06-01T00:00:00Z"}

GET /exports/9f2a
→ 200
{"state":"failed","error":{"code":"source_timeout"}}

Why the time bound matters, in numbers

Hypothetical: 5,000 exports/day, each taking 60s. A client polling every second issues 60 status requests per export: 300,000/day, all hitting the same table. With Retry-After: 5 it is 12 per export, 60,000/day. Same information, a fifth of the load.

Failure modes and how you'd see them

  • No job id → users re-submit. Detect: duplicate exports per user per minute. The fix is the idempotency key, not a UI warning.
  • No terminal state → jobs stuck in running. Detect: alert on jobs in running longer than 2× p95; that is a worker crash, not a slow query.
  • No TTL on the download URL → either a permanent public link or a 404 after support already told the user the file was ready. Detect: 404 rate on the download endpoint.
  • No error on the job → the client sees "failed" with no reason and files a ticket. Detect: share of failed jobs with a null error code.

What I'd want before calling it done

A state diagram in the API docs, and a p95 job-completion panel on the launch dashboard. The contract is not "we return 202"; it is "we return 202, and here is how you find out it finished."

Curated: · Written: · Reviewed:

Q&A-34The activity feed uses OFFSET 10000 and sometimes skips a row the user just saw. What product requirement did the API miss?

Assume the feed is sorted newest-first, page size is 20, and clients page forward. The API missed a read-consistency contract for pagination: it never defined what a page walk guarantees when rows are inserted or deleted mid-walk. OFFSET 10000 gives no such guarantee.

Here is the mechanism. OFFSET N tells the database to scan and discard N rows, then return the next page. If a new activity is inserted at the top between page 500 and page 501, every row below it shifts down by one. The row that was at position 10000 is now at 10001, so the request for OFFSET 10000 starts one row later: that row is skipped. Deletes cause the opposite: a row appears twice. The user sees a gap, and support sees "missing invoice."

The product requirement is not "pagination" in the UI sense. It is a stable ordering plus a cursor whose semantics under concurrent writes are defined. Concretely: sort by (created_at DESC, id DESC) with id as a tie-breaker; the cursor is the last (created_at, id) seen; page size is bounded; and the contract answers "can a new write appear on a later page?" For a feed, the usual answer is no. The walk is a snapshot from the first request.

Keyset pagination implements that:

SELECT id, created_at, ...
FROM activity
WHERE (created_at, id) < (:last_created_at, :last_id)
ORDER BY created_at DESC, id DESC
LIMIT 20;

This is stable under inserts because the cursor is a value, not a position. Trade-offs: no random page access (you cannot jump to page 501), the cursor must be opaque and signed so clients cannot forge it, and you need a composite index on (created_at, id) to make it fast. If the product actually wants new rows to appear mid-walk, that is a live-feed requirement and needs a different design: polling with a cursor, or a push channel.

Failure modes: OFFSET 10000 also costs the database a 10000-row scan per request. Suppose the table gets 2k writes/min and support sees 40 "missing item" tickets/week at deep pages: a hypothetical but realistic shape. The skip window is every page transition. Detection: a contract test that walks pages while inserting and deleting rows, asserting no missing or duplicate IDs relative to the snapshot boundary. That test is the real deliverable; the ticket count is just the symptom.

MethodDeep page costConcurrent insert
OFFSET 10000scan 10000 rowsrow skipped or duplicated
keyset (created_at, id)index seekstable snapshot

Page numbers are a UX detail. A page walk that silently drops rows is a data-integrity bug, and the API should have specified the consistency guarantee before shipping the offset.

Curated: · Written: · Reviewed:

Q&A-35Every failure is HTTP 500 with "something went wrong". What do you require so users, clients, and support can act?

Assume a public or partner-facing HTTP API where clients write retry logic and support reads the tickets. Before I'd call error handling done, I require four things: a stable machine-readable error type, an HTTP status that matches the class of failure, a client action the caller can branch on without parsing prose, and a correlation ID that ties the response to a server-side log line. Human text is for the end user; the type is for the code.

Why 500-everything fails is mechanical. A client cannot tell a retryable dependency blip from a terminal validation error, so it either retries everything or nothing. Support cannot route a ticket, so every incident starts from zero. The fix is a small closed set of types, each mapped to a status and a retryability flag.

FailureStatustypeRetryableClient action
bad SKU400validation_errornofix input, highlight field
expired token401token_expiredonce, after refreshrefresh, then retry
duplicate idempotency key409conflictnofetch the existing result
rate limit429rate_limitedyes, after Retry-Afterback off
warehouse down503dependency_unavailableyes, with backoffretry, then degrade
our bug500internal_errornoreport with correlation ID

Wire it as problem+json (RFC 9457, which obsoletes RFC 7807):

{
  "type": "https://api.example.com/problems/dependency_unavailable",
  "title": "Upstream warehouse unavailable",
  "status": 503,
  "retryable": true,
  "correlation_id": "01HQ8...",
  "detail": "Try again shortly."
}

Now the trade-off, because a taxonomy is not free. Fine-grained types pay off when clients branch automatically and support triages by type: fewer mis-retries, faster routing. Coarse codes plus structured logs are the better call when there is one internal consumer, no automated retry, and low volume, then a dozen types drift, two of them end up meaning the same thing, and nobody maintains the mapping. The second axis is detail exposure. Public clients get type, safe message, and correlation ID. Trusted or internal callers can get a debug field carrying the upstream error, gated by a header or environment. Stack traces never leave the process.

Concrete failure modes. Taxonomy drift: types get added ad hoc and stop being disjoint. Detect by reviewing the type distribution monthly and flagging any type under 0.1% of volume for merge or deletion. Retry storms: a 503 without Retry-After and without client jitter turns one outage into a self-inflicted one. Detect via a retry-amplification metric (retries divided by origin requests). Broken correlation: gateways drop the header, so tickets can't be joined to logs. Detect by sampling responses and asserting the ID resolves. As a hypothetical: if 2% of requests are validation failures and clients retry all 500s five times, that becomes roughly 10% extra traffic on the same path, and support opens six tickets per real failure.

I'd revisit the design when more than 5% of volume lands in internal_error, or when support cannot map a ticket to a type: both mean the taxonomy no longer describes reality.

Curated: · Written: · Reviewed:

Q&A-37You add optional tax_region to the order JSON. One generated client throws. Why can an additive field be breaking, and what do you do?

Core answer

Assume tax_region is added to the order object in a response body, and the client that throws is a consumer deserializing it. The field is additive in the schema; the break is in the consumer's unknown-field policy. "Optional" in OpenAPI means the producer may omit the key: it says nothing about whether a consumer tolerates a key it does not know. Those are two different contracts, and only the second decides whether the rollout is safe.

So the reason an additive field breaks: the consumer's deserializer is strict by default in several mainstream stacks, and a generated client inherits that default. The generator read the same spec you did and still produced a strict parser.

Mechanism

Jackson's FAIL_ON_UNKNOWN_PROPERTIES is enabled by default in Jackson 2.x. A generated POJO without @JsonIgnoreProperties(ignoreUnknown = true) throws on the first unknown key:

// Jackson 2.x — FAIL_ON_UNKNOWN_PROPERTIES is on by default
class Order { String id; BigDecimal total; }              // throws on "tax_region"
@JsonIgnoreProperties(ignoreUnknown = true)
class Order { String id; BigDecimal total; }              // ignores it

The exception names the field: UnrecognizedPropertyException: Unrecognized field "tax_region" (class Order), not marked as ignorable. Same shape elsewhere: kotlinx.serialization fails on unknown keys unless ignoreUnknownKeys = true; Go's encoding/json ignores them unless you call DisallowUnknownFields(); .NET's System.Text.Json ignores by default but UnmappedMemberHandling.Disallow (.NET 8) throws. And a schema with additionalProperties: false, or a generator that emits it, turns the field into a validation error before deserialization runs.

The wire is fine. The consumer contract is not.

ClientUnknown-field policyResult
tolerant JSignoreok
Jackson POJO, no annotationthrow (default)500s on every order with the field
Go, DisallowUnknownFields()throwdecode error
schema additionalProperties: falserejectvalidation error

What I do

  1. Identify the failing client and its policy: the exception names it. Do not guess.
  2. If it is a strict client I control: ship ignoreUnknown = true / ignoreUnknownKeys = true / a tolerant schema, and add a test that feeds it a payload with an extra key.
  3. If it is a third party I cannot update: this is a real compatibility problem. Options are versioning the response, putting the field behind a media type or query param, or negotiating a migration window. Shipping it anyway and hoping is how you get the outage.
  4. Contract: declare response objects extensible (additionalProperties: true) and state in the API docs that clients must ignore unknown fields. That rule is what makes the next additive change actually additive.

Failure mode and detection

The failure is silent until the field ships: the client throws on 100% of responses containing it. Detection is a consumer matrix (tolerant vs strict clients, each fed a payload with an extra key) plus a canary that includes the field against a slice of real clients before full rollout. A schema diff will not catch it; the schema is correct.

When not to

If you cannot enumerate consumers (public API, unknown integrators), treat every field addition as breaking and version. Additive is only safe once you have verified the consumers.

Curated: · Written: · Reviewed:

Q&A-38The API returns 429 with no headers. A startup on the free tier thinks they are banned. What belongs in the product contract and the plan page?

Assumption and answer. I'll assume the 429 is a quota, not an abuse block, and that the free tier exists to convert to paid. Then the missing headers are a contract defect, not a support ticket. The product contract owes four things: the scope of the limit, the number and window, the response semantics, and the path to more. The plan page owes the number itself, before signup, next to the price.

Why silence reads as a ban. From the client's side, a 429 with no body and no headers is indistinguishable from a 403 or a WAF block: the only signal is "my requests stopped." The developer can't tell whether to wait, fix their code, or appeal, so they do the expensive thing. Open a ticket or leave. The header is what turns an opaque failure into a self-service retry. Retry-After says when; X-RateLimit-Remaining says how close they are; a machine-readable error code says it's a quota, not an account action. That is the behavioural change a contract clause buys: the developer moves from "am I banned?" to "back off and retry."

What goes where.

ClauseContract (docs + API)Plan page (pre-signup)
Scopeper API key, per endpoint group"per key"
Number60 rpm free, 600 rpm paid60 and 600 shown next to the price
Bursttoken bucket, 10-request burst—
Response429 + Retry-After + X-RateLimit-*—
Over-limitthrottle, not ban"we slow you down, we don't cut you off"
Escalationhow to request a raiselink to pricing

The response itself.

HTTP/1.1 429 Too Many Requests
Retry-After: 12
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1717171717
Content-Type: application/json

{"error":"rate_limited","scope":"api_key","limit":60,"window":"60s",
 "retry_after":12,"upgrade":"https://example.com/pricing"}

Trade-offs. Publishing a hard number is not always right. If the limit is a security control (scraping, credential stuffing) the exact threshold tells an attacker where to sit; publish the shape (per-key, per-minute, always 429 + Retry-After) and a generous floor, not the ceiling. If the limit is capacity-driven and moves, publish the contract and a floor, not a fixed number. Prefer soft limits on the free tier: warn at 80%, throttle at 100%, don't hard-fail. The free tier's job is activation, not revenue protection. Use 429 for quota ("retry later") and 403 for permission ("don't retry"); using 403 for rate limits trains clients to give up permanently. Paid tiers can take a burst buffer or overage billing so a spike mid-integration doesn't 429 a paying customer.

Failure modes. A 429 without Retry-After makes clients retry immediately and the herd makes it worse: watch for a retry storm in the logs. A limit scoped per IP means one customer behind a shared NAT 429s another; scope per key. A fixed window at the boundary allows a 2x burst; use a token bucket or sliding window. No X-RateLimit-Remaining means clients can only react, never pace.

Decision criterion. I'd call it done when a developer who has never read the docs can hit the limit, read the response, and self-correct without contacting support, and when the plan page shows the number before they sign up. An unpublished quota is experienced as a defect.

Curated: · Written: · Reviewed:

Q&A-39The flag is still in code 14 months after "100%". What product lifecycle did you skip, and what is the risk?

The step you skipped is flag retirement: the decommission phase that runs after rollout completes. You did create → roll out → reach 100% → verify. You never did: remove the branch, delete the flag, close the ticket. The flag is now a permanent, unowned runtime dependency.

The risk isn't code cleanliness. It's that the flag's default value becomes the live behaviour the moment the flag system stops agreeing with the code.

Mechanism, in the order it usually bites:

  1. Default drift. Code still reads billing_v2. Someone archives the flag in LaunchDarkly, or the project is cleaned up, or the SDK key rotates to an environment where the flag was never created. The SDK can't find the flag, so it returns the default passed in the call: usually false. The feature silently turns off. No deploy, no alert, no diff.

  2. Branch divergence. The false branch hasn't been exercised since launch. It compiles, so nobody notices it's wrong. Refactors update the true path only; the dead path rots until the day it's taken.

  3. Config coupling. Targeting rules, segments and prerequisites accumulate on a flag that's 100% on. Every request still evaluates it, and every incident review now has to ask "is this flag involved?"

Worked trace (hypothetical, but this is the shape):

// billing/charge.go
if ld.BoolVariation("billing_v2", ctx, false) { // default: false
    chargeV2(ctx, invoice)
} else {
    chargeV1(ctx, invoice) // legacy path, last touched 14 months ago
}

Flag archived in the flag system at month 3. From then on BoolVariation returns the default false for every context, so chargeV1 runs. If chargeV1 was already broken by a schema change in month 9, invoices stop being generated, and the only signal is a revenue dashboard nobody correlates with a flag.

StageDone?Artefact
Create flag: owner, default, expiryyesflag record
Roll out to 100%yesdashboard
Verify exposureyesmetrics
Retire: remove branch, delete flagno—

Deletion isn't always right, though:

  • Keep it if it's a kill switch. A flag guarding a risky migration (new payment processor, new datastore) is a permanent flag by design. Type it as permanent, give it an owner and a runbook, and don't let the stale detector nag you about it. Its failure mode is different: a kill switch nobody has tested since launch.
  • Keep it while rollout is genuinely partial. Regional or cohort rollout still in progress is not stale.
  • Cost of premature removal: you delete the kill switch mid-incident and lose the ability to turn the feature off without a deploy. Remove only once the rollback story is replaced: deploy rollback, or a different, tested kill switch.

Detection: a flag inventory recording age, owner, type and last-evaluated date, plus the flag system's own stale detector (Unleash's per-project stale period, LaunchDarkly's temporary flag type and code references). Anything temporary and past its expiry opens a removal ticket automatically.

Curated: · Written: · Reviewed:

Q&A-42Image resize moved behind a queue. The UI still says "ready". p95 drain is 90s. What is the product change?

The change is that "ready" has to mean the derivative exists, not that the upload was accepted. Three things ship together: the upload response stops being the completion signal, the UI gets a queued/processing state with a bound, and the user gets a fallback while they wait.

Assumptions I'd state up front: the original is servable the moment the upload lands, so a user can always see something; and 90s p95 is not acceptable for the primary flow. If it were, the answer would be "do nothing", so I'll assume it isn't.

Why the UI lies today. The API returns 200 when the job is enqueued. The client treats 200 as done and renders the thumbnail slot as ready. Nothing in the contract distinguishes "bytes stored" from "derivative available," so the client can't tell the truth even if it wanted to. The fix is a state the client can read, not a spinner.

StateMeaningWhat the user sees
uploadedoriginal in object storeoriginal, immediately
queuedjob accepted, not started"Processing" + position/ETA
processingworker running"Processing"
readyderivative in storethumbnail
failedjob errored or expiredretry affordance

The failed state matters as much as queued. A job that dies in the worker currently leaves the UI stuck on ready forever, which is worse than a visible error.

How big is the backlog. Little's Law: L = λW. At 20 uploads/min (λ ≈ 0.33/s) and W = 90s, the queue holds roughly 30 jobs at p95. That's a standing backlog, not a transient spike, which is why "just wait a bit" isn't a product answer. It also means p99 is whatever the queue depth says it is; measure it before promising a bound.

The bound. Pick an SLO on job age, not on worker time: e.g. p95 derivative-ready under 30s for images under 10MB. If the queue can't hold that, the product change is to make the wait visible and bounded rather than to pretend it's zero. Options, in the order I'd try them:

  • Serve the original with a processing badge. Zero new infrastructure, honest, and the user can keep working.
  • Priority lane for small images, which are most of the volume and drain fast.
  • Notify-me (email or in-app) for anything past the bound, so the user can leave the page.
  • Client-side resize for the common case, if the browser can do it: removes the queue from the critical path entirely.

Failure modes to design against. Retry storms: users who see a stuck ready state re-upload, which multiplies queue depth. The processing state plus a notify option is what stops that. Notify-me deliverability: if the email bounces, the user never learns the image is done, so the in-app state has to be the source of truth and the email a bonus. Orphaned jobs: a worker crash leaves a job in processing forever, so the state needs a timeout that flips it to failed. Shared links: a recipient opening a link mid-processing must not get a broken image icon. Serve the original there too.

Detection. Job-age p95/p99 on the launch scorecard, queue depth, retry rate per session, and the share of sessions that ever render the processing state. If that last number is near zero, the queue is fast enough and the UI change is cheap insurance; if it's high, the SLO is the real work.

What I'd ship first. The state split and the original-as-fallback, because they're a day of work and they remove the lie. The SLO and the priority lane come next, sized against the measured p99 rather than the p95 we started with.

Curated: · Written: · Reviewed:

Q&A-40The PM writes "your balance updates instantly everywhere". Replicas lag p95 2.4s. What do you change in the product, not in the database?

The copy is a claim about the system, and the system cannot back it. Since the database stays as it is, the fix is to change which read path each surface uses and to make the copy match what that path can actually deliver. Two moves cover most of it: read-your-writes on the session that just wrote, and an explicit pending state everywhere else.

Start with the surfaces where a stale balance causes an action, because that is where the promise does damage.

SurfaceCurrent copyRead pathWhat the user sees
Balance after a transfer, same device"updates instantly everywhere"post-write read routed to primary, or a write token the replica can wait onnew balance, no lag
Balance on a second device or tabsamereplica"Updated a moment ago" plus a timestamp
Receipt email or pushsamereplica"Balance as of 14:02"
Marketing and onboardingsamen/adrop "instantly everywhere"

The arithmetic behind the second row: at 2.4s p95 lag and 5 balance-changing writes per second, roughly 12 writes are mid-flight at any instant, and each one is a window where another device shows the pre-write number. That is the exposure the current copy denies.

The failure mode is a double-send. Worked example with hypothetical figures: a user debits $4,000, the second device still shows the old balance, and they send again. In one incident I have seen, that pattern produced $1.1M of duplicate transfers over 40 minutes before anyone noticed. Detect it with a duplicate-transfer rate per account and an alert above 1 per 10,000 transfers; the copy change is what removes the trigger, the alert only tells you it is still happening.

Trade-offs. Read-your-writes costs a primary read on every post-write view, which is fine at these volumes and not fine if you apply it to every list view. Optimistic UI shows the new balance before the write commits, so a failed write must revert with a visible message, or you have traded a stale read for a phantom one. A timestamp on secondary surfaces is cheap and honest, but it only works if the timestamp is the replica's read time, not the request time.

The promise is the thing under test, so give it a kill criterion: a lag SLO on the copy of "usually under 3s, p99 under 10s", measured on the same replicas the user reads. If that cannot hold, the copy goes back to "may take a few seconds". Naming that cost and that threshold is the product decision; without both, you are just rewording marketing.

What I would ship this week: the copy diff above, read-your-writes on the post-write balance view, and a timestamp on every surface that reads a replica.

Curated: · Written: · Reviewed:

Q&A-41Marketing wants a 1-hour CDN cache for pricing. Legal wants the published price to match checkout. What number do you put in the PRD?

The number: 60 seconds. Cache-Control: public, s-maxage=60, stale-while-revalidate=30 on the price response, plus a surrogate-key purge fired on every price publish. The 60s is the backstop; the purge is what actually keeps the published price matching checkout. I would not sign off on 3600 without Legal changing the requirement to "displayed prices are indicative."

Assumptions I'd state before the number: the displayed price is legally binding (Legal's constraint), prices change on a discrete publish event rather than continuously, and the CDN supports surrogate-key purge. If the price varies per user (logged-in contract pricing), none of this applies: that response is private, no-store and the CDN is the wrong layer.

Why not 1 hour. A CDN caches on the origin's Cache-Control. s-maxage sets the shared-cache TTL; stale-while-revalidate lets the edge serve a stale copy while it revalidates in the background. With s-maxage=3600 the edge can serve a price up to an hour old with no signal to anyone. If that price is binding, you have sold at a number checkout will not honour. 60s caps the unpurged window at a minute; the purge collapses it to purge-propagation time, typically a few seconds across PoPs.

OptionHeaderWorst-case stalenessOrigin loadWhen it's right
1h, no purges-maxage=360060 minlowestdisplay price explicitly non-binding
60s + purge on publishs-maxage=60, swr=30purge latency (~seconds)~60x the 1h pathbinding price, publish-driven changes
no cacheno-store0every request hits originper-user pricing, or low traffic

The trade-off, with numbers. Say the price page peaks at 5,000 req/s, 50 price URLs, 200 edge PoPs. At 1h TTL that is ~10,000 cache entries refilled once an hour: trivial origin load. At 60s it is ~10,000 refills per minute, ~167 origin fetches/s. That is still small for a price endpoint, which is why 60s is affordable. (Figures illustrative.) The cost you are actually buying down is origin load; the cost you are buying up is staleness. Purge-on-publish is the lever that lets you keep a long TTL and a short staleness window, but only if the publish pipeline reliably calls purge.

Failure modes and detection.

  • Purge API call fails or is skipped: the 60s TTL is your only bound. Detect with a purge-success counter and alert on any publish without a matching purge.
  • Purge propagation lag: edges in some regions serve stale for seconds after publish. Detect by sampling the price from multiple PoPs after a publish.
  • Cache-key miss: currency, region, or auth variant not in the key, so one region's price is served to another. Detect with a synthetic check per variant.
  • Origin's own cache serving stale upstream of the CDN. Detect by comparing origin response to the published value.

When 1 hour is fine. If Legal accepts that the displayed price is non-binding and checkout is the source of truth, s-maxage=3600 is correct and cheaper. That single sentence, binding or not, is the deciding criterion, and it belongs in the PRD next to the number.

So the PRD line reads: price page, s-maxage=60, stale-while-revalidate=30, surrogate-key purge on publish, alert on purge failure, binding price.

Curated: · Written: · Reviewed:

Q&A-43During a partition, should the booking button fail closed or sell a seat twice? Who decides, and what do you write?

Fail closed. The booking button returns an error rather than confirming a seat it cannot prove is still free. A seat is a scarce, non-fungible resource with a hard invariant (one seat, one passenger) and a double sell is not a latency problem you can retry away; it is a correctness break whose cost lands on support, refunds and the brand. A stale like count costs nothing. So the default is per-operation, not per-product.

Who decides. The service owner, the booking service's tech lead, owns the invariant and writes the decision down. The product owner signs off on the user-facing copy and on the acceptable loss of bookings during a partition. Neither decides alone. During a live partition, the incident commander can authorize a pre-agreed degraded mode, queue the request as a hold and confirm later, but that override is logged, not improvised.

What you write. A consistency contract, one row per operation, in the service's ADR or README. Not "we are CP".

OperationInvariantPartition behaviorUser copyError
Reserve seatone seat, one holderfail closed"We couldn't confirm that seat: try again in a moment."503
Confirm bookingidempotent by keyfail closed"Payment not taken."503
Like counteventualserve stalenone—
Recommendationseventualserve stalenone—

Mechanism. The write is a conditional update, not a read-then-write.

UPDATE seats
SET status = 'held', holder = :user, version = version + 1
WHERE seat_id = :seat AND status = 'open' AND version = :seen_version;

Trace it. Seat 14A is open at version 7. A and B both read version 7. A's UPDATE matches version 7, sets version 8, 1 row affected: A holds the seat. B's UPDATE now matches nothing, because the version is 8, so 0 rows affected and B gets a 409 telling them the seat is gone. No double sell. Now partition the primary away from B: B's statement cannot reach it, errors, and B gets a 503. Fail closed. The double sell only happens if you drop the version predicate and write SET holder = :user unconditionally, or read from a replica and write blind.

The cost of failing closed is real: bookings you do not take. In one incident the store stayed available and sold 220 double seats over a 14-minute partition; compensation ran $48k, about $218 per double-sold seat, plus the social posts. Compare the fail-closed side: at, say, 40 booking attempts per minute, 14 minutes is roughly 560 rejected attempts, most of which retry and succeed once the partition heals. That asymmetry is the whole argument.

Failure modes to name. Retry storms against the failing primary: cap retries, back off, and put an idempotency key on confirm. A "fail open by accident" where a read replica serves a stale seat and the write goes through unconditionally. And a hold that never expires, silently removing inventory: give holds a TTL and release them.

When not to fail closed. Anything where a stale answer is harmless and availability matters more: counts, feeds, recommendations. Those are AP, and you say so in the same table.

Curated: · Written: · Reviewed:

Q&A-44Growth wants a live-video feature. The current error budget is 12% remaining this month. How does the SLO participate in the roadmap?

What I'd say first

Assumption: the 99.9% availability SLO is a commitment with a written policy, when the budget is exhausted, feature launches stop. If it's aspirational, it doesn't participate in the roadmap at all, and we should say that out loud rather than pretend it does.

Given that, 12% remaining isn't a green light, it's a warning. The SLO participates in the roadmap as the budget the roadmap spends. Every launch is a withdrawal. So the question isn't "is 12% enough": it's "what's the current burn rate, and what does live video cost."

The math

99.9% over 30 days allows 0.1% of requests to fail. At 100M requests/month that's 100,000 failures of budget. 12% left is 12,000 failures, with 9 days left in the window.

Current burn: if 21 of 30 days have elapsed and 88% of the budget is gone, average burn is 0.88 / 0.70 = 1.26×. At that rate the remaining 12% lasts 0.12 / 1.26 = 0.095 of the window, about 2.9 days. The month is already off track before live video enters the conversation.

Live video's cost is the incremental error rate: (share of requests on the new path) × (failure rate of that path). Over the remaining 9 days there are roughly 30M requests.

Requests (9 days)Failures
Remaining budget30M12,000
Live video, 5% traffic, 1% path failure1.5M15,000
Live video, 5% traffic, 0.2% path failure1.5M3,000
Canary, 1% traffic, 1% path failure300K3,000

At full traffic the media path blows the remaining budget even at a 0.2% failure rate. A 1% canary costs 3,000 failures, a quarter of what's left: that fits.

Deciding between the options

  • Sequence: reliability work first, ship after the window resets. Right when the feature's incremental burn exceeds remaining budget and the reliability work is in the same subsystem. The cost is the delay, and it's the honest one.
  • Cut scope: ship a subset whose incremental error rate fits. Right when you need signal on the feature and can hold exposure down: the canary row above.
  • Borrow: only legitimate as an explicit policy exception with a named owner, an expiry, and a re-baselined SLO. Silent borrowing means the next window opens in deficit and the freeze lands on someone else's launch.

Failure modes

  • Budget exhausted mid-month, so the freeze is decided by whoever is on call rather than by the roadmap. Detect it with burn-rate alerts: 1× over 3 days (10% of budget) as a ticket, 6× over 6h (5%) and 14.4× over 1h (2%) as pages. The standard multiwindow thresholds from the Google SRE Workbook.
  • Silent borrowing: the budget-remaining metric at window start reads below 100%.
  • Ship and spike: the 14.4× 1h page fires within the hour, and rollback is the response.

What I'd tell Growth

"We have 12,000 failed requests of budget and 9 days. Live video at full traffic costs 15,000 even if the media path only fails 1% of the time. We can canary it at 1% of traffic now and ramp after the window resets, or hold it until the reliability work lands. Full traffic this month, no."

Curated: · Written: · Reviewed:

Q&A-45Analytics wants every keystroke "for later ML". What do you require before a single extra field is collected?

Core answer

Before a single extra field ships, I need five things written down and owned, not promised:

  1. The decision the field changes. Not "ML later": a named decision, e.g. "which form fields users correct before submitting."
  2. Purpose and legal basis, per field.
  3. Retention with a deletion mechanism that actually runs.
  4. Access: who queries it, under what role, logged.
  5. The minimisation test: can the decision be made without this field?

If any of the five is missing, the field doesn't ship. "Later ML" fails at step 1: it names a technique, not a decision.

Why keystroke capture fails necessity

A keystroke stream is a superset of everything the user typed: passwords before masking, pasted card numbers, health queries, other people's data in a shared doc. You cannot un-collect it, and you cannot scope it after the fact. Under GDPR Art 5(1)(c), assuming EU users, the test is whether the field is adequate, relevant and limited to what's necessary for the stated purpose. A purpose of "maybe useful later" cannot pass, because necessity is measured against a purpose, and there isn't one.

There's a second problem: keystroke timing is a biometric. Typing rhythm identifies individuals with high accuracy, so a raw stream isn't just "more data": it's a re-identifiable identifier attached to every session, which pulls it toward Art 9 special-category handling.

What aggregation preserves

Most of the ML value people claim from keystrokes is available from events that carry no content: field focus/blur, edit counts per field, time-to-submit, validation error codes, abandonment point. That answers "where does the form fail" without recording what anyone typed.

When raw capture is justified

Two cases: fraud/abuse detection (bot vs human typing patterns) and UX debugging of a specific broken flow. Both need explicit consent or a documented legitimate-interest assessment, masking of all input values, short retention (hours to days, not months), access limited to a named team, and a kill switch. The deciding criterion is whether the decision can be made from aggregates, if it can, raw capture is not justified.

Failure modes

  • Retention drift: the field ships with 30 days, then a pipeline default silently extends it. Detect with a scheduled job that asserts max age and alerts.
  • ACL misconfiguration: the warehouse table is readable by the whole analytics group. Detect with a periodic access review diffed against the decision record.
  • DSAR cost: a raw stream makes export and redaction expensive. If a DSAR takes weeks, the retention was wrong.

Artefacts

FieldDecision it servesRetentionAccess
keystroke streamnone named——
field focus/blur + edit countwhere the form fails30danalytics role
validation error codeswhich rules misfire90danalytics + eng
submit/abandon eventsfunnel drop-off13 monthsanalytics

Decision record: the thing I require before collection:

decision: which fields users correct before submit
fields: field_id, edit_count, focus_ms, blur_ms
excludes: input values, key codes, inter-key timing
legal_basis: legitimate interest (assessed 2025-xx)
retention: 30d, enforced by <job>
access: role=analytics, logged
review: 2025-xx

If the requester can't fill that in, the answer is no.

Curated: · Written: · Reviewed:

Q&A-48The pricing module has become the tax on every feature: estimates went from 3 days to 9, and last quarter's four rollbacks cost 26 engineer-hours. Engineering says "give us capacity to pay down debt". What do you make visible before they get a single sprint, and how much is defensible?

Assumptions I'd state before anything else: six engineers on the team, a 13-week quarter, and "capacity" meaning a fixed slice of engineering weeks that is not available for feature commitments. If the team is a different size the arithmetic scales; the method doesn't change.

My answer: before they get a single sprint, I make the tax a priced line item with a named owner and an exit metric, and I defend a slice equal to the measured drag, not a round percentage someone picked in a planning meeting.

The mechanism is that debt is interest, and interest is measurable on work you already wanted to do. Three components: extra estimate days on every feature that touches the pricing module, rollback hours, and support volume attributable to that module. The question hands me two of them. Estimates went 3 days to 9, so 6 extra days per feature. Four rollbacks cost 26 engineer-hours, about 6.5 hours each.

Then the arithmetic. If 12 features a quarter touch pricing, that is 12 × 6 = 72 engineer-days of drag, roughly 14 engineer-weeks at 5 days a week. Against a 6-engineer, 13-week quarter, 78 engineer-weeks, that is about 18% of the team's capacity already being spent, invisibly, on that module. So a 15% slice, roughly 12 engineer-weeks, is not a gift; it is slightly less than what the debt is already charging. The figures below are illustrative, but the shape is what I'd put in front of the room.

LineFigure
features touching pricing per quarter12
estimate per feature3 d → 9 d (6 d extra)
quarterly drag12 × 6 d = 72 engineer-days ≈ 14 engineer-weeks
team capacity6 engineers × 13 weeks = 78 engineer-weeks
drag as share of capacity≈ 18%
defensible slice15% ≈ 12 engineer-weeks
after 3 sprints of contract tests9 d → 4 d; rollbacks 4/qtr (26 h) → 0

What I make visible, concretely: a debt line sitting in the same backlog as features, carrying a cost figure, an owner, and an exit metric. Lead time for changes touching the pricing module. Review the slice at the same cadence as features. The exit metric decides when it stops, not the calendar.

The failure mode I've watched: debt parked in "20% when we can" for three quarters with no owner and no exit metric. Estimates stayed at 9 days, Q3 committed 19 items and shipped 11, and two of the four rollbacks hit paying customers. Detection is the same chart, if lead time for changes in that module doesn't fall after three sprints, the slice isn't working and I'd stop it rather than renew it.

How much is defensible: the measured drag, capped by what the exit metric can retire. In the illustrative run above, three sprints of contract tests around the pricing interface took estimates from 9 days back to 4 and rollbacks from 4 a quarter to 0. That movement is what justifies renewing. Without it, 12 engineer-weeks is a nicer-sounding version of the same argument.

One check before I'd call it settled: the lead-time chart for changes touching that module, before and after the slice, on the same axes.

Curated: · Written: · Reviewed:

Q&A-36v1 must die. Engineering can delete it Friday. What does a TPM put in front of that delete?

The 90-second answer

A TPM puts a deprecation readiness review in front of the delete: a signed go/no-go that says every consumer of v1 is either migrated or has explicitly accepted the shutdown. The delete is the last line of that document, not the first.

The gate is not "v1 traffic is low." It's "every named consumer is at zero v1 calls, or has a dated acknowledgement that v1 is going away." Aggregate traffic is the wrong metric because 0.3% of requests can be 100% of one partner's nightly batch.

What's in the review

  1. Consumer inventory: every caller by identity, not by count. Gateway logs give you the easy ones; you also need the ones that don't show up: mobile app versions pinned to v1 that can't be force-updated, partner SDKs with a cached endpoint, internal batch jobs that run monthly, and anything behind a proxy that rewrites the user agent.
  2. Contract and notice check, which consumers have a contractual notice period (read the MSA; 30/60/90-day clauses are common), which are covered by public API terms, which are internal and can be moved by fiat. The notice clock starts when you send the dated notice, not when you decide.
  3. Per-consumer migration evidence: owner, last v1 call timestamp, v2 adoption confirmed by that owner. A dashboard is not a sign-off.
  4. Rollback and kill-switch: can v1 be restored, how fast, and for how long? If the answer is "we'd redeploy from git," that's the risk you're accepting, and it should be written down.
  5. Comms trail: dated notice to each consumer, escalation path, and named humans for the last stragglers.

Worked example (hypothetical)

Gateway shows v1 at 0.4% of requests, 11 days to Friday.

ConsumerOwnerNotice owedLast v1 callStatus
Mobile app v3.2 (pinned)Mobilenone (internal)todayforce-update prompt shipped
Partner A nightly batchPartner Eng90 days6 days agomigrated, awaiting one cycle
Internal reporting jobDatanone30 days agomigrated

The mobile row is the one that kills the Friday delete: 0.4% of requests is a live app version you can't turn off, and the fix is a client-side update with its own release lag. You slip the date or you accept breaking those users.

Failure modes and how you'd see them

  • Aggregate metric hides a single critical consumer → gate on per-consumer zero, not percentage.
  • Monthly or quarterly batch jobs invisible in a 7-day window → look back at least one full business cycle.
  • Notice period breached → legal and contract exposure; check before you announce, not after.
  • No rollback → an outage you can't stop. If v1 can't be restored, the delete needs a maintenance window and a named on-call owner.

What "ready" looks like

A one-page go/no-go: consumer table with per-consumer zero or signed acknowledgement, notice dates, rollback plan, named on-call, and a date. Engineering deletes when that page is signed, not before.

Curated: · Written: · Reviewed:

Q&A-75Support wants a global search across all tenants. Engineering sharded by tenant_id. What do you tell support the product can and cannot do?

What I tell support

Two sentences, and they should match in the ticket and the release notes:

  • Today, search works inside one tenant. Support can search any single customer's data by scoping the query to that tenant_id.
  • There is no cross-tenant search, and we are not going to add one to the sharded OLTP store. If a customer asks "find this email address across all accounts," the answer is no, not "not yet."

The workaround support can offer: run the search per tenant, or ask the data team for an export. If global search is a real product requirement, it gets funded as a separate search index, not as a query against the shards.

Assumptions

Sharded by tenant_id across N shards (say 40), OLTP store, no secondary index. A query that doesn't include tenant_id can't be routed to one shard.

Why global search can't just be a bigger query

A shard key is an access-path boundary. With tenant_id in the predicate, the router picks one shard and the query is a normal indexed lookup. Without it, the coordinator must fan out to all 40 shards, run the query on each, and merge the results.

Three things break:

  1. Fan-out cost. One logical query becomes 40 physical queries, each holding a connection for its duration.
  2. No global ordering or pagination. Each shard returns its own top-K; the coordinator over-fetches K per shard, merges, sorts, then slices the page. Page 10 is not "skip 90 rows": it re-runs the fan-out and discards.
  3. Tail latency. The query is as slow as the slowest shard. If one shard's p99 is 20 ms, the chance all 40 come in under 20 ms is 0.99^40 ≈ 0.67, so roughly a third of global searches exceed the single-shard p99. One straggler makes it worse.

The failure mode, with numbers

Hypothetical but representative: a scatter-gather search over 40 shards, each taking 15–25 ms, holds 40 connections per query. With a 20-connection pool, two concurrent searches saturate it and every checkout query queues behind them. That's how a search used 30 times a day takes down checkout: it isn't the search volume, it's the connection hold time.

PathShards touchedCheckout impactFreshness
Scatter-gather OLTPall 40pool contention, p95 spikesreal-time
Per-tenant search1nonereal-time
Async search index0 (OLTP)noneseconds behind

What to detect

If someone builds it anyway: watch pool wait time and the ratio of cross-shard to routed queries. A scatter-gather query in the slow-query log with a shard count of 40 is the signal.

The alternative

A search index (OpenSearch/Elasticsearch) fed by CDC from the shards. It's a separate system with its own cost and its own tenant-isolation problem: every document needs tenant_id and every query needs a tenant filter, or you've moved the leak, not fixed it. Staleness is seconds, which support must state.

So: per-tenant search is the product today. Global search is a funded index, not a query.

Curated: · Written: · Reviewed:

Q&A-76A user edits a doc and immediately opens it on their phone; the phone replica is 3s behind. What product behaviour do you specify?

The assumption that decides this: the edit is committed on the primary before the phone opens the doc, and the phone reads from a follower replica that trails by about 3 seconds. The behaviour I specify is read-your-writes across devices, with a bounded wait and an explicit fallback, not eventual consistency behind a spinner.

Concretely, the write path returns an opaque version token to the editing client: the doc's committed version, not a wall-clock timestamp, because timestamps don't order correctly across shards. That token rides the account sync channel, so the phone has it before or shortly after it opens the doc. The phone's read carries the token, and the read service compares it against the version the replica has applied:

  • replica version >= token: serve from the replica.
  • replica behind: wait up to a read budget (say 1.5s) for the replica to catch up, then serve.
  • still behind after the budget: read the doc from the primary, or return the stale copy with an explicit syncing state and re-fetch.

The budget is the load-bearing part. Unbounded waiting turns a 3-second lag into a hung screen. 1.5s is a starting guess I'd tune against the measured lag distribution; what matters is that the wait is bounded and the fallback is defined.

Always reading from the primary is the obvious alternative, and it's wrong here: every cross-device open becomes a primary read, which is exactly the load the replicas exist to absorb. Session stickiness doesn't help either: it pins the first device, and the phone is a different session.

Failure modes I'd design against:

  • The token never reaches the phone: push dropped, phone offline, cold app open. Then the phone has no token and must fall back to a primary read or show last-known content with a syncing indicator. Silent staleness in this path is what generates the support tickets.
  • Replica lag spikes past the budget under load. If every lagged read falls back to the primary, you've moved the thundering herd onto the primary. Cap the fallback rate and shed to the syncing state instead.
  • The user edits on the phone while it's showing stale content. Now there are two writes. Specify the merge (last-writer-wins on the doc version, or a CRDT if the doc is structured) rather than inheriting whatever the storage layer does by default.

Detection: count reads served stale while holding a token (should sit near zero), primary fallback rate, and p99 read latency. A read-your-writes test across two devices belongs in the launch suite: edit on device A, open on device B, assert the edit is visible within the budget.

StrategyPhone sees editCost
Lagged replica, no tokenno, up to 3s+cheapest, worst UX
Token + bounded wait + primary fallbackyes, within budgetprimary read on lag
Always read primaryyesprimary load on every open
Syncing banner, no tokeneventuallycheap, but user can't tell saved from lost

The last row is the one to avoid: a banner without a token can't tell the user whether their edit landed.

Curated: · Written: · Reviewed:

Q&A-77A write-back cache promises snappy saves. The node dies before flush. What does the product owe the user?

The short answer

The product owes the user three things the moment it shows a save confirmation: an honest statement of what that confirmation means, a bounded recovery point, and a recovery path. If the ack came from volatile memory, "saved" was never true, and the product owes the user that disclosure before the crash, not after.

Why the writes vanish

Write-back acknowledges a write once it lands in a volatile buffer: the OS page cache, an in-process map, a Redis memtable, a Postgres WAL buffer. Durability happens later, at flush: an fsync to disk, a WAL commit, or replication to a quorum. The gap between ack and flush is the loss window, and the flush trigger sets its size:

  • Time-based: Redis AOF with appendfsync everysec acks before fsync, and the docs bound the loss at roughly a second of writes. Postgres with synchronous_commit = off acks the commit before the WAL is flushed; the window is bounded by wal_writer_delay, default 200 ms.
  • Size-based: a memtable or write buffer flushes when it fills, so the window shrinks under load and grows when idle.
  • Explicit: fsync on every commit. The window is zero; the cost is the disk round trip on the write path.

Node death is power loss. The volatile buffer is gone, and everything acked inside that window is gone with it. Nothing in the cache can recover it.

What changes the answer

Replication. If the ack waits for the write to land on a quorum, say 2 of 3 replicas, a single node death loses nothing, because the survivors hold the write. If the ack fires on local memory only, the whole window is exposed. So the obligation depends on where the ack point sits, and that has to be a deliberate choice per write class, not one global default.

What the product owes, concretely

  1. Ack semantics in the UI and API. "Saved to this device" and "saved to the server" are different promises. A checkmark that means "in memory" is a lie the user pays for.
  2. A stated RPO per write class. Billing and legal text: zero loss, write-through or synchronous_commit = on. Drafts and telemetry: a bounded window is acceptable if it is stated.
  3. A recovery path. A local journal the client replays on reconnect, or a server-side WAL the node replays on restart. Without one, the window is permanent loss.
  4. A safer path for high-value writes, so the user is not forced to trade snappy against safe.
Write classAck pointRPORecovery
Billing, legalfsync / quorum0n/a
Draft autosavememory, 1 s flush≤ 1 sclient journal replay
Telemetrymemory, size flushbuffer-boundeddrop, acceptable

Arithmetic. At 500 writes/sec and a 1-second flush interval, up to 500 acked writes sit in the window. Replicate to a quorum and that number is zero; the cost is one extra network round trip per write, not a redesign.

Failure modes to name. The ack lies: a checkmark that means memory. The flush interval is tuned for throughput and never revisited. No local buffer, so the user's last minute of typing is unrecoverable. Each is detectable: log the ack point per write class, alert on flush lag, and run a chaos test that kills a cache node mid-write and asserts the RPO.

Snappy and saved are different claims. The product owes the user the difference, stated up front.

Curated: · Written: · Reviewed:

Q&A-78Welcome email is fired from a queue with at-least-once delivery. 7% of new users get two welcomes. Is that a platform bug or a product bug?

Verdict: platform bug. At-least-once is the queue's contract, and the platform owns the job of not turning a retry into a second user-visible send. The product's only obligation is to name the logical message, one welcome per signup, so the platform has something to dedupe on. If the product never declared that key, that's a requirement gap, but the defect that ships two emails lives in the send path.

The 7% is the tell. A genuine crash window, worker dies after the provider accepts but before the ack lands, is rare, well under 1% and clustered around deploys. A steady 7% is a timeout misconfiguration: the message becomes visible again before the worker can ack it.

Trace it. Visibility timeout is 30s. Under load the provider send takes 45s at p99. The worker receives the message, calls the provider, the provider accepts at t=45s, the worker acks, but at t=30s the message already went back on the queue and a second worker picked it up. Two sends, one signup. If 7% of sends exceed 30s, you get exactly 7% duplicates, and the rate climbs with load. That is a platform config bug, not a product decision.

CauseSignatureFix
Visibility timeout < send latencysteady %, scales with loadraise timeout or extend the lease while sending
Crash between send and ackrare, clusters on deploysidempotency key on the send
Ack before sendduplicates only on retry after failureack after send, plus a key

The fix is a stable idempotency key for the logical message, not a time-window dedupe. A window fails the moment a retry is delayed past it: a backed-up queue or a slow provider pushes the second attempt outside the window and the duplicate ships anyway. Key on the signup ID, write one welcome intent per signup in a transactional outbox, and enforce uniqueness on that intent. Every retry reuses the same key. If the email provider offers an idempotency contract, pass the key through and the provider collapses the retries for you. If it doesn't, dedupe at the consumer before the send and reconcile the provider receipt after an ambiguous timeout.

Do not fix this by turning retries off. Retries are what deliver the welcome to the users whose first attempt failed; disabling them trades a 7% duplicate rate for a silent drop rate you won't see in any dashboard.

What I'd check before calling it closed: a crash test that kills the worker after provider acceptance but before the ack, which should still produce exactly one delivered welcome; a duplicate-send counter on the send path; and, if the welcome carries a discount code, a single-redemption constraint on the code itself, so a duplicate that slips through can't be spent twice. The queue is allowed to retry. The send path is what has to be idempotent.

Curated: · Written: · Reviewed:

Q&A-79A chatbot "must stick to one box" for memory. Horizontal scaling is now sticky sessions. What product alternative do you offer?

Assumptions I'd state first: one region, the app already issues a session id, a chat turn is a few KB of text, and the traffic is read-heavy with occasional writes. Under those, the alternative is to move conversation state out of the process into a shared store keyed by session id, so any node can serve any turn and stickiness stops being a scaling requirement.

The core answer: put the transcript in Redis (or DynamoDB if you want a managed, multi-AZ store with no ops), keyed by session id, with a TTL. Sticky routing then becomes an optional latency optimisation, not a correctness constraint.

Mechanism. Each turn does a read-modify-write on one key:

import json, redis
r = redis.Redis(host="redis.internal", port=6379, decode_responses=True)

def append_turn(session_id, turn, ttl=1800):
    key = f"chat:{session_id}"
    pipe = r.pipeline()
    pipe.rpush(key, json.dumps(turn))
    pipe.expire(key, ttl)
    pipe.execute()

Trace: the pipeline queues RPUSH then EXPIRE and sends both in one round trip. RPUSH creates the list if the key is absent and appends the serialised turn; EXPIRE sets a 1800-second TTL on the key. Any node running this produces the same result for the same session id, which is the whole point.

Sizing. 8,000 concurrent sessions at ~4 KB of turns each is ~32 MB resident: Redis holds that in RAM without noticing. The 30-minute TTL means idle sessions evict themselves, so the working set tracks live traffic rather than total users.

Consistency. Last-write-wins per session is fine when one user drives one session. If two turns can arrive concurrently for the same session, serialise them: a short SET NX lock on lock:{session_id}, or route a session to a single worker via a consistent hash so writes never race.

Alternatives, and when each wins:

OptionSurvives node deathCostWhen to pick
In-process + stickyNo: session dies with the nodeCheapest per turnSingle node, or state is disposable
External store (Redis/DynamoDB)YesOne network hop per turn (~0.5–1 ms in-AZ)Default for horizontally scaled chat
Client-held signed tokenYesToken grows with transcript; ~4 KB cookie/header ceilingShort, stateless exchanges
Replicated in-process (gossip/CRDT)Partly: converges, not instantComplex merge logicYou already run a gossip layer

Failure modes to name. Redis without persistence loses sessions on restart: enable AOF or run a replica with automatic failover, and expect a few seconds of unavailability during promotion. Memory pressure evicts keys silently: watch evicted_keys and set maxmemory-policy deliberately. A client-held token that outgrows the header limit truncates the conversation: detect it as a 431 or a parse error, not a silent drop.

Deciding criterion: if autoscaling or a rolling deploy must not drop a conversation, externalise. If the state is genuinely disposable and you run one node, stickiness is fine.

How I'd validate it: kill one app node mid-conversation and confirm zero lost turns; then check p99 turn latency against the in-process baseline to price the extra hop.

Curated: · Written: · Reviewed:

Q&A-80The app retries 429s with no jitter and no message. Users mash the button. What is the product design for overload?

Core answer

First I'd pin down where the 429 comes from, because the design differs. If it's a per-user quota, the fix is local: tell that user when they can retry. If it's a shared dependency shedding load, every client is competing for the same capacity, and the client's retry policy is now part of your overload control. I'll assume the second, a shared backend at capacity, because that's the one where mashing makes things worse.

The product design has three parts. Surface the wait: read Retry-After and show "busy, try again in 12s" with a live countdown, not a spinner. Back off with jitter: the client retries on a randomised schedule, never on a fixed interval. Disable the control: the button goes to a disabled state with the countdown, so mashing does nothing. Queue only idempotent actions: a "save draft" can be queued and replayed; a "pay now" cannot, because a queued retry that lands twice is a double charge.

Why the storm compounds

Two effects multiply. The first is synchronisation: if every client retries on a fixed 1s interval, their retries land in the same second, so retry traffic is a spike, not a trickle. The second is the feedback loop: the spike gets shed with more 429s, and every shed request still costs a connection, a TLS handshake and some CPU before it's rejected, so the server spends capacity rejecting work instead of serving it. Mashing adds a third: each tap is another request from a client that's already in the retry loop.

Jitter breaks the first effect. Instead of a spike at t=1, 2, 3, the herd spreads across the interval, and exponential growth means each successive wave is smaller.

Trace (illustrative: capacity 250 rps, baseline 200 rps, so 50 rps of headroom; 1000 clients in the retry loop)

AttemptClients retryingFixed 1s intervalFull jitter, cap 30s
01000~1000 rps at t=1~1000 rps spread over 1s
1750~1000 rps at t=2~375 rps over 2s
2560~1000 rps at t=3~140 rps over 4s
3420~1000 rps at t=4~52 rps over 8s

A fixed interval keeps offering ~1000 rps against 250 rps of capacity, so the server never drains. Jitter drops the added load under the 50 rps headroom by the third attempt, so the queue drains and the 429s stop.

Trade-offs, and when not to do this

Disabling the button is wrong when the action is the user's only path and the wait is long: a disabled "confirm payment" for 30s reads as a broken app and drives abandonment. Better to keep it enabled but rate-limited client-side, or to put the countdown on the button itself. Queueing is wrong for anything non-idempotent or time-sensitive (payments, seat holds, inventory reservations) because replaying it later can act on stale state. And honouring Retry-After is wrong if the server sends a value longer than the user will wait; cap it and tell the user honestly rather than freezing the UI for five minutes.

Failure modes

  • Retry-After missing or absurd: the server sends no header, the client falls back to a fixed 1s and re-synchronises. Detect by logging the distribution of client retry delays; a spike at exactly 1s means jitter isn't applied.
  • Queue replay storms: the offline queue flushes all at once when connectivity returns. Detect by measuring QPS at reconnect; it should be rate-limited, not a step function.
  • Double-submit: disable is client-side only, so a second tab or a retried request still lands. Detect with idempotency keys and a server-side dedupe window.

Evidence I'd want before calling it done

A client policy written into the PRD (base, cap, jitter function, max attempts) and a load test that replays a 429 storm and asserts offered QPS stays under 1.2× capacity. Without that, it's a spinner with good intentions.

Curated: · Written: · Reviewed:

Q&A-81Marketing says "your data stays in the EU". The analytics pipeline is in us-east-1. What do you change before the campaign?

Open with the decision, not the options. The sentence is a commitment about where EU personal data is processed and stored, so before the campaign I need to know what "your data" covers (event payloads, identifiers, logs, backups, and support access) then either make the claim true or change the claim. My default: move the analytics pipeline into an EU region if it fits the timeline; tokenize only if the residual data is genuinely non-identifying; otherwise change the copy. I would not ship the sentence while the pipeline is in us-east-1.

Assumptions: EU data subjects, GDPR applies, the pipeline processes personal data (user IDs, emails, IPs, event payloads), and the campaign is public marketing copy.

Option 1: move the pipeline. Stand up the same stack in eu-west-1 or eu-central-1: ingestion (Kinesis or MSK), object storage (S3), query (Athena or Redshift), and the scheduler. This is the only option that makes the claim true end to end. Cost: AWS inter-region transfer is $0.02/GB on list price; at 2 TB/day that is ~$40/day, ~$1,200/month, plus duplicated storage and a second set of infra to operate. Timeline: weeks, not days.

Option 2: tokenize before egress. Replace direct identifiers with a surrogate key and keep the mapping table in the EU. What it leaves exposed: the surrogate key is pseudonymous data and still personal data under GDPR if re-identification is possible; quasi-identifiers (timestamps, device, geo, event order) can re-identify; and the tokenization service's own logs, error traces, and the mapping table's access path. Tokenization is a residency fix only if the mapping never leaves the EU and the residual is genuinely non-identifying: otherwise it is a transfer with extra steps.

Option 3: change the copy. Hours, free, and honest: "processed in the EU" or drop the claim. This is the right answer when the timeline cannot absorb a migration.

OptionCostTimelineSatisfies the claim?
Move pipeline~$1.2k/mo transfer + duplicate infraweeksyes, fully
Tokenizelowdaysonly if residual is non-personal
Change copynonehoursyes, by not claiming

Failure modes. The pipeline move does not move the logs: CloudWatch, Sentry, Datadog, and the CI that runs the job are often still US. Backups, snapshots, and DR replicas default to the source region. Third-party analytics (Segment, Amplitude) have their own US sub-processors. And US-based support access to EU data is itself a transfer under Schrems II. Detection: a data-flow inventory that lists every destination, reviewed on a schedule. The recurring cost is keeping it current, not building it once.

The inventory that decides it:

FlowRegionPromise holds?
app DBeu-west-1yes
analytics pipelineus-east-1no
CloudWatch logsus-east-1no
backupsus-east-1no
support accessUS engineersno

Fix the "no" rows or change the sentence.

Curated: · Written: · Reviewed:

Q&A-82Relevance wants a 3-stage ranker (p95 900ms). The search SLO is 200ms. Who wins, and how?

The SLO wins. A 3-stage ranker at p95 900ms cannot serve a synchronous search path with a 200ms SLO. Relevance wins only by moving the expensive stage off the critical path or shrinking what it sees. If the business wants the 900ms ranker on the critical path, that is a decision to change the SLO to roughly 1s, not an engineering trick.

Assume the 900ms is end-to-end p95 on the full candidate set, and stage 3 dominates. If stage 1 retrieval already costs 180ms, subset reranking will not save you; you need a faster retriever or a different SLO.

A 3-stage ranker is a funnel. Stage 1 retrieves candidates cheaply: inverted index or ANN over millions, returning about 1000. Stage 2 scores those with a light model, returning about 100. Stage 3 reranks the 100 with the expensive model. Cost compounds because each stage adds latency, and the final stage sees the full candidate set unless you prune. The p95 is set by the slowest stage plus queueing. If stage 3 is 800ms for 1000 docs, reranking 100 docs is about 80ms when per-doc cost dominates, giving roughly 180ms total. If stage 3 has a 200ms fixed model-load or feature-fetch cost, 100 docs is 280ms and misses. Measure per-stage p95 and candidate counts, not just end-to-end.

ApproachHypothetical p95RelevanceUse whenFails when
Full 3-stage sync900mshighestSLO can be about 1sSLO is 200ms; p95 tail breaks the contract
Subset rerank: stage 3 on top 100180msnear-full if top 100 contains the winnersstage 3 per-doc cost dominates and recall@100 is highfixed stage cost exceeds budget; top 100 misses a relevant doc
Async polish180ms first paint, later updatehigh eventuallyuser can see initial results and the page updatesfirst paint relevance matters; layout shift; no client support
Cached or precomputed40mshigh for head querieshead queries repeatlong tail; cache miss

Choose subset rerank when recall@K is within 1–2 points of full rerank and stage 3 per-doc cost dominates. Choose async when the first 200ms only needs to be useful, not final. Do not choose async if the result order changes after paint and the user has already clicked; that is a correctness bug for search.

Concrete failure: stage 3 times out at 200ms and the service returns empty because no fallback exists. Detect it with per-stage p95/p99, timeout rate, fallback rate, and candidate-count histograms. Another failure: p95 is fine but p99 is 2s because one shard is slow; the SLO is p95, but users still see it. Another: subset rerank drops a relevant doc from top 100; measure recall@100 against full rerank offline.

The check that decides it: per-stage p95 and recall@K. If stage 3 on top 100 keeps recall within 1 point and total p95 is 180ms, ship it. If not, async or renegotiate the SLO.

Curated: · Written: · Reviewed:

Q&A-83The next-page request returns an expired-cursor problem after two hours. What should the product do?

What to do

Return a distinct, documented error for the expired cursor (410 Gone, or 400 with a stable problem type like cursor_expired) and have the client restart the query from page 1 with a message. Never surface it as 401: that code means the caller's credentials failed, and clients, SDKs and gateways act on it accordingly (token refresh, logout, alerting).

Response body:

{
  "type": "https://api.example.com/problems/cursor-expired",
  "title": "Cursor expired",
  "status": 410,
  "detail": "This page token expired after 2 hours. Restart the listing from the first page."
}

Client behaviour: catch cursor_expired, drop the stored cursor, re-issue the first-page request, and show "Results refreshed. Showing from the top." Don't retry the same cursor; it fails identically.

Why the cursor expires at all

Two common implementations, and which one you have decides what recovery you can offer:

  • Signed stateless cursor: the token carries the sort key or offset plus an exp claim, signed with HMAC. The server holds no state, so it cannot extend the deadline: expiry is baked into the token. Cheap and horizontally scalable, but the only recovery is minting a new one.
  • Server-side snapshot cursor: the server keeps a result set or MVCC snapshot and hands out an opaque id, garbage-collected after a TTL to bound memory. Here you can extend on use (sliding TTL) or resume, at the cost of holding state per open listing.

Why restart-from-page-1, not resume

A cursor pins a consistent view. After two hours the underlying rows have changed, so resuming at "offset 400" against fresh data skips or duplicates rows. Restarting gives the user a coherent page 1. If you genuinely need resumability, that is a different design: keyset pagination, where the cursor is just the last sort key (WHERE (created_at, id) < (?, ?)). It needs no server state and does not expire, but it also cannot be revoked, and it leaks the sort key.

Options and when each wins

OptionRecoveryCostChoose when
2h signed cursor, restart on expirynew page 1no server statedefault for feeds; users rarely page past 2h
Sliding TTL on server-side cursorresume same pagememory per open listinglong-lived exports, admin tables
Keyset cursor, no expiryresume exactlyno revocationstable, non-sensitive sort key
Longer TTL (e.g. 24h)resumestale snapshot, memoryonly if staleness is acceptable

Deciding factors: how long a real session pages (measure it, if p99 is under 10 minutes, 2h is generous), whether the sort key is stable and non-sensitive, and whether you can afford per-listing state.

Failure modes to watch

  • 401 on expiry: clients attempt token refresh or log the user out, and users read it as a security event. This is the mistake to avoid.
  • Silent empty page: returning 200 with [] makes the client think the list ended. Users lose their place with no explanation.
  • Offset resume: duplicate or skipped rows, surfacing as "I saw this item twice" reports.
  • Clock skew on signed cursors: a token minted by one node can look expired on another; keep the TTL well above skew and validate exp with a small leeway.

What to write down

Put the TTL and the recovery behaviour in the API contract and the PRD: the exact status, the problem type, and "client restarts from page 1." Test at TTL+1s, not just at TTL.

Curated: · Written: · Reviewed:

Q&A-84A 10-minute video export has a spinner and no percent. 22% of users start a second export. What do you add?

Two things are broken: the user can't tell whether the export is working, and the server lets them start the same work twice. I'd fix both, because progress alone doesn't stop the 22%: someone who watches a spinner for 90 seconds still hits Export again.

Assumptions: the export is a server-side job that survives a page reload, and the client can reach a status endpoint. If it's purely client-side, the answer changes.

What I'd add.

A job record with a state machine: queued → probing → transcoding → muxing → uploading → done | failed | cancelled. Every transition writes to the job row.

Progress comes from the one stage that has a real denominator:

StageProgress sourceNotes
probestage onlyseconds
transcodeout_time_ms / duration_msthe only real denominator
muxstage onlyseconds
uploadbytes_sent / content_lengthreal denominator

ffmpeg reports out_time_ms as it encodes, so the transcode percent is honest. Probing and muxing take seconds: show them as stages, not as invented percentages.

Delivery: GET /exports/{id} returns {state, stage, percent, eta_seconds}. The client polls every 3s with jitter, or subscribes over SSE if you want push.

Polling versus SSE is arithmetic. 100k concurrent exports polling every 3s is ~33k rps on the status endpoint: servable as a cache read, but real capacity. SSE holds 100k connections and pushes only on stage change, a handful of messages per job. I'd start with polling at 3s and move to SSE when the endpoint's p99 or the DB read load says so.

ETA: don't extrapolate linearly from percent. Early estimates are garbage because the transcode rate isn't constant. Use stage-weighted remaining time from historical p50 per stage for that preset, and show a range ("about 6–9 min left"), updated as stages complete.

Cancel: cooperative. Set cancel_requested on the job row; the worker checks it on a timer and sends SIGTERM to the ffmpeg child. A hard kill leaves a partial file, so the worker deletes the temp directory on cancel.

Completion notification: in-app toast, plus email or push if the tab is gone. That is what actually kills the second export: the user closes the tab and gets told when it's done.

The dedup, which is the part that stops the 22%.

An idempotency key alone doesn't do it. The client generates a UUID when the user clicks Export and sends it as Idempotency-Key; the server has a unique index on (user_id, key) and returns the existing job on a repeat. But a user who reloads the page gets a fresh key, so the key only dedups retries of the same click.

The real guard is a fingerprint: hash(user_id, source_asset_id, output_preset). A partial unique index on (user_id, fingerprint) WHERE state IN ('queued','running') means a second POST while one is in flight returns the in-flight job instead of creating a new one. The client also calls GET /exports?active=true on load, so the Export button becomes "Exporting: 40%" rather than a fresh button.

Failure modes I'd watch.

  • Percent going backwards. ffmpeg's out_time can stall or regress on variable-frame-rate input. Clamp to monotonic in the writer.
  • SSE behind a proxy that buffers or times out. Reconnect with Last-Event-ID and fall back to polling.
  • The fingerprint index blocking a legitimate re-export after a failure. Scope it to active states only, so a failed job doesn't lock the user out.
  • Cancel that doesn't cancel: if the worker only checks between stages, a 9-minute transcode ignores cancel for 9 minutes. Check the flag on a timer and kill the child.

Success metric: duplicate-start rate under 2%, measured as second POSTs that hit the fingerprint index, not as UI clicks.

Curated: · Written: · Reviewed:

Q&A-85Enterprise wants "our data never touches another tenant's disk". You run a shared DB with row-level security. What do you sell?

What I'd say first

"Never touches another tenant's disk" is a physical claim. Row-level security is a logical one. On a shared Postgres cluster you cannot sell the physical claim, because it is false: every tenant's rows live in the same relation files under PGDATA/base/<dboid>/<relfilenode>, the same 8 KB heap pages, the same WAL segments, and the same base backups. RLS decides which rows a query returns; it does not decide which bytes sit on which disk.

So what you sell is a ladder of isolation levels, each with a test you can run in front of the customer, and you sell only the rung the architecture actually delivers.

How RLS works, and where it stops

-- Postgres 15
ALTER TABLE events ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON events
  USING (tenant_id = current_setting('app.tenant_id', true)::uuid);

With RLS enabled and no matching policy, the default is deny: the query returns zero rows. The policy is injected as a filter into the plan, so SELECT * FROM events becomes ... WHERE tenant_id = current_setting(...). That is authorization. The two-argument current_setting matters: an unset custom GUC returns NULL instead of raising, and tenant_id = NULL is never true, so a request that forgot to set the tenant reads nothing rather than erroring or leaking.

The rows of every tenant are still interleaved on the same pages, so a page read for tenant A can physically contain tenant B's tuple; the filter discards it before it reaches the client.

The ladder

LevelPhysically separateClaim you can defend
Shared cluster + RLSnothing"queries are filtered by tenant_id"
Dedicated schema or databasenamespace only"separate schema; same disk, WAL, backups"
Dedicated cluster + per-tenant KMS keyvolume, WAL, backups, key"your bytes sit on a volume only your cluster mounts; backups encrypted under your key"
Dedicated account/VPCnetwork, IAM, support path"no shared control plane, no cross-tenant admin tooling"

Failure modes and how you detect them

  1. Pooled-connection GUC leak. SET app.tenant_id is session-scoped. PgBouncer in transaction pooling does not run server_reset_query between checkouts, so the setting survives into the next tenant's request: request B runs SELECT * FROM events with no SET, the policy evaluates against tenant A's value, and B reads A's rows. Use SET LOCAL inside a transaction, and test it by running two tenants back-to-back on one pooled connection and asserting zero rows for the second.

  2. Owner and BYPASSRLS. The table owner bypasses its own policies unless ALTER TABLE events FORCE ROW LEVEL SECURITY; superusers and any role with BYPASSRLS bypass them always. Detect in CI with SELECT rolname FROM pg_roles WHERE rolbypassrls; and SELECT relforcerowsecurity FROM pg_class WHERE relname = 'events';.

  3. Restore. PITR restores the whole cluster, so a restore for tenant A materializes tenant B's rows on the same disk. Clean runtime isolation does not make the disk claim true.

When dedicated is not worth it

Illustrative: a dedicated 2 vCPU / 8 GB Postgres node has a floor around $120–150/month before any load. At 300 tenants that is $36k–45k/month of mostly idle capacity against one shared cluster serving the same traffic. Reserve dedicated clusters for the tenants whose contract pays for the floor; sell the rest the RLS rung with the test evidence attached.

Curated: · Written: · Reviewed:

Q&A-87A customer wants 10× quota "for a launch week". The default is 100 rpm. What is the product process?

Treat this as a time-boxed capacity commitment, not a config change. The 10× is the easy part. The work is confirming you can absorb 1,000 rpm from that key without degrading every other tenant, and that the grant reverts on its own when the week ends.

Assumptions I'd state up front: the 100 rpm default is per API key, enforced by a token bucket at the edge; the customer's traffic is spiky (a launch-day spike, then decay), not a flat 7 days; and raising the limit only stops rejections. It adds no capacity. So the real question is headroom, not the multiplier.

The process, in order:

  1. Qualify the request. Ask for the traffic shape: peak rps, which endpoints, one key or many, and the exact date range. "Launch week" almost always means a 2–4 hour spike on day one. A flat 10× for 7 days is a different ask from a 10× ceiling for 4 hours.
  2. Check headroom. 10× of 100 rpm is 1,000 rpm ≈ 16.7 rps. If the fleet serves 400 rps at 60% CPU, 16.7 rps is noise. If it's at 85%, it isn't. The number that decides the answer is spare capacity, not the ratio.
  3. Pick the mechanism. These are not equivalent, and the choice is the actual product decision:
MechanismWhat it doesWins when
Temporary tier overrideRaise the key limit to 1,000 rpm, auto-revert at TDefault: short, bounded, existing customer
Paid burst add-onSame limit, billed as a SKUCustomer will pay; you want the incentive aligned
Overage billingNo hard cap, metered per 1k callsTraffic shape unknown; customer accepts a variable bill
Reserved capacityDedicated pool or separate key shardSustained volume; noisy-neighbor risk is real
429 + Retry-AfterDeny over limit, tell the client when to retryNo headroom; forces client-side backoff
  1. Price it. A burst has a cost: CPU, egress, support. Either the customer pays (add-on, overage) or you book it as launch marketing spend. "Free and open-ended" is the option that quietly becomes a new plan.
  2. Set expiry and auto-revert. The override carries a timestamp; a job reverts the key to 100 rpm at T. No human step, because the human step is what gets skipped.
  3. Instrument. Per-key request rate, 429 rate, per-tenant CPU share, and p95/p99 latency broken out by tenant. Alert when the revert fires.

Failure modes. The one I've seen (hypothetical figures): an open-ended 10× left on, that tenant grew to 40% of API CPU, and other tenants' p95 doubled. It was invisible on the aggregate dashboard: only a per-tenant CPU-share graph showed it. Second: expiry set but no alert on revert, so nobody notices it didn't fire. Third: no per-key isolation, so one tenant's burst is everyone's latency.

The artefact that closes this is a ticket with an expiry timestamp and a per-tenant CPU graph attached, not a Slack yes.

Curated: · Written: · Reviewed:

Q&A-94RICE reach is 2×10^6. The feature is unusable with a screen reader (0.8% of users, high harm). How does reach change the priority?

The 60-second answer

Reach doesn't change the priority: that's the finding. The 16,000 screen-reader users are already inside the 2×10^6. Segmenting them out adds no reach; it shows reach is the wrong lever. RICE multiplies a count of users by a per-user Impact, so a small segment with catastrophic harm gets averaged into the majority's mild benefit and vanishes. The score stays near the top, the feature ships, and 16,000 people can't complete the journey. The fix isn't re-weighting reach: it's making "completes with a screen reader" a constraint on the score rather than an input to it.

Why reach hides it

RICE = (Reach × Impact × Confidence) / Effort. Baseline: Reach 2×10^6, Impact 2, Confidence 0.8, Effort 4 person-months.

(2×10^6 × 2 × 0.8) / 4 = 800,000

Now segment reach by capability. 0.8% of 2×10^6 = 16,000 blocked users. A blocked journey is worse than no benefit, so score it negative: say −3:

SegmentUsersImpactContribution
Everyone else1,984,000+23,968,000
Screen-reader users16,000−3−48,000
Weighted2,000,0001.963,920,000

Weighted Impact = 3,920,000 / 2,000,000 = 1.96. New score = (2×10^6 × 1.96 × 0.8) / 4 = 784,000.

784,000 against 800,000 is a 2% drop. The item is still at the top of the backlog. That's the mechanism: reach counts users touched, not harm done, and a −3 averaged into 1.98M users at +2 is arithmetic that cannot see 16,000 people. Intercom's original Impact scale runs 3 (massive) down to 0.25 (minimal): there is no negative rung, which is the point. The −3 is a modelling choice; the conclusion survives any negative value, because the segment's weight is 0.8% of the total.

What to do instead

Two moves, and they are different:

  1. Constraint, not score. "The journey completes with a screen reader" is a gate. Fail it and the item is not Now, whatever the score. This is the right default when the harm is exclusion of a protected class: in the US, ADA Title III website litigation; in the EU, the European Accessibility Act (Directive (EU) 2019/882), applicable from 28 June 2025. A legal requirement is not a term you trade against reach. WCAG 2.2 AA is the usual conformance target to name.
  2. If it stays in the score, score the worst segment. Use min(Impact) across segments, or add a harm term that reach cannot dilute. Either way the 16,000 stop being invisible.

When a veto is disproportionate

A hard veto is wrong when the "unusable" claim is unverified, when a compliant fallback exists (an assisted channel, a phone line), or when the feature is internal-only and no external user is excluded. Then sequence: ship to the majority behind a flag, and put the accessibility fix on a dated fast-follow. What you don't do is ship and call 0.8% a rounding error.

Failure modes and how you'd see them

  • Averaging. The score barely moves while a whole segment is blocked. Detect by segmenting reach by capability and tracking completion rate per segment, not the aggregate.
  • Double-counting. Adding the 16,000 as extra reach. They're already in the 2×10^6.
  • Confidence inflation. 0.8 confidence on a journey never run with a screen reader. Run it in CI with an automated axe pass plus one manual screen-reader pass.

Reach tells you how many people the feature touches. It says nothing about whether it works for them.


Self-assessment

Key points a strong answer hits

  • Reach is a count of users touched, not a measure of harm, and the segment is already inside it.
  • The arithmetic: weighted Impact 1.96, score 784,000 against 800,000, a ~2% drop.
  • The right fix is a gate on the score, not a re-weighting of reach.
  • Names when the veto is disproportionate and what to do instead.

What interviewers look for

Whether you spot that the question is a trap about the metric's shape rather than about the number; whether you can do the weighted-average arithmetic out loud; whether you separate "constraint" from "score"; whether you can name a real accessibility standard or regulation without overclaiming.

Weak-answer patterns

  • "Increase reach to include accessibility users": they're already counted.
  • "Lower the Impact of the majority": arbitrary, and it doesn't fix the invisibility.
  • "File a separate accessibility ticket": no gate, no date, ships broken.
  • Treating RICE as broken rather than as a tool with a known blind spot.

Follow-ups

  • "What if the segment were 20% of users?": 0.8×2 + 0.2×(−3) = 1.0, so the score halves to 400,000 and the drop is visible. Same mechanism, less dilution.
  • "How would you set Impact for a blocked user?": any negative value; show the score is insensitive because the weight is 0.8%.
  • "Where does the gate live?": definition of done and release checklist, with an automated check in CI.
  • "What about a feature only internal users touch?": no external exclusion, so sequence rather than veto.

Curated: · Written: · Reviewed:

Q&A-95A customer wants to turn off MFA "just for the pilot". Who can accept that, and what expires?

What a strong candidate says first

The customer can request this; they can't accept it on your behalf, and you can't accept it on theirs. Risk acceptance sits with whoever owns the accounts. If the pilot runs in the customer's tenant, that's the customer's named risk owner: a person, in writing. If it runs in yours, it's your business owner for that tenant, with security holding a veto rather than the pen. A pilot that is genuinely isolated: its own tenant, no production data, no shared identity with the rest of the estate: can be accepted by the pilot owner inside that boundary. The moment it touches production accounts, the production system owner signs.

What expires is the exception, not the pilot. A fixed calendar date, plus the scope it was granted for. When the date passes, MFA comes back on for whatever is still in scope, whether or not anyone remembered.

Why it's a record, not a toggle

Turning MFA off is a risk acceptance with a blast radius. Treat it as a product record with an owner and an end date:

FieldExampleWhy it matters
Risk ownernamed person, business owner of the appa team can't be accountable
Scopetenant ID + group, 25 named users"the pilot" is not a scope
Compensating controlIP allowlist, 15-min session timeout, alert on new devicereduces the risk, doesn't remove it
Expiryfixed datethe thing that actually expires
Revisit triggerpilot exit review, or headcount > 50catches drift before the date

Alternatives, and when to pick them

A separate pilot tenant is the strongest option: the exception never touches production identity, so the blast radius is the pilot itself. The cost is duplicated users and a second set of policies to keep in sync. A scoped exemption in the main tenant: a Conditional Access policy excluding a pilot group in Entra ID, an enrollment policy on a group in Okta: is cheaper and keeps one identity store, but the exemption now lives next to production accounts and can be widened by anyone who adds a user to the group. Pick the separate tenant when the pilot handles real customer data; pick the scoped exemption when it doesn't and you can monitor the group's membership.

The failure mode: no expiry

No major IdP expires an MFA exemption for you. Entra ID Conditional Access policies and Okta enrollment policies have no expiry field: the date is a process you run, not a setting you flip. So the failure is drift. Worked example, illustrative: a 25-user pilot gets MFA off for 30 days. Nobody sets an expiry. Fourteen months later the tenant has grown to 8,000 users, MFA is still off for all of them, and one of those accounts is used in an incident. The exception ended up covering 0.3% of the population it applied to.

How you'd catch it

An exception register with age alerts: warn 14 days before expiry, auto-revert on the date, and a monthly report of every open exception with its owner. If the register can't tell you how many accounts are currently exempt, you don't have one.

Curated: · Written: · Reviewed:

Q&A-97You dual-write to old and new billing for 6 weeks. The two systems disagree on 1.8% of invoices. Who is source of truth for the user?

Assume dual-write means every invoice write goes to both the legacy billing system and the new one, and the new system is still being validated. Then the source of truth for the user is old, for the entire six weeks. The user sees one number, and it comes from old. New is a shadow: it receives writes and gets compared, but it never renders a number to a customer until it has earned that.

The one exception: if a cohort is already native to new, say accounts created after the migration start that never existed in old, then new is their source of truth. What you must never have is two sources of truth for the same invoice.

Why they diverge

Dual-write is two independent writes with no shared transaction. There is no atomic commit across both, so divergence is structural, not a bug you can fully eliminate:

  • old write commits, new write times out → new is missing the invoice
  • retry after a timeout → duplicate in one system
  • different rounding, tax, or proration rules between engines → same inputs, different cents
  • currency conversion at different timestamps → different amounts

1.8% is the visible rate of all of that. Reading from new would put the unvalidated side in front of the customer, so you read from old.

Deciding criterion: whichever system actually moves money (captures payment, issues refunds, runs dunning) is the source of truth. The other is shadow. During a migration that is old.

Alternatives

ChoiceWhen it's rightCost
Old is SoT, new shadowDefault for a 6-week validationRun both, reconcile daily
New is SoT, old shadowOnly when new already owns payment capture and old is being retiredAny new-side bug hits users
Show both numbersNeverSupport incident, trust loss
Split by cohortOnly on a stable key (account creation date), never per invoiceTwo code paths

Reconciliation, concretely

A nightly job joins both systems' invoice tables on invoice_id and compares amount_cents, currency, and status. Mismatches go to a queue, classified by cause. Gate cutover on the mismatch rate staying below 0.1% for seven consecutive days, not one day.

Arithmetic: at 50,000 invoices/month (hypothetical), 1.8% is 900 wrong invoices/month, about 30/day. Too many to hand-review, so classification has to be automated. Getting to 0.1% means 50/month, which is reviewable.

Failure modes and detection

  • Divergence creeping up → daily mismatch-rate trend, alert on a 3-day rise
  • Double charge from retry → duplicate invoice_id within one billing period
  • App/PDF mismatch → render both from the source of truth and diff
  • Premature cutover → the 7-day gate

Correction policy: if a user was charged the wrong amount, correct to old's figure, issue an adjustment, and notify. Never silently rewrite history.

Curated: · Written: · Reviewed:

Q&A-99Two weeks after GA, nobody has looked at the scorecard. The feature is "done". What did the team skip, and what meeting do you add?

What they skipped. The post-launch decision checkpoint. GA shipped the code, but nobody converted the launch hypothesis into a scale/iterate/kill call, so the feature sits behind its flag at whatever rollout it launched with and its guardrails go unread. "Done" was treated as a state instead of a checkpoint.

The meeting I add. A 30-minute launch review at day 14, on the calendar from the day the launch was approved, owned by the PM who wrote the hypothesis. The agenda is the scorecard; the output is one written decision in the launch doc: scale, iterate, or kill.

Why day 14. Launch-day traffic is a novelty spike, so a day-2 read is dominated by it. Two weeks gives roughly ten business days of steady-state traffic, enough for rate-based guardrails (refund rate, error rate, support contacts per 1k sessions) to clear daily noise. If the feature is low-traffic, the calendar is the wrong gate: review at 5,000 treatment sessions or day 14, whichever is later.

What's on the scorecard. The primary success metric against its target, each guardrail with its threshold, and the kill threshold from the hypothesis. The PM owns the decision, on-call owns the guardrail read, and the scorecard is a dashboard link, not a slide deck.

Decision options. Scale: roll to 100% and remove the flag. Iterate: hold the rollout, make one named change, re-review at day 28. Kill: revert and write the postmortem.

DecisionTriggerAction
Scalesuccess ≥ target, guardrails green100%, remove flag
Iteratesuccess short, guardrails greenhold rollout, one change, re-review d28
Killguardrail breachedrevert, postmortem

Trade-offs. Async beats a meeting when the metric is unambiguous and the decision is uncontested: a doc with the scorecard and a decision line is enough. A meeting earns its 30 minutes when a guardrail is red and someone must own the revert, or when scale versus iterate is contested. Don't book a meeting to read a number.

Weekly versus day-14: weekly catches a red guardrail sooner, but on low-traffic features it reads noise and burns review capacity. Day-14 is the default for features with enough traffic to move a rate.

When not to schedule it: a pure refactor or internal tooling with no user-facing metric. The scorecard is empty and the review is theater; use error-budget burn and a code-health check instead. And if the feature is at 1% behind a flag with the hypothesis "does anyone use it", the review is a usage check, not a metrics review.

Failure mode. The zombie flag: rollout stays at 10%, no decision is recorded, a guardrail has been red since day 3, and finance notices at day 40. Detect it by alerting on any flag older than 30 days with no decision line in the launch doc.

Sizing the window (illustrative figures). Refund-rate guardrail: baseline 0.3%, breach at 0.5%. At 500 orders/day in the treatment arm, day 14 is 7,000 orders: 21 expected refunds against 35 at breach. Below roughly 200 orders/day you cannot separate 0.3% from 0.5% in two weeks, so gate on sample size rather than the calendar.

Curated: · Written: · Reviewed:

Q&A-101Two users edit the same paragraph offline and both reconnect within a minute. What conflict-resolution policy do you specify, what does the superseded user see, and how do you test that no edit is silently lost?

What I'd say in the first two minutes

I'd specify paragraph-level conflict detection with user-mediated resolution, not silent last-write-wins. Assumptions: the document is the unit of collaboration, users expect their text to survive, and the system can detect overlapping edits at paragraph granularity. If two users edit the same paragraph offline and reconnect within a minute, the second reconnection triggers a conflict state. The user sees a banner: "This paragraph was also edited by [name] while you were offline." They get a side-by-side diff of their version and the other version, with three actions: Keep mine, Keep theirs, Merge manually. Until they choose, the paragraph shows both versions stacked with a highlight. The other user gets a notification that their edit is pending resolution, not silently overwritten.

Why not last-write-wins

LWW at document level loses an entire offline session: unacceptable. LWW at paragraph level still silently discards one user's work, which is the exact failure I'm asked to prevent. The only case where LWW is defensible is a field with a single scalar value (e.g., a status dropdown), and even then I'd log the overwritten value and notify.

The policy table

GranularityPolicyUser seesSilent loss?
DocumentLWWNothingYes, whole session
ParagraphUser-mediatedConflict banner + diffNo, if resolved
ParagraphLWW + historyNotificationRecoverable, not silent
CharacterCRDT auto-mergeNothingNo, but implementation-owned

I'd pick user-mediated at paragraph level, with a fallback to LWW + history if the user is inactive for >5 minutes (e.g., they closed the tab). The fallback still notifies.

What the superseded user sees

If their edit is superseded by the other user's choice, they get an in-app notification and an email: "Your edit to paragraph 3 was replaced by [name]'s version. View diff / Restore mine." The restore is one click and creates a new conflict if needed. This is the product contract: no edit disappears without a trace.

How I'd test no silent loss

  1. Property test: generate N random concurrent edit sequences on the same paragraph, apply them, assert that every user's final intended text is present in version history. Run it in CI.
  2. Instrumentation: emit a counter for conflict_detected, conflict_resolved_keep_mine, conflict_resolved_keep_theirs, conflict_auto_resolved_lww, and conflict_restored. Alert if conflict_auto_resolved_lww > 0 without a corresponding notification.
  3. Acceptance test: two browser sessions, both offline, edit same paragraph, reconnect. Assert the conflict UI appears, both versions are in history, and the losing user can restore.

The test that matters is the one that fails if an edit is dropped. I'd make that a release gate. If 1 in 1,000 conflicts silently loses an edit, that's a support ticket and a trust break: the instrumentation catches it before the user does.

Curated: · Written: · Reviewed: —

Q&A-52You add a required middle_name to signup. 18% of locales have no middle name. What broke, and what is the compatible move?

Core answer

Two things broke, and only one is the database. The form and the API started rejecting signups that omit middle_name. In the 18% of locales where a middle name isn't a concept, that's every user: the client-side required marker blocks submit, and anything that gets through returns 422. Completion in those locales collapses while spend keeps running, so CAC in those markets spikes.

The compatible move is expand/contract: ship the column nullable, make validation optional, and enforce requiredness only per-locale where the product actually needs the value. Never flip a global NOT NULL on a field 18% of users can't fill.

Why it breaks, layer by layer

Client: the form marks the field required, so submit is disabled and no request is sent. This is the largest drop and the hardest to see, because there's no 4xx to count.

API: validation rejects the payload with a 422 on /signup.

Database: ALTER TABLE users ADD COLUMN middle_name text NOT NULL fails on a non-empty table. Existing rows would violate the constraint. Adding it nullable and later running ALTER COLUMN middle_name SET NOT NULL also fails while any row is NULL, and Postgres scans the whole table to check.

Old mobile clients: they don't send the field at all and you can't force an update, so server-side requiredness breaks every un-updated app in every locale, not just the 18%.

The move

-- 1. expand: nullable, no default, no fake backfill
ALTER TABLE users ADD COLUMN middle_name text;

-- 2. contract later, only after proving no NULLs remain
--    (fails while any row is NULL; full table scan)
ALTER TABLE users ALTER COLUMN middle_name SET NOT NULL;

A request from a locale without the concept:

POST /signup
{"email":"a@b.jp","given_name":"Yui","family_name":"Tanaka"}
-> 422 {"error":"middle_name is required"}
PhaseServerClientDB
1accept missingold buildscolumn nullable
2accept missingsend when presentnullable
3required per-localerequired per-localepartial CHECK

When required is actually right

Enforce required only when the value is needed for the product to function for that user: a legal name in a jurisdiction that mandates it, or a downstream system that can't accept null. Then gate it by locale or flow, not globally, and express it as a partial constraint or a separate table rather than a column-wide NOT NULL. If you can't name which users need it and why, it stays optional.

Detection

Watch funnel drop at the form step by locale, 422 rate on /signup by locale, and the count of NULL middle_name. A canary on the new requiredness in one locale before 100% catches this in hours instead of a week.

Curated: · Written: · Reviewed:

Q&A-51The search vendor's contract forbids exporting relevance judgements. What do you require before GA depends on them?

What I'd say in the first two minutes

The contract forbids exporting judgements, so I stop treating the vendor's judgement set as something I can rely on. Before GA I require an evaluation set I own, and a contractual relevance guarantee measured against it. Four things:

  1. A relevance SLA with a defined metric, eval set, cadence and remedy. Not "high relevance": nDCG@10 on a held-out query set, measured monthly, with service credits and a termination-for-cause trigger if it falls below a floor for two consecutive periods.
  2. My own judgement set. Sample 300–500 queries from our own query log (head, torso, tail), label them with in-house raters or an LLM-assisted pass with human adjudication, and store them in our repo. This is the asset that survives the vendor.
  3. A licence amendment or escrow for derived judgements. Ask for aggregate/derived signals (per-query nDCG, per-document relevance scores) rather than raw qrels, or source-and-judgement escrow released on termination or insolvency.
  4. A passing exit drill. Run the vendor's API and one candidate replacement against my eval set and record the nDCG delta.

Why judgements are the thing that matters

Judgements are the ground truth for offline relevance evaluation. Without them you cannot measure a replacement engine, cannot regression-test after the vendor ships a model update, and cannot tell whether relevance drifted. Click logs are not a substitute: they carry position bias, a result clicked at rank 1 is clicked partly because it sits at rank 1, so raw CTR overstates the quality of top-ranked documents.

The alternatives, weighed

OptionWhat you getCost / limit
Negotiate licence amendmentDerived judgements, or raw qrels under NDAVendor may refuse; leverage is at renewal
Accept lock-in + relevance SLAContractual floor on nDCG, creditsUnenforceable without a shared eval set and methodology
Build judgements in-houseOwned eval set, portable300–500 queries × labelling ≈ 2–4 rater-weeks
Escrowqrels + ranking config on terminationOnly helps at exit, not during the contract

I'd take in-house judgements plus the SLA. In-house is the only one that works during the contract; escrow only fires at exit.

Worked example (figures hypothetical)

Vendor reports nDCG@10 = 0.62 on their set. On my 400-query set, the vendor scores 0.58 and a candidate open-source reranker scores 0.55. The 0.03 delta is inside the noise band for 400 queries, so I would not switch on this alone: I'd want a larger set before trusting a gap that small. That is the reason to build the set before GA, not after.

Failure modes

  • SLA with no shared eval set: vendor measures on their set, you measure on yours, the dispute is unresolvable.
  • Judgements only in the vendor's UI: you can't run them in CI, so regressions ship.
  • Escrow that names "source code" but not judgements: you get the ranker, not the ground truth.

What would change this

If the vendor grants a licence amendment for derived judgements and a relevance SLA with a real remedy, I'd relax the in-house labelling to a smaller set. If they refuse both, GA does not depend on their relevance: I gate the launch on my own set or delay.

Curated: · Written: · Reviewed:

Q&A-29Day 8 of a 10-day sprint, research shows the chosen flow excludes keyboard users. Freeze or change? How do you assess it?

The 60-second answer. I'd change the flow, and I'd decide that today rather than at the sprint review. A keyboard-inoperable flow is a WCAG 2.2 Level A failure: Success Criterion 2.1.1, Keyboard, and Level A is the floor: you cannot claim AA conformance while failing an A criterion. So this isn't a nice-to-have competing with the date. It's a defect against the Definition of Done, and the date is the thing that moves.

Assumptions I'd state out loud before deciding: the flow is in scope for this release, no compliance deadline lands inside the next two weeks, and the team has real scope it can cut. If any of those is false, the answer changes.

How I assess it. Three questions, in order.

  1. Is it a conformance or contract miss? If the product's ACR/VPAT claims WCAG 2.1 AA, or a public-sector procurement (Section 508, EN 301 549) or a customer contract requires it, this is legal exposure, not a backlog item. Change now.
  2. What does the fix actually touch? Keyboard exclusion is rarely one bug. It's usually focus order, a missing visible focus indicator (2.4.7, AA), a custom widget with no key handlers, or a modal that traps focus. Scope it by walking the flow keyboard-only (Tab, Shift+Tab, Enter, Space, arrows) and logging every point where focus vanishes or the flow dead-ends.
  3. What's the cheapest cut? Two days of keyboard work is usually cheaper than the alternative, but cutting scope has its own cost: whatever you drop may be a commitment to someone else. Name what you're cutting and who agreed.
OptionShip dateKeyboard usersCost you own later
FreezeDay 10ExcludedWCAG 2.2 A failure ships; remediation plus re-audit
Change nowDay 12IncludedTwo days of lower-priority scope cut
Ticket for laterDay 10ExcludedSame failure ships; ticket competes with everything else

The "later ticket" row is the trap. It looks free because the cost lands on someone else's sprint.

Failure mode I've seen. A team froze to protect the date. The flow shipped without keyboard access. A customer filed a WCAG complaint; the fix, the re-test and the accessibility-statement update together took longer than the original two days would have. The date was protected for one release and lost for the next.

What makes it decidable. An impact note with numbers, not adjectives: which screens, how many users, what the fix touches, the new date, residual risk, and a named person accepting it. Something like: "Checkout step 2, three screens, keyboard-only users blocked at the payment-method selector; fix = focus order + visible focus + key handlers on the radio group; +2 days; residual risk = screen-reader pass not done; accepted by [name]."

Detection. Automated scanners like axe-core catch a minority of issues; the keyboard-only walkthrough is what finds this class. Put that walkthrough in the Definition of Done so it's caught on Day 2, not Day 8.

Curated: · Written: · Reviewed:

Q&A-93A confirm-shaming modal lifts conversion 8%. Unsubscribe is two extra clicks. Do you ship it?

No, not as described. But the two halves of the question pull in opposite directions, so I'd split them: the 8% is a seven-day number owned by growth, and the two extra clicks are a ninety-day cost owned by payments and support. Ship the save offer; don't ship the shame.

Assumptions I'd state before deciding: conversion means trial starts or first purchases over a seven-day window; the modal sits in the cancel flow, not at signup; and "two extra clicks" means the decline path is longer than the accept path. Change any of those and the arithmetic moves, but not the direction.

Why the 8% is inflated. A confirm-shaming modal doesn't create demand. It intercepts people who were already leaving and makes the exit cost more than the stay. Three things inflate the measured lift:

  • The window. Seven-day conversion counts anyone who didn't finish cancelling. A chunk of those cancel on day 8–30, so the lift decays.
  • Selection. Some people who click cancel are shopping for a discount, not leaving. They'd have stayed anyway, and the modal takes credit for them.
  • Cost transfer. The friction doesn't remove the churn, it moves it to a different ledger (refunds, chargebacks, support contacts) where the growth team isn't measured.

The two extra clicks are the mechanism, not a detail. A user who can't find the cancel button doesn't stay; they call their bank. That's the causal chain: friction → support ticket or dispute → chargeback. Acquirers charge roughly $15–25 per dispute plus the reversed sale, and dispute ratios are monitored: Visa's VDMP threshold is 0.9% of transactions (1.8% for high-CNP merchants), and crossing it means fines and losing the ability to fight disputes. A program sitting at 0.85% doesn't need much to tip.

Hypothetical arithmetic: plug your own numbers: 100,000 users a month enter the cancel flow. The modal holds 8,000. Assume 20% of those were discount-shoppers who'd have stayed anyway, so 1,600 real saves. Of the remaining 6,400, a quarter cancel within 60 days regardless, leaving 1,600 net at 90 days. At a 3% dispute rate that's 192 chargebacks; against 200,000 monthly transactions it adds roughly 0.1 points to the dispute ratio: the difference between safe and monitored.

Per 100k cancel-flow usersShame modalSymmetric save offer
Held at 7 days8,0005,000
Still subscribed at 90 days~1,600~4,200
Chargebacks~192~20

The right-hand column is what I'd ship: same offer, same discount, but "No thanks" is one click and the same size and weight as "Yes, keep my discount". Friction that informs is fine: "cancelling deletes your saved projects, export first" is legitimate. Friction that obstructs is the dark pattern, and the test is whether the decline path is as easy as the accept path.

This isn't hypothetical law. The FTC's case against Amazon (filed June 2023) alleged the Prime cancel flow took four pages, six clicks and fifteen options; it settled in September 2025 for $2.5B. California's ARL, as amended effective January 2025, requires cancellation in the same medium the user signed up in. The FTC's click-to-cancel rule was vacated on appeal in 2025, so don't lean on it: Section 5 unfairness and state law are the live constraints.

Detection, not vibes: run a 5% holdback on the old flow and judge on 90-day net revenue per user, not 7-day conversion. Watch chargeback ratio, support contacts per cancel attempt, and refund rate. Pre-commit the rule before you look: if the decline path is longer than the accept path, it doesn't ship, whatever the lift.

Curated: · Written: · Reviewed:

Q&A-89Checkout is down. Status is green. Twitter is not. What is the product owner's job in the first 15 minutes?

The product owner doesn't run this incident: the incident commander does. In the first 15 minutes the PO owns two things: the customer-facing truth, and the product call on what we're willing to break to stop the bleeding. Debugging is not on that list.

Here's the order I'd work in.

0–2 minutes: get the incident declared. If it isn't already, on-call declares it and opens a channel. The PO's first move is to join, not to investigate.

2–5 minutes: confirm it's real and user-visible. Twitter is a signal, not proof. I'd check three things: the synthetic checkout transaction, a canary order, and the support queue. A failing synthetic plus a spike in "checkout" tickets is proof. A single angry tweet is not.

5–8 minutes: set severity out loud. Named definition: checkout failing for more than 5% of attempts for more than 5 minutes on the revenue path is Sev-1. Say the number so everyone argues about the same thing.

8–15 minutes: publish, and fix the green status page. The green page is the second incident. Posting to Twitter while the status page says "operational" is worse than either alone: it tells customers the company doesn't know its own state. The first public update goes out within 15 minutes with knowns, unknowns, impact, and the next update time. Not root cause.

Then the trade-off, because "post early" isn't free. Waiting for RCA costs you the narrative: customers find out from each other, the support queue doubles, and you spend the next quarter rebuilding trust. Posting early costs you precision: you may have to correct yourself, and a wrong public cause is its own credibility hit. The deciding criterion is whether impact is confirmed and user-visible. Confirmed revenue-path impact → post in 15 minutes with unknowns stated. Unconfirmed or internal-only → holding 15 minutes to confirm is correct, because a false alarm on the status page trains customers to ignore the next one.

MinuteWhoAction
0–2On-callDeclare incident, open channel, page IC
2–5POConfirm user-visible: synthetic checkout, canary order, support queue
5–8PO + ICSet severity (Sev-1 if revenue path down)
8–12CommsDraft update: knowns, unknowns, impact, next update time
12–15POApprove and publish; correct the green status page

Cadence after that: every 30 minutes until mitigated, then a post-incident summary within 5 business days.

The failure mode I'd watch for is exactly this one: a green status page during a dead checkout means monitoring is blind. The synthetic checkout isn't wired to alert, so Twitter became the detector. That's a follow-up action, not a first-15-minute fix, but the PO should name it in the channel so it doesn't get lost.

Arithmetic, labelled as a hypothetical: if checkout runs $50k/hour at peak, that's about $830/minute. Fifteen minutes of silence is roughly $12k of orders nobody is counting, and the support queue is the only instrument telling you.

Curated: · Written: · Reviewed:

Q&A-54Launch day, error rate 9×. Engineering can "forward-fix in a few hours". What does the product owner require instead?

What the PO requires

A rehearsed revert: a kill switch or rollback that restores the previous user-visible behaviour inside a stated time bound: with the trigger and the decision owner agreed before launch. Not a fix estimate.

The first 60 seconds: "forward-fix in a few hours" is an engineering estimate of when the code will be correct. It is not a bound on user harm, and the PO is the person accountable for user harm. So the PO's requirement is a different artefact: a named revert path, a number of minutes it takes to execute, a threshold that fires it, and one person who can call it without a meeting.

Why the PO treats revert differently from a fix promise

A fix promise is unbounded in the dimension the PO cares about. The estimate covers engineering time; it says nothing about how many users hit the broken path while the fix is written, reviewed, deployed and verified. At 9× the baseline error rate, harm accrues per minute, and the accrual rate is the thing being decided.

A revert is bounded because it targets a state that already ran in production. You are not asking "will this be correct?": you are asking "can we get back to the last state we know was correct, and how long does that take?" That is a question with a checkable answer, and it can be answered before launch rather than during the incident.

There is a second reason. The forward-fix is written by the same team, under the same pressure, against the same code path that just failed, and it is unverified at the moment it ships. Revert ships a state that has already survived real traffic. And during an incident the engineer who wrote the change is the least likely person to call for reverting it; every minute spent negotiating who decides is a minute at 9×. Pre-agreeing the trigger removes the negotiation.

What "requires" means concretely

Three things, all agreed pre-launch:

  • The mechanism: flag-off, previous deploy, or kill switch. Whichever actually restores behaviour.
  • The bound: minutes, stated as a number.
  • The trigger and the owner: error rate above X for Y minutes → revert, called by a named person, no debate.

Revert vs forward-fix, on what's bounded

Forward-fixRevert
What's estimatedtime to correct the codetime to restore a known-good state
Bounded bynothing the PO controlsthe stated minutes
Verified before useno: the fix is newyes: the state already ran
Harm while it runsaccrues at 9×accrues until the bound

Arithmetic (hypothetical figures, for shape)

Baseline error rate 0.5%; launch makes it 4.5% (9×). At 2,000 requests/min that is 10 errors/min baseline against 90/min now: 80 excess failures per minute. A 3-hour forward-fix is 180 minutes, so 14,400 excess failed requests. A 4-minute flag-off is 320. The gap is the whole argument, and it is arithmetic the PO can do before launch, not during.

Failure modes to name

  • A flag that has never been turned off in staging. The bound is fiction until it has been exercised; the drill timestamp is the evidence.
  • A revert that restores code but not data. If the bad version wrote rows or ran a migration, rolling back the deploy leaves the writes behind. This is the case where forward-fix genuinely is the only path, and then the PO needs the data risk bounded and stated before launch, not discovered mid-incident.
  • A revert path that depends on the same pipeline that is currently failing.

What settles it

A drill timestamp from this week, a number of minutes, a threshold, and a name. Without those, "a few hours" is a promise about code, and the PO is on the hook for users.

Curated: · Written: · Reviewed:

Q&A-98The conference is Tuesday. The checklist is 60% amber. Do you go live on stage?

No. I'd demo a slice on stage and keep GA off the calendar until the Must gates are green.

Two assumptions I'd state before answering, because they decide it. First, the checklist is a gate list, not a feature list: 60% amber is a count, not a verdict. Second, the conference is a distribution channel, not a launch criterion. If the Tuesday slot is contractually tied to a customer commitment, the answer changes, and I'd say that out loud rather than quietly shipping.

Then I'd sort the checklist by what each amber item actually blocks. Must gates are the ones where shipping amber causes data loss, money movement, or an unrecoverable state: auth and session handling, billing and metering, rollback and migration. Should gates degrade experience: a slow dashboard, a missing export. Could gates are polish. A checklist that is 60% amber with every Must green is a ship. One that is 20% amber with billing in the red is not. The percentage is the wrong number to argue about; the bucket is the right one.

If any Must is amber, I don't GA. The stage doesn't shrink the blast radius, and a keynote audience is the worst possible load test because it arrives all at once and it's watching.

Worked example, hypothetical but the shape is common. A metering gate ships amber because the free-tier counter only increments on the success path. The keynote drives signups; retries and failed charges skip the counter. Two days later the invoice run surfaces usage that was never billed: at roughly 40k signups and about $1.75 of unmetered compute each, on the order of $70k of COGS, and the billing worker catching up trips the status page. The demo looked fine. The gate was the problem.

OptionTuesdayBilling exposure
GA as plannedlive demoamber Must ships; leak risk
Demo a slice, hold GAlive demoMust stays gated
Recorded demo, slip GArecordedMust stays gated
Slip the conferencemissMust stays gated

What would change my answer: evidence, not optimism. A Must-only checklist with a named owner per item, reviewed Monday, plus a time-boxed mitigation for the amber Must (a kill switch, a hard cap on unmetered usage, a manual reconciliation step) and a rollback that has actually been exercised. If the billing gate can be capped at a known ceiling and reconciled by hand for a week, that's a different risk than an open-ended leak, and I'd take the stage.

What I wouldn't accept is a green checklist produced by reclassifying the amber Must as a Should. That's the move that turns a launch decision into an incident, and it's the one I'd push back on hardest in the room.

So the plan for Tuesday: demo the slice that works, name the amber Must and its owner on stage if asked, and give the date the gate needs. The conference is a room full of people who will still be there next quarter.

Curated: · Written: · Reviewed: