Skip to content
Tech Interview Prep

Top 100 Product Manager Interview Questions and Answers

The questions most likely to come up in your Product Manager interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 6

106 questions

Q&A-25What belongs on a single page that starts a project?

Start with the problem, not the feature. A one-page brief is a decision-forcing document: it exists so ten people build the same thing, and so disagreement surfaces before code rather than in week five. Assumptions: one team, one to two quarters, reversible decisions, no external compliance gate. If any of those is false, a one-pager is the wrong artefact and I say so below.

What goes on the page

Five sections, each one or two sentences:

  1. Problem and customer, who is hurt, doing what, and how you know. Two sentences.
  2. Outcome: one metric, its current baseline, a target, and a date. "Reduce checkout abandonment from 68% to 55% by 30 June" is a brief; "improve checkout UX" is a wish.
  3. Scope boundary: what is in, and explicit non-goals. The non-goals are the load-bearing half.
  4. Constraining decisions: the one or two choices that everything else follows from (e.g. "reuse the existing payments service, no new datastore"). Not a design.
  5. Open questions: each with an owner and a date.

Circulate before kickoff, collect disagreement in writing, and update the same page rather than issuing a second document.

Problem:  <who> can't <do what> because <why>; today they <workaround>.
Outcome:  <metric> from <baseline> to <target> by <date>.
In:       ...
Out:      ... (non-goals)
Decisions that constrain us: ...
Open:     <question> — <owner> — <date>

When a one-pager is the wrong tool

ArtefactAnswersRight whenWrong when
One-page briefWhy, for whom, what success is, what's outOne team, ≤1 quarter, reversibleRegulated change, multi-team interface, irreversible architecture
RFCHow, and which alternatives were rejectedCross-team interface, contested designYou only need agreement on the problem
Design docMechanism, data model, failure modesMulti-week build across subsystemsProblem still unsettled: you'll write it twice
ADROne decision, its context and consequencesDecision is made and needs a recordYou need to align people before deciding

Concretely: regulated work needs traceability to a named control, so the brief is an input to a compliance artefact, not the artefact. A multi-team interface needs an RFC with sign-off from each team. An irreversible choice (a schema, a public API, a vendor) needs a design doc that lists the alternatives. And if the problem itself is still unknown, the right page is a discovery plan with a timebox, not a brief.

Failure modes, and how I'd detect them

  • Brief becomes a status doc. Detect: no edits after kickoff, or edits that only append progress.
  • Outcome is unmeasurable. Detect: ask two people for the success number; if they differ, the page failed.
  • Non-goals missing. Detect: two workstreams touching the same service. I've seen a 14-page spec that 3 of 9 people had read, where the scope boundary sat on page 11 and two engineers built overlapping services for five weeks.
  • Ownerless open questions. Detect: the same questions still open at the first demo.

How I'd check it worked

Ask two people who missed the kickoff to state the goal and the non-goals from the page alone. If they can't, the page is not done. Illustrative figures from one team's retro, not a benchmark:

DocumentRead fullyStated goal correctly
14-page spec3 of 94 of 9
1-page brief9 of 98 of 9
kickoff onlyn/a3 of 9

A brief is measured by the disagreements it surfaces before the work starts.

Curated: · Written: · Reviewed:

Q&A-2How do you write a problem statement that an engineering team can argue with?

A problem statement an engineering team can argue with is a falsifiable claim about observed behaviour: who is affected, what they cannot do today, how often it happens, what it costs, and what evidence would make you drop it. If it contains a noun that names something to build, it is a plan wearing a problem's clothes, and the argument is over before it starts.

The test is social, not editorial. Hand the statement to two engineers who were not in discovery. If they propose two different approaches, it is a problem. If they both propose the thing you already had in mind, you wrote a spec.

The five slots

Figures below are illustrative.

SlotWeakArguable
Who"users"on-call admins at 40-200 person SaaS shops
Cannot do"need better visibility"cannot tell which escalation is already being handled
How often"often"~4 missed escalations per tenant per week, from the escalation log
Cost"hurts retention"each one is a ~20 min customer call, ~1.3 h/week of CS time
Kill criterionabsentif the log shows under 1/week, drop it

Worked rewrite

Start: "We need a notification centre so admins don't miss escalations."

Strip the nouns that name a build: notification, centre. What is left is admins miss escalations. Attach a frequency from a named source, a cost with a unit, and a kill criterion.

Rewritten: "On-call admins miss roughly 4 escalations per tenant per week (escalation log, 6 weeks, 12 tenants) and find out when the customer calls. Each costs about 20 minutes of CS time and one apology. If the log shows fewer than 1/week, we drop this."

Now the team can argue: is 4/week real, is the log complete, is CS time the right cost, is auto-assign cheaper than a digest, and, the argument you actually want, should these escalations need a human at all. That last question is invisible in the original statement, because the original already answered it.

Failure modes

  • Solution as a noun. Notification centre, dashboard, migration, self-serve. Test: delete the noun. If the sentence collapses, the noun was the solution.
  • Unfalsifiable framing. Improve reliability, reduce friction, make it faster. No observation kills these, so nobody can argue with them.
  • Frequency with no source. "Users complain constantly" is usually one loud customer. Name the query, the window, the population.
  • Cost with no unit. "Bad UX" is not a cost. Minutes, dollars, churned accounts, tickets per week.
  • Scope so wide it cannot be wrong. "Users want better search": nobody can be incorrect about it, so nobody can argue.

When not to do this

If the problem is externally fixed (a compliance deadline, a CVE, a contractual SLA) the problem is not in question. Write the constraint and the date, and spend the week on the fix. Discovery theatre on a known constraint is the expensive version of doing nothing. Same for small reversible changes: the statement costs more than the change.

How I would know it worked

Two engineers who were not in discovery propose two different solution families, and someone can name the observation that would kill the statement. If nobody can propose a different solution, you wrote a plan and called it a problem.

Curated: · Written: · Reviewed:

Q&A-4How do you ask about a problem without teaching the customer the answer you want?

What a strong candidate says first

Ask about the last time the problem happened, not what they would do about it. Before writing a single question, name the decision it feeds and who lives with that decision: that is what tells you whether you need behaviour or opinion.

Three rules:

  1. Never say the noun of your solution in the question. "Would you use a scheduling assistant?" teaches the respondent that a scheduling assistant is the answer you want, and they will help you give it.
  2. Ask about a specific past event, with a date and a sequence. "Walk me through the last time you rescheduled something: what did you do first?"
  3. Ask what it cost: time, money, a workaround they built. Cost already paid is the only signal that survives contact with a launch.

A question about the past returns behaviour; a question about the future returns a courtesy.

Why hypotheticals fail

A "would you" question asks someone to predict the behaviour of a future self in a context that does not exist yet. They answer a different question instead (what kind of person am I, what does this interviewer want to hear) and that answer is cheap, so they give it. Past-event questions retrieve episodic memory: the tool they opened, the person they messaged, the twenty minutes it took. That memory is reconstructed and imperfect, but it is anchored to something that happened. People predict their behaviour badly and report it well.

When hypotheticals are the right tool

Past-event questions are useless when the behaviour does not exist yet, and they are the wrong instrument for four jobs:

  • Concept testing a new category. Nobody has the past behaviour to recall. Show the concept, then ask for a commitment (a pilot slot, a deposit, a calendar invite) not a rating.
  • Pricing. People cannot recall what they would pay. Use forced trade-offs: conjoint, or a Van Westendorp price ladder. Treat the output as a ranking of price points, not a forecast.
  • Usability. Do not ask whether it is easy. Put the prototype in front of them and watch the task; the observation is the data.
  • Positioning and messaging. "Which of these two sentences describes your job?" is a legitimate hypothetical, because you are testing comprehension, not predicting behaviour.

The rule that keeps them clean: run behavioural interviews first, concept tests second, never in the same session. A concept shown before the interview contaminates every answer after it.

Failure modes

  • Courtesy bias. Detected by attaching a cost to the ask. If the yes rate collapses when the ask becomes a deposit, the earlier yeses were social.
  • Recall bias. Frequency claims ("I reschedule twice a month") drift. Cross-check against instrumented logs, calendar data or receipts.
  • Sampling from the interested. Recruiting from a list of people who said yes selects for politeness. Recruit people who did the behaviour in the last 30 days.
  • Atypical last event. One vivid story is not a pattern. Ask for the last three and see whether the shape repeats.

Worked example (illustrative numbers, not a real study)

30 target users, two question sets.

QuestionResult30-day adoption
"Would you use a scheduling assistant?"27 of 30 said yes1 of 30
"Walk me through the last time you rescheduled"3 described doing it twice or more last monththose 3 were the adopters

The yes rate measured politeness. The event question measured the behaviour.

How I would know it worked

Count the questions in the guide: if the past-event questions do not outnumber the hypothetical ones, the guide is still leading the witness. Then read the recruiting screen, if it admits anyone who expresses interest, the sample is selected for courtesy.

Curated: · Written: · Reviewed:

Q&A-3You have time for eight customer interviews before a planning cycle. How do you pick who to talk to?

I'd pick the eight by sampling, not by who is easiest to reach. The default list (whoever complains loudest, whoever the account team likes) is a biased sample, and a roadmap built on it looks validated right up until churn shows up in a segment the list never covered.

Two assumptions I'd state before choosing. First, the planning cycle is about a problem we already believe exists; if it isn't, the eight slots should go to problem discovery and segment selection is premature. Second, we have usage data good enough to segment accounts by behaviour rather than by contract size.

Then I'd split the eight across four segments:

  • current heavy users: 2
  • current light or lapsed users: 2
  • churned in the last two quarters: 2
  • prospects who evaluated us and chose a competitor: 2

Recruit from product usage data, not from the account team's suggestions or the customer advisory board. Cap any single segment at a third of the slots so no one group can carry the answer on its own.

The churn and lost-deal slots are the ones that earn their place. Those people are the only ones who can tell you what the product failed to do, and they are exactly who a default list never contains.

The failure I've seen: a workflow product ran 12 interviews sourced entirely from its customer advisory board, shipped the roadmap those 12 endorsed, and the following quarter churn concentrated in the self-serve tier. 61% of accounts, zero seats on that board.

Here is the same eight slots filled two ways:

SourceSlotsSegments coveredChurned accounts heard
advisory board810
usage-sampled842
inbound complaints820

Before booking anything, I'd check the list against the revenue and account mix and confirm no segment is more than a third of the slots. If the churn slots can't be filled, churned customers often won't take the call, I'd record that gap and treat the findings as covering only the segments I actually reached, rather than quietly letting the sample collapse back to happy users.

The customers easiest to reach are the ones whose problems you already know.

Curated: · Written: · Reviewed:

Q&A-13When is a survey the right instrument, and what can it not tell you?

I would start surveys as a supporting instrument from the customer problem it is supposed to change, not from the feature already on the roadmap.

A survey measures the distribution of an opinion across a population you already understand, and it cannot discover a problem you did not think to ask about. Using it as the first instrument produces precise answers to the wrong questions.

Concretely, run interviews first to learn what to ask, then survey to size how common each finding is. Report the response rate and the non-response profile alongside every result. Never let a free-text box stand in for a conversation.

The reason for that specificity is a failure I have seen: A team surveyed 4,200 users with a 7 percent response rate, built the top-ranked request, and discovered afterwards that respondents skewed to accounts older than 2 years, who were 14 percent of the base and had the opposite need to the newest cohort.

Respondents against the base.

GroupShare of baseShare of respondents
accounts over 2 years14 percent61 percent
accounts under 90 days38 percent8 percent
trial accounts22 percent3 percent

I would not consider it settled without evidence: Compare the respondent profile against the whole user base and report where they diverge.

Surveys size a finding; they do not find it.

Curated: · Written: · Reviewed:

Q&A-5When is a jobs-to-be-done framing useful, and when does it get in the way?

The 90-second answer

Jobs-to-be-done replaces "what features do our competitors have" with "what progress is the customer trying to make, and what do they hire today to make it." It earns its keep exactly when it changes the competitive set or a roadmap decision. It gets in the way when the job statement is so abstract that no feature decision could violate it, or when the workshop output is a renamed backlog.

The test I apply: name the decision the job statement changed. If I can't, the framing cost a workshop and bought nothing.

Why it works

A job statement has three parts (situation, motivation, expected outcome) and the situation is what makes it falsifiable. "When I'm paged at 3am and don't know which service broke, I want the failing error signature in under two minutes, so I can decide whether to roll back." That names a trigger, a metric, and a decision. It also forces you to list what the customer hires today, including doing nothing, because non-consumption is usually the real competitor. Tony Ulwick's outcome-statement format (direction of improvement + unit of measure + object of control + context) is the version that survives contact with a roadmap, because each clause is checkable.

Alternative the customer hiresCost per useSwitching friction
grep over the cold S3 archive20 min of on-call timenone: the bucket already exists
incumbent APM vendoralready paid, per-seatannual contract, re-instrumentation
do nothing: guess and roll backwrong rollback roughly 1 in 3 timesnone

Figures are illustrative.

Worked example

A log-search team's roadmap said "add 12 more log parsers." The job statement above says the constraint is time-to-first-answer, not coverage. Arithmetic on the alternative: 20 min per search × 3 searches per incident × 40 incidents per quarter = 40 hours of on-call time per quarter, about $4,000 fully loaded at $100/hr. That number, not the parser count, is what the customer is paying to avoid. The decision changed: keep the last 30 minutes hot-indexed, cold-archive the rest. One roadmap item moved. That is the framing working.

When it gets in the way

  1. Abstraction collapse. The job is stated so broadly ("help me ship faster") that every feature satisfies it. Detect it by trying to write a feature that violates the statement. If you can't, it isn't a job.
  2. Vocabulary drift. The workshop produces 40 job stories that map one-to-one onto existing epics. Detect it by counting how many job statements changed a roadmap item. Zero means stop spending on the exercise.
  3. Buyer/user conflation. In enterprise, the user's job ("find the error") and the buyer's job ("pass the security review without onboarding a new vendor") are different jobs with different alternatives. A study of users only will miss the deal-killer.
  4. Survivorship bias. Interviews run only with people who already bought. The job that explains non-consumption is the one your own customers cannot tell you.

When not to use it

When the constraint is execution, not direction: a compliance deadline, a migration off a deprecated API, a reliability target. The job is already known and the alternatives don't change the plan. Same when there's no decision pending and no access to non-consumers, then it's a research hobby with a vocabulary.

A framing earns its keep by changing a decision, not by renaming a backlog.

Curated: · Written: · Reviewed:

Q&A-24Usage data and customer interviews disagree. Which do you trust?

Short answer: neither by default. Trust whichever source is measuring the thing you actually asked about, and assume that is not what you think until you have read the event definition and the interview sample. Usage data is evidence about behaviour; interviews are evidence about intent. They only conflict when one of them is answering a different question than the other.

What I'd say in the first two minutes. A disagreement is a signal that one source has a definition problem, not that one source is more trustworthy in general. So I don't pick a favourite; I look for a mechanism that predicts both the number and the story. If I can't find one, I don't have an answer yet.

Assumptions I'd state out loud: the metric is instrumented rather than a survey proxy, the interviews were drawn from the measured population, and the question is about shipped behaviour rather than a future decision.

Resolution procedure, in order.

  1. Read the event definition. Most conflicts are an event firing in a case nobody intended: wrong trigger, wrong platform, wrong account cohort, or a client version that stopped sending it.
  2. Check the denominator. A rate over "all accounts" and a rate over "accounts that reached step 2" are different numbers wearing the same name.
  3. Replay one real session end to end. Cheapest way to see whether the event fires where you think it does.
  4. Segment. If the two sources agree inside a segment and disagree overall, the conflict is composition, not measurement.
  5. Only then decide which one to act on.

Worked example. A team saw a 23% completion rate on the dashboard while interviews had 9 of 10 users describing the flow as easy. The completion event had been firing only for accounts created after a release five months earlier: 31% of the users in the report. The dashboard was reporting a rate over that cohort and calling it the product's rate.

SourceCompletionPopulation actually measured
Dashboard23%accounts created after the release (31% of users)
Interviews9 of 10long-tenured accounts
Corrected event71%all accounts

The interviews were right about their own population and wrong as a description of the product; the dashboard was right about its arithmetic and wrong about its denominator. Back out the older cohort from the corrected figure and it completes at roughly 93%, which is the population the interviews were actually sampling. Neither source was lying. The fix was the event, not the roadmap.

Failure modes, and how each is detected.

  • Event fires in unintended cases. Detect by replaying a session and diffing the event count against a manual count for one day.
  • Survivorship in interviews: you interview the people who stayed. Detect by checking the interview roster against churned users, if nobody who churned is in the sample, the sample cannot speak to churn.
  • Leading questions: "did you find it easy?" gets yes. Detect by asking what they did last time instead of how they felt about it.
  • Small n: 9 of 10 is a wide interval. Detect by stating the interval rather than the fraction.
  • Proxy metrics: a click is not a completion. Detect by asking what the event would look like if the user succeeded by another path.

Where interviews win. When the metric is a known-bad proxy, when the behaviour is new enough that instrumentation is immature, when the population is small enough that a rate is noise, or when the question is why a number moved. Data tells you a drop happened; only a user tells you the drop is because a required field moved.

The reverse is narrower. Any question about how many, how often, or which segment goes to the data. Recall is unreliable and self-report bends toward the socially acceptable answer, so "users say they'd pay for it" is not a demand signal.

Curated: · Written: · Reviewed:

Q&A-14How do you use support data without letting it set the roadmap?

What I'd say first

Support data is a demand signal from people who are still here. It ranks known friction well and is blind to everyone who left or never got far enough to complain. So I use it to price the cost of not fixing something, never as the sole evidence that something should be built.

How I'd actually use it

Cluster tickets by the workflow they interrupt, not the feature named in the subject line. "Export formatting" and "CSV has the wrong delimiter" are one cluster; "billing" and "export" are not, even when both mention invoices. Read twenty tickets verbatim before trusting any dashboard: the cluster labels are usually wrong the first time.

Then weight by distinct accounts, not ticket count. Tickets per account is heavy-tailed: one integration-heavy customer with a broken script can file hundreds. Postgres, 90-day window:

SELECT workflow_cluster,
       count(*) AS tickets,
       count(DISTINCT account_id) AS accounts,
       round(count(*)::numeric / count(DISTINCT account_id), 1) AS tickets_per_account
FROM support_tickets
WHERE created_at >= now() - interval '90 days'
GROUP BY workflow_cluster
ORDER BY accounts DESC;

Then join the cluster to an outcome number (activation, churn, expansion) before it earns a roadmap slot. Illustrative figures from a case I've seen:

ClusterTicketsAccountsTickets/account
export formatting380663.3
first import fails942100.4
billing confusion120881.4

The team spent two quarters on export formatting: 380 tickets, 6 accounts, 63 tickets each. First-import failures were 94 tickets across 210 accounts: 5.25% of that quarter's 4,000 signups, and those accounts activated 12 points lower. Same quarter, two rankings, and the ticket count picked the wrong one.

Tickets measure the path to a workaround, not the value of a fix. That is why the count and the account-weighted ranking disagree, and why the account-weighted one is the one that survives contact with the funnel.

Where support data should drive

Severity is a separate axis from volume. Data loss, security, compliance, billing correctness and outages go to the top on a single ticket. Volume weighting is for the long tail of friction, not for those. And when the product is young or the segment is small, support is often the only signal you have, then it drives, but you say so out loud and revisit once funnel data exists.

Failure modes and how I'd catch them

  • Ticket-count ranking. Detect: tickets/account above roughly 5 in a cluster means a handful of accounts are speaking for everyone.
  • Loudest-account capture. Detect: top 3 accounts' share of a cluster's tickets. Above ~30% and it's a customer-relationship problem, not a product one.
  • Silent-churn blindness. Detect: diff ticket topics against cancellation reasons and activation-funnel drop-offs. Topics that dominate churn but never appear in tickets are the blind spot support structurally cannot see.
  • Roadmap drift. Detect: audit roadmap items whose only supporting evidence is ticket volume. If you can't name the outcome metric the fix moves, it's drift.

The test I'd apply

Recount the top clusters by distinct accounts and compare against the count by tickets. If the ranking flips, the ticket count was measuring who complains, not what's broken.

Curated: · Written: · Reviewed:

Q&A-1A large customer asks for a specific feature by name. How do you decide whether to build it?

Treat the request as a hypothesis about a problem, not as a specification. The customer named a solution; the job is to recover the problem it is meant to change, then price the cheapest thing that changes it.

In the first minute of an interview I'd say: I ask three questions before anyone estimates. What do you do today to work around this? How often does it happen? What does it cost you when it goes wrong? Then I restate the problem in one sentence and read it back. If the customer adds a new requirement to my restatement, I don't have the problem yet.

Then I compare. The named feature is one candidate; I want at least two cheaper ones that solve the same stated problem, and I want to know which of them the customer would actually adopt.

Why the specificity matters: a failure I've seen: three enterprise accounts asked for a CSV export by name. A billing team built it. Eleven weeks later two of the three still reconciled by hand, because the real problem was a missing invoice identifier, not the file format.

Reading of the problemWorkAccounts that stopped reconciling by hand
Build the named CSV export11 weeks1 of 3
Add the invoice identifier2 weeks3 of 3
Do nothing, document the workaround0 weeks0 of 3

The named feature was the most expensive of the three and the least effective.

Quantify the problem in the customer's units before comparing: hours per week per person, error rate, or revenue blocked. A request that saves one analyst 30 minutes a month is a doc page; one that blocks a renewal is a roadmap item. The units matter more than the number.

Failure modes to name:

  • The request is a solution in disguise. "Add a bulk-edit endpoint" is often "our import is slow" or "we re-key data from a spreadsheet". Ask what they do today; the workaround names the real problem.
  • The requester isn't the user. The person who wrote the email may be a manager relaying a workaround their team invented. Confirm with someone who does the task.
  • The problem is real but not ours to solve. If the fix is a config change, a doc page, or a partner integration, say so and don't build.
  • Silent non-adoption. Ship the named feature and watch usage; if the accounts keep the workaround, you built the wrong thing. Detect it by instrumenting the workaround path, not just the new feature.

When I'd just build it: the request is a contractual or compliance obligation, the cost is small relative to the relationship and the maintenance burden, or the customer is a design partner whose workflow we're deliberately shaping. Those are decisions to make explicitly, not defaults.

What settles it: I read the problem statement back to the requesting customer and they accept it without adding a new requirement. That agreement, plus the cost comparison, is the evidence I'd bring to the build/no-build call.

Curated: · Written: · Reviewed:

Q&A-20How does discovery change when your customers are internal engineering teams?

Core answer (first 90 seconds)

Assume "internal customers" means other engineering teams who consume your platform and cannot buy elsewhere. Discovery doesn't change in method (you still interview, observe, and validate cheaply) but it changes in which signal you trust. External discovery leans on revealed preference through money: people pay, churn, or pick a competitor. Internal teams do none of that. They stay, they're polite in interviews, and they tell you what they think you want to hear. So the signal you trust shifts from what teams say to what they do: the workaround they built, the time it takes them to get value, and how many of them route around you.

The practical consequence: your discovery instrument is their repo, their CI config, and their on-call rotation, not a survey.

What that looks like

  • Shadow a team's on-call or a deploy. You see the manual step they never mention.
  • Read the workaround. A maintained script is a funded competitor; a stale one is noise.
  • Instrument the golden path: time to first successful deploy for a new team, and how many teams onboarded without a platform engineer's help.
  • Count parallel paths. That count is your churn number.

Trade-offs and when to pick which

MethodSignalCostFails when
Interviewsstated preferencelowcaptive users are polite; says yes, adopts never
Shadowingrevealed workflowhigh (one team at a time)you over-fit to that team's bespoke need
Telemetry on golden pathrevealed behavior at scalemediumyou measure what's easy, not what hurts
Thin slice to one teamrevealed preference for a changemediumn=1; the team isn't representative

Deciding criteria: how many teams share the pain, whether the workaround is actively maintained (a cost signal), and whether the platform can serve it without a bespoke path. If only one team has the pain, it's a feature request, not a platform investment.

Failure modes

  • Registration-as-adoption: dashboards show 100% because every service is registered, while teams keep their own deploy scripts. Detect by counting parallel paths, not registrations.
  • Loudest-customer capture: you build for the team that complains most, not the median. Detect by checking how many teams adopt the new path.
  • Politeness trap: interviews return unanimous enthusiasm, adoption stays flat. Detect by comparing stated intent to time-to-first-success.

Worked example (hypothetical)

A platform team reports full adoption. Counting parallel paths: 14 of 31 teams maintain their own deploy scripts. Time to first successful deploy: 11 days on the platform vs 2 hours with the script. The real adoption number is 17 of 31, and the deciding metric is the 11-vs-2 gap, not the registration count.

The line to remember

A captive customer expresses churn as a workaround. Discovery for internal teams is finding and measuring that workaround.

Curated: · Written: · Reviewed:

Q&A-21How do you tell whether a problem is severe enough to pay for?

"Pay for" has two halves: someone has to feel the problem, and someone has to hold a budget line that can fix it. Severity is mostly the second half. So my first move in any discovery call is to find out what the customer already spends on the workaround, and who owns that spend.

Stated pain is cheap to say in an interview. Spend is a revealed preference: it survived a budget review. A problem nobody has paid to route around (no contractor, no overtime, no spreadsheet maintained by hand every Friday) is usually tolerable, whatever the interviewee's tone suggests. The corollary matters as much: if the spend exists but nobody owns it, nobody can sign, and you have a research finding rather than a market.

In practice I ask three things per interview, and record them per account rather than per person:

  1. What do you do today when this problem shows up?
  2. What does that cost you in hours, money, or headcount per month?
  3. Whose budget does that come out of, and have they funded anything new this year?

Hypothetical results from one round of 25 interviews:

ProblemCalled frustratingMedian monthly spendOwner
duplicate records22 of 250 hoursnone
month-end close14 of 259 hourscontroller
access requests11 of 251 contractor, ~10 hrs/moIT ops manager

The ranking inverts the frustration ranking, which is the whole point. Duplicate records wins the interview and loses the deal.

The spend also sets a price ceiling. Nine hours a month at a loaded $85/hour is $765. A $500/month product is defensible against that; a $2,000/month one needs a claim about revenue or risk, not time saved. I would rather find that ceiling from the customer's own numbers than from a pricing survey.

Three ways this test misleads, and how I catch each:

  • The spend is mandatory, not discretionary. A workaround driven by a compliance rule has a budget line but no room to move. Detection: ask what the owner cut last quarter to fund something new. If the answer is nothing, the spend is not addressable.
  • The median hides concentration. If 2 of 25 accounts carry 80% of the spend, you have a two-customer market, not a segment. Detection: plot the distribution, not the median, and count accounts above the threshold you'd need to charge.
  • The workaround dies quietly. I have watched a team build for a problem that 22 of 25 interviewees called frustrating, then find at launch that the median account spent zero hours on it: a default setting shipped 18 months earlier had made the workaround rare. Detection: re-ask the spend question at the end of the sales cycle, and ask specifically whether the workaround was used in the last 30 days.

Where the test doesn't apply: genuinely new categories, where nobody has spent anything yet because the workaround doesn't exist. There the substitute is the cost of the status quo in a metric the buyer already tracks (audit findings, error rate, churn) plus one design partner who pays something, even a token amount, before you build. A signature is a stronger signal than a survey, and it's the only one that survives contact with a budget cycle.

Follow the spending, not the sighing, and then check who signs the cheque.

Curated: · Written: · Reviewed:

Q&A-6An executive asks how big this opportunity is. How do you answer without inventing a number?

The 90-second answer

"I can give you a range and the one assumption it hangs on, not a single number. Bottom-up: accounts that can actually buy this, times seats per account, times the price we actually close at. That's roughly $20M fully penetrated, and $0.3M–$2M in year one depending on penetration. It moves most if the account count is wrong, and I can test that in two weeks from our own win-loss data."

Then stop. The executive will push on whichever assumption they doubt, which is the point.

Why bottom-up, not a share of an analyst number

Every factor in a bottom-up estimate is either a countable unit or an observed price, so a reviewer can attack one factor at a time and see what changes. A top-down figure, analyst TAM times "we'll get 2%", has no mechanism behind the 2%, so nothing in it can be falsified. It survives the meeting and dies in the planning cycle.

Worked example (hypothetical figures, labelled as such)

Say we sell a CI-integration product to mid-market engineering orgs.

  1. Accounts. 41,000 companies in the segment, from a source I can name: a company registry, a vendor's published customer list, or our own CRM. Filter to those running a CI system we integrate with: 2,100.
  2. Seats. Median engineering headcount in those accounts is 40. We sell per seat, but only about a fifth of engineers touch the pipeline, so 8 seats.
  3. Price. List is $1,800/seat/year; realized ACV over the last four quarters averaged $1,200 after discounting.

2,100 × 8 × $1,200 = $20.2M annual, fully penetrated. That is a ceiling, not a forecast.

Year one is the ceiling times a ramp: 2,100 × 8 × $1,200 × 2% penetration ≈ $400K. Range $0.3M–$2M if we land 1% or 10% of the qualified set.

MethodNumberWhat has to be trueAttackable?
1% of a $12B analyst TAM$120Mnothing statedno
2,100 accounts × 8 seats × $1,200$20.2Mthe 2,100 filteryes
Year-one ramp at 1–10% penetration$0.2M–$2Mpenetration rateyes

Failure modes to name before someone else does

  • Double counting. Counting a parent and its subsidiaries, or counting seats and accounts as separate revenue.
  • List price instead of realized. Using $1,800 when the median deal closes at $1,200 overstates by 50%.
  • Registry ≠ addressable. The 2,100 filter is doing all the work. If the integration requirement is wrong, the estimate collapses: a team once sized a bet off 1% of a $12B market and shipped to 340K in year one, because only 2,100 accounts in that market had the integration the product required.
  • TAM/SAM/SOM conflation. The $20.2M is SAM at full penetration. The executive is usually asking about SOM in year one, which is the $0.3M–$2M line.
  • Currency and geography. A global account count priced in USD.

Sensitivity, and the cheapest test

Ask which single assumption halves the number. Here it is the 2,100. Test it from data we already have: pull the integration requirement from the last 50 closed-won and closed-lost deals. If 30% of losses cite a missing integration, the count is too high and the range shifts down.

What to hand over

One page: the range, the assumption it hinges on, the cheapest test, and the date the test lands. That turns the number into a decision input rather than a claim.

Curated: · Written: · Reviewed:

Q&A-16How do you test a solution idea without engineering time?

The decision this feeds is whether to spend engineering time at all, so the test has to produce a kill signal, not a compliment. Without engineering you cannot test the mechanism. You can test three things: whether anyone wants the outcome, whether they understand the proposed solution, and whether they will do the work it requires. Pick the method by which of those is the riskiest assumption, because each method is blind to the other two.

Fake door (smoke test). A real button, ad, or landing page that promises the outcome and leads to a waitlist or a "coming soon" page. It measures demand at the moment of intent. Cost is hours to a day of design and copy, no engineering. It cannot tell you whether people will pay or change their workflow. The failure mode is curiosity: a click is cheap. Detect it by putting a commitment step behind the click (an email, a deposit, a booked call) and comparing the two rates.

Landing page with a price. Same as above but with a price and a pre-order or deposit. Measures willingness to pay. A day of work. The failure mode is that a $0 pre-order is not a pre-order; detect by requiring a card or a deposit.

Concierge. You deliver the outcome manually, behind the scenes, with no product. Measures whether people will do the work the product requires. Cost is your own hours per user, not engineering. Failure mode: the value comes from your judgment, not the software. Detect by logging minutes of human judgment per delivery; if that number does not fall when you remove yourself, the product is you.

Wizard of Oz. A clickable front end with a human behind it. Measures the interaction and the workflow. Cost is a mock plus an operator. Failure mode is human latency and error rate. Detect by measuring the operator's response time and error rate at ten concurrent users; if the human cannot keep up, the product cannot either.

Concept test. A static mock or a written description, shown to five to eight target users, who are asked to explain it back unaided. Measures comprehension, not demand. Thirty minutes per participant. Failure mode is praise without comprehension. Detect by counting unaided restatements, not preference ratings.

MethodMeasuresCostBlind toKill signal
Fake doorDemand at intentHours–1 dayPayment, workflowClick rate high, commitment rate near zero
Priced landing pageWillingness to pay1 dayWorkflowDeposits below threshold
ConciergeWhether they do the workYour hours/userScaleHuman judgment per delivery flat
Wizard of OzInteraction, workflowMock + operatorDemandOperator cannot hold latency at 10 users
Concept testComprehension30 min/userDemandFewer than half restate it unaided

A worked example, hypothetical: a fake door gets 2,000 visitors, 3% click the call to action (60 people), and 20% of those leave an email (12 qualified). If the bar for building is 25 qualified, you kill it for the cost of a landing page. If 60 leave an email, you have a demand signal and the next test is a priced page.

The failure mode across all five is testing preference instead of behaviour. Preference is free to give and costs the participant nothing; behaviour has a price. If the riskiest assumption is demand, run a fake door with a commitment step. If it is the workflow, run a concierge. If it is comprehension, run a concept test. Running all five is a way of not deciding.

Curated: · Written: · Reviewed:

Q&A-17How much fidelity does a prototype need?

Enough to answer the question you're asking, and no more. Fidelity isn't a quality dial you turn up as the deadline allows; it's a cost you pay to buy a specific answer. So the first move is to name the decision the prototype informs, then pick the cheapest artefact that can inform it.

The mechanism is that fidelity changes what people respond to. A grey-box clickable flow gives testers nothing to look at except the sequence, so their confusion lands on the sequence. Add visual polish and attention moves to the surface: typography, colour, spacing, copy. That isn't testers being shallow: a finished-looking screen reads as a finished decision, so they critique it as one. You also pay for that polish twice, in build time and in the cost of changing your mind. A paper flow you redraw in an afternoon; a pixel-complete build you can't.

Match the artefact to the question:

ArtefactBuild timeQuestion it answers
paper flow~2 hoursis the sequence understood
grey-box clickable~2 dayswhere do people get lost
styled mock~1 weekdoes this look credible and appealing
working slice on real data~2–4 weeksdoes the data support the flow

Those times are typical, not laws: a working slice over a well-understood API can be a day.

Then say out loud which feedback you're collecting. "Today I want to know where you get stuck, not what you think of the colours" is one sentence that saves a session.

Failure mode, and how you'd catch it: a team spent four weeks on a pixel-complete prototype to test a three-step flow. Nine of eleven testers spent the session on typography, and the step where five of them got lost went undiscussed. The tell was in the notes: almost nothing about the sequence, plenty about the surface. If your synthesis is mostly aesthetic commentary, you built the wrong artefact for the question.

The test before you build: write the question down, write down what a useful answer looks like, and confirm the artefact is the cheapest one that produces it. If you can't state the question in a sentence, you aren't ready to pick a fidelity.

Curated: · Written: · Reviewed:

Q&A-18A usability test shows users struggling. How do you decide whether that changes the product?

Core answer (first 90 seconds)

A usability finding tells you a task is hard, not that it is worth doing. So I separate two questions: can users complete this task, and do enough users attempt it for the completion rate to matter. Then I apply a rule. Fix the interface when the task sits on a path with real demand and failure has a cost. Change or drop the feature when demand is thin. Fix it regardless of demand when the failure is accessibility, compliance, or a high-stakes low-frequency task like account recovery.

The assumption I would state out loud: the test measured the right task with a representative sample, and I have production data for the same step. If I do not have the second, I get it before scheduling work.

Mechanism

A usability test measures observed task success under a moderator's framing. That is a measure of the interface, not of value. Two independent axes decide whether the finding changes the product: demand (share of accounts that attempt the task) and impact (cost of failure per attempt). A finding earns engineering when demand × impact clears the cost of the fix.

It also matters which kind of failure it is, because the fix differs: comprehension (users misread the label), affordance (they cannot find the control), and model mismatch (they are trying to do something else entirely). The third is usually a product problem wearing an interface costume.

Worked example (hypothetical figures)

RoundTask completion (n=25)Accounts attempting monthly
baseline10/25 = 40%4%
after 5 rounds of fixes (7 weeks)22/25 = 88%4%
after removal—0%

Arithmetic: 4% of accounts attempt the builder monthly, so at 40% completion 1.6% of accounts get value from it. Five rounds of fixes over 7 weeks took completion to 88%, giving 3.5% of accounts: a 1.9-point gain. The same 7 weeks on the checkout step, attempted by 60% of accounts at 70% completion, moving to 85% would take 42% to 51%, a 9-point gain. Same effort, roughly five times the effect. The builder was removed and its users migrated to export.

Failure modes and how I detect them

  • Fixing a low-demand flow. Detect: pull monthly attempt counts per task before writing the ticket.
  • Over-reading a small sample. With n=5, one participant moves the number 20 points; a 40% vs 60% gap is noise. A conservative 95% bound is about 2/√n. Detect: report n, and treat any gap under that bound as unresolved.
  • Lab success is not production success. Moderated tasks overstate completion. Detect: compare lab completion against the analytics funnel for the same step; a 20-point gap means the test is measuring the moderator.
  • Making a bad task easier. Password-reset completion rising is not a win if the real fix is SSO. Detect: ask what the user is trying to achieve, not what the screen asks for.

When the demand gate does not apply

Accessibility failures, legal and compliance requirements, and low-frequency high-stakes tasks: account recovery, billing disputes, exporting data before deletion. Demand is low by construction; impact per occurrence is high. Same for pre-launch features with no usage data: use segment size and support-ticket volume as the demand proxy, and say that is what you are doing.

Close

Demand × impact against the cost of the fix, with accessibility and high-stakes paths exempt. If I cannot get the demand number this week, I still fix anything blocking a task on the critical path, and I say the demand check is pending rather than pretend it is settled.

Curated: · Written: · Reviewed:

Q&A-8How do you test whether customers will pay for something before you build it?

Pick the cheapest test that makes the customer give up something they cannot get back (money, a signature, a slot in their own roadmap) then read the conversion rate, not the survey average. State three things up front: who signs (the user or the budget holder), what price you are testing, and what counts as a yes.

Cheaper tests first, paid pilot last.

TestCost / time to signalWhat it provesWhat it cannot prove
Fake-door landing page + ad spendDays, low hundreds in adsWhether the problem statement pulls clicksNothing about price; clicks are free
Pre-order or deposit1–2 weeksThat money moves at a stated priceWhether they renew, or that the product exists
Concierge MVP (you do it by hand)2–4 weeks, your hoursThat the outcome is worth paying for at allThat software can deliver it at that price
Letter of intent1–3 weeksThat someone will put their name on itThat they will pay; LOIs are free to sign
Paid pilot4–12 weeks, procurementPrice, buyer identity, renewal intentAnything if the pilot is discounted to zero

Run the ladder in that order. A fake-door test that pulls 4% click-through on a problem statement is worth more than a survey, and it costs a weekend. Only escalate to a paid pilot when the cheaper signals agree and the remaining question is price.

A paid pilot is the wrong tool when: the buyer is not the user (bottom-up adoption. Test a gated beta and measure week-4 retention instead); the product cannot be evaluated without being built (infrastructure, APIs. Test with a design-partner contract that pays for engineering time); procurement cycles exceed your runway (a six-figure enterprise deal is a sales process, not a demand test); or you are testing a feature inside an existing product, where usage of a gated rollout is the honest signal.

Worked example (illustrative figures). A team surveyed 140 accounts; the reported average acceptable price was $240/month. They opened a paid pilot at $99 and converted 3 of 60. They then ran a second cohort of 60 at $49 and converted 14.

SignalPopulationConvertedRate
Survey at $240140not asked to pay—
Pilot at $996035%
Pilot at $49601423%

The forecast was 45%. Even at n=60, 3/60 rejects it: the 95% Wilson interval is roughly 2–14%, nowhere near 45%. The $49 cohort's interval is roughly 15–36%, so the two prices are not statistically distinguishable at this sample size: the honest read is "somewhere between 5% and 36%, and $240 is unsupported." That is enough to kill the $240 assumption and not enough to set the price.

Failure modes to name. Survey bias: people answer a price question as a hypothetical, and the answer tracks what they think is fair, not what their budget allows. Free LOIs: if signing costs nothing, you are measuring politeness. Discounted pilots: a pilot at $0 converts at 100% and tells you nothing. Selection bias: recruiting from your own waitlist inflates conversion versus cold outbound. Small n: report the interval, not the point estimate, or you will chase noise.

Ask for the money before you write the code.

Curated: · Written: · Reviewed:

Q&A-23How do you forecast adoption for a product with no comparable predecessor?

What I'd say first

"No comparable predecessor" is almost never true at the level that matters. There is no product that does this exact thing, but there are products that asked the same people to change the same behaviour. I'd forecast the barrier, not the category.

Three things to pin down before producing any number: what decision the forecast drives, the horizon, and the unit. A forecast that sizes a 12-month support contract needs a commitment-safe lower bound. A forecast that decides whether to keep building needs a midpoint and a kill threshold. Same product, different numbers.

Mechanism

Adoption of a novel product is a funnel, and every factor in it can be anchored to something observable:

reachable population × awareness × trial × paid conversion × retention

Reachable population comes from your own data (registry, signup list, telemetry) so it is the one factor that is not an estimate. Awareness comes from launches through the same channels. Trial and conversion come from a smoke test you buy. Retention is the least knowable factor and the one that widens the range most, because novelty spikes decay.

Analogue selection is where most forecasts go wrong. Pick analogues by adoption barrier, not by category. If the product requires instrumenting production code, the analogues are other tools that required a code change, not other vendors in the same market.

Worked example (hypothetical figures)

FactorLowBaseHigh
Reachable developers400,000400,000400,000
Awareness, year 14%8%12%
Trial15%25%35%
Paid conversion8%12%18%
12-month retention50%60%70%
Retained paying accounts965762,117

Base case: 400,000 × 0.08 = 32,000 aware; × 0.25 = 8,000 trials; × 0.12 = 960 paid; × 0.6 = 576 retained at 12 months.

The spread is 22x. That is the finding, not a failure of the method: five factors each moving by a plausible amount multiply. A single point estimate here is a claim you cannot defend.

Buy the signal

A landing page measures intent. Charge something (a $20 deposit, a signed LOI, a paid pilot) and you measure commitment. 1,000 visitors producing 30 deposits is a 3% rate. If the funnel above assumed 12% paid conversion, that factor is 4x too high and the base case drops to roughly 144 retained accounts (8,000 trials × 0.03 = 240 paid; × 0.6 = 144).

Pre-register the threshold before running it: "if deposit conversion is under 4%, we don't staff for more than 200 accounts." A test with no pre-committed decision rule gets rationalised after the fact.

Failure modes

  • Single-analogue anchoring. A launch forecast of 20,000 first-year accounts came from one optimistic analogue; actual was 2,900, and a 12-month support contract at 41,000 a month was already signed. Detect it by requiring at least three analogues and reporting the spread between them.
  • Survivorship in analogue selection. You remember the successes. Detect it by listing analogues that failed and checking whether you can name why they failed.
  • Intent mistaken for demand. Detect it by charging money rather than counting signups.
  • Novelty decay. Detect it by cohort retention at week 4 against launch week; if week 4 is under half of week 1, the year-one curve is front-loaded and the annual number is wrong.
  • Commitment ratchet. Any contract with a lead time longer than the forecast horizon converts a range into a liability. Detect it by asking what the company is already obligated to pay.

When not to do this

If the decision is reversible and cheap (a two-week prototype, a small ad buy) skip the forecast and run the test. Forecasting earns its cost when the commitment is hard to unwind.

Curated: · Written: · Reviewed:

Q&A-12What evidence would convince you a product has found its market?

The short answer

Three things, and I'd want each to survive the removal of whatever is pushing demand: retention that flattens instead of decaying to zero, new accounts arriving without paid acquisition, and unit economics that still work once the push stops. The assumption I'd state up front is what "market" means here: a repeatable segment that pays enough to cover the cost of reaching it, not a spike in signups.

Why retention, and why the shape matters

A cohort retention curve tells you whether the product is a habit or a novelty. Novelty decays: every cohort slides toward zero, and the aggregate looks healthy only because new cohorts keep arriving. A habit leaves a tail: some fraction of each cohort is still there at month 12. The signal is the flat part, not the height of month 1.

Illustrative monthly cohorts, share of the original cohort still active:

CohortM0M1M3M6M12
Jan100%42%28%24%22%
Feb100%40%27%23%—
Mar100%41%26%——

The tail sits near 22–24% and stops falling. That is a durable segment. A curve reading 100 → 42 → 28 → 18 → 9 has the same month-1 number and is a different business.

Why the channel split matters

Retention by acquisition channel separates fit from spend. Paid acquisition buys a first session; it does not buy a second. If the paid cohort retains at a fraction of the organic cohort, growth is a function of budget, not of the product.

Illustrative composite from a company that read 40% month-on-month growth as fit:

SourceShare of new accountsMonth-6 retention
paid78%9%
organic referral14%44%
partner8%31%

Blended month-6 retention is 0.78×9 + 0.14×44 + 0.08×31 ≈ 16%: respectable-looking, and carried almost entirely by a channel that stops the moment the budget does. When paid spend paused for six weeks, new accounts fell 82%.

The test that actually decides it

Pause or cut paid acquisition for a defined window, long enough to cover the product's natural repeat cycle (for a weekly-use tool, six to eight weeks) and watch organic signups and cohort retention. If they hold, the demand was real. If they collapse, you were measuring spend. I'd also check the payback math: if CAC payback runs longer than the retention tail, no amount of fit saves the unit economics.

What wouldn't convince me

Survey scores on their own. The Sean Ellis test: the share answering "very disappointed" if the product disappeared, with 40% as the usual bar: is a leading indicator worth tracking, but it is attitudinal and cheap to game. Same for NPS, a single anchor logo, or revenue that only moves when a salesperson is in the room. None of those survive the pause test.

Failure modes and how I'd catch them

  • Aggregate retention hiding cohort decay → plot per cohort, not blended.
  • Paid masking organic weakness → split every retention number by channel.
  • Retention propped up by onboarding support or a CSM → remove the touch and re-measure.
  • Expansion that is really seat inflation from one buyer → check whether expansion revenue comes from accounts that also retained.

Fit is what is left when you stop pushing.

Curated: · Written: · Reviewed:

Q&A-7How do you decide which segments your product should serve first?

What I'd say first

"Serve first" is two decisions, not one: which cut of the user base actually separates behaviour, and which of the resulting segments gets the next two quarters of engineering. I'd answer the first with data and the second with a scored trade-off, and I'd name the segment I'm deliberately not serving.

The cut: behaviour beats firmographics, but only if it's actionable

Firmographic cuts (industry vertical, headcount, geography, contract size) are cheap and available before a deal exists, which makes them good for territory and quota. They're bad for product decisions because they average behaviour. A vertical view can hide two populations with opposite retention.

Behavioural cuts, the workflow a user runs and how often, separate the thing the product has to do. The cost is that you can only observe behaviour in people already using you, so a segment you've never served looks empty. That's the circularity to watch.

The deciding criterion between them: can you classify a prospect before they buy? If you can't route a lead into the segment, you can't price, staff or target for it, and it's a reporting category, not a segment.

Worked example (hypothetical numbers)

A product reported by industry vertical for five quarters. Same cohort, cut three ways:

CutGroups90-day retention spreadClassifiable pre-sale?
industry vertical644–49%yes
weekly active editors222–71%yes, via signup survey plus first-week behaviour
contract size341–52%yes

The vertical cut averaged two roughly equal groups, 71% and 22%, into a single 46.5% line. The behavioural cut separated them. That's the test: cut the same cohort two ways and keep the cut that separates curves the other merges.

Sequencing: which segment gets funded first

Once you have a cut that separates behaviour, score the segments on:

  • Willingness to pay and expansion evidence: net revenue retention by segment, not logo count.
  • Cost to serve: support tickets per account, infra per account. A segment with great retention and 4× support load can still lose.
  • Shared primitives: does building for this segment produce the thing the next two segments need? Serving the smallest segment first is right when it forces the general abstraction.
  • Reachability: can you acquire them through channels you already have, or does it need a new motion?
  • Referenceability: does winning here unlock the next segment's buyers?

The usual answer is the segment where behaviour differs most and the primitives you build are reusable. The usual mistake is picking the largest segment, which is often the one whose behaviour is most averaged.

Alternatives I'd reject and why

  • ACV or contract size: good for prioritising sales investment, poor for roadmap. Two accounts of the same size can need different products.
  • Technographic or stack-based: strong for integration-led products, weak when the stack doesn't predict the workflow.
  • Jobs-to-be-done framing: good for messaging, hard to measure, so it can't be validated the way retention curves can.

Failure modes

  • Firmographic-only segmentation: averaged curves hide bimodal retention; you find out a year later.
  • Behavioural cut that's circular: you only see behaviour of current users, so the segment you should serve looks small.
  • Too many segments: each gets 60% of a product and none gets a whole one.
  • Segment chosen for logo value with negative unit economics: the reference customer costs more to serve than they pay.

A segment is real when the two halves need different products, and worth serving first when the primitives you build for it are the ones the next segment needs.

Curated: · Written: · Reviewed:

Q&A-10How do you position a product whose main competitor is a spreadsheet?

The 90-second answer

Ask one question before writing any copy: what does this buyer do on Monday if we don't exist? For most products whose real competitor is a spreadsheet, the answer is "the same thing they did last Monday, in a file they built themselves." That changes the job. You are not displacing a vendor; you are asking someone to give up a tool that is free, infinitely malleable, and already trusted. So the positioning has to name the spreadsheet, price against the labor it consumes, and anchor to a budget line that already exists: usually not "software."

Assumptions I'd state out loud: the buyer is a team lead or ops manager, not procurement; the spreadsheet is homegrown; there's no incumbent contract to displace.

Why the spreadsheet is a different kind of competitor

Three structural differences drive everything else.

No referent. "Better than Vendor A" only works if the buyer has Vendor A on a shortlist. If they've never heard of it, the sentence is spent on a comparison they can't make.

Invisible cost. A spreadsheet has no invoice, so its cost never appears in a budget review. It shows up as overtime, rework, and key-person risk. Your positioning has to make that cost visible and numeric, or the buyer has nothing to compare your price against.

No category. Buyers can't request budget for a thing with no name. Either you attach to an existing line (FP&A tooling, revenue operations) or you spend a year or more creating a category. Attaching is faster and usually right for a first product.

Trade-offs

Naming the spreadsheet is cheap and credible when the pain is already felt; the buyer nods. It also anchors you to a free product, which caps your price unless you've made the labor cost explicit. Category creation avoids that anchor but costs a year of content and analyst work, and it's the right call only when the manual process is so normalized that naming it insults the buyer.

Naming a vendor competitor is right in one case: the buyer is already running an evaluation and that vendor's name is the budget line you're trying to take. Otherwise it's noise.

Don't ask them to abandon the spreadsheet. Position as the system of record the spreadsheet reads from. That removes the rip-and-replace objection and keeps the flexibility they value.

Failure modes and how you'd catch them

Feature-parity trap. A spreadsheet does pivot tables and VLOOKUP, so you end up competing on formula features and losing. Detect it in call recordings: count how often a prospect asks "can it do X formula?" If it happens in most deals, your messaging is inviting the comparison.

Free-anchor discounting. Win/loss where price is the top objection and the buyer never mentions labor hours means the cost framing didn't land. Track that share per quarter.

Wrong-alternative messaging. Instrument the signup survey: "what did you use before today?", and compare the top answer to what your homepage names. If they diverge, the copy is aimed at a buyer you don't have.

The artefact I'd bring to the positioning review

A table of the buyer's real alternatives, with the cost of each, and a positioning statement written against the top row.

AlternativeAnnual cost to buyerWho signsNamed in our positioning
Spreadsheet (homegrown)$0 license; ~$280k loaded labor (hypothetical)team leadyes
Vendor A~$120kVP Financeno
Do nothing$0—no

The arithmetic behind that labor figure, for a 12-person ops team: 6 hours/week each on manual reconciliation × 52 weeks × $75 loaded hourly = $280,800. Add one error per quarter at $15k rework = $60k. Call it $340k a year. A $40k product is 12% of the status quo: that's the sentence the buyer repeats to their VP.

Curated: · Written: · Reviewed:

Q&A-9A competitor ships a feature you do not have. How do you respond?

The 90-second answer

First I'd pin down what "respond" means, because it's four different decisions: build it, buy or partner for it, fast-follow with a thinner version, or decline and say so out loud. My default is to price the decision against the deals the capability actually decides, not against the competitor's announcement.

Assumption I'd state up front: we're a B2B product where deals are won or lost on a handful of criteria, and we have win-loss records to check.

A competitor shipping something is evidence about their strategy. It's weak evidence about your buyers' problems. So I'd go to the last 10-12 closed-lost deals and count how many name this capability as the deciding factor. If it's zero or one, the honest answer is usually "decline, and tell sales why." If it's a third or more, it's a real gap and I'd scope it.

Counting beats arguing, because the pattern is usually not what the loudest voice in the room thinks:

Reason namedLossesWork scheduled
onboarding time7none
dashboard builder29 weeks
price3none

That's a pattern I've seen: a team rebuilt a competitor's dashboard builder in 9 weeks and won one additional deal, while onboarding time was named in 7 of 12 losses and nothing was scheduled against it. The build was cheap to justify in a planning meeting and expensive to maintain forever.

Then the trade-off, because "never copy" is as lazy as "always copy."

  • Build. Right when the capability is table stakes in a segment you can't afford to lose, when the competitor's version is a wedge into your accounts, and when the build cost is small relative to the ARR at risk. Wrong when it's a demo feature nobody buys on.
  • Buy or partner. Right when the capability is adjacent to your core (a reporting layer, an integration) and a vendor already does it better. The cost is a per-seat or per-account fee plus an integration you now own.
  • Fast-follow. Ship the 20% that covers the buying criteria, not the competitor's full surface. This is usually the correct middle.
  • Decline. Right when the capability pulls you off the dimension you actually win on. Say it explicitly to sales so they stop asking.

Arithmetic makes it concrete (hypothetical figures): if the capability is named in 3 of 12 losses, average deal size is $40k, and win rate on those deals would go from 0% to 50%, that's roughly $60k/yr of recovered revenue. Two engineers for 9 weeks plus a permanent maintenance tail doesn't pay that back. If it's named in 8 of 12 losses on $200k deals, it does.

Failure modes worth naming: the checkbox trap, where you match the feature but not the workflow around it and still lose; the maintenance tail, where every feature you copy is one you now own; and roadmap capture, where a competitor's release sets your priorities for a quarter.

I'd detect all three with win-loss reviews that name the dimension, plus a quarterly check on whether the copied feature actually moved win rate.

I'd rather decline loudly than build quietly.

Curated: · Written: · Reviewed:

Q&A-11How do you decide what stays out of a first release?

The 90-second answer. Ask what the release is for, then cut against that. Three framings are defensible and they cut differently:

  • Learning question: the release exists to answer one question ("do pilot users get through import without help?"). Cut anything that cannot change the answer.
  • Risk: the release exists to retire the biggest unknown or the biggest blast radius. Cut by what is reversible and what is not; ship the irreversible thing last, or not at all.
  • Date: the date is a commitment (a customer contract, a conference, a regulatory deadline). Fix the date, then cut scope to fit it, in reverse priority order.

Most first releases are a mix, and the honest move is to say which one is driving. If you can't name it, scope gets set by whoever argues last.

Assumptions I'd state out loud: who the first users are, what "released" means (internal, pilot, GA), and whether anything downstream is contractually promised.

Cutting vs shipping behind a flag. A flag is not free scope avoidance; it's deferred scope with interest. Cutting removes the code path; a flag keeps both paths alive and every later release has to test them. Three flags that touch the same code path give 8 states; if you test 2, you ship 6 untested ones. So: flag when a specific cohort needs the feature and you'll flip it within a release or two; cut when nobody needs it yet and the flag would outlive the quarter. Permanent flags are the worst outcome: scope you never admitted to.

When the learning-question framing fails. It's the wrong tool when:

  • The release is a commitment, not an experiment: a migration with a hard cutover, a contractual deliverable. There the question is "can we do it safely", and safety work (audit logs, permissions) is not optional.
  • The cost of being wrong is asymmetric: payments, auth, data deletion. You don't learn your way into correctness there; you cut the feature instead.
  • You already know the answer. If the question is settled, the release is delivery, and delivery is scoped by the date and the risk.
  • The question can't be answered inside the release window. "Will users pay?" takes months; a two-week release can't answer it, so don't pretend the cut is evidence-driven.

A worked example (composite, not a specific company). A team spent 5 months on role permissions, audit logs and theming for a first release, then found in week 2 of the pilot that 8 of 10 pilot users abandoned at the import step: a step none of that work touched. The cut table they should have written:

ItemWeeksCould change the answer
import flow3yes
role permissions6no
audit logs4no
theming2no

Detection matters as much as the cut: instrument the funnel and name the abandonment threshold before launch, or you learn the same thing late and call it a surprise.

What I'd actually do. Write the driving framing (learning / risk / date) at the top of the release plan, list what's cut and why, and set the date from the framing. Then check each included item: can it change the answer, retire the risk, or is it required by the date? If not, it's next release.

Curated: · Written: · Reviewed:

Q&A-26How do you use a scoring framework like RICE without letting it make the decision for you?

Core answer. RICE is an input to a decision, not the decision. Score the candidates, then read the ranked list as a set of claims to argue with: where the order contradicts informed judgement, fix the input that is wrong rather than quietly reordering the list. Write the decision rule down before you score (what gets funded, and what would make you override the ranking) so an override is a named disagreement about a specific estimate, not a silent edit.

Assumptions. More candidate work than capacity; reach, impact, confidence and effort each estimable within an order of magnitude; the team owns the roadmap and can act on the result. If any of those fails, the framework is the wrong tool.

Mechanism. RICE = (Reach × Impact × Confidence) / Effort. Reach is the term most often wrong, because it is measured against the wrong denominator: total accounts instead of accounts that reach the screen. Confidence is the term that hides disagreement: 0.5 and 0.9 on the same item is two people disagreeing about evidence, and averaging them buries it. Effort is the term that varies most between teams.

Worked example. Three items, same impact (2), confidence (0.8) and effort (3), so the score moves only with reach.

ItemEstimated reachActual reachScore at estimateScore at actual
bulk edit12,0001,3006,400693
import retry3,00011,4001,6006,080
saved views8,0006,9004,2673,680

bulk edit ranked first and was 9× overstated; import retry ranked last and was understated by roughly 4×. The ranking was not wrong because RICE is wrong: it was wrong because reach was estimated from total accounts rather than accounts that reach the screen. That is the failure mode to detect: a top-ranked item whose reach denominator nobody can name.

Alternatives, and when to pick them.

  • ICE: drops reach, so it is faster and weaker; use when reach is roughly equal across candidates.
  • Weighted shortest job first (cost of delay ÷ duration): use when duration varies by more than about 3×, which RICE ignores entirely.
  • Cost of delay / CD3: use when delay has a defensible money figure (revenue, penalty, churn); it forces the argument into units the business already uses.
  • Opportunity scoring or Kano: use for feature gaps in an existing product, where importance beats reach.
  • No framework: use when the candidate set is not comparable (a compliance fix against a growth experiment), when one item is a bet that invalidates the rest, or when the real question is strategy rather than ordering.

Failure modes. Reach estimated from the wrong denominator: detect by asking each top item's author to name the population. Confidence used as a tie-breaker: detect by having two people score the same item and flagging gaps over 0.2. Effort estimated by the person who wants the item: detect by comparing the estimate to the last three similar items shipped. And the meta-failure: a ranked list nobody re-checks, so the same overestimate survives four quarters.

What I would do. Score, read the top and bottom three, ask whether the order matches judgement, fix the input where it does not, record every estimate with its author, and compare estimates against actuals for the next quarter's shipped items. The score is an argument written down.

Curated: · Written: · Reviewed:

Q&A-15What do you write down before asking for a team's quarter?

Before asking for a team's quarter, I write down five things: the problem, who has it (a number pulled from product data, not a persona), the behaviour that must change, the value if it changes, and the result that would prove the bet failed. With a date attached.

The falsifier is the part people skip. Without it the document is advocacy, and advocacy can't be stopped once funded.

How heavy should the document be? Match it to the cost of reversing the decision, not to how important the bet feels.

  • One-pager, half a day. Reversible bet, one team, one quarter, no new system, no other team's interface changes. Problem, population, behaviour, value, falsifier, date. This is the default and it covers most quarters.
  • RFC or design doc, a few days, reviewed by the affected teams. The bet requires someone else to change an API, schema, or shared service. The RFC is where the interface and migration get argued; the opportunity framing rides in the summary section.
  • Full opportunity assessment with a sizing model. Expensive or hard to reverse: multi-team, multi-quarter, a pricing change, a data migration. Here the sizing model and the falsifier have to survive scrutiny, because being wrong costs several teams a quarter.
  • PRD. Not a substitute at any weight. A PRD answers how we build it after the bet is funded. Writing one before a falsifier exists is the most common way a team ends up with a quarter it can't cancel.

If you can turn the thing off in a week, a full assessment is overkill: it delays the start and buys rigour you could have bought by shipping.

Failure modes I've watched. No falsifier: a 3-team quarter funded on a memo with no failure condition ran 9 months, goal redefined each quarter, 0 recorded checks against the original claim. A falsifier that can't be measured: "users are happier" is not one. A sizing model nobody can audit: the number gets quoted for years after the assumptions rot. A date with no owner: the check never happens.

Worked example, hypothetical figures:

FieldValue
affected accounts3,400
behaviour that must changeweekly imports per account, 1 → 3
value if it holds$1.2M annual
falsifierunder 1.6 imports/account/week by day 90
checkday 90, owner named in the doc

Before I say yes, I check one thing: does the document name a date and a number that would count as failure, and is that check on someone's calendar? If not, it's a one-pager that hasn't finished being written.

Curated: · Written: · Reviewed:

Q&A-30An executive asks for a feature that is not on the roadmap. What do you do?

The 90-second answer

I don't answer yes or no in the room. I ask two questions, price the request in the currency the executive already tracks, committed dates, and hand the choice back to whoever owns the roadmap.

The two questions: what event does this have to land before, and what breaks if it lands a quarter later? Half the time the second answer is "nothing," and the request is a preference, not a requirement. That reframes the conversation from "can you do this" to "when do you need it."

Then I size it. A range, not a point: "3 to 5 team-weeks, assuming the data is already in the warehouse; if it isn't, add 2." State the assumption out loud so it can be challenged.

Then the part that actually decides it: the roadmap is a queue, not a wish list. Work added at the front pushes named items out. So I bring the executive a choice, not a refusal: illustrative numbers:

OptionShipsSlipsNew date
Aexec dashboard (4 team-weeks)import retryMar 14 → Apr 11
Bimport retryexec dashboardQ3

"Which do you want?" If they say both, that isn't a scheduling answer, it's a staffing one, so I ask for the headcount or accept the slip. Either way the decision is theirs, because they own the company-level trade-off and I own the team-level one.

Then I write it down: the decision, the displaced item, the new date, in the roadmap doc and the decision log. The affected team hears it from me before they hear it from a date moving.

Failure modes

Silent absorption is the one that gets people. A PM I worked with took three executive requests in a quarter without touching the published plan: a dashboard (5 team-weeks), a partner export (4), a logo refresh (2). Eleven team-weeks, roughly a quarter of one team's capacity. Three committed dates slipped, and the operating review logged it as a delivery problem, not a scope decision. The team took the blame for a call they never made.

The mirror failure is escalating everything. If every request goes to the exec, I've outsourced prioritization and I'm useless. Requests under a day or two I absorb and note in the log; the threshold is roughly "does this move a committed date." If it doesn't, it isn't a decision, it's work.

And "the exec asked for it" as the reason given to the team is corrosive: it tells them the plan is fiction and their commitments don't matter. The reason is always the trade-off, never the title.

If they insist

If it's the CEO and they've heard the cost and still want it, it ships. They own the trade-off at that level. What I don't do is pretend it was free: the displaced item gets a new date, its stakeholders get told, and the log records who decided.

Curated: · Written: · Reviewed:

Q&A-31How specific should a roadmap be six months out?

I would treat roadmaps as commitment versus direction as a question about behaviour, so I would look for the behaviour before the opinion.

Confidence decays with distance, so a roadmap should carry decreasing specificity: dated commitments near, themes with intent in the middle, and problems without solutions at the far end. Publishing distant work with dates converts every normal learning event into a broken promise.

Concretely, split the roadmap into committed, planned, and exploring, and say what moves an item between them. Attach dates only to the committed band. Republish on a fixed cadence so movement is expected rather than announced as a failure.

The reason for that specificity is a failure I have seen: A team published 11 dated items covering 3 quarters, delivered 4 on their original dates, and the account team had already promised 7 of them to customers in writing, generating 23 escalations when the plan changed.

Three bands and what they may carry.

BandHorizonCarries a dateRepeatable to customers
committed0 to 6 weeksyesyes
planned1 to 2 quartersnowith caveat
exploringbeyondnono

I would not consider it settled without evidence: Check what the sales team has repeated to customers and confirm it matches the committed band only.

Precision you cannot honour is a promise you did not mean to make.

Curated: · Written: · Reviewed:

Q&A-44How do you plan a quarter when part of the team's time is unpredictable?

Assumptions: a team of 6–8 engineers, a 13-week quarter, one primary on-call rotation, and no hiring or reorg inside the quarter. Under those, the answer is: reserve the share of capacity that unplanned work actually consumed over the last two quarters, commit only the remainder, and make the reserve a line item rather than a hope.

Measure first. Pull the last two quarters of tickets and classify each as planned or unplanned: support, incidents, compliance, security reviews, and the follow-up work they generate. The unplanned share is the reserve. If it ran 31% and 36%, reserve a third and plan against two-thirds.

QuarterCommitted to planned workConsumed by unplanned workCommitments met
Q1100%31%4 of 5
Q2100%36%3 of 5
Q366%34%4 of 4

Q1 and Q2 planned at 100% of capacity, so total demand was 131% and 136%: the plan was arithmetically impossible before the quarter started. Q3 planned at 66%, unplanned landed at 34%, total 100%, and every commitment landed. (Illustrative figures.)

Alternatives, and when each is right:

  1. Full reserve: plan at measured capacity. Right when unplanned work is broad and uncorrelated: many small interrupts from many sources. Cost: you commit less than the org wants, and you have to defend the number in the planning meeting.

  2. Partial reserve with commitment tiers: commit a hard core at, say, 50% of capacity and a best-effort tier above it. Right when some commitments have external deadlines and others don't. Cost: the best-effort tier gets read as a commitment anyway unless you name the trade explicitly.

  3. Reduce the unplanned load at the source: on-call restructuring (fewer pages, better runbooks, a secondary rotation), fixing the one flaky service, automating the manual compliance step. Right when the last two quarters show unplanned work concentrated in a few recurring causes. Cost: it's a project, so it competes for the same capacity.

A full reserve is wrong when the unplanned work is dominated by one or two fixable causes, or when the reserve would starve a commitment with a hard external date. In that case take the risk explicitly: name the commitment you're protecting and the reserve you're spending.

Failure modes to watch:

  • The reserve becomes a dumping ground. Anything unowned gets charged to it and it grows every quarter. Detect by tracking what lands in the reserve and whether the same causes recur.
  • Sandbagging. The reserve is padded beyond the measured share. Detect by comparing the reserve to the measured unplanned share; if it's consistently larger, that's the signal.
  • The reserve gets spent on planned work in week 3. Detect by checking at mid-quarter whether the reserve still exists.

The one number to defend in the planning meeting is the reserve, and the evidence for it is the last two quarters' classification.

Curated: · Written: · Reviewed:

Q&A-45Two weeks into a quarter the company changes direction. What do you do with the plan?

What I'd say in the first two minutes

Assumptions first: the change is a real, durable redirection from the people who own the quarter, not a one-week spike, and the quarter's capacity is fixed. If either is false the answer changes: a spike gets absorbed, and a capacity increase gets bought rather than traded.

Given that, a direction change is a trade, not an addition. My first move is to make the trade explicit with whoever owns the decision: "If this starts, what stops?" I don't accept the new work until something is named as displaced. That question is most of the answer; the rest is execution.

Then I execute it in writing:

  • Stop the displaced work explicitly. Not "deprioritize": stop. Record its state in the backlog so it can resume without archaeology.
  • Restate the quarter's commitments with new dates. The old dates are now fiction.
  • Tell every downstream team and customer-facing function the same day. A plan change that reaches engineering but not sales fails in week 6.

Why the trade is the point

The failure mode is adding without removing. It looks like flexibility and it is the most common way a quarter ends with nothing shipped. A team that adds a priority in week 2 and removes nothing doesn't get 20% more done; it gets the same throughput spread across more work in progress, so everything finishes later and some things never finish.

Illustrative, not measured: a team I worked with added a new priority in week 2 without stopping anything. By quarter end it had four half-built initiatives, and three were abandoned the next quarter. Call it roughly 22 team-weeks that produced nothing released. Treat those numbers as a shape, not a measurement.

The same trade, three ways, again illustrative:

ResponseInitiatives in flightReleased by quarter end
add only51
add and stop two43
defer the change44

The table is the argument. "Add and stop two" ends with fewer initiatives in flight than "add only" and more shipped. Deferring ships the most this quarter but ignores the direction change, which is only right if the change is not durable.

When not to do this

If the new direction is genuinely additive (new budget, new headcount, a separate team) nothing needs to stop, and forcing a trade is theater. If the change is a one-week spike, absorb it and protect the plan. The trade is for durable redirection against fixed capacity.

How I'd know it's settled

Not by a meeting. By evidence: the restated plan names what stopped, the backlog entries for stopped work carry their state, and the downstream teams can state the new dates back to me. If any of those is missing, the change hasn't landed.

Curated: · Written: · Reviewed:

Q&A-46An executive wants a date before the design exists. How do you answer?

In the room I say three things: a range, the assumptions that set its ends, and a date for a decision rather than for delivery.

Something like: "Six to fourteen weeks to a working prototype. Six if the vendor's API does what the docs claim and we can reuse the existing auth service; fourteen if we have to build the ingestion path ourselves. I can tell you which end we're on by the 12th, once the spike is done."

That's the whole answer. The rest is defending it.

Why a range and not a date: before the design exists, the width of the range is the only honest measurement of what's unknown. A single number is a point estimate of a distribution nobody has sampled. The executive remembers the number, not the caveat, and the number travels without you.

Here's the mechanism. The hallway answer becomes the plan. A PM was asked for a date in a corridor, said "eight weeks," and that number was in a board deck nine days later as a commitment. The team's internal range was six to fourteen weeks the whole time. They shipped in fifteen.

ArtefactFigure
internal range6–14 weeks
hallway answer8 weeks
board deck8 weeks, committed
actual15 weeks

Nobody lied: the range was compressed to its midpoint by the act of being asked, and the midpoint hardened into a promise.

So the concrete moves:

  1. Give the range with the assumptions attached to each end. "Six if X, fourteen if Y" is falsifiable; "six to fourteen" alone is not.
  2. Offer a date for a decision point. "I'll know which end we're on by the 12th" is a commitment you can keep, and it's the thing the executive actually needs in order to plan.
  3. Set a cadence. Weekly, same channel, and each update says what narrowed the range and what didn't.
  4. Close the loop on what gets repeated. Ask the person who asked to read back what they'll tell their own boss. If they say "eight weeks," correct it then, in the room, not in the deck.

Failure modes worth naming:

  • The low end gets quoted. Executives plan against the optimistic end. Say the high end out loud too, and say what would have to be true for the low end to hold.
  • The range never narrows. If it's still six to fourteen after three weeks, the spike isn't answering the question that matters. That's a signal to change the spike, not to keep reporting the same range.
  • The decision date slips silently. If the 12th arrives and you don't know, say so on the 12th with a new date and the reason. A missed decision date is recoverable; a missed delivery date that was never a commitment is not.
  • The range is treated as a commitment anyway. If the org genuinely needs a committed date, that's a scope conversation, not an estimating one. Commit to a smaller scope with a date, and keep the rest as a range.

An early date is a rumour with your name on it.

Curated: · Written: · Reviewed:

Q&A-27Two initiatives both look worthwhile. How do you choose?

The 90-second version

First clarify what's being decided: which of the two we do first, or which one we do at all. Those have different answers, and the deadline forcing the choice usually tells you which one you're in.

"Both are worthwhile" is not a comparison: it's two separate comparisons against zero, and anything beats zero. Convert both to the same unit: expected value per engineer-week of the same team. Then rank, don't approve. The winner has to name the item it displaces.

If the two land within noise of each other, break the tie on reversibility and information: do the one that's cheap to undo and tells you something you can't learn otherwise. If one is expected-value work and the other is tail-risk work, don't force them onto one axis: ask whether the tail is survivable, and fund the risk buy from a risk budget rather than the roadmap.

Mechanism

The constraint is the team's capacity, not the supply of good ideas. The cost of an initiative is the next best thing that team could have done, so the comparison is between alternatives.

Value has to be stated in a unit you can defend: orders, revenue, incidents avoided, engineer-weeks unblocked elsewhere. And it has to carry an evidence tier: measured beats instrumented estimate beats belief.

Worked example (hypothetical figures)

Team of 5, 13-week quarter = 65 engineer-weeks; reserve 20% for on-call and support → 52 engineer-weeks of project capacity.

A: search latencyB: multi-region failover
Cost8 eng-weeks20 eng-weeks
Claimed effectp99 900 ms → 200 msRTO 4 h → 15 min
Basis3% of sessions hit the slow path; conversion 1.8% there vs 2.4% on the fast path2 region incidents in 3 years, ~$40k each
Value0.03 × 0.6 pp = 0.018 pp; 2M sessions → 360 orders; at $80 AOV = $28.8k/quarter(2/3) × $40k ≈ $27k/yr, ~90% removed ≈ $24k/yr
Per eng-week$3.6k$1.2k

A wins 3:1 on the same axis. But B's number is an average over a fat tail: the same 4 hours during peak season is a different figure. So the real question isn't A vs B on $/engineer-week; it's whether the worst plausible version of B is survivable. If a 4-hour payments outage in December is a $2M quarter, B is a risk buy and comes out of a different budget.

Failure modes

  • Approving against capacity. Seven items each with positive return, 41 engineer-weeks needed against 26 available: three shipped, four unfinished, 19 engineer-weeks burned with nothing releasable. Detect it by comparing committed engineer-weeks to delivered at the end of every iteration; two iterations of committed > capacity means the plan is fiction.
  • Sunk cost. "We're 60% through" is not a reason unless the remaining 40% buys the whole outcome.
  • WIP. Two half-done initiatives deliver zero. Cap in-flight work at what the team can finish.
  • Loudest sponsor. The ranking is only real if the loser is written down.

When this doesn't apply

If B depends on A, the order is forced: that's sequencing, not selection. If one is a contractual or compliance deadline, it's a constraint: it comes out of capacity before anything is ranked. And if both are cheap to reverse, do the one that buys information first.

Curated: · Written: · Reviewed:

Q&A-28How do you order work when one team's output blocks another's?

The 90-second answer

First classify the dependency. Hard: the consumer can't compile, run, or test without the producer's artifact. Soft: the consumer needs a decision or a data shape, not the running thing. Hard dependencies get sequenced; soft ones get decoupled with a written contract.

For a hard dependency, make the blocking item earlier and smaller. Split the producer's work into two milestones: the interface (schema, endpoint signature, error codes, versioning) and the implementation behind it, and ship the interface first as a stub the consumer can build and test against. The consumer's start date is set against the stub date, not the producer's real date.

Size the producer's slack from their own history. If their last five deliverables landed at +0, +2, +1, +9, +3 days against estimate, the p80 is about +3, so the consumer's integration date carries +3 to +5 days of buffer, and the consumer's non-dependent work fills the gap.

If you can't stub it, you don't control the date. Then either reorder so the consumer isn't on the critical path, or accept the serialization, staff the consumer with other work, and track the producer weekly.

Why the joint risk is worse than either estimate

Two independent estimates become one risk. If each team has a 30% chance of slipping, the chance at least one slips is 1 − 0.7² = 51%. And the consumer's idle time is the producer's slip, not the consumer's own variance: the consumer's estimate doesn't protect them.

Worked example, hypothetical figures. Producer estimates 10 days; history says 60% on time, 30% +3 days, 10% +10 days. Consumer needs 8 days of dependent work.

  • Consumer waits for the real API: unplanned idle = 0.6×0 + 0.3×3 + 0.1×10 = 1.9 days expected, 10 days worst. Feature lands day 18 expected, day 28 worst.
  • Contract defined in 3 days, consumer builds days 3–11 against a stub, integration 1–2 days after both finish: idle 0, rework 1–2 days. Feature lands ~day 13 expected, day 22 worst.

Three orderings of the same pair

OrderingUnplanned idleReworkFeature lands (expected / worst)
Wait for the real API1.9 d / 10 d0day 18 / day 28
Contract stub first01–2 dday 13 / day 22
Build blind, integrate late08 dday 18 / day 28

Building blind trades idle for rework and lands on the same date as waiting, with 8 days of discarded work. Stub-first is the only row that moves the date.

Failure modes, and how you catch them

  • Stub drift. The stub is a guess and the real implementation diverges. Catch it with consumer-driven contract tests (Pact or equivalent) run in the producer's CI. If the producer's build doesn't run the consumer's contract, drift is invisible until integration.
  • Permissive stub. A stub that accepts any payload lets the consumer build against a fiction the real endpoint rejects. Make the stub enforce the same schema and validation as the contract.
  • The interface milestone gets deprioritized because it ships no user value. Give it its own acceptance criteria and tracked date; its slip is the early signal, not the implementation slip.
  • Hidden serialization. Both teams report "started" while the consumer is blocked on a decision, not an artifact. Ask what the consumer will commit code against this week.

When not to stub

If the interface is genuinely unknown (a novel protocol, or a dependency on the producer's production data) a stub is a fiction that costs more rework than waiting. Run a timeboxed spike with both teams in the room, then sequence.

Curated: · Written: · Reviewed:

Q&A-29Engineering wants a quarter for refactoring. How do you decide?

What I'd say first

"A quarter for refactoring" bundles two decisions: which debt, and whether it needs a contiguous block. I answer the first with evidence and let the second fall out of the shape of the change.

For each candidate I ask three things: which named roadmap items it blocks, how many incidents trace to it in the last 12 months, and whether the change is structural or localised. Structural means partial states are worse than either end (a schema migration, a service split, a serialization format swap) you cannot ship half and stop. Localised means each slice ships independently and you can walk away at any point.

The evidence, for a hypothetical 6-engineer team

AreaIncidents (12 mo)Roadmap items blockedChange shapeRight unit
reporting service20localisedincremental, if at all
checkout path93schema + service splitdedicated block
admin tooling11localisedincremental

The arithmetic that decides it

A full-team quarter is 6 engineers × 13 weeks = 78 engineer-weeks. A half-team quarter is 3 × 13 = 39.

Checkout's drag: 9 incidents × roughly 4 engineer-hours of response and follow-up = 36 hours, about 0.9 engineer-weeks a year. Under one engineer-week, so incidents alone never buy a quarter. The real cost is the three blocked roadmap items: at 8 engineer-weeks of downstream work each per quarter they stay blocked, that is 24 engineer-weeks of delayed value per quarter, plus 18 engineer-weeks of one-time workaround when they are finally built.

So the full-team ask pays back in 78 / 24 ≈ 3.25 quarters: over a year, too slow. A half-team block pays back in 39 / 24 ≈ 1.6 quarters. That is the number to put in front of the roadmap owner, and it says fund three engineers, not six.

Quarter or incremental: the trade-off

A dedicated block is right when the change is structural, so a half-migrated state is worse than either end; it has one owner and a measurable end state; and the roadmap can absorb the delay without missing a committed date. It buys focus and avoids the dual-code-path tax that drags for months when you slice a structural change. It costs a quarter of feature delivery with no external forcing function: no customer deadline keeps a refactor honest, which is why these blocks overrun.

Incremental allocation, a standing 20% of each sprint, is right for localised debt: it keeps a shipping cadence, gives a natural stop point, and catches a wrong estimate in week two instead of week ten. Its failure mode is the opposite: it gets raided the first time a launch slips and the debt never moves.

The unit follows the drag. If the debt is eating half the team's capacity, 39 engineer-weeks a quarter, a full-team quarter pays back in two quarters and is worth it. Below that, slice.

Failure modes and how I'd detect them

  • The quarter that never ends. Detect by milestone slippage: if the first two milestones land late, the estimate is wrong by a factor, not a week. Kill criterion up front, if the new path is not carrying 50% of traffic by week 8, stop and revert to incremental.
  • Behaviour regression in the rewrite. Detect with the 90-day incident count and error-budget burn in the touched area against the same window before, plus p99 latency on the migrated path.
  • Debt the metric cannot see. Onboarding time and deploy frequency do not show up in incident counts. Track lead time and change failure rate before and after; if neither moves, the refactor bought nothing measurable.

What I'd commit to

Fund checkout as a three-engineer, one-quarter block with a named end state and a kill criterion; keep reporting and admin on incremental allocation; re-measure at the end. Fund the debt standing in front of something you want, and pay for a contiguous quarter only when the change cannot be sliced or the drag is at least half the team's capacity.

Curated: · Written: · Reviewed:

Q&A-35You can fund an initiative at half the requested size. Should you?

Short answer: only if the half-size version is a complete thing someone can use. Half the money is a question about the shape of the value curve, not the size of the ask. If value is roughly linear in spend, fund half and ship the smaller slice. If value is a step function, nothing until cutover, half funding buys the full carrying cost and none of the outcome, and the right move is to cut scope to a smaller complete thing or decline.

The test I'd apply. Three questions, in order:

  1. What ships at the half point, and who uses it? Name the customer and the deliverable. "Half the migration" is not a deliverable; "the new path for one account segment, with the old path still authoritative for the rest" is.
  2. Is the half point on the critical path or beside it? If the remaining work is a cutover, a compliance deadline, or a security fix, partial delivery is worth zero.
  3. What does stopping at the half point cost? Dual-run, two systems to maintain, a partial migration you can't roll back.

Half the people is not half the duration. This is where "half the cost" usually turns out to be false. Cutting a team from six to three rarely halves the calendar time; coordination, ramp-up and the fixed parts of the work (design, review, the cutover itself) don't shrink. You often pay more in elapsed time and dual-run than you save in salary.

Illustrative arithmetic, hypothetical but the shape is what matters. A migration estimated at 6 engineers × 4 months = 24 engineer-months. Funded at half, it runs 3 engineers × 7 months = 21 engineer-months, so you save 3 engineer-months, about $60k fully loaded at $20k/month. But the old and new systems now run side by side for 7 months instead of 2, at $24k/month, so dual-run costs 5 extra months × $24k = $120k. The "saving" is a $60k loss before you count the two teams' attention.

Value curve shapes.

ShapeHalf funding deliversFund half?
Linearroughly half the value, as a usable sliceYes, if the slice ships
Step / thresholdzero until cutoverNo: cut scope or fund fully
Diminishing returnsmost of the value earlyYes, and stop at the knee
Superlinear (platform, network effects)less than halfUsually no

Failure modes and how I'd catch them. The half-funded state becomes permanent: the old path is never deleted, and the dual-run cost is now a line item nobody owns. Detect it by putting an expiry on the half state, a date and an owner for deleting the old path, before the work starts. Second, the half version is unusable but shipped: a partial migration that reads from both systems and is correct in neither. Detect it by requiring the half version to pass the same end-to-end test as the full one, on a real customer. Third, attrition: engineers who watch a project get defunded mid-flight leave, and the replacement cost exceeds the saving.

What I'd say. "I can fund half. Here's what ships at that number, here's who can use it, and here's the date we delete the old path. If none of those three is true, the honest options are to fund it fully, cut scope to a smaller complete thing, or not start."

Curated: · Written: · Reviewed:

Q&A-36How do you decide whether to build a capability or buy it?

The 90-second answer. Two questions decide it. First: would a customer pick us because of this capability, or would they notice only if it were missing? Second: how fast does the capability move. Is the frontier shifting every quarter, or is it a solved problem behind a stable interface? Build when it differentiates and moves fast. Buy when it is necessary but undifferentiated and the vendor's roadmap moves at least as fast as yours. Partner when you need the capability and the relationship but not the control: a new market's payment rails, a joint go-to-market.

Rate of change is the axis people skip. Buying a fast-moving capability means inheriting the vendor's release cadence; if they ship twice a year and you need weekly, you will end up building anyway, having paid for the detour.

Then price it honestly: three-year total cost of ownership, not licence fee against first-release estimate. Include integration, the headcount that stays after launch, and the cost of switching later.

The arithmetic. Illustrative figures: a vendor search licence at $60k/year, one engineer-month to integrate. The team builds instead. First version takes 14 engineer-months; at a loaded $150k per engineer-year that is about $175k, and it ships at month 8. Two engineers stay on relevance tuning permanently: $300k/year loaded, so $100k of that lands in year 1.

PathYear 1Years 2–33-year total
Buy$60k licence + ~$12k integration$60k/yr~$192k
Build~$275k$300k/yr~$875k
Partnerrevenue share, no licencerevenue share + joint roadmapdepends on volume

Build costs roughly 4.5x buy over three years. That is a bargain if search relevance is why customers choose you, and a write-off if it isn't. The ratio doesn't decide; whether the capability sits on the critical path of the purchase decision does.

What actually goes wrong.

  • Counting licence fee against first release. The first release is the cheap part of building; the permanent team is the expensive part. Detect it: if the build team's headcount hasn't dropped 12 months after launch, you didn't ship a feature, you bought a product org.
  • Buying something on the critical path without exit terms. Check before price: can you export your data in a usable format, what does egress cost, is there a self-host or source-escrow option, what is the price-protection clause at renewal. A vendor acquired in year 2 and repricing at renewal is the common version of this.
  • Partnering without a named owner. Revenue share plus a joint roadmap means two companies' priorities have to stay aligned; when they diverge you have neither control nor a clean exit. Detect it: if the joint roadmap has slipped twice and nobody on your side owns the relationship, it is already failing.

When not to. Don't build to save a licence fee: the licence is rarely the cost. Don't buy a capability that is the reason customers pick you, however cheap the vendor looks. Don't partner for something you could buy off the shelf; partnership is for when the relationship itself is the asset.

Curated: · Written: · Reviewed:

Q&A-38How do you justify platform investment that no customer asked for?

"No customer asked for it" isn't the test. The test is whether I can name the features this makes cheaper and put a number on the difference. Without that list, platform work competes as an act of faith against work that has customers attached, and it loses.

So the first move is to write down the next four to six roadmap items that depend on the change and estimate each one twice: as it would be built today, and as it would be built on the platform. The delta is the return. Fund only the slice that serves the nearest two items: the rest is a bet on roadmap items that may never arrive.

Upcoming featureWithout platformWith platform
tenant branding4 weeks1 week
regional pricing6 weeks2 weeks
partner fields3 weeks3 weeks

Two things keep that table honest. A row where the numbers are equal is a signal, not noise: it says the platform is being sized along an axis the roadmap doesn't move on, and I should shrink scope until every funded row shows a real delta. And the estimates are guesses until delivery, so the commitment isn't the estimate: it's the recheck. After the two features ship, I compare actual against projected. If the delta didn't materialise, the next slice doesn't get funded.

The failure I've watched: a team spent 9 weeks on a configuration framework justified by faster feature delivery. Over the following two quarters, 1 feature used it and 6 bypassed it, because the framework assumed a data shape the product had already moved off. The estimate was never wrong in the room; it was wrong in the code, and nobody measured. Detection is cheap: instrument adoption. Count features that go through the abstraction against features that route around it. A bypass rate climbing past a third means the abstraction's assumptions are wrong, and the fix is to change the abstraction, not to mandate it.

Not every platform investment is a feature-velocity bet, and the two arguments shouldn't be mixed. Work that cuts operational load (fewer pages, lower error-budget burn, shorter incident recovery) is justified on those numbers instead: pages per week before and after, p99 latency, MTTR. Answering a reliability question with a feature-velocity table is answering the wrong question.

When the list is thin, the honest answer is to not build the platform yet. Build the feature-specific version twice, and extract the shared piece on the third use, when the shape is known. That costs more in the short term and is usually right.

If I can't name the features, the answer to "why are we building this" is "we can't say yet", which is a reason to wait, not a reason to fund.

Curated: · Written: · Reviewed:

Q&A-40How do you decide which features belong in which tier?

The decision tiering feeds is how revenue grows without renegotiating every contract, so I start by naming the value metric: the unit that grows as the customer gets more value from the product. Seats, API volume, events processed, records under management, and governance scope are the usual candidates. A good metric passes three tests: it grows when the customer succeeds, you can meter it accurately and cheaply, and the customer accepts it as a fair proxy for what they get. Fail any of those and the boundary built on it will leak.

Value-based boundaries hold for a specific reason. When the bill tracks the customer's own growth, you can raise price without a negotiation, because their usage already moved. That is what makes expansion revenue automatic and net revenue retention above 100% possible. The upgrade trigger is also self-inflicted: the customer hits a limit they created by succeeding, so upgrading reads as their decision rather than a hostage situation. Feature-gated boundaries invert this. The trigger is a capability the customer already assumed was included, so the upgrade feels like a penalty and the sales team starts granting exceptions to close.

To pick the metric, rank candidates on correlation with value, metering cost and accuracy, and customer acceptance, then check each boundary against the segment. Governance features are the trap here: SSO, audit logs and role controls only matter at scale in a mid-market account, but in a regulated segment they are table stakes at every tier. The same capability can be a legitimate top-tier feature in one segment and a churn driver in another, so the boundary has to be drawn per segment, not per feature list.

Here is a worked example with illustrative numbers, not a real account. A product puts its audit log behind the top tier. A compliance rule then makes audit logging mandatory for every customer in its mid-market segment, so the capability is table stakes there. Suppose 34% of mid-tier renewals raise it as a blocker, the team grants 19 one-off contract exceptions, and 6 deals are lost before repackaging.

Tier boundaryExceptions grantedDeals lost
audit log at top tier196
seats at mid tier10
API volume21

The pattern in that table is the diagnostic: boundaries on the value metric (seats, API volume) produced almost no exceptions, while the boundary on a table-stakes capability produced nearly all of them. The numbers are made up to show the shape of the signal; the shape is what you look for in your own data.

The trade-offs matter here. Pure usage-based pricing tracks value closely but makes revenue volatile and forecasting hard, which finance will push back on. Seat-based pricing is predictable but decouples from value for automation-heavy customers who get more out of the product with fewer people. Most companies land on a hybrid: a platform fee that covers baseline cost plus a usage or seat component that scales. That is a reasonable default, not a law.

To check whether the packaging is working, I'd count contract exceptions and discount requests per quarter, tag renewal conversations where a feature was raised as a blocker, and measure upgrade conversion at each boundary. A boundary with a high exception rate and low upgrade rate is mispriced; a boundary with high upgrade conversion and no exceptions is doing its job. I would not call the packaging settled until two quarters of that data agree.

Curated: · Written: · Reviewed:

Q&A-41How do you decide between fixing a known bug and shipping the next feature?

Core answer

Bug-versus-feature is the wrong frame. Both are work items drawing on the same capacity, so the question is which has the higher cost of delay per engineer-week. Estimate both, compare, and let the number decide, then apply two overrides: irreversible damage (data corruption, money already sent, a security hole) jumps the queue regardless of size, and externally committed dates (contract, regulator) turn this into a scope question, not a priority question.

State three assumptions before estimating: is the damage reversible, is there a workaround and what does it cost per week, and is the feature's value contingent on the bug being fixed. A feature built on corrupted data is worth nothing.

The arithmetic

Hypothetical: an invoice-rounding defect.

InputValue
accounts invoiced monthly1,100
mean error per invoice$9.20 (max $40)
months deferred6
mis-billed total1,100 × $9.20 × 6 = $60,720
support tickets3% of accounts = 33/month × 6 = 198 × 25 min = 82.5 h
credits issued~$61,000

Cost of delay ≈ $10,120/month direct, plus ~14 h/month of support, plus the finance audit.

Now the feature: it unblocks 200 accounts at $50/month expansion, live one month after ship. Cost of delay = $10,000/month, starting in a month.

The two are within a couple of percent, so the tie-breaker is reversibility. The bug's cost is already incurred and compounding: every month of delay adds another 1,100 wrong invoices that must be credited and audited. The feature's cost is deferred, not lost. Fix the bug.

Alternatives, and when they beat the estimate

  • Error budget policy. For reliability defects, an SLO with a burn-rate policy settles this without a meeting: if the budget is exhausted, reliability work wins by rule. Use it when the bug is an availability or latency defect and you already have an SLO; don't stretch it over correctness bugs that never show up in the SLO.
  • WSJF (cost of delay ÷ job size). Matters when the bug is a two-hour fix and the feature is six weeks. Dividing by size flips the ranking that raw cost of delay gives you.
  • Fixed dates. If a regulator or a contract sets the date, priority is settled; negotiate scope instead.

Failure modes

  • Severity labels as a proxy. "P2" says nothing about accounts affected. Detect by sampling closed bugs and checking whether the label predicted the blast radius.
  • Escalation-driven priority. The loudest customer wins. Detect by counting how many decisions in a quarter were driven by a single escalation rather than the estimate.
  • Silent compounding. Each week the bug looks cheap, so it never clears the bar. Detect by tracking open-bug age × weekly cost; a $2,300/week bug open 26 weeks is a $60k decision nobody made.
  • Feature value with no denominator. "Strategic" with no metric means the cost of delay is unknown, not infinite. Detect by requiring a named metric and an expected delta before the feature enters the comparison.

Small is not a property of bugs; it is a property of their consequences.

Curated: · Written: · Reviewed:

Q&A-47How do you slot regulatory work into a roadmap that is already full?

Regulatory work isn't a roadmap item competing for a slot; it's a constraint with a date someone else set. So the first move is to put the deadline on the plan, compute the latest start date, and decide what comes off, not to score it against features and hope the arithmetic works out.

Why it can't be scored like a feature. Three properties break the comparison. The deadline is fixed and doesn't move when you're late. The consequence is non-linear: a missed attestation can block a market or a renewal, which no feature's value compares to. And the scope is elastic: the same requirement is a two-week control or a two-quarter program depending on interpretation. Scoring it against features hides all three behind a single number.

The four real options. Pick by what's actually scarce: cash, calendar, or a specific skill:

OptionRight whenLead time / costWhat it risks
Descope featuresWork is small relative to a quarter and cut items are restartable$0 cash, immediateDeferred roadmap value, morale
Hire or contractBottleneck is a scarce skill (a specific control, a migration) and deadline is under a quarter away2–4 week ramp, premium rateKnowledge walks out with the contractor
Buy a vendor solutionRequirement is commodity (evidence collection, a regional cloud region, tax engine)4–12 week procurement + security reviewLock-in; the vendor's own certification timeline becomes yours
Negotiate deadline or scopeCounterparty is a customer or internal compliance owner, not a regulatorUnknownYou're relying on someone else's discretion

A fifth, cheaper than all of them: shrink the interpretation. Get the accountable owner (legal or compliance, not the engineer reading the regulation) to confirm the minimum acceptable scope in writing. The strictest reading routinely inflates scope several-fold.

What comes off the roadmap. Rank candidates for removal by four questions: does it have its own external date? Is it reversible if paused? Does its value decay with delay (launch windows, seasonal demand)? Is it a prerequisite for something else? Cut reversible, non-dated, non-decaying, non-prerequisite work first. Sunk cost is not a reason to keep anything.

Worked example (hypothetical figures). An 8-engineer team, quarter fully planned, learns of a data-residency requirement with a 5-month deadline. Costed options:

OptionCashCalendar
Descope two features$0Fits with slack
Two contractors~$340k6-week ramp, 9-day feature freeze for reviews
Vendor regional hosting~$60k/yr8-week procurement, then migration
Phased attestation$0Depends on the counterparty

At 5 months, descoping is sufficient and cheapest. The contractor route is what you buy when you discover the requirement at 6 weeks: same outcome, 340k and a freeze later.

Failure modes, and how you'd see them. Interpretation drift: scope grows after kickoff because nobody confirmed the minimum in writing. Detect by diffing the control list at week 2 against week 6. Procurement lead time: the vendor's security review is the critical path, not your integration. Detect by tracking the vendor's review queue, not your own tickets. Contractor ramp: the 2–4 weeks is real and unpaid. Detect by measuring first-commit date, not contract signature date. Negotiated deadline: assumed but never confirmed. Detect by requiring the concession in writing before you plan against it.

The operational check is one number: required start date = deadline − (work estimate × your team's historical variance). Alert when today crosses it. A deadline you did not set is a constraint, not a priority.

Curated: · Written: · Reviewed:

Q&A-37How do you retire a feature that a small number of customers still use?

What I'd say first

Retiring a feature is a migration with a deadline, not an announcement. Before anything else I'd pin down what "still use" means: twelve accounts clicking through the UI is a different problem from one partner's cron job polling an endpoint every minute. Then the answer: I keep the feature until every caller has a named replacement, I instrument the old path so "still used" is a number rather than an opinion, and I set a removal date that only moves for a written exception with an owner and an expiry.

Why small usage doesn't mean small cost

Cost is per-change, not per-account. If the feature sits on a shared code path, every future change to that path has to reason about it, and every new engineer has to learn it. Illustrative arithmetic: a feature with 8 monthly users on the billing schema can add a day of review to each of 20 changes a year, 20 engineer-days, while those 8 users generate maybe 2 support tickets. The users are cheap; the coupling is not. That asymmetry is the argument for retiring, and it's also why "only 8 users" is not by itself a reason to keep it.

The plan

  1. Inventory callers, not users. Query access logs or telemetry for distinct principals: human accounts, service accounts, API keys, cron jobs, exports. Product analytics misses the cron job.
  2. Attach revenue and contract terms. A top-10 account by revenue gets a hand migration; a dormant account gets an email and a date.
  3. Ship the replacement first and make it the default for new usage.
  4. Instrument the old path to emit a per-caller counter, and gate removal on that counter being zero.
StageExit criterionGate to next stage
Announcereplacement GA, notice sent—
Assisted migrationtop accounts migrated by handusage by revenue-bearing accounts = 0
Self-serve windowdocs and migration script publishedweekly distinct callers ≤ 5
Removalcode deletedold-path counter = 0 for 14 consecutive days

The counter is the whole mechanism:

# Python 3.12
def legacy_export(account_id: str) -> bytes:
    metrics.increment("legacy_export.calls", tags={"account": account_id})
    return _export(account_id)

Failure modes and how I'd catch them

  • The invisible caller: a partner's cron job authenticates with a service account that never shows up in session analytics. Caught by counting distinct principals on the old path, not sessions.
  • The date that moves: each extension resets the clock and the migration stalls. Caught by treating the removal date as a fixed field; an extension requires a named owner and an expiry.
  • The exception that becomes permanent: one large account gets a "temporary" carve-out that outlives the team. Caught by an expiry date and a quarterly review of open exceptions.
  • The silent data dependency: the feature writes rows another system reads. Caught by checking downstream consumers before deleting the writer, not after.

When not to retire

If the feature is contractually committed, regulated, or the replacement costs more to operate than the old path, the honest answer is to keep it and pay the maintenance cost deliberately, or price the migration into the renewal. Retiring something a customer's compliance depends on is a contract negotiation, not an engineering task.

How I'd know it's done

Old-path counter at zero for 14 consecutive days across all principals, code deleted, docs removed, support runbook entry deleted. Until the counter is zero, the feature is not retired, whatever the announcement said.

Curated: · Written: · Reviewed:

Q&A-42How do you plan for a product whose usage is shrinking?

Answer first

"Shrinking" is three different problems, and they take different plans. Usage can fall because the market is contracting, because a competitor took the segment, or because we stopped investing and the product decayed. Before proposing anything I'd find out which, then name one of three strategies out loud (harvest, turnaround, or exit) and make the funding match the name. The one thing I won't do is keep last quarter's funding level and call that a plan.

Diagnose before you choose

Split the base by cohort (signup month, plan, geography, integration) and look for a cohort that is flat or growing while the total falls. A flat cohort inside a shrinking total means the problem is acquisition, not retention, and that is a different fix from churn. If every cohort decays at the same rate, the cause is the product or the market, not one bad segment.

Three signals decide it:

  • Retention curves by cohort: parallel decay points at product or market; one collapsing cohort points at a broken segment.
  • Win/loss on renewals: losing to a named competitor is a market problem; losing to "no budget" is a contracting market.
  • Support ticket mix: rising defect and workaround volume is neglect, and neglect is the one cause you can fix with engineering.

The three strategies

StrategyFundingEngineeringCustomer messageExit criteria
harvestmaintenance onlybug fixes, security, no new features"stable, no new features"revenue per engineer holds
turnaroundfull team2-3 named bets with dates"here is what is coming and when"cohort retention inflects within 2 quarters
exitmigration teamexport tools, data portabilitydated end of life, 12+ months outmigration rate

Harvest is not "do nothing." It has a quality floor: security patches, data integrity, and the top support drivers still get fixed, or the stable customers funding the harvest start leaving.

Worked example (hypothetical figures)

A product at $8M ARR with 40 engineers, usage down 12% year over year. No strategy is declared, so it gets 2 engineers a quarter (14 engineer-quarters over 7 quarters, 5% of the org's engineering capacity) while revenue falls 41% to $4.7M. At the end, shutdown notice goes out with 60 days, and 3 of the 10 largest accounts, roughly $600K of the remaining ARR, churn inside the notice window because they cannot migrate that fast.

The arithmetic: 14 engineer-quarters of undeclared maintenance bought nothing, and the short notice cost more than the migration tooling would have. A declared exit 4 quarters earlier, with a 12-month migration window and 4 engineers on export tooling, is 16 engineer-quarters, about the same spend, and keeps those accounts.

Failure modes

  • Drift: no strategy declared, funding continues at the old level. Detected by the absence of a written strategy and a budget line that matches none of the three.
  • Harvest without a floor: quality slips, the stable cohort starts churning, and the revenue you were harvesting disappears.
  • Turnaround funded like a harvest: named bets with 2 engineers slip, and you lose credibility with the customers you gave dates to.
  • Exit with short notice: customers cannot migrate, they leave angry, and they tell the market.

What I'd leave with

A one-page decision: the strategy named, the funding line that matches it, the customer message and its date, and the metric that would make me revisit. If I cannot get the strategy named, I escalate, because the default is drift, and drift spends the same money as harvest while delivering none of its returns.

Self-assessment

Rubric: what the interviewer is scoring

  1. Diagnosis before prescription: separates market contraction, competitive loss, and neglect, and names the evidence that distinguishes them.
  2. A named strategy with funding that matches it, not "invest more" or "cut costs."
  3. Exit criteria: a metric and a date that would change the decision.
  4. The customer and contractual side: message, notice period, migration path.
  5. Quantifies the cost of drift rather than asserting it.

Weak-answer patterns

  • "Double down on growth" or "add features" with no diagnosis of why usage fell.
  • Treating all shrinkage as churn and jumping straight to retention tactics.
  • Harvest described as "do nothing" or "maintenance mode" with no quality floor.
  • A strategy with no exit criteria: nothing that would make you revisit it.
  • Ignoring migration and notice cost when proposing exit.
  • Confusing cost-cutting with strategy: cutting headcount is a funding decision, not a strategy.

Likely follow-ups

  • "How do you tell market contraction from competitive loss?" → win/loss on renewals, cohort shape, competitor win rate on the segment.
  • "What is the quality floor in harvest?" → security patches, data integrity, top support drivers; SLA unchanged.
  • "How long a migration window?" → 12+ months for enterprise data, bounded by contractual notice terms.
  • "What if leadership won't name a strategy?" → escalate with the drift arithmetic; propose a default with a review date.
  • "How do you fund a turnaround without new headcount?" → reallocate from harvest-mode products; cap at 2-3 bets.
  • "What metric makes you revisit?" → cohort retention inflection within 2 quarters; revenue per engineer.

Curated: · Written: · Reviewed:

Q&A-43What makes a product strategy useful rather than decorative?

A useful strategy predicts decisions you haven't made yet. If it can't tell you which of two live requests to decline, it's a mission statement with a roadmap stapled to it.

So the test isn't how the document reads. It's whether it beats a coin flip on the next ten decisions. Take three live requests, ones with real money or headcount attached, write down what the document implies, then compare against what the team actually decided. Silent on two of three means it isn't driving anything.

What makes a document predict:

  • A named customer and the job they're hiring you for. "Mid-market ops teams, 5–50 seats" rules out the enterprise SSO request before anyone debates it.
  • The value you're betting on, stated as a mechanism rather than a slogan. "We win because setup takes under an hour, so we optimise time-to-first-value over depth of configuration" tells you what to cut.
  • Capabilities you'll build, in order, and the ones you won't.
  • Explicit non-goals. This is the load-bearing part.

The trade-off. The alternative is a directional strategy: a thesis about where the market goes, with no commitments. That's the right call under specific conditions: pre-product-market-fit, a team under roughly 15 people, or a market moving faster than your planning cycle. If you're wrong about the segment, a constraining strategy costs a quarter of misdirected work; a directional one costs a week of realignment. The deciding criteria are how many teams must act in parallel (coordination cost rises with the count), how reversible the decisions are (cheap to reverse, don't over-specify), and how fast the underlying assumption decays.

A middle option is a portfolio of bets with explicit kill criteria. Stronger than a vision doc for coordination, weaker than a constraining strategy, and the right shape when you have several plausible segments and can run cheap tests.

Failure modes.

  1. Can't say no. Detect it by counting the proposals it rejected last quarter. Zero means decorative.
  2. Says no to everything. Over-constrained, kills option value. Detect it by counting live bets: one, and failing, means you have no portfolio.
  3. Stale. Compare the date on the doc with the date of the last assumption that changed.
  4. Used as a political weapon, cited to relitigate a decision already made. Detect it by whether the doc is cited before or after the call.

Worked example. A 22-slide deck justified both a self-serve push and an enterprise push in the same quarter. Two teams built conflicting onboarding flows; reconciling them took five weeks with two engineers and a designer at half time: 12.5 person-weeks, roughly $25k fully loaded at $2k/week (illustrative arithmetic). The deck wasn't wrong about the market. It just didn't decide.

RequestStrategy predictsActualMatch
enterprise SSOnoyesmiss
free tier expansionnoyesmiss
partner marketplaceuncleardeferred—

Two misses out of three means something other than the document is setting direction.

If it can't say no, it isn't a strategy. But a document that says no to everything is a different failure, not a stronger version of the same one. The useful shape is narrow enough to predict the next decision and loose enough to survive being wrong once.

Curated: · Written: · Reviewed:

Q&A-64Assume engineering's doubled estimate is right. What gives: scope, date, or quality?

Scope gives. Quality is a loan; the date is someone else's promise.

That's the 90-second answer. Cutting quality borrows from the same team that has to repay it: the defects land in the next sprint, on the same on-call rota. Moving the date is a promise to someone outside the team (a customer, a dependent team, a regulator), so it can only move if that someone agrees. Scope is the only lever whose cost lands on the people who chose it, and the only one you can reverse: you can add a slice back next quarter.

Assumptions I'd state out loud: engineering's 20 weeks is right, the team is fixed (adding people to a late project makes it later), and the date is a commitment to a named party. The doubling is itself information: something was discovered after the first estimate; find out what, because it usually points at the expensive slice.

The arithmetic. 20 engineer-weeks estimated, 10 available before the date. You have to remove 10 weeks. Which 10 is a question about outcome per week, not about what's easy to cut.

SliceEngineer-weeksShare of target usage
create and edit core flow668%
import from spreadsheet417%
approval routing58%
audit log34%
bulk export23%

The top two slices are 10 weeks and carry 85% of target usage. The bottom three are the other 10 weeks and carry 15%. So the answer is: ship the top two, defer the bottom three, and say so in the plan. That's a scope cut with a number attached, which is the only kind that survives contact with a stakeholder.

Why not quality, concretely. Quality here means load testing, error handling, reconciliation, accessibility, security review. Skipping it doesn't remove the work; it moves it to production, where it's more expensive and visible. Illustrative numbers: a billing migration holds its 15 March date by skipping load testing and shipping a reconciliation job that ran in 30 seconds in staging. At launch volume it takes 26 minutes, double-charges 1,400 invoices, and the rollback plus manual re-billing costs six engineer-weeks: against the two weeks the skip saved. The interest rate on that loan is 3x, and the same team pays it. The one quality cut I'd accept is deferring cosmetic polish, not tests, not error handling, not security.

When the date moves. Only against a named commitment: a contractual SLA, a regulatory filing date, a dependent team's launch, a marketing spend already committed. "The date" is not a reason; "the date because the payments team ships against our API on the 20th" is. If the date is immovable and the thin scope is below the minimum that delivers the outcome, you don't have a plan: you have a request for capacity or a decision to ship something that doesn't work. Say that out loud rather than absorbing it silently.

Failure modes to watch.

  • Cutting quality shows up as incident rate, on-call pages, and rework in the next two sprints. If you can't name the metric you're watching, you're not managing the trade, you're hoping.
  • Moving the date without telling downstream teams: they keep planning against the old date and the integration breaks at the seam.
  • Cutting scope without a usage figure: you cut the slice one customer actually needed, and you find out from them.

The ask. Before arguing about any of it, get the estimate broken into slices with a cost per slice. Then name the slice you'd drop first and the usage figure that says the thinner version still works. If nobody can produce that, the estimate isn't the problem: the plan is.

Curated: · Written: · Reviewed:

Q&A-79Engineering's estimate is twice what you hoped. How do you respond?

The first 90 seconds

I check what the estimate actually is before reacting to it. "Twice what I hoped" usually means I was carrying a number nobody on the team agreed to. So I separate the estimate, which belongs to the engineers doing the work, from the target, which is mine or the business's. They are allowed to differ, and the gap is the thing to work on.

Then I ask which part carries the uncertainty. I don't ask "can you do it faster," because that moves the reported number and not the work. If the answer is "the legacy migration, because we don't know how dirty the data is," I have something actionable: a spike, a decision, or a scope cut.

I argue with the scope, not the arithmetic. If the date is real, I ask what we drop to hit it. If the date is soft, I take the estimate and re-plan around it. What I don't do is push the number down and call the result a plan.

Assumptions I'd state out loud

  • The estimate comes from the engineers who will do the work, not from a manager relaying it.
  • The date is a business constraint, not a contractual commitment to a customer.
  • I control scope, or can get a scope decision within a day.

If any of those is false the conversation changes. A contractual date with no scope flexibility is a staffing or escalation problem, not an estimating problem.

What the estimate is made of

Ask for the breakdown. An 8-week estimate against a 4-week hope is not one number, it's three:

ComponentEstimateUncertaintyWhat reduces it
CRUD work, well understood3 weekslownothing needed
Legacy data migration4 weekshightwo-day spike on a data sample
Third-party rate limits1 weekhighread the vendor's documented limit, or make one test call

Two of the three lines are high-uncertainty. That tells me the estimate is honest and the hope was optimistic, and it tells me where to spend the next two days.

The failure mode

Pushing the number is the failure. A PM I worked with took an 8-week estimate down to 5 in a planning meeting, on the argument that the team was padding. The work took 11 weeks, including 2 weeks of rework after the migration assumptions turned out wrong. The lasting damage was worse than the three weeks. For the next two quarters that team gave estimates at their worst case, and planning got less accurate, not more.

The detection signal is cheap and early. If estimates start arriving as single numbers with no range and no stated uncertainty, the team has learned that uncertainty gets punished.

What I actually say

"Eight weeks is more than I hoped for. Which line are you least sure about? If it's the migration, can we spend two days on a data sample and re-estimate Friday? If the date is fixed, I'd rather cut the reporting feature than have you commit to a number you don't believe."

Name the driver, buy information, trade scope. The number follows from those, not from the conversation.

Curated: · Written: · Reviewed:

Q&A-106Engineering comes back with an estimate for your top-priority feature that is three times what you planned for. What do you do?

The number is information, not a verdict. Three times is big enough that the feature as specified probably isn't the thing to build, so I don't start by arguing the estimate down or by moving the date. I start by finding out what the 3x is made of.

First: decompose the estimate, don't contest it. I ask the tech lead to walk me through the breakdown and, critically, which line items are committed work and which are unknowns priced at a guess. A 3x is rarely uniform: it's one or two items that exploded.

Line itemPlannedEstimateWhy it moved
Core feature logic45understood
Integration with billing service16no existing API; new service contract
Backfill of 400k existing accounts03never in the original ask
Staging parity / test data0.52staging can't reproduce prod data shape
Rollout, flags, support runbook0.52
Total618

Figures are illustrative engineer-weeks.

Two things fall out. The backfill and staging work were never in my ask: they're consequences of it, and they're negotiable. The billing integration is the real 3x, and it's an unknown, not a commitment. So I ask what would have to be true for it to be two weeks instead of six, and whether a version exists that doesn't touch billing at all. I also ask what the estimate assumes about staffing: a 3x often quietly assumes one engineer at 50% while they carry on-call.

What I don't do is ask "can you do it in six?" That's an instruction to negotiate down, and it teaches the team to pad the next estimate.

Then: the smallest version that still produces the outcome. Not the smallest version of the feature. The smallest thing that moves the metric I claimed this feature would move. I write the ladder down with the outcome each rung buys:

  • v0 (5 weeks): core logic, manual backfill by support for the 40 accounts that asked, no billing integration. Outcome: the retention signal, on a sample.
  • v1 (11 weeks): automated backfill plus billing integration. Outcome: works for everyone, no manual ops.
  • v2 (18 weeks): full estimate, staging parity, self-serve.

If v0 tests the same hypothesis, the honest move is to ship v0, learn, and decide on v1 with data. If v0 doesn't test the hypothesis, if the value only exists at full scale, I say that out loud, because it changes the decision from "trim" to "is this worth a quarter at all?"

Then: re-check priority against what the time buys. 18 engineer-weeks is roughly one engineer for a full quarter plus change. I put the 12-week delta next to the next two roadmap items and ask which produces more of the outcome. Sometimes the answer is still this feature: a 3x on the top priority is often still the top priority. Sometimes the 12 weeks buys the other initiative outright, and the right call is to shelve this until the unknowns are cheaper.

Then: make the trade explicit, in writing, with the team. The failure mode is a PM who quietly moves the date and lets engineering absorb the blame, or who says "make it fit" and buys quality debt. What I actually say: "We're keeping the date and cutting to v0. No billing integration, manual backfill for the first 40 accounts. I own that trade and I'll tell the customer why." Or: "We're keeping scope and moving to Q3; here's what I told the exec." Either is fine. Silence is not.

Escalation. I go to my director only after I've exhausted scope, and only with options: "the feature is 3x; here are three versions and what each costs; I recommend v0 on the original date. Do you want the full version at the cost of X?" Escalating the raw number, with no options attached, is how PMs teach engineering to sandbag.

Curated: · Written: · Reviewed: —

Q&A-105You planned the quarter and everything you asked for does not fit. How do you decide what to cut?

The rule first, then the list. I cut against the quarter's stated outcome, not against the list of asks. Before I touch anything I write the one or two outcomes at the top of the page: take a hypothetical self-serve product where the outcome is "cut time-to-first-value from three days to under four hours", and then rank every ask by expected contribution to that outcome per engineer-week. Two floors sit above the ranking: anything already promised externally with a date, and anything that multiplies other work (performance, a shared API). Everything else is ranked, and I cut from the bottom.

Worked example. Three engineers, thirteen weeks, minus 15% for on-call and interrupts: 33 engineer-weeks against 40 asked for.

AskEng-weeksTied toDecision
SSO for enterprise10Named deal, dated commitmentKeep
Onboarding rewrite6The quarter's outcomeKeep
p95 load 2.1s → 1.2s4Multiplies every flowKeep
Audit log5Contract language, one accountKeep, scoped to read-only export: 2 ew
Bulk import812 accounts, no dateKeep CSV path only: 4 ew
Push notifications7Nothing on the outcomeCut

The two floors did most of the work. SSO was promised to a named deal with a date, so it stays regardless of its ratio. The performance work multiplies every other flow, so it stays. Push notifications had no outcome attached: that's the cut. Bulk import and the audit log were both real asks, so I scoped them rather than deleting them: CSV-only import covers most of the twelve accounts, and a read-only export satisfies the contract language. That leaves 26 engineer-weeks committed and 7 as buffer, roughly the 20% I want for interrupts.

Sequencing, not dropping. Most "cuts" are slips. Bulk import goes to the top of next quarter's list, written down, with the twelve accounts named. The trigger that brings it back is explicit: if the CSV path doesn't cover at least eight of the twelve by week six, it comes back into this quarter and something else moves.

Who hears it, and when. The team whose work is dropped hears it from me first, in person, before the plan is published, not in the all-hands, not from a slide. I give them the ranking, the reason, and the trigger. Then the affected sales or support people, with the same three things. The exec review gets the trade-off stated as a sentence, not a table.

The trade-off I'm accepting, out loud. "We accept that twelve accounts wait a quarter for bulk import and we may lose one of them. We accept that push notifications don't ship this year and that the mobile work is now unowned." If I can't say those sentences in the planning review, I haven't made a decision: I've made a list.

Failure modes I watch for. Cutting the cheapest items instead of the lowest-value ones leaves you with a quarter of small things and no outcome. Cutting something that unblocks another team moves the cost onto someone else's plan, which is how you get a dependency crisis in week eight. Announcing to the exec before the team means the team learns their work is gone from a slide. And a slip with no written trigger never comes back: it quietly disappears.

One more thing: if the ask exceeds capacity by more than about 30%, prioritization is the wrong tool. That's not a plan that needs cutting, it's a plan that needs renegotiating: go back to whoever set the outcome and say the outcome and the capacity don't match.

Curated: · Written: · Reviewed: —

Q&A-78Two weeks before launch the work will not fit. What do you cut?

Before I cut anything I ask one question: is the date fixed by something external (a contract, a launch event, a customer commitment) or is it a target we set? If it's a target, moving it two weeks is on the table and usually cheaper than shipping a journey nobody can finish. If it's fixed, the only variable left is which capabilities ship whole.

Assume it's fixed. Then I cut whole capabilities, not percentages, and I protect the narrowest journey a real customer can complete end to end.

The arithmetic usually forces the decision. Say six capabilities remain, with these remaining costs and the journey each one completes:

CapabilityEng-days leftJourney it completes
Bulk CSV import6new-customer onboarding
Field-mapping UI4new-customer onboarding
Saved filters3daily triage
Scheduled reports5weekly review
SSO8enterprise onboarding
Audit-log export2compliance sign-off

28 eng-days of work, 10 eng-days of capacity. Trimming 20% from every capability leaves 22.4 days: still 12.4 over. Proportional trimming isn't a compromise here, it's arithmetically impossible. And the only subset that both fits 10 days and completes a journey for a customer who doesn't already have data in the system is import + field mapping, exactly 10. Everything else goes: saved filters, scheduled reports, SSO, audit-log export.

That's the ranking rule: journeys completed per remaining eng-day, not features touched. Ties break toward cutting the capability whose absence is loudest. A missing export button produces a support ticket; a filter that silently ignores a date range produces wrong decisions.

Alternatives, and when each is right:

  • Move the date. Right when the missing capability is the reason customers buy. Costs credibility once; a partial launch costs it every week.
  • Flag it to a cohort. Right when the journey is complete for that cohort: 10 design partners on the full path beats 10,000 on a broken one.
  • Ship with known bugs. Almost never. Only for cosmetic defects with a documented workaround; anything else converts engineering debt into support load.
  • Cut the audience instead of the scope. Right when the journey is complete but unproven.

Failure modes I've watched happen:

  • Cutting the cheapest capability rather than the least valuable. You drop the 2-day audit export, keep the 8-day SSO, and still miss the date.
  • Cutting a dependency. Drop field mapping, keep import, and import ships unable to map fields: the feature is present and useless.
  • The silent cut. Nobody tells sales or support what's out, and the first 9 days after launch produce 214 support contacts about three missing behaviours. Detection is a named cut list with an owner per line, reviewed by support before launch.
  • Trimming every feature to hold the date. Same team shipped an import that couldn't map fields, an export that ignored filters, and a report with no date range: four partial features, zero completable journeys.

Before I'd call it done: walk the surviving journey as a customer, on the release build, with no workarounds, and confirm it completes. Then publish the cut list with the date each item lands.

Curated: · Written: · Reviewed:

Q&A-98You join a team with no roadmap, unclear goals, and a large backlog. What do you do first?

First week, I read the pager and the pipeline before I read the backlog. The backlog is a wish list; the on-call rotation, the deploy path and the repo are what's actually true. So the first move is to find out which of three problems I've inherited (delivery, reliability, or direction) because the fix is different for each.

Four things in the first five working days:

  1. Pull the last 90 days of incidents and pages and count them. If the team is paged more than about twice a week, or MTTR is over an hour, reliability is the constraint and the roadmap is a distraction until it's fixed.
  2. Time the deploy path end to end. If merge-to-prod is 40+ minutes and roughly one in five deploys needs a rollback or hotfix, delivery is the constraint.
  3. Find the hotspots. Churn plus bug density usually concentrates: a handful of files carry most of the incidents.
git log --since="6 months ago" --name-only --pretty=format: | sort | uniq -c | sort -rn | head -20

Cross-reference that list against closed bug tickets. If the top 5 files account for a third of the incidents, that's where the first fix goes.

  1. A 30-minute 1:1 with each engineer: what breaks, what you'd fix, what you're afraid to touch. The last question is the one that surfaces the real risk.

Then ship one thing inside 30 days. Something visible and low-risk: delete the alert that fires daily and gets ignored, add the missing test on the top hotspot, fix the deploy step that fails intermittently. The point isn't the fix, it's that proposals from someone who has shipped get read differently from proposals from someone who hasn't.

When I'd skip the tour. If there's an active data-loss or security exposure, fix it now: you don't need discovery to justify stopping a bleed. And if the team already has DORA metrics and error budgets and the problem is genuinely prioritization rather than discovery, then a framework-first move is correct and my two weeks of reading is wasted time. The listening tour is for when you can't yet tell which of the three problems dominates.

Two failure modes. The endless listening tour: six weeks of interviews, nothing shipped, the team writes you off as another observer. Detect it by putting a date on the first ship before you start and holding it. The second is treating the backlog as ground truth: I've watched someone re-propose three items that had already been tried and rolled back, because the reasons lived in postmortems nobody re-read. Detect it by grepping closed tickets and postmortems for each item before you propose it.

Pager and pipeline first, backlog last, one shipped fix inside 30 days.

Curated: · Written: · Reviewed:

Q&A-32How do you choose a single metric for a product team to steer by?

The 60-second answer

Pick the metric that counts a customer completing the job they came for, that the team can move inside one quarter, and that a cynical team cannot raise without actually doing the work. Then bolt on one or two guardrails that catch the cheap way to move it. The north star is the steering wheel; the guardrails are the lane markings. One without the other is how teams optimise a number into the ground.

State the assumptions out loud: the team owns a surface with a direct customer, it ships at least every couple of weeks, and the metric has enough weekly volume that a real change clears the noise.

Deriving it

Write the customer's job as verb + object + completion criterion, then count the completion event, not the intent event.

  • Intent: report created, account created, search performed.
  • Completion: report delivered to a recipient who opened it, first real task completed, search that led to a click.

Intent events are cheap to produce and cheap to fake. Completion events require the customer to have got something. Then run three tests.

  1. Movable. Can the team change it with the levers it owns? If the team owns checkout and the metric is company revenue, they cannot steer. Pick the largest metric they can actually move.
  2. Timely. Does it respond within the team's iteration cycle? Revenue lags a quarter; steering by it is driving on the rear-view mirror.
  3. Expensive to fake. Ask how a cynical team raises it without helping anyone. If the answer comes easily, it is not a north star.

Guardrails

The guardrail must be the thing that degrades when you game the north star. Pairing rule: for each cheap way to move the north star, name the number that would fall.

CandidateCheap way to move itGuardrail
accounts createdfrictionless empty signups, bulk invitesfirst real task completed within 7 days
reports createdauto-create a draft on every loginreport opened by a recipient
pageviewssplit one page into threetask completion time
revenuediscounting, one-off enterprise dealsnet revenue retention
weekly reports delivered (chosen)send each report to more recipientsrecipient open rate, error/return rate

Worked example (hypothetical figures)

A team owns a reporting product. North star: weekly reports delivered to a recipient. Guardrail: share of delivered reports opened by a recipient within 7 days.

Steered instead on reports created, they auto-create a draft on every login. The count rises 34% over two quarters. Recipient opens fall from 44% to 19%. The north star never moved down, because it was counting the wrong event. Detection: the guardrail sits on the same dashboard with an alert at a 10% week-over-week drop.

Noise arithmetic

Before committing, compute the metric's week-over-week standard deviation over the last 12 weeks. If the weekly count averages 500 with sd 60, a 5% lift is 25 units: inside one sd, so a single week tells you nothing. Assuming independent weeks, detecting that 5% lift at 80% power needs roughly 90 weeks per arm. If that is the answer, pick a higher-volume metric, aggregate to a monthly window, or steer on a leading proxy and validate it against the lagging metric quarterly.

Failure modes

  • Goodhart. North star up, guardrail down. Detect by alerting on the guardrail, not just reporting it.
  • Denominator shrink. "Share of sessions with a search" rises when you delete non-search entry points. Track the absolute count beside every ratio.
  • Unmovable. A quarter of shipping, no movement, no plausible mechanism. That is a company metric; hand it back.
  • Double counting. One customer action incrementing the metric twice: a report delivered to five recipients counted as five. Define the unit (per customer, per recipient, per session) and hold it.

When not to use one

Pre-product-market-fit discovery: the job is to find the value moment, not steer by it. Platform and infra teams: the customer is another team, so the value moment is downstream adoption or an SLO such as a p99 latency budget, not a user action.

The test before committing

Name the cheapest way to raise the number and confirm a guardrail catches it. If a cynic cannot move it without doing the work, it is the right metric.

Curated: · Written: · Reviewed:

Q&A-33Your north star moves slowly. What do you give the team to steer by week to week?

I would frame input metrics a team can actually move around the segment it affects, because an average across segments hides the decision.

A lagging outcome cannot guide weekly decisions, so it has to be decomposed into inputs the team changes directly and whose relationship to the outcome has been checked. Inputs chosen without checking that relationship become busywork with a dashboard.

Concretely, decompose the outcome into a small number of inputs and test each one's historical correlation with the outcome. Set weekly targets on the inputs and review the link every quarter. Drop an input when the link stops holding rather than keeping it for continuity.

The reason for that specificity is a failure I have seen: A team drove an input of demos booked for 3 quarters, raised it 58 percent, and revenue per demo fell by half because the added demos came from a segment that converted at 4 percent against 19 percent for the original one.

The input moved and the outcome did not.

QuarterDemos bookedRevenue per demoRevenue
baseline2104,100861k
plus two quarters3322,050681k

I would not consider it settled without evidence: Recheck the correlation between each input and the outcome using the last two quarters of data.

An input metric is a hypothesis about the outcome, and hypotheses expire.

Curated: · Written: · Reviewed:

Q&A-53How do you tell whether a metric is worth reporting?

The 90-second answer

A metric is worth reporting when some value of it would change a decision you would otherwise make differently. So the test is to name the threshold and the action: "if weekly active rate drops below 35%, we pause the onboarding experiment and roll back." If you cannot fill in both halves, the metric is decoration.

Two things follow from that. Prefer rates and cohort views over cumulative totals, because a cumulative counter can only rise and therefore can never warn you. And a metric that has never been cited in a decision is a candidate for deletion, not a candidate for a dashboard.

When the test does not apply

Three cases where "no threshold, no metric" is the wrong rule:

  • Leading indicators you are still calibrating. You do not yet know what value predicts churn. Report it anyway, but time-box it: give it a quarter to earn a threshold, and cut it if it has not. Otherwise you never discover the leading indicator.
  • Compliance and contractual metrics. Uptime SLAs, audit counts, data-residency checks. These are reported because someone else requires them, not because they change your decision. The decision was made when you signed.
  • Exploratory instrumentation. When you are debugging an incident or hunting for a new signal, you instrument broadly and narrow later. The failure mode is leaving the exploratory set in place forever.

Failure modes

  • Cumulative counters cannot fall, so they cannot warn. Signups, total users, cumulative revenue.
  • Averages hide tails. Mean latency looks fine while p99 is on fire. Report the percentile you would act on.
  • Metrics that move only with the thing they measure. A "deploys per week" chart that tracks team size tells you nothing about reliability.

Worked example (hypothetical)

A monthly review carried 31 metrics, 22 of which had never been cited in a decision. Over three months the weekly active rate fell from 41% to 35%, a 14% relative drop, and nobody raised it, because the cumulative signup chart dominated the page and kept climbing.

MetricThresholdActionCited in a decisionCan fall
cumulative signupsnonenone0no
weekly active rate<35%pause onboarding experiment9yes
support contacts per account>2.5add self-serve docs4yes

The first row fails the test on every column. The other two pass.

What I would do

Ask the owner of each reported metric for its threshold and action. Delete the ones nobody can supply, unless they fall into one of the three exceptions above, and for those, write down who requires it and until when.

Curated: · Written: · Reviewed:

Q&A-63How do you handle accounts, users, and devices in one metric?

The 90-second answer. Pick one canonical unit per metric and tie it to the decision that metric feeds. Keep the other units as separate metrics joined by an explicit key: never blended into one rate. For an authenticated product the canonical unit is the account, because accounts are what you bill, message and reason about. Devices are a proxy that only holds while one person uses one device.

Assumptions worth stating out loud: users log in at least once, so a stable account id exists; anonymous pre-login traffic is a real but separate population; and the decision in front of you is about people or money, not about hardware.

The alternatives, and when each is right.

ApproachCostCorrect when
Canonical account unitCheap: deterministic join on account_id at loginAuthenticated product; billing, retention, cohort decisions
All three units side by side, joined by a documented keyMedium: needs identity resolution plus a published join keyOne pipeline must serve device-level ops and account-level business metrics
Device-first countingCheapest: no resolution at allNo login exists. Pre-auth funnel, install attribution, push reach, app-version adoption, device capacity planning
Probabilistic identity graphHighest: stitching compute and storage, precision/recall tuningCross-device behaviour matters and login coverage is low

Device-level counting is not a mistake in itself. It is the right unit when the decision is about the device: how many push tokens are live, which app version is still in the field, how many concurrent streams a CDN edge must hold. It becomes wrong the moment you attach a per-seat price or a per-user rate to it.

What identity resolution actually costs. Deterministic resolution, join on the account id captured at login, is exact and nearly free, but only covers post-login events. Probabilistic stitching buys coverage and pays for it twice: compute and storage for the graph, and errors in both directions. Over-merge (a family tablet, a retail kiosk) collapses several people into one, undercounting. Under-merge (cookie clear, IDFA/GAID reset, reinstall) splits one person into several, overcounting. A 5% over-merge on 780k accounts is 39k phantom users: enough to move a pricing tier.

Failure modes and how they show up.

  • Late identity binding. A device event is anonymous until the user logs in, so last month's number changes after the fact. Detect by snapshotting each metric as-of and diffing the snapshot against the recomputed value.
  • Join fan-out. Joining device events to accounts multiplies rows. Detect with a row-count assertion: expect roughly the observed devices-per-account ratio, alert when it jumps.
  • Shared devices. Detect by watching the tail of the accounts-per-device distribution, not the mean.
  • Device id resets. Detect by comparing the new-device rate against the new-account rate; they should track, and divergence means resets.

Worked example (illustrative figures). A product reported 2.1M monthly users from device identifiers. Deduplicating by account gave 780k; paying workspaces were 41k.

UnitMonthly count
devices2.1M
user accounts780k
paying workspaces41k

2.1M / 780k = 2.7 devices per account, so a device-level "monthly active" overstates the billable population by 2.7x. 780k / 41k = 19 accounts per paying workspace, which is the ratio a seat-based pricing model should actually be built on.

The check that keeps it honest. Run the ratio daily and alert on change, in PostgreSQL 16:

SELECT date,
       count(DISTINCT device_id)  AS devices,
       count(DISTINCT account_id) AS accounts,
       count(DISTINCT device_id)::float
         / nullif(count(DISTINCT account_id), 0) AS devices_per_account
FROM sessions
GROUP BY date
ORDER BY date;

A stable ratio means the units are behaving; a step change means identity resolution broke, not that the business doubled.

Curated: · Written: · Reviewed:

Q&A-52Two dashboards disagree about active users. How do you settle it?

The disagreement is almost never in the data. It's in the definition, and the definition was never written down, so both dashboards are correct implementations of different sentences. Settle it by writing the definition out clause by clause, reconciling the two queries against the same snapshot until the remaining difference is zero, and then deciding who owns the number.

Four clauses decide almost every active-user metric: population (which accounts count), qualifying event (what the user has to do), window (the time range, and whether it's rolling or calendar-aligned), and exclusions (bots, internal traffic, API-only accounts, test tenants).

Worked example from a real reconciliation: a weekly active figure differed by 19% between two teams for four months. A pricing decision had already been made from the higher number. The clause-by-clause diff:

ClauseTeam ATeam B
Populationall accountsexcludes API-only accounts
Qualifying eventany requesta session
Window7 days rollingcalendar week (Mon–Sun UTC)
Exclusionsnoneinternal IPs

Reconciling meant running both queries against the same snapshot and diffing the account sets. The API-only exclusion accounted for most of the gap; the rolling-vs-calendar window accounted for the rest, concentrated on Mondays.

Then the trade-off. Two defensible resolutions:

  1. One canonical metric, one owner. Everyone reads the same number. Cost: teams whose question genuinely differs (billing wants API-only accounts counted, product engagement doesn't) now have to argue with the owner every time, and the owner becomes a bottleneck.
  2. Keep both metrics, label them explicitly, publish the mapping. Cost: two numbers in circulation, and someone will screenshot the wrong one into a deck.

The deciding criterion is whether the two numbers feed the same decision. If they do (same decision, same audience) collapse to one canonical metric with a named owner and route changes through them. If they feed different decisions, keep both, but rename them so the label carries the definition ("Weekly Active Accounts (incl. API)" vs "Weekly Active Sessions (excl. API)") and put the mapping in the same doc. A number whose label doesn't state its population and window will be misread.

Failure modes to check before declaring it settled:

  • Calendar-week metrics break at timezone boundaries. A "Monday" in UTC is Sunday afternoon in Pacific. Pin the timezone in the definition.
  • Late-arriving events change yesterday's number. If the dashboard is a live query, the number moves; if it's a batch table, it's frozen at the batch time. State which.
  • Backfills and schema changes silently shift the population. A new account type added to the enum changes "all accounts" without anyone editing the definition.
  • Bot and internal-traffic filters drift. One team adds a filter, the other doesn't.

Detection: run both queries against the same snapshot, diff the row sets, and attribute the difference to a clause. If the difference survives clause-by-clause reconciliation, it's a pipeline bug, not a definition dispute: go find the dropped rows.

I wouldn't call it settled until the reconciled difference is zero and the definition lives next to the query, not in a wiki page nobody reads.

Curated: · Written: · Reviewed:

Q&A-54What do you look for in a cohort retention chart?

The 90-second answer

First I check what the axes actually are: rows are acquisition cohorts (usually signup week or month), columns are periods since acquisition, and each cell is the share of that cohort still active in that period. Then I ignore the first column and the level, and read the shape.

I look for three things, in order:

  1. Where the curve flattens. A curve that decays and then goes flat has a durable core: the floor is the size of the group that found recurring value. A curve that keeps decaying toward zero describes a product people finish with, and no amount of acquisition fixes it. The floor, not the week-1 number, is the retention number.
  2. Whether recent cohorts flatten higher or lower than older ones. That is the only signal that a product change worked. A curve that flattens at 11% after a redesign is worse than one that decays to 30% and then flattens at 20%.
  3. Whether the flattening is real. Bounded vs unbounded retention, the definition of "active", and cohort size all move the floor without the product changing.

Why the shape behaves that way

Retention is a mixture: a fraction of each cohort finds a recurring need and stays; the rest churn out at a roughly constant hazard rate. The hazard rate sets the slope; the size of the durable fraction sets the asymptote. The first-week drop is mostly onboarding and novelty, and the slope after the knee is the product. That is why I read the tail.

The failure mode I look for first

A blended average. Illustrative numbers, equal cohort sizes:

CohortMonth 2Month 6
pre-redesign44%39%
post-redesign33%11%
blended38%25%

The blended month-6 of 25% is more than double the post-redesign 11%. The average was held up by the pre-redesign cohort while every cohort after the redesign flattened at 11%. A team can read that 38% month-2 number as healthy for three quarters and never see the product dying underneath it. So: never read the blended line; read the cohort rows.

What else I check before believing a curve

  • Cadence. If the product is used weekly, plot weekly retention. Daily retention for a weekly-use product decays to zero by construction and tells you nothing.
  • Bounded vs unbounded. Unbounded retention (active on day N or later) flattens by construction and can hide decay. Bounded (active on exactly day N) is honest but noisier.
  • Definition changes. If "active" changed mid-flight (a new event, a new session threshold) the cohorts either side are not comparable.
  • Cohort size and seasonality. A 200-user cohort is noise; a holiday-signup cohort is a different population.
  • Mix shift. If the channel mix changed, the curve moved because the population moved, not the product.
  • The smile. A dip then a rise is real in marketplaces and seasonal products: resurrection, not a bug.

What I would ask for

The last six cohorts plotted separately, the definition of "active" and its history, and cohort sizes. If the recent cohorts flatten rather than continue to decay, the change worked. If they do not, the average was hiding it.

Curated: · Written: · Reviewed:

Q&A-55How do you define activation for a product?

The short version. Activation is the earliest user behaviour that predicts durable retention. I don't choose it in a meeting; I test candidate behaviours against retention and adopt the one that separates best while still being reachable.

Assumptions I'd state before doing any of this: we have enough new accounts per quarter for the split to be powered (a few thousand, not a few dozen), we can join first-week product events to month-3 retention, and "durable retention" means the account is still active in month 3. Without that volume the test is underpowered, and I'd say so rather than quote a number.

How I'd run it

  1. List candidate first-week behaviours: the things a user could plausibly do that signal they got value.
  2. Split new accounts into did / didn't for each candidate, and measure month-3 retention in both groups.
  3. Rank by separation, then filter by reachability: a behaviour with a huge gap that 2% of accounts ever perform is not an activation metric, it's a niche.
  4. Check the behaviour is upstream of value, not a symptom of it.
  5. Recheck on a schedule, because product changes move the moment.
First-week behaviourMonth-3 retention if doneIf notSeparation
Completed profile34%31%3 pts (1.1x)
Invited a teammate62%24%38 pts (2.6x)
Ran two imports58%22%36 pts (1.8x)

On those numbers, profile completion is out: a 3-point gap is noise at most sample sizes. Inviting a teammate and running two imports both separate hard. The tiebreak is reachability and causality, not the size of the gap.

Cause or symptom. Inviting a teammate may be a consequence of already having got value, in which case prompting it moves the metric and not retention. The test is a holdout: prompt the behaviour in a random half of new accounts and see whether month-3 retention rises. If it doesn't, the behaviour is a marker, not a lever, and I'd keep looking.

A failure I've seen. A team defined activation as completing the profile. Prompts took it from 51% to 78% over two quarters. Month-3 retention didn't move, because profile completion separated retention by 3 points: they had optimised a step, not the moment. The cost was two quarters of roadmap and a metric everyone had learned to distrust.

Other failure modes. Gaming: any activation metric that is a product action can be driven by nagging, so pair it with the retention check. Survivorship: accounts that churn in week one never appear in the did/didn't split, which flatters both groups. Segment drift: a definition that holds for self-serve can fail for sales-led accounts, so confirm the separation holds within each segment before adopting one number. Staleness: recheck quarterly while the product is changing fast, annually once it settles.

When not to use this. Pre-product-market-fit, or in enterprise with six-month sales cycles, month-3 retention is too slow and too noisy to define activation against. There I'd use a shorter horizon (say, week-2 usage depth) or a qualitative signal from customer calls, and label it as provisional.

Activation is discovered in the data, not chosen in a meeting.

Curated: · Written: · Reviewed:

Q&A-56Signups are flat but traffic is up. How do you find the problem?

Assume signups and traffic use the same definitions and window. If traffic is up 20% and signups are flat, overall conversion fell from 3.15% to 2.63%: that is the number to explain. Pull signups and sessions by source, device, geography, and landing page for the same period and the prior period. Compute conversion within each segment, not just the aggregate.

Hypothetical figures:

SourceShare beforeShare afterConv beforeConv after
organic62%56%3.1%3.1%
paid social8%24%0.4%0.4%
referral30%20%4.0%4.0%

Weighted before: 0.623.1 + 0.080.4 + 0.304.0 = 3.15%. Weighted after: 0.563.1 + 0.240.4 + 0.204.0 = 2.63%. Same within-source rates, different mix. That is a mix shift, not a broken signup form.

The decision rule: if within-source conversion is flat within noise and the weighted average using old rates and new mix matches observed, the mix explains the drop. If any source's conversion fell, that source or the shared flow is the problem, and the mix is not the cause. If all sources fell at once, suspect instrumentation, bot filtering, or a release before suspecting demand.

Alternatives to rule out:

  • Instrumentation: signup event deduped, cookie consent blocking analytics, or a new bot filter. Check raw server-side signup count against analytics. If server count is flat and analytics signups are flat but sessions jumped, the extra traffic may be bots.
  • Campaign quality: paid social volume up but conversion 0.4% vs 3.1% organic. Check landing page match, creative, audience. If only paid social conversion fell, it is campaign or landing page, not the form.
  • Flow breakage: if organic and referral conversion both drop, test the signup flow end-to-end by device and browser. Look at step-level rates: email submit, password, verification. A drop at one step localises the break.
  • Seasonality or attribution: compare same week last year and check attribution window changes. If the drop disappears when using a 7-day vs 1-day window, it is measurement.

What would change the decision: a within-source drop that survives segmentation, or a server-side signup count that also fell. If server-side signups are flat and within-source rates are flat, the problem is traffic quality or measurement, not the product.

Curated: · Written: · Reviewed:

Q&A-57What do you watch to be sure a win is not causing damage elsewhere?

A guardrail is the metric that measures the cost you are least willing to pay for the win. Pick two or three before launch, set a stopping threshold on each, and read them on the same cadence as the primary metric. If a guardrail breaches, the default is to stop the rollout even while the primary metric is winning, but that default is a policy choice, not a law, and the useful part of the answer is which policy you would pick and why.

Three assumptions matter here: the change sits behind a ramp or holdback so it can actually be stopped; there is enough traffic to read the guardrail inside the experiment window; and the guardrail is a leading indicator of the harm you care about, not the harm itself.

The mechanism is that most product changes move their target by moving something else. An upsell prompt raises trial-to-paid by pushing harder on the same users, so the cost lands on support load and on the quality of the conversions it produces. The guardrail is where that cost shows up first.

Two decisions have to be made explicitly.

Per-metric versus composite. Per-metric guardrails (support contacts per 1,000 trials, p99 latency, 90-day churn) each carry their own threshold and each can stop the rollout. Composite guardrails fold several into one score, which is easier to monitor and easier to game: a large regression in one metric can be averaged away by a small improvement in another. Use per-metric when the harms are not substitutable, since churn is not offset by latency; use composite when they are substitutable and you want a single decision rule.

Auto-stop versus human review. Auto-stop halts the ramp the moment a threshold is crossed, and it is the right default when the harm is irreversible or compounding: churn, data loss, a compliance breach. Human review, where you page the owner, hold the ramp and decide within a day, fits reversible harms where a false stop costs more than a day of exposure and where the guardrail is noisy at low sample sizes.

A breach can sometimes be tolerated: when the guardrail is a proxy rather than the harm itself, when the breach sits inside the pre-registered confidence interval, or when the harm is bounded and the fix ships this week. That tolerance rule belongs in the plan too, or it becomes an excuse invented after the fact.

The failure mode I have seen: an aggressive upsell raised trial-to-paid 9% and was kept for a quarter. Support contacts rose 31%, and 90-day churn in the converted cohort ran 2.4x baseline. Neither was listed as a guardrail, so nothing stopped it.

MeasureChangeGuardrail before launch
trial to paid+9%primary
support contacts+31%no
90-day churn, converted cohort2.4x baselineno

For detection, alert on the guardrail at the same cadence as the primary and read it on the holdback rather than the whole population; a guardrail that only moves in the treatment arm is the signal.

The artefact that makes this real is the experiment plan: guardrail names, thresholds, the stop/hold/review action for each, and the tolerance rule, all written before the first user sees the change.

Curated: · Written: · Reviewed:

Q&A-62A team is optimising a proxy metric. What do you watch for?

What I'd say in the first two minutes

Before anything else I'd pin down three things: what the real outcome is, how it is measured, and who is held to the proxy. Then I'd watch four things.

  1. Divergence between proxy and outcome. A proxy is a stand-in for something expensive to measure, and it holds only while the relationship holds. Once it becomes a target, effort flows to the cheapest way of moving it, which is rarely the path that moves the outcome.
  2. Effort migrating to the cheapest lever. Ask what the cheapest way to move this number is, and whether that is the behaviour you actually want. If the answer is "send an automated acknowledgement", the proxy is already broken.
  3. The proxy leaking into incentives. If it appears in a performance review or a bonus formula, it is a target, whatever the dashboard calls it.
  4. The cost of keeping both measured. Paired monitoring is not free, and there are cases where it is not worth it.

Why it drifts

The proxy is a cheap regression of the outcome on a signal you can collect continuously. The residual, the part of the outcome the proxy does not explain, is exactly where gaming lives, because that is the space where you can move the number without moving the thing. Goodhart's law and Campbell's law are the same observation from two fields.

The alternatives, and when each is wrong

ApproachWhat it buysWhat it costsWhen it's the wrong call
Paired monitoring (proxy and outcome in one report)Divergence visible within a reporting cycleTwo pipelines, two definitions, ongoing analyst timeThe outcome is slow or destructive to measure: quarterly revenue, churn, anything you cannot observe for months
Composite metric (weighted blend)Harder to game any single leverWeights are arbitrary and drift; a bad component hides inside a good scoreYou need to know which component moved
Randomised audit or holdoutUnbiased read on the true outcomeNeeds volume for significance; politically hard to hold a group outLow-volume teams, where a holdout never reaches significance
Leave the proxy untargeted (measure, don't manage)Cheapest; no gaming pressureYou lose the ability to steer at allYou genuinely need to move the outcome

Paired monitoring is the default, but it is the expensive option when the outcome takes a quarter to observe or the volume is too low for a holdout. There, a periodic audit (sample a cohort, measure the real outcome, accept the noise) beats a continuous pairing that nobody reads.

Failure modes and how I'd detect them

  • Divergence. Track a rolling correlation between proxy and outcome per segment; alert when it drops below a threshold you set from your own history, not a round number.
  • Gaming. Watch the input distribution, not the score. Automated acknowledgements show up as a spike in first-response events with near-zero handling time.
  • Simpson's paradox. The aggregate improves while every segment worsens. Detect by segmenting the report by cohort, region, or plan tier.
  • Silent decay. A product change redefines the outcome and the proxy quietly stops tracking it. Re-derive the relationship after any change to the outcome definition.

Worked example (illustrative numbers)

A support team measured on first-response time:

MeasureBeforeAfter
first response4.1 hours22 minutes
resolution time19 hours25 hours
satisfaction7462

The proxy improved by an order of magnitude; the outcome got worse. The cheapest lever was an automated acknowledgement, and the paired report is what made it visible.

Every proxy is a measure until it becomes a target.

Curated: · Written: · Reviewed:

Q&A-73Your satisfaction score fell twelve points. What do you do first?

First 90 seconds

Ask which score: which survey, which population, what cadence, and whether the instrument changed in the last period. Assume it's a monthly composite across all users.

Then I don't touch the roadmap. A 12-point move in a composite is a measurement event until I've ruled that out. The first hour goes to three checks, in order: response rate, respondent mix, and whether the survey itself changed.

Why mix comes first

A composite is a weighted average over segments with different baselines. Move the weights and the average moves even when every segment is flat. Trial users score lower than paying users on almost any product survey: they haven't got the value yet. So a shift in who answers dominates a shift in what they think.

Response rate matters for the same reason. A drop from 12% to 5% doesn't sample the same people. The ones who leave are usually the indifferent middle, so the remaining respondents skew to the extremes.

Worked example (illustrative numbers)

SegmentShare beforeShare afterScore
Trial users6%38%41
Paying users94%62%78
Blended100%100%75.8 → 63.9

Blended before: 0.06×41 + 0.94×78 = 75.8. After: 0.38×41 + 0.62×78 = 63.9. That is the 12-point drop, with both segment scores unchanged. The cause was a survey prompt newly shown to trial users. The team that ran a retention campaign off the headline number spent a quarter on a problem that didn't exist.

What would change my mind

Reweight the current period to the previous period's mix. If the mix-adjusted score is flat, it's measurement. If it's still down 8–10 points, it's real. Then cut by segment, tenure, and recent support contact, and read free text only for the cut that moved.

If every segment is down by a similar amount, that's a product or reliability event, and I escalate the same day: release log, error rates, latency, and support volume for the same window.

Failure modes

  • Acting on mix. Detection: mix-adjusted score flat while the headline moves.
  • Dismissing a real drop as mix. Detection: segment scores down across the board, or a segment with stable share and a falling score.
  • Small-sample noise. Treating the score as a proportion of satisfied respondents: a 12-point move on 150 responses is about three standard errors, so it's real. The same move inside a segment of 20 respondents is noise: the 95% interval there is roughly ±22 points. Check n per cut before believing any of it.

Cost of delay

If it's real, a week of investigation costs a week of the bad experience. So I timebox: mix and instrument checks inside a day, segment cuts inside two, and I tell stakeholders what I'm checking and when I'll have an answer.

Curated: · Written: · Reviewed:

Q&A-75Nobody looks at the team dashboard. What is wrong with it?

The short answer

A dashboard nobody reads is almost never a chart problem: it's a missing decision. If no recurring decision changes based on what's on the board, looking at it is pure cost, and the team correctly stops paying it. The fix is either to attach the board to a decision someone already makes on a cadence, or to delete it.

Why it happens

Dashboards are pull. Everything competing for attention (pages, alerts, PR review requests, tickets) is push. Pull loses unless the reader has a reason to arrive. Three reasons they don't:

  1. No decision attached. If the number moves and nothing follows, the panel is decoration. Ask of each panel: what action does a bad value trigger, and who takes it?
  2. No cadence. A board gets read when it's embedded in a ritual the team already keeps: standup, weekly ops review, incident triage, sprint planning. A board with no slot in the calendar has no reader.
  3. No owner. If nobody is accountable for the number, nobody notices when it's wrong, and once one panel is wrong people stop trusting the whole board. Trust in a dashboard is close to binary.

There's also a category error underneath: "is anything wrong right now" is an alerting job, not a dashboard job. If a condition is urgent enough to need a human, it should page or post, not sit on a wall waiting to be noticed.

What to do

Name the audience, the recurring decision, and the cadence. Keep only the panels that bear on that decision, put the primary metric and its target at the top, and delete the rest. A workable rule: any panel with zero views in 90 days and no citation in a doc, incident, or ticket gets deleted. Deleting is the point: a smaller board gets read more, not less.

Failure modes and how to detect them

  • Dashboard blindness: nobody looks because nothing ever changes. Detect with per-panel view counts.
  • Stale or wrong data: one broken panel poisons the rest. Detect with a freshness check on the data source.
  • Vanity metrics: the number moves, no action follows. Detect by asking who acted on the last change.

Getting the evidence

Grafana Enterprise and Grafana Cloud expose dashboard usage insights; on OSS Grafana you can get view counts from your ingress or reverse-proxy access logs. Prometheus can log queries with --query.log-file, which shows which panels actually fire. Either way, the number you want is views per panel, not views per dashboard.

Worked example (illustrative figures)

BoardPanelsWeekly upkeepViews/quarter
everything403 h11
weekly decision620 min180

The 40-panel board costs about 156 hours a year to keep accurate and returns 44 views. The 6-panel board costs about 17 hours and returns 720. Upkeep scales with panel count; readership scales with decision relevance. They move in opposite directions.

Curated: · Written: · Reviewed:

Q&A-66How do you know whether a year of feature work made any difference?

The only clean answer is a long-lived holdout: randomly keep a slice of users on the old experience for the whole period, then compare the primary outcome at the end. Summing the lifts from individual experiments does not answer the question, because interactions, decay, and overlapping users make the sum wrong.

Assume you can pre-register one primary metric, keep assignment stable, and prevent spillover between holdout and treated users. If any of those fail, say so before you argue about the result.

Size the holdout from the effect you care about. For a two-proportion test at 5% two-sided alpha and 80% power, per-arm sample is roughly:

n = 7.85 * [p1(1-p1) + p2(1-p2)] / (p1-p2)^2

With a 10% baseline conversion and a 1 percentage point absolute lift (10% relative), that is about 14,700 users per arm. If you get 100,000 eligible users per month and hold out 10%, the holdout arm fills in about 1.5 months. For a 0.5 pp lift, it is about 59,000 per arm, or roughly 6 months. For a 0.35 pp lift, it is about 120,000 per arm, or 12 months. That is the real trade-off: smaller effects need either a larger holdout or a longer duration. A 20% holdout at 100,000 users per month can detect about 0.25 pp over 12 months, but it withholds twice as much revenue or learning.

Duration should cover the effect you are measuring. If the business has annual seasonality, run at least 12 months. If effects plateau in 4-6 weeks and there is no seasonality, 3 months may be enough. For a year of feature work, the holdout has to be held out from all of it, not just one launch, or you are measuring something narrower.

A holdout is the wrong tool when withholding causes harm or violates a compliance or security requirement. It is also weak under network effects: holdout users in a marketplace or social product interact with treated users, so the comparison is contaminated. With very low traffic, the required holdout is too large to be practical. In those cases use a switchback, stepped-wedge, or synthetic control, and state that they answer a weaker question.

Failure modes to watch: spillover through shared backend changes, novelty effects that fade, metric drift, and multiple testing across the year. Detect them with pre-period trend checks, covariate balance, leakage monitoring, and sequential alpha spending. If you cannot run the holdout, the honest answer is that you do not know the aggregate effect; you have local estimates and a list of assumptions.

Curated: · Written: · Reviewed:

Q&A-71How do you evaluate an optional feature's success?

What I'd say first

An optional feature has two questions inside it: do people use it, and does using it change anything. Only the second justifies the maintenance cost, so I'd define the outcome the feature was meant to move, get a counterfactual for the people who adopted it, and pre-commit to the adoption level below which we remove it. Usage is an input to that decision, not the decision.

Why adoption alone can't carry it

Opting in is a choice, and the people who choose a feature are the ones most likely to benefit from it. A 34% adoption number can be pure selection effect. So the comparison has to be against what the same kind of users did without the feature. The cleanest version is a randomised availability test: offer the feature to a random half, withhold it from the other half, and compare the outcome metric rather than the usage metric. When randomisation isn't available (a small population, or a feature users will demand once they see it) fall back to a matched cohort (propensity-score matched on prior activity) or a switchback, and label the estimate as weaker in the write-up.

A worked example (hypothetical figures)

Saved views, 2,000 users, randomised availability test, primary outcome = task completed within 7 days.

ArmnAdoptedTask completion
Offered1,00034%71%
Withheld1,000—70%
Offered, heavy adopters (≥5 uses/week)120100%88%

The 71 against 70 is the answer: no effect. The 88% row is the trap: heavy adopters look great because they're the users who were already completing tasks, and they're a subset of the offered arm, not a comparison. The feature carried about 2 engineer-weeks a quarter of maintenance, so the decision was to remove it.

What I'd measure, and how I'd catch the failure modes

Primary outcome plus guardrails (support tickets, p95 latency, error rate), and a threshold written down before launch. Novelty is the most common false positive: check week 4+ usage, not week 1. Control contamination is the second: if the withheld arm builds a workaround, the measured effect shrinks toward zero and you'll kill something that works. On a small population, run the power calculation first: under 200 users per arm you usually can't detect anything smaller than a large effect, and a null result there is not evidence of no effect.

When this doesn't apply

If the feature exists for compliance, security, or platform parity, adoption isn't the success criterion; the requirement being met is, and the measurement question becomes maintenance cost against the cost of the alternative.

The decision rule I'd state out loud: if the outcome moves and adoption clears the threshold, keep it; if the outcome doesn't move, remove it however good the adoption looks.

Curated: · Written: · Reviewed:

Q&A-72How do you avoid launching something you cannot measure?

What "cannot measure" means here: you cannot answer the question that decides whether to keep, fix, or kill the launch. That is usually one or two numbers, not a dashboard. So the answer is not "instrument everything": it is "make the decision number a launch gate, and buy everything else cheaply."

Why post-launch instrumentation fails. Events are not retroactive. If the address step never emitted, no later work recovers who dropped there. Deploy lag adds a second gap: even a same-day patch leaves the first hours unmeasured, which is exactly when the worst bugs surface. So the events and their definitions belong in the spec with the feature, but only the ones the decision depends on.

Trade-offs. Upfront spec-writing is not free, and treating it as always-right is how teams end up with 60 events nobody queries.

ApproachCostAnswersBreaks when
Full event spec in the design doc~15–30 min per event to define, review and own; 20 events ≈ a day of eng timestep-level funnels, segment attributionevents rot after refactors; unused events still cost storage and a query someone maintains
Proxy from existing telemetry (logs, DB rows, existing events, support tickets)near zerocoarse "did it move"cannot localise the drop or split by segment
Sampled events (e.g. 10%)cheap at volumehigh-volume totalsa 0.5% segment yields ~0.05% of traffic: too thin to read
Ship behind a flag with a minimal event set, expand afterlow upfrontthe one decision numberflag removed before the expanded set lands

Decision rule: if the keep/kill call hinges on a number existing telemetry cannot produce, spec it. If the change is reversible and low-blast-radius, a proxy plus a rollback is enough.

Failure modes, and how each is detected.

  • Event defined but never emitted: a staging assertion, then a day-one production query returning zero rows.
  • Event emitted with a null or wrong property: a cardinality check (country should have more than one distinct value).
  • Sampling silently applied to a decision-critical event: compare sampled totals against a raw count for one hour.
  • Event rot after a refactor: alert on volume dropping to zero.

Worked example (hypothetical figures). A checkout redesign shipped without funnel events. Five weeks later the review had a support-ticket count (~40 tickets mentioning "payment") and nothing else. It could not tell whether users dropped at address entry or at payment, so the team could not decide whether to fix the form or the payment integration. Backfill was impossible: the intermediate steps had never emitted. One checkout_step_completed event with a step property would have answered it.

Verification. Run the analysis query against production data within the first day and confirm it returns usable rows: non-zero, expected cardinality, expected property distribution. A query that returns zero rows on day one is a launch blocker, not a follow-up. A feature without events ships blind and stays that way.

Curated: · Written: · Reviewed:

Q&A-70Your outcome takes six months to measure. How do you steer before then?

The 90-second answer. I'd run two loops at different speeds. The slow loop is the outcome itself: keep measuring it on the same definition, so you never lose the ability to check yourself. The fast loop is a proxy you can read weekly, and the only thing that makes a proxy legitimate is that you measured its relationship to the outcome on your own historical data, not borrowed it from a blog post. Then I'd say the assumption out loud: "we believe X at week 2 moves Y at month 6, and we'll re-fit that belief every quarter." If I can't get a defensible proxy, I'd rather run short-cycle experiments that produce their own signal than steer on a number I can't tie back to the outcome.

Four ways to steer, and when each is right.

  1. Leading–lagging pair. Pick a behaviour that historically predicts the outcome, quantify the relationship, steer on it, keep measuring the outcome. Right when you have several cohorts that have aged fully and the product isn't changing shape. This is the default for a mature product.

  2. Short-cycle experiments. When the outcome is too slow and no proxy is trustworthy, run experiments whose own result is the signal: a pricing test, an onboarding A/B, a concierge test with 20 users. Right when you're early, when the mechanism is unknown, or when the proxy's validity is in doubt. Cost: you learn about the experiment's metric, not the outcome, so you still need a theory linking them.

  3. Cohort interleaving. Alternate the change across cohorts, geographies or time windows so a treated and an untreated group age in parallel. Right when you can't wait six months for a clean read and can tolerate running two versions. Cost: operational complexity, and it isn't always possible or ethical to withhold the change.

  4. Instrument the mechanism. If you can name the causal chain, activation → habit → retention, measure each link. Right when the chain is well understood. Wrong when you're guessing at the chain, which is most of the time.

When a proxy is too risky to steer on. I'd refuse when: the correlation was never measured on our data, only assumed; the sample is one cohort, so you can't separate signal from seasonality; the proxy is gameable, meaning a team can move it without moving the outcome; or the product is about to change materially, which is exactly when relationships break. In those cases, fall back to experiments or to measuring the mechanism.

The failure mode, and how you'd catch it. Illustrative figures: a subscription product steered on first-week logins as a predictor of annual renewal. After a mobile launch the relationship collapsed while the team kept optimising logins for three quarters.

IndicatorCorrelation before launchAfter launch
first-week logins0.610.14
second-week import0.550.58
teammate invited0.490.52

The tell is not the outcome: that arrives too late. The tell is the proxy's correlation drifting against the most recent fully-aged cohort. So detection is a standing job: every quarter, re-fit each candidate indicator against realised outcomes and watch for the one that moved. Here, logins fell off a cliff and the team would have seen it within one cohort cycle if anyone had been re-fitting. The fix is to hold a second indicator (import, teammate invited) that survived the change, and to treat a proxy as expired the moment its correlation drops below the threshold you set for steering.

The one-line version: the six-month outcome is ground truth; the proxy is a loan against it, and you keep the loan only while you can service it with fresh data.

Curated: · Written: · Reviewed:

Q&A-103How do you decide what your product's success metrics should be, and what do you do when one of them improves while the user outcome gets worse?

What I say first

Metrics come from the decision, not the dashboard. Before naming any, I write down the decision the number is meant to inform and who makes it. If nothing changes when it moves, that's reporting, not a metric.

Then I pick a small set in three layers, and I never ship a primary metric without a counter-metric next to it.

LayerExample (self-serve analytics tool, $49/seat)CadenceOwner
Outcome% of accounts still active at day 90monthlyPM + GM
Driver% of new accounts that publish a shared dashboard in week 1weeklyteam
GuardrailSupport contacts per 100 active accounts; p95 dashboard loadweeklyteam

Four tests for any candidate: it moves within a quarter if the work is right; it can't be moved without the user getting value; it's leading enough to steer by; and one team owns it end to end. When two candidates pass, I take the one closest to the moment the user gets value.

Metric up, outcome worse

Treat it as a definition bug until proven otherwise. Three checks, in order:

  1. Did the metric itself change? Instrumentation, definition, or denominator. A ratio whose denominator shrinks looks like growth.
  2. What did the guardrails do? If a guardrail moved the wrong way in the same window, the primary is measuring the wrong thing.
  3. Segment it. A flat mean hides two opposite segments: Simpson's paradox, usually from a channel or plan mix shift.

Worked example, illustrative figures. Support contacts per 100 accounts fell from 4.1 to 2.8 over a quarter: the deck said "support load down 32%." But day-90 retention fell 61% → 54%, and CSAT on the tickets that remained fell 4.4 → 3.9. Cause: we'd moved the contact link behind two menus and removed the in-app chat entry point. Tickets fell because friction rose, not because problems fell. Searches for "contact support" tripled, and 40% of users who searched it churned within 30 days.

Fix: keep the metric, pair it with an unresolved-problem rate, sessions where a user hits an error and doesn't finish the task, and put the contact path back. Contacts returned to 3.9 and retention recovered. The metric got worse and the product got better.

The general rule

When a metric and the user outcome disagree, the outcome wins and the metric is wrong: mis-defined or mis-gamed. Then I do one of three things: redefine it so it can't move without user value, demote it to a diagnostic, or keep it and add the counter-metric that makes the gaming visible.

Failure modes I watch for

  • Ratio metrics with shrinking denominators: contacts per account while accounts churn out.
  • Averages hiding the tail: median task time flat, p95 doubled.
  • Survivorship: cohort retention looks strong because the worst users never activated and dropped out of the denominator.
  • Segment mix: a new low-intent acquisition channel lifts signups and drops activation, and the blended number looks unchanged.

When not to lean on metrics: pre-product-market-fit, or a genuinely novel bet where the leading signal is qualitative. There I'd rather run five interviews a week than watch a number that can't move yet.

Curated: · Written: · Reviewed: —

Q&A-58When would you not run an experiment?

I would not run an experiment when any of four conditions fails: the result cannot change the decision, randomisation is impossible or contaminated, the cost of a wrong call is irreversible, or the test cannot be powered to detect an effect worth acting on. The default is not “always test”; it is “test when the causal question is answerable and the answer is actionable.”

Assume we mean online controlled experiments for product changes, with a metric and a decision owner. The four conditions are necessary, not sufficient.

SituationWhy not experimentWhat to do instead
Cannot randomise (e.g., pricing in a marketplace with shared supply)SUTVA violated; treatment leaks between armsSwitchback or geo holdout, or model-based causal inference
Irreversible or one-shot (brand campaign, layoffs, major pricing)Can’t undo; no second chancePre-registered decision rule, staged rollout in low-risk segment, qualitative + guardrails
Traffic too low for MDEUnderpowered; result is noiseCompute MDE; if not feasible, use qualitative or ship with monitoring
Decision already made or no actionExperiment is theatreDon’t run; document assumptions
Ethical/legal (withholding treatment for vulnerable users)HarmEthics review; observational study; no experiment

Also skip when the cost of the test (engineering, delay, opportunity cost) exceeds the value of the information.

The power condition is the one teams skip most often. Compute the sample size before designing the test. Example (hypothetical): baseline conversion 10%, smallest effect worth acting on is 2 percentage points (10% -> 12%). With alpha 0.05 two-sided and 80% power, n per arm = (1.96 + 0.84)^2 * (0.100.90 + 0.120.88) / 0.02^2 = 7.84 * 0.1956 / 0.0004 ≈ 3,834. Total 7,668. At 400 visitors/week, that’s ~19 weeks. If the pricing decision must ship in 4 weeks, the experiment cannot answer the question; use a staged rollout with a holdout and pre-registered guardrails.

Failure modes: An underpowered test produces a p-value that is not significant, but teams call a 2% lift a win anyway. Detection: pre-register MDE and sample size; if you peek, use sequential testing. Network effects: treatment in one geo changes supply in another; detection: check for spillover by comparing treated and control geos on pre-period trends. Novelty effect: early lift fades; detection: run longer and look at week-over-week effect. One-shot decisions: a Super Bowl ad cannot be A/B tested; detection: pre-register decision rule and use synthetic control.

When not to experiment is not a single rule. It is a checklist: can I randomise? can I measure? can I power? is the decision reversible? If any answer is no, pick a different method. An underpowered test is an opinion with error bars.

Curated: · Written: · Reviewed:

Q&A-59An experiment looks significant on day three. Do you ship it?

No, not on day three alone. Ship only if the pre-registered plan said day three is the readout. If the plan said "run two weeks" and you are looking on day three, the p-value you are reading is the p-value of the data so far, not of the experiment. The experiment's false-positive rate is the rate over every look you take, and it is much higher than 0.05.

Why: each look is another chance for noise to cross the line. If the looks were independent, five peeks at α = 0.05 give a 1 − 0.95⁵ = 22.6% chance of at least one false positive. Daily looks on accumulating data are correlated, so the real inflation is smaller, but it is still well above 5%, and it grows with the number of looks. A day-three "significant" result is the maximum over the looks so far, not a single test. If you look often enough, noise will eventually agree with you; that is the whole problem.

What to do instead:

  • Fixed horizon. Pre-register the duration and sample size, read once at the end. Day three is a readout only if the plan said day three. This is the simplest rule and the one most teams should use.
  • Sequential design. If you need to look early, use a method built for it: group-sequential (Pocock, O'Brien-Fleming) or always-valid confidence sequences. These control the error rate across looks. Pocock uses the same boundary at every look: about 2.87 for five looks at two-sided α = 0.05, so a day-three z of 2.0 still does not stop. O'Brien-Fleming spends less alpha early and more at the end, so it is harder to stop early and easier to stop late.
  • If you must act now. Ship behind a flag to a small percentage and keep measuring, or extend the test. Do not call it a win.

The failure mode is peeking and stopping on favourable days. Hypothetical, illustrative:

ExperimentsStopped earlyConfirmed by later holdout
9 run62 of 6
3 run to full duration33 of 3

The six early stops were decided on good days; four of them had no effect. The three that ran to the pre-registered end all held up. The cost was 14 engineer-weeks on features that did nothing.

The decision rule: write the stopping rule and duration before launch, and check the reported result against that rule, not against a favourable day.

Curated: · Written: · Reviewed:

Q&A-61An experiment shows no overall effect. Is it a dead end?

No, but only after you rule out that the test could have detected the effect at all. A flat average is one of three things, and they call for different next steps:

  1. A true zero. The effect is smaller than your minimum detectable effect (MDE), and the confidence interval is tight enough to bound it.
  2. An underpowered test. The interval is wide; you learned nothing.
  3. Two real effects cancelling. Averages are linear, so a +22% segment and a −9% segment can sum to zero.

Only case 1 is a dead end, and even then it is a useful one: you now have a bound you can quote.

First, is the null informative? Read the CI, not the p-value. If the primary metric's 95% CI is [−0.3%, +0.4%] and you would only ship at +1%, stop: that is a real answer. If it is [−8%, +9%], the test was inconclusive, not null. Two mechanical checks catch most false nulls: a sample-ratio mismatch (chi-square on assignment counts; p < 0.001 there means the split or the logging is broken) and a trigger/dilution analysis (what share of assigned users actually saw the feature. A 20% exposure rate dilutes a 5% effect to 1%).

Power is the other half. At a 5% baseline conversion, detecting a 5% relative lift (0.25 pp) at 80% power and α = 0.05 needs roughly 120,000 users per arm. If you ran 20,000, the null is uninformative by construction.

Then, and only then, look at segments: pre-registered ones. The right test is the interaction, not a scan of subgroup p-values. Pre-register the two or three cuts you had a reason to expect to differ, and report the interaction effect with its CI alongside the pooled result. An unplanned segment that looks interesting is a hypothesis for the next test, not a finding.

The reason for that discipline is multiplicity. With 20 independent segments at α = 0.05, the chance of at least one false positive is 1 − 0.95²⁰ ≈ 64%. A scan will always hand you a winner; it just won't be real.

Worked example (illustrative figures). A team killed a feature on a null pooled result. A later pre-registered rerun split by account size:

SegmentShare of usersEffect
over 5 seats21%+22%
2–5 seats16%+9%
single seat63%−9%
pooled100%+0.4%

The weighted sum checks out: 0.21×22 + 0.16×9 + 0.63×(−9) = 4.62 + 1.44 − 5.67 = +0.39 ≈ +0.4%. The pooled number was real and useless; the split was the finding.

When it is a dead end. Adequate power, a tight CI around zero, no pre-registered interaction, and guardrails flat. Then stop, and record the bound: "we can rule out any effect larger than 0.4 pp" is a result you can reuse.

Two failure modes to name. Novelty/primacy: a short test can show a null that reverses or grows over a longer holdback, so check the metric's time series before concluding. Proxy metrics: if the primary metric is a click and the real goal is retention, a null on the proxy says nothing about the goal. Look for a downstream metric with a longer window.

So: not a dead end by default, but not a licence to fish either. Rule out the mechanical explanations, then test the one or two interactions you predicted.

Curated: · Written: · Reviewed:

Q&A-60A new design wins for a week and then the gap closes. What happened?

The 90-second answer

The week-1 win was mostly not value. Two effects produce it and both decay. Novelty: any visible change draws attention, so engagement rises because the interface is new, not because it is better. Change aversion: habitual users penalize a change that breaks a learned routine, so the treatment arm is dragged down by the people who used the old design most. They run on different clocks and different populations, which is why the pooled curve can look like a clean win for a week and then flatten.

The third possibility is that nothing happened: you stopped at the peak of a noisy early sample. A week of data on a low-frequency metric often cannot separate a 2% effect from zero.

How to tell them apart

  • Plot the effect by day with confidence intervals, not the pooled number. Novelty decay is monotone: the effect shrinks toward zero as the treatment arm's exposure ages.
  • Split by prior-session count. Novelty and change aversion live in returning users; brand-new users have no habit to break and no baseline to compare against, so their effect is the closest thing to a clean read.
  • Check whether the metric is a one-time action. "Tried the new layout", "dismissed the banner", "completed onboarding" can only fire once per user; the treatment arm burns through that action in week 1 and then has nothing left.
  • Check the exposed population over time. If the treatment causes churn, the users left in the arm by week 6 are a different, more tolerant population, and the effect moves for compositional reasons.

Hypothetical figures, illustrative only:

WeekNew usersReturning usersPooled
1+19%−2%+14%
3+8%−2%+5%
6+2%−6%−3%

The pooled column is the trap: positive for two weeks, then crossing zero, and the crossing is driven by returning users getting worse, not by new users getting bored.

Arithmetic on duration. To detect a 5% relative lift on a 20% baseline at 80% power, two-sided 5%, you need about 16 × 0.16 / 0.01² ≈ 26,000 users per arm. A week on a low-traffic surface will not get there, which is why the week-1 number is usually a peak, not an estimate.

What I would do instead

  • Pre-register the duration and the decision rule; do not stop at the peak.
  • Run a long-run holdout: keep 1–5% of traffic on control for 4–8 weeks after launch and compare against the shipped treatment. This is the only design that measures the steady state, because the treatment arm's novelty has already decayed by then.
  • If you cannot hold out, use a reverse test: after launch, flip a slice of the treatment population back to the old design and see whether the effect reappears. If it does, you are measuring novelty in both directions.

Failure modes and detection

  • Stopping at the peak: detected by plotting the effect over time; the fix is sequential-aware analysis or a fixed horizon.
  • One-time-action metric: detected by checking whether the metric can fire more than once per user per period.
  • Composition shift: detected by comparing the covariate distribution of the arms week over week, not just at assignment.
  • Interference: if the feature is social, treated users change control users' behaviour; detected by cluster-randomized or switchback designs.

When not to run long. If the change is a bug fix or a compliance requirement, ship it. If the metric is a one-time action, a long test adds nothing. And a long test costs you the ability to iterate and exposes you to seasonality, so pair it with a holdout rather than extending every test.

Measure the habit, not the reaction to the change.

Curated: · Written: · Reviewed:

Q&A-67A result is statistically significant at 0.3 percent lift. What do you do?

Pin down the units first. "0.3 percent lift" could be relative or absolute, on conversion, revenue, or a proxy like clicks, over a day or a month. A 0.3% relative lift on a 3% conversion rate is 0.009 percentage points: nine extra conversions per 100,000 sessions. That absolute number is what the decision turns on; the p-value is not.

Significance answers one question: is this effect distinguishable from noise at the sample size we ran? It says nothing about whether the effect is worth carrying. Power grows with traffic, so at high volume a 0.3% lift can clear the 5% level and still be smaller than the cost of the code path, the flag, and the interaction risk it adds to every future test.

So the rule: write down the minimum effect worth shipping before the test starts. The smallest lift whose expected value covers the ongoing cost of owning the change, and read the confidence interval against that bar, not the point estimate. Ship only when the interval's lower bound clears the bar.

Three results against a 2% shipping bar:

Observed lift95% intervalShip?
+0.3%+0.1% to +0.5%No: interval sits entirely below the bar
+2.4%+1.1% to +3.7%Not yet: point estimate clears, lower bound does not
+5.0%+3.1% to +6.9%Yes: lower bound clears the bar

Then do the arithmetic, because it usually decides the marginal case. Hypothetical: 10M sessions a month, 3% baseline conversion, $8 margin per conversion, and the change costs about 2 engineer-weeks a quarter to own. 80 hours at $100/hour fully loaded is $8,000 a quarter, or $2,667 a month. Break-even is 333 extra conversions a month, which is 0.0033 percentage points absolute, or 0.11% relative. A 0.3% lift (900 conversions) clears that; the interval's lower bound of 0.1% (300 conversions) does not. The honest call is to keep running until the interval tightens, or to ship on the lower bound only if the ownership cost is genuinely lower than estimated, not to ship because the result is significant.

Before any of that, rule out the ways a small significant result lies to you:

  • Optional stopping. Checking daily and stopping on the first p < 0.05 inflates the false-positive rate well past 5%; fix the horizon in advance.
  • Novelty and primacy effects. A new button gets clicked because it is new. Re-read the metric two weeks after launch, or keep a long-lived holdout.
  • Segment heterogeneity. A +0.3% aggregate can be +4% for new users and −3% for returning ones. If the segments disagree, the aggregate is not the finding.
  • Metric proxy. If the primary metric is clicks and revenue is flat, you moved the proxy, not the business.
  • Multiple comparisons. Twenty metrics at p < 0.05 produce one "winner" by construction. Correct for the number of looks, or pre-register one primary metric.

Detection for each: a pre-registered minimum detectable effect and a fixed horizon, a holdout that persists after launch, per-segment readouts, and a primary metric named before the test.

If the interval's lower bound clears the bar, ship. If it doesn't, extend the test or kill the change: significance alone doesn't move it either way.

Curated: · Written: · Reviewed:

Q&A-68When does randomising individual users give the wrong answer?

Randomising individual users gives the wrong answer whenever one user's treatment changes another user's outcome: a violation of the stable unit treatment value assumption (SUTVA). The estimator still returns a number; it just measures a blend of the direct effect and the interference, and that blend can be larger, smaller, or opposite in sign to what a rollout would produce.

Three settings where this bites:

  • Shared supply. Marketplace, ad auction, inventory, delivery capacity. Treating some users pulls a scarce resource away from control users.
  • Shared social graph. Referrals, messaging, feeds. A treated user's posts change what control users see.
  • Shared physical or logical resource. Same server pool, same queue, same workspace, same city's traffic.

Worked example (hypothetical, arithmetic only). One city, 200 buyers, one scarce item with 60 units in stock. Each buyer wants one unit.

Individual randomisation: 100 treated, 100 control. Treated buyers see the item first and 60 of them buy it; stock is gone before control buyers arrive, so 0 control buyers convert.

  • Treated conversion: 60/100 = 60%
  • Control conversion: 0/100 = 0%
  • Measured lift: +60 percentage points

Cluster randomisation by city: the treated city sells 60/200 = 30%; the control city also sells 60/200 = 30%, because the item sells out to whoever arrives first regardless of the boost.

  • Measured lift: 0 percentage points

Same product, same 60 units sold. The individual-randomised design reported a 60-point win that a city rollout would not reproduce, because the "lift" was control users losing access to stock the treatment arm consumed.

Detection: plot control-unit outcomes against treatment intensity nearby. Share of treated users in the same city, seller, or time bucket. If control conversion falls as treated density rises, SUTVA is broken. The cheapest confirmation is to run the same change under cluster randomisation and compare; divergence between the two estimates is the tell.

Analysis consequence: once you randomise clusters, the effective sample size is the number of clusters, not users. Design effect = 1 + (m − 1)ρ. With cluster size m = 200 and intra-cluster correlation ρ = 0.05, that is 1 + 199 × 0.05 ≈ 11, so you need roughly 11× the users for the same power. Budget the duration for that before committing.

DesignHandlesCost
Individualno interferencenone
Cluster (city, seller, workspace)shared supply, shared graphpower, ~1+(m−1)ρ
Switchback (alternate time windows)congestion, time-varying loadcan't measure long-run effects
Saturation (vary treated share within cluster)estimates spillover slopeneeds many clusters

Individual randomisation is the right default when there is no shared resource and no cross-user coupling: a copy change, a per-user ranking model with no capacity constraint, a pricing page. The test is whether moving one user's treatment changes any other user's potential outcome. If it does, randomise at the level that contains the interaction.

Curated: · Written: · Reviewed:

Q&A-65Users who do X retain better. Should you push everyone to do X?

No, not on the correlation alone. A behaviour that correlates with retention usually marks users who were already going to stay. The behaviour is a symptom of the segment, not the cause of the retention. The way to tell the two apart is to induce the behaviour in a random group and compare that group's retention against a control.

Why the naive read fails. Users who invite a teammate tend to be on team-shaped use cases: more seats, more intent, a reason to come back. Users who never invite may be solo evaluators who churn regardless. Nothing about the invite itself has to be doing work.

The test. Randomise at account creation. Prompt half the accounts to invite; leave the other half alone. Measure 90-day retention on an intent-to-treat basis: everyone assigned to the prompt counts, whether or not they complied. Compare induced against control. Comparing induced against natural adopters re-imports the selection effect you were trying to remove.

Take a worked example. Inviters retained at 62% against 24% for non-inviters, so the team prompted every new account to invite a teammate. The prompt lifted invitations by 40%.

Retention by group in the invite experiment:

GroupInvited a teammate90-day retention
Natural adoptersyes62%
Never invitedno24%
Induced by promptyes26%
Control (not prompted)mixed25%

The 62/24 gap was selection. The prompt moved the behaviour and left retention flat: 26% against 25%.

When you can skip the experiment. If the action is free, reversible, and carries no downside for the user (a one-line tooltip, a default that is easy to undo) ship it and read the result as a rollout. The experiment earns its cost when the action spends something: user attention, money, or the goodwill of the people being invited.

Failure modes and how to catch them.

  • Comparing the wrong groups. Induced vs natural adopters. Detect by checking the randomisation held and that the comparison is control-based.
  • A prompt that didn't move the behaviour. If invitations rose 40% but the manipulation check is weak, the test says nothing. Check the first-stage effect before reading retention.
  • Underpowered test. A 1-point move on a 25% baseline needs roughly 30,000 accounts per arm at 80% power (n ≈ 7.85 × (0.25×0.75 + 0.26×0.74) / 0.01² ≈ 29,800). Most teams run a few thousand and then argue about noise.
  • Harm to the recipient. Invites cost the invited person's attention. Watch unsubscribe and block rates, not just retention.

What I'd say. Don't push everyone to do X because X-users retain. Run the inducement test, and roll out only if the induced group beats control.

Curated: · Written: · Reviewed:

Q&A-69An experiment produced an effect you cannot explain. What next?

In the first two minutes

Clarify what "cannot explain" means before picking a method, because it is three different situations. The metric moved but randomisation may be broken. The aggregate moved but the effect is concentrated somewhere. Or the effect is real and durable and you have no story for why. The first two are quantitative questions and cheap to settle. Only the third justifies qualitative work.

Assumptions: the experiment ran to plan, the primary metric moved beyond its confidence interval, and no guardrail is screaming.

Answer: run the cheap quantitative diagnostics first, in cost order, because each can turn "unexplained" into "explained" or "invalid" in minutes. Go qualitative only when they come back clean, and then use it to generate a mechanism, not to confirm one.

Quantitative diagnostics, cheapest first

  1. Sample ratio mismatch. Chi-square the assignment counts against the planned split. p < 0.001 means the experiment is broken, not surprising: the effect is a selection artefact, and no amount of watching users will fix it.
  2. Segment the metric. Re-cut by new vs returning, platform, geo, entry point. An aggregate +8% that is +30% in one segment and flat elsewhere already has a mechanism, and it is segment-specific.
  3. Novelty and primacy decay. Plot the effect by days since first exposure. +8% in week one decaying to +1% by week three is a novelty effect; the mechanism is "the change was noticed", and it predicts the effect disappears.
  4. Counter-metrics. Check latency, support contacts, refunds, error rates. A win that coincides with a 200 ms p95 regression has a mechanism you can name.

When to go qualitative instead

Session recordings and five interviews with users from the winning arm are the right tool when the diagnostics are clean, the effect is durable, and the why is missing. They are cheap per hypothesis and good at generating mechanisms. Their limit: they do not bound the effect size, they do not tell you where the effect stops being true, and five users cannot separate a mechanism from a plausible story. Use them to produce a hypothesis, then test it quantitatively.

Then test the mechanism, not the change

Write the mechanism so it makes a prediction about a second application: direction and rough size. If it cannot be falsified, it is not a mechanism.

Worked example (illustrative numbers):

FlowChangeResult
checkoutremove shipping-cost surprise+8%
signupsame pattern−2%
upgradesame pattern−4%

The team shipped the checkout win without a mechanism, reapplied the pattern to two other flows, and both lost. The mechanism was "the gain came from removing a cost surprise at the point of payment": signup and upgrade had no cost surprise to remove, so it predicted no gain there. Stated up front, it would have saved two experiments.

Failure modes

  • Treating a segment effect as universal. The aggregate is an average over segments, and the average is what hides the decision.
  • A mechanism that explains the direction but not the size: "users liked it more" fits any result.
  • Confirming the mechanism with the same data that produced it. The second application is the test.

Curated: · Written: · Reviewed:

Q&A-34How do you set a quarterly target that is neither sandbagged nor fantasy?

A target is sandbagged when the team hits it with the mechanisms removed, and fantasy when the mechanisms can't ship and be measured inside the quarter. Both are the same failure: the number isn't decomposable. So build it from parts you can each falsify by mid-quarter.

Target = counterfactual baseline + Σ(mechanism × expected effect), each effect carrying a range, with the committed number set at the bottom of the summed range.

Three things make that work.

The baseline is the counterfactual, not last quarter's number. If Q4 brings a promo that shifts signup mix, activation moves without anyone doing anything. Start from the trailing 8 weeks, then adjust for known mix and seasonality. That adjustment is what stops sandbagging: a target you'd hit by doing nothing isn't a target.

Every mechanism has a ship date and a measured effect size. A randomized pilot gives you an effect with a confidence interval; an analogue from another region or product gives you a discounted one. Never take a round number out of a planning doc.

Commit at the low end of the range. Plan at the middle, stretch at the top. The commit is what you're accountable for; the range is what you're honest about.

Worked example: B2B SaaS, quarterly activation (signup → first value action within 7 days):

PartValueSource
Trailing 8-week baseline31.0%4,200 signups
Q4 mix/seasonality adjustment−2.0 ptsprior two Q4s
Counterfactual baseline29.0%
CSV import fix+5 pts (range +3 to +7)randomized pilot, 4 wks, ~1,050/arm, +6.0 pts (95% CI +2.1 to +9.9), discounted for novelty
Guided setup checklist+2 pts (range +1 to +3)EU analogue, +4.0 pts on n=2,600, discounted 50%
Commit / plan / stretch33 / 36 / 39sum of ranges

Mid-quarter trace, week 6 of 13:

  • Import fix shipped week 3 to 100%. Post-ship cohorts (weeks 4–6, n≈1,600) activate at 33.4% against the 29% counterfactual: +4.4, inside the +3 to +7 range.
  • Guided setup slipped from week 5 to week 8. Effect so far: zero.
  • Tracking: 29 + 4.4 + 0 = 33.4. Commit is safe, the plan of 36 is at risk.
  • The gap is attributable: a schedule slip on one mechanism, not a wrong effect size on the other. That's the point: you can act on it. Either ship the checklist by week 8 (five weeks left, ~2,600 signups) or cut it to its first step.

Failure modes and how you catch them:

  • Baseline drift. Mix shift or seasonality moves the counterfactual. Track a holdout or a matched prior-period cohort weekly, not just the headline number.
  • Mechanism not shipped. Compare ship date against exposure: the share of signups actually hitting the new flow.
  • Effect size wrong. The pilot was unrepresentative (small n, novelty, different segment). Re-measure the first two weeks of full rollout against the pilot's CI.
  • Effect too small to confirm in-quarter. A +2-point lift read against a 10% holdout over five weeks is about 0.7 standard errors. You can't call it this quarter; keep the holdout and read it next quarter rather than declaring victory on noise.

At quarter end the review asks which part was wrong (baseline, mechanism, or effect size) not whether the number was hit. A target you can't debug that way is a wish with a deadline.

Curated: · Written: · Reviewed:

Q&A-39A competitor halves its price. How do you respond?

What I'd say first: don't match on the announcement. Match on evidence, and only in the segments where price is actually deciding deals.

A 50% cut is a claim about the competitor's cost structure or their funding, not just a price. Three cases, and they call for different responses. They have a genuinely lower cost base: you can't win a price war, so compete on something they can't copy or exit the segment. They're buying share with investor money: their price isn't sustainable, and matching just burns your margin until their funding runs out. They're defending an installed base: match selectively where you'd otherwise lose a reference account.

The diagnostic. Pull the last 20–30 losses and separate "price was the stated reason" from "price was decisive." Reps name price because it's the easiest thing to say, so the decisive count is usually well below the stated one. Also pull your own discount data: if your reps are already discounting 20% to close, the competitor's list price isn't the real comparison. Your street price is.

The arithmetic. Worked example, illustrative figures: list price $100, COGS $40, contribution $60 per unit. Competitor cuts to $50. Matching leaves $10 per unit, so you need 6x the volume to hold the same gross profit. At 1,000 units a month you were making $60,000 of contribution; at the matched price you need 6,000 units to make the same $60,000. That multiple is the number to put in front of the pricing decision, not the percentage.

What happened in one case. A team matched a competitor's 50% cut across the board. They needed 6x volume, got 1.2x, and the post-mortem win/loss review found price had been decisive in 4 of the last 20 losses, not the 14 where reps had named it. The cut cost them margin on deals they were already winning.

Options, in order of preference:

  1. Hold price, change the comparison. Repackage (bundle support, SLA, migration help) or move the conversation to total cost of ownership. Works when the buyer's real risk is switching cost, not sticker price.
  2. Match selectively. Only where price is decisive and the account is strategic: a reference, a logo, a beachhead for expansion. Approve at deal level with an expiry date so it doesn't become the new list price.
  3. Match across the board. Only if your cost structure can absorb it and the segment is one you must hold. Rarely true.

Failure modes and how you'd see them. Across-the-board matching shows up as falling price realization, net revenue per unit, while win rate barely moves: you paid for deals you'd have won anyway. Selective discounts leak; watch the discount approval log for the same concession reappearing in segments you didn't intend. And if the competitor's cut is funded rather than structural, their gross margin disclosures or funding news will show it within a couple of quarters. A price war you can outlast is one you shouldn't join at their tempo.

First week: recount losses for decisiveness, compute the break-even multiple at your actual margin, and set a deal-level approval threshold above which matching needs a named reason. Then decide segment by segment.

Curated: · Written: · Reviewed:

Q&A-48How do you split investment between acquisition and retention?

I don't set a fixed ratio. I allocate the next dollar to whichever side has the shorter payback, and I hold a retention floor: if the cohort curve hasn't flattened by month six, retention gets the next dollar regardless of what acquisition's payback looks like. Acquisition spend into a cohort that leaks is unrecoverable; retention spend compounds across every cohort already acquired.

Assumptions I'd state out loud. A subscription or repeat-purchase product, cohorts I can track by signup month and acquisition channel, a blended CAC I trust, and a planning horizon of about a year. If any of those is missing, the split is a guess and I'd say so rather than produce a number.

Why payback, not a ratio. A ratio like 60/40 is a planning artefact. What decides the split is the marginal return on the next dollar on each side. Average CAC and average retention tell you where you've been; marginal numbers tell you where the next dollar goes. So I compute two things: cohort retention by month, and payback period, CAC divided by monthly contribution margin, by channel.

A worked trace. Take a hypothetical subscription at $30/month with 80% gross margin, so $24 of contribution per account per month. Blended CAC is $310, which is 12.9 months of payback: already past a 12-month target. A quarter of acquisition spend added 9,400 accounts at $310 each: $2,914,000 out. Month-six retention was 18%, so 1,700 accounts survived, and each had realised $190 by then: $323,000 back. The cohort is underwater by about $2.6M at month six.

Now compare the marginal dollar. $50,000 into onboarding and lifecycle messaging that lifts month-six retention from 18% to 26% keeps 752 more accounts, worth $142,880 realised by month six: about 2.9x on the spend. The same $50,000 into acquisition at $310 buys 161 accounts, of which 18% survive: 29 accounts, $5,510 realised. Same money, roughly 26x difference in month-six return. That is the argument for weighting retention while the curve is still falling.

Failure modes I'd watch. Blended retention hides a channel that never retains: segment the curve by acquisition channel before trusting it. Retention spend that buys discounts rather than activation shows up as margin erosion with a flat curve; check discount depth per cohort. A curve that flattens early but at a low level, say 20%, is a product problem, not a marketing one, and more retention spend won't move it.

When the rule flips. Pre-product-market-fit, or in a one-time-purchase business where retention is structurally capped, acquisition is the only lever and the floor doesn't apply. Once the curve flattens above the level that makes payback work, I'd move the split toward acquisition until marginal CAC rises.

I'd revisit the split quarterly against the cohort curve, not annually against the plan.

Curated: · Written: · Reviewed:

Q&A-49How do you prioritise between supply and demand in a marketplace?

Start by finding which side is binding, then fund that side until the constraint moves. A marketplace clears locally, so the answer is per market, not global.

Assumptions I'd state up front: there is a measurable match event, supply capacity is roughly fixed in the short term, and the two sides are not interchangeable. Under those, priority is a measurement question, not a strategy preference.

The binding side is the one whose scarcity caps match rate. Measure match rate (matched requests / total requests) and time to match, split by segment and geography, over a window long enough to cover the demand cycle. Fund the side with the larger marginal effect on match rate until match rate recovers, then re-measure. The constraint moves, so a fixed preference for supply or demand is wrong within a quarter.

Worked example, hypothetical figures. A city has 8,000 weekly buyers and supply that can serve 3,280 matches a week, a 41% match rate. A demand campaign adds 12,000 buyers. Supply does not move, and existing buyers are matched first, so the 12,000 new buyers capture none of the 3,280 matches.

StageBuyersSupply capacity (matches/wk)Match rateUnmatched new buyersReturn rate of unmatched
before8,0003,28041%——
after demand campaign20,0003,28016%12,0006%
after supply push20,00011,60058%——

3,280 / 8,000 = 41%. 3,280 / 20,000 = 16%. 11,600 / 20,000 = 58%. The demand push bought 12,000 buyers and a 25-point drop in match rate; 6% of the unmatched ever returned. The supply push that followed added 8,320 matches of weekly capacity to reach 58%. The order was wrong: the same supply push before the campaign would have held match rate near 41% while demand grew.

Failure modes I'd watch for:

  • Measuring globally. A national match rate can sit at 60% while three cities are at 20%. The market clears locally, so the shortage is invisible until you split by geography.
  • Counting only matched requests. Time to match looks healthy if you drop the unmatched from the denominator. Track unmatched requests and their return rate explicitly.
  • Supply quantity without supply quality. Adding capacity that cannot serve the demand segment raises the count and not the match rate. Check match rate within the segment, not just total supply.
  • Subsidising the side that is already long. Discounts on the unconstrained side convert into unmatched demand and churn, which is the campaign above.

When not to apply this: in a cold start both sides are short, and you seed supply first because demand without supply churns and rarely returns. If the real constraint is trust, safety or payment friction rather than volume, adding either side does not move match rate and the priority is the trust mechanism instead.

Growth on the wrong side of a shortage manufactures disappointment.

Curated: · Written: · Reviewed:

Q&A-50How do you argue for accessibility work against revenue features?

The 90-second answer

Don't argue accessibility against revenue. That framing loses, because revenue arrives with a number and accessibility arrives as a value. Argue it as revenue protection and defect cost, in the same currency the roadmap already uses: deals gated on a conformance report, legal exposure, and remediation cost that grows the longer it sits.

Three moves.

  1. Name the standard and the gate. WCAG 2.2 AA is the current W3C recommendation; contracts and procurement questionnaires typically name 2.1 AA or 2.2 AA. The European Accessibility Act applies to in-scope products and services from 28 June 2025. US federal buyers require Section 508 conformance, which points at WCAG 2.0 AA. When a customer's procurement review asks for an ACR, a filled-in VPAT, you either have one or you spend weeks answering follow-ups.
  2. Price the exposure, not the principle. Ask sales how many open deals have an accessibility question in the RFP. That count is the argument; a values argument is not.
  3. Shrink the ask. Don't request an "accessibility project". Request that issues blocking task completion be fixed as defects, and that the standard go into the definition of done so the backlog stops growing.

Worked example (illustrative figures)

An enterprise deal worth $480k stalls in procurement because the buyer wants an ACR. Three workflows block keyboard-only use. The retrofit takes 11 weeks across 3 teams plus a $40k external audit. Compare that to carrying it inline: if the team ships 40 features a year at roughly 2 engineer-weeks each, an 8% accessibility overhead is about 6.4 engineer-weeks spread across the year.

Inline, in definition of doneRetrofit under a deal deadline
Engineering~6.4 engineer-weeks, spread~33 engineer-weeks, concentrated
External auditnone$40k
Dealunaffected$480k at risk

The concentrated version also has a date attached that you don't control.

Failure modes

  • The audit that becomes a deliverable. A 400-page report with 900 untriaged issues is worse than no audit: it is written evidence you knew. Triage by whether the issue blocks task completion, and fix those as defects.
  • The overlay. A JavaScript widget that claims compliance doesn't change the underlying DOM, and it doesn't produce an ACR a procurement team accepts. The FTC fined accessiBe $1M in January 2025 over deceptive claims that its overlay made sites WCAG-compliant.
  • The "next quarter" that never lands. Track blocking issues open per release. If that count isn't falling, the definition-of-done change didn't take, and you're back to arguing values.

What I'd actually say in the room

"Three open enterprise deals have an accessibility question in the RFP and we can't answer it today. Fixing the two workflows that block keyboard-only checkout is about X engineer-weeks. The alternative is answering that question under a deal deadline, which is what the last retrofit cost us."

Then commit to two numbers you'll report next quarter: blocking issues open per workflow, and whether the standard is in the definition of done.

Curated: · Written: · Reviewed:

Q&A-90You are raising prices. How do you handle customers who are already paying?

Core answer

Two things decide this before any comms plan: what the contract permits, and whether the customer is getting anything new.

Pull the MSA and read the price-change clause. Most B2B agreements either fix price for the term, allow an increase with 30–90 days' written notice before renewal, or carry a CPI escalator. If the clause says 60 days and you give 14, you haven't raised prices: you've breached the contract, and the customer can exit mid-term. Verify that per segment first, because enterprise paper and self-serve terms differ.

Second: a price lift on an unchanged offer is a different animal from a repackage. Repackaging (new tier, added usage or seats at a higher price) gives the customer a reason to say yes and gives sales something to sell. A pure lift is take-it-or-leave-it, and should be sequenced as such.

Then the plan: notice ahead of the contractual window, one clear explanation of what changed, a grandfather or phase path for long-tenured accounts, support and sales briefed with exact per-segment numbers, and cohort-level tracking afterwards.

The arithmetic that decides how aggressive to be

Break-even churn for a price increase is increase/(1+increase) of the cohort's revenue:

  • 10% increase → 9.1%
  • 20% increase → 16.7%
  • 50% increase → 33.3%

Worked example, illustrative: 1,000 accounts at $500/mo is $6M ARR. A 20% lift takes that to $7.2M, +$1.2M gross. Each churned account now costs $7,200/yr. Lose 90 accounts (9%) and you give back $648k, netting +$552k. Lose 167 (16.7%) and the gain is gone. That number, not a feeling about churn, is what tells you whether to phase or grandfather. If your largest accounts churn disproportionately, the account-count break-even understates the damage; run it on revenue, not logos.

Grandfathering options

OptionYear-1 revenueChurn riskFits when
Hard cutover at renewalFull increaseHighestLow switching cost, self-serve, price well under market
Time-boxed grandfather (12 months)Increase from month 13ModerateMost B2B; gives customers a decision point
Phase-in (half now, half next renewal)HalfLowerLarge accounts where a step change triggers re-procurement
Indefinite grandfatherNone from that cohortLowestOnly accounts you are deliberately defending

Indefinite grandfathering is the one that quietly costs most: it creates a permanent two-tier price list, and new customers find out.

Failure modes and how you see them

  • Silent application at renewal. Shows up as a spike in cancellation requests and support tickets in the 30 days after renewal dates; in some contracts it is a breach.
  • Sales sandbagging. If comp doesn't change, AEs discount the increase away to protect their number. Compare list increase to realized increase per rep.
  • Inconsistent saves. Save-offer acceptance rate varies by agent: brief support with one approved offer and a floor.
  • Grandfather with no end date. Shows up as a cohort whose ARR never moves.

Track NRR and downgrade rate by cohort against an unaffected cohort for two renewal cycles. Aggregate numbers hide the affected segment, which is the only one that matters here.

Curated: · Written: · Reviewed:

Q&A-74Finance wants a number for next year. How do you produce one you can defend?

The 90-second answer

Clarify three things before modelling: which metric and its exact definition, what decision the number feeds, and how much error that decision tolerates. Then build the number bottom-up from drivers, publish it as a range with named assumptions, and make every component independently measurable so next month's variance decomposes into "the baseline moved" or "initiative X didn't land". A single blended number can't be debugged, and finance will treat it as a commitment whatever you call it.

Building it

Baseline first: roll existing cohorts forward with observed retention and no new work. That is the number you get if nothing ships. Then each initiative as its own line, with its own assumption and a confidence. The total is the sum; the range comes from the low/high on each line, not from a fudge factor applied to the total.

Worked example (hypothetical): 50,000 paying accounts on 1 January, $10 ARPU, 1.5% monthly churn.

baseline = sum over m=1..12 of 50,000 x 0.985^(m-1) x $10
         = 50,000 x (1 - 0.985^12) / 0.015 x $10
         = 50,000 x 0.1659 / 0.015 x $10
         = 552,900 account-months x $10
         = $5.53M

Add price uplift $0.35M and new segment $0.60M: total $6.48M, range $5.9M-$7.1M. Twelve months later the decomposition looks like this.

ComponentForecastActualVariance
Existing cohorts5.535.41-0.12
Price uplift0.350.12-0.23
New segment0.600.15-0.45
Total6.485.68-0.80

The miss is attributable: the base held, the two initiatives did not. That is the whole point of the structure.

What the sensitivity tells you

Churn at 2.0% instead of 1.5% gives 50,000 x (1 - 0.98^12) / 0.02 x $10 = $5.38M, a $0.15M move, about 3%. Over a single year retention moves the number far less than people assume, because the base decays slowly; the new-business assumption dominates. So that is where scrutiny belongs and where I would want the tightest evidence.

Failure modes

  • Blended total: a plan committed to 8.4M on one line landed at 6.1M and the post-mortem could not say whether the baseline had decayed or the three initiatives had underdelivered, so the next plan repeated the structure.
  • Hockey stick from unshipped features: an initiative with no owner and no date is not a forecast line.
  • Definition drift: if "paying account" changes mid-year (trial conversions, annual prepay recognised upfront versus monthly), the variance is an accounting artefact, not a business result.
  • Survivorship in cohort data: averaging retention across cohorts that have already churned out overstates the base.
  • One-off deals: a Q4 enterprise contract in last year's base is not recurring revenue.
  • Anchoring: last year's number plus a growth percentage is a top-down guess wearing a bottom-up costume.

When not to do this

For a product with no cohorts, bottom-up is fiction. Use a range from comparable launches and gate the number on a milestone. For a mature, stable metric a seasonal-naive with trend is cheaper and about as accurate; the driver model earns its keep when you need to explain a miss.

Defending it

Reconcile the bottom-up total against a top-down sanity check, market growth and share, and explain any gap. State the assumptions, give P50 and P90, say which one you would sign up to, and name what would falsify it. Re-forecast monthly and correct the component that moved. The forecast is the process, not the number.

Curated: · Written: · Reviewed:

Q&A-91A feature would work better with more personal data. How do you decide?

"Better" is the claim to pin down first. I'd ask what the feature does today, which metric moves, and how much of the gap the extra data actually closes, because that answer decides the case more than the principle does.

Then I'd run the request through five tests, in this order:

TestQuestionIf it fails
NecessityCan aggregate, on-device, or a coarser field reach the same outcome?The field is convenience, not need
ExpectationWould the person be surprised to learn you hold this?Surprise drives complaints, and complaints become incidents
SensitivityIdentifiers, location, health, biometrics, anything about minors?Bar rises from "state a purpose" to explicit opt-in, documented, reviewed
ReversibilityCan you delete it cleanly from every system that copied it?You've bought a temporary gain with a permanent liability
BeneficiaryDoes this improve the user's own experience, or someone else's targeting?The second needs a much stronger case

More personal data is justified when the outcome genuinely can't be reached without it and the person would expect you to have it. Fraud and abuse detection is the clearest case: per-user history is the signal, and aggregate counts can't distinguish a compromised account from a busy one. Opted-in personalisation is another: recommendations need per-user behaviour, the user can see why, and they can turn it off. Safety features, accessibility, and anything where the user is the direct beneficiary clear the bar more easily than internal analytics.

The cost side is real and usually understated. Every added field widens the breach blast radius, enlarges the erasure and access-request surface, adds a system that must honour deletion, and lengthens the retention pipeline you have to maintain. It also costs the feature: more fields means more consent friction, measurable drop-off at the permission prompt, and slower iteration because each schema change now goes through review. Under GDPR Art. 5 the constraints are purpose limitation, minimisation and storage limitation; under CCPA/CPRA you owe notice at collection and a stated purpose. Neither forbids collecting: they forbid collecting without a reason you can write down.

A failure I've seen: a team logged full page URLs including query strings for analytics, which captured customer record identifiers for 14 months. Cleanup meant purging 3 systems and disclosing to 31 enterprise accounts. Nothing about that collection was malicious; it was just never tested against necessity.

Minimum data against what was collected:

FieldNeeded for the outcomeCollected
page templateyesyes
full URL with querynoyes
account identifieraggregate onlyraw

Before building, name the purpose, the minimum field set, the retention period, and the user-facing explanation, and get the review from whoever is accountable for privacy rather than assuming it. Prefer aggregate or on-device processing where it reaches the same outcome.

I wouldn't call it settled without evidence: sample the collected records and confirm no field falls outside the stated purpose. If one does, either drop it or amend the purpose and tell the people affected.

Collect for a purpose you can state to the person it is about.

Curated: · Written: · Reviewed:

Q&A-92How do you prepare support for a new feature?

A strong answer names the decision first: how much support prep this feature earns, not whether to do prep at all. Two inputs set that: blast radius (how many users hit it, and can they route around it) and reversibility (can a user undo the damage themselves, or does it need a human). State your assumptions out loud: the feature ships behind a flag, the first cohort is opt-in, and support is a shared queue rather than a dedicated team.

Then the mechanics. Walk the flow with a support lead before the flag flips and list the questions and failure states you find. Each failure state gets a macro and an article. Instrument contact reasons as a required field at ticket close, with a taxonomy coarse enough that agents fill it in consistently: five to eight reasons, not thirty.

That is the heavy path. It is not always the right one:

Prep levelWarranted whenWhat you skip
Full enablement: macros, articles, in-app guidance, agent trainingIrreversible actions (billing, data deletion), or >10% of users in week onenothing
Macros and an article, no trainingReversible, opt-in, self-servetraining, in-app copy
Beta cohort with a named support engineerFailure modes are novel and you can't predict themmacros, articles
Internal users only, behind a flagYou don't yet know the failure statesall of the above

The lighter rows aren't laziness. When the failure modes are genuinely unknown, writing macros for states you haven't seen produces docs that are wrong on day one. The heavy path has a real cost too: a training session for 40 agents is roughly 20 hours of queue capacity, and a macro that goes stale after the first config change is worse than no macro.

Failure modes worth naming, with how you'd catch each:

  • Stale macro after a config change. Macro usage drops but AHT doesn't; the macro's last-updated date is older than the last deploy.
  • Taxonomy too coarse. More than 30% of tickets land in one bucket, usually "other."
  • Deflection hiding a bug. Article views are high and contacts are low, but the error rate in logs is unchanged.
  • Support answering from a doc that contradicts the product. Reopen rate climbs on tickets that reference the article.

Here is the case that made me take this seriously. A launch went out with no prepared answers: 380 contacts in the first fortnight at an average handling time of 11 minutes, about 70 support hours, and the top reason was one unclear error message.

ReasonContactsFixed in product
unclear error message180week 3
pricing question140article only
permission confusion60week 5

What settles it: instrument contact reasons for the first two weeks, confirm the top reason has a named owner and a ship date, and check that its share is falling by week three. If it isn't, the prep was aimed at the wrong failure, and the contact reasons, not the launch checklist, are what tell you.

Curated: · Written: · Reviewed:

Q&A-93A new team is taking over an area you own. How do you hand it over?

Core answer (first 90 seconds)

First I'd pin down what's actually moving. "The area" could mean the code, the pager, the roadmap, the vendor contracts, or all four. A handover that transfers the repo but leaves me answering the alerts isn't a handover. So I'd agree the scope in writing before anything else, and set an end date for my involvement.

Then the answer: hand over the decisions and their reasons, not just the artefacts. The code and the backlog are recoverable from the repo; the reasons are not. A team that inherits a service without its decision history re-litigates settled questions and re-introduces failures that were fixed on purpose.

What I'd actually produce

  • A decision record: for each non-obvious choice, what we chose, what we rejected, and the constraint that forced it. One paragraph each, not a design doc.
  • The failure modes and how they're detected: which alerts fire, what the runbook says, and the last two incidents with their root cause.
  • The dependency map: upstream and downstream services, who owns each, and the SLOs we're held to.
  • The customers who care most and why: the accounts whose usage shapes the design.
Handover artefactWhat it carriesFailure it prevents
Repo + backlogCurrent workNothing on its own
Decision recordWhy the design is this wayRe-litigating settled choices
Runbooks + alert listHow failures are detected and handledLearning the system only on the happy path
Dependency mapWho to callBlocked on an unknown owner mid-incident
Shadow on-callOperational judgementFirst page handled alone, at 3am, cold

Worked example of the verification

I don't treat the handover as done when the documents are written. I ask the receiving team to explain three non-obvious decisions and to predict what happens in a scenario. Take this service: it has no per-tenant rate limiter. The decision record says we removed it because the API gateway already enforces one per tenant, and keeping both double-counted bursts and caused a 40-minute outage. If the new team, after reading, says "we should add a rate limiter here," the handover failed: they have the code but not the reason. If they say "the gateway covers it; adding one here would double-count," it worked.

How the transfer runs

Run the first planning cycle jointly: they drive, I sit in. They take the pager while I'm still reachable, so their first incident is a learning event rather than a crisis. I stay available for a defined window, say one quarter, with a stated end date, because an open-ended "ping me anytime" keeps me the real owner and stops them from building their own judgement.

Failure modes of the handover itself

  • Document dump: a wiki page nobody reads. Detected by the verification above.
  • No end date: I stay the de facto owner; the new team never commits to a decision.
  • Pager transfer without shadowing: they meet the system for the first time during an incident.
  • Transferring the happy path only: they know how it works when it works.

Exit criteria

They've handled a page solo, shipped one change end to end, and explained the three decisions correctly. Then I stop.

Curated: · Written: · Reviewed:

Q&A-94You need work from a team that does not report to you and has its own priorities. How do you get it?

You don't persuade a team that has its own roadmap: you buy capacity from it. Their backlog is a queue at fixed capacity, and your request is an insertion that competes with everything already in it. So the work is: find the metric they're measured on, cut the cost of the insertion as far as you can, and make the ask in their planning cadence with a date attached. Escalation is for when the dependency is genuinely shared, not a fallback for when you were ignored.

Assumptions worth stating out loud: the work is real and not something you could do yourself; the other team has capacity you can't command; and there's something you can give back. Engineering time, on-call cover, a dependency they need later. If none of those hold, the honest answer is to build the workaround or drop the request.

What buying capacity looks like in practice:

  1. Find their metric before you write anything. Their OKRs, their on-call rotation, their error budget, their migration deadline. A request that reduces their page volume is a different object from one that adds a feature to their surface.
  2. Cut the cost of the insertion. Do the design, write the ticket with acceptance criteria, provide the test fixtures, offer to own the on-call for the new path. A request that costs a senior engineer three days of design plus implementation is a hard sell; one that costs four hours of review is not.
  3. Ask in their planning cadence, not in a DM. A verbal yes in Slack is not a commitment; a ticket in their sprint with a named owner is.
  4. Write down what you asked for and when. This is what makes escalation a fact rather than a complaint.

Two framings of the same request (figures illustrative):

FramingTheir metric it movesCost to themOutcome
"our launch needs this by the 14th"none3 days of a senior engineerslotted after their migration, delivered 4 weeks late
"this removes 200 pages/quarter from your on-call"page volume4 hours of review, you own the runbookin the next sprint

The second framing isn't nicer, it's cheaper, and it points at something they already report upward.

Failure modes and how you see them:

  • Escalating after two unanswered messages. You get the work eventually and a team that deprioritises your next three requests. Detect it by watching your request latency trend upward across quarters.
  • A verbal yes with no ticket. Detect it at their next planning meeting when your item isn't in the sprint.
  • Framing around your deadline. You get a low-priority slot and a "we'll get to it" that never lands.
  • Asking a team whose metric your request doesn't touch at all. No framing fixes this; you need a different team or a different design.

When escalation is right: if the dependency is a shared platform obligation (a security fix, a schema change both teams need, a contractual deadline) the shared owner should hear about it early, not after you've exhausted the direct route. The test is whether the other team's own goals are served by doing it. If they are, escalation is a scheduling problem. If they aren't, escalation buys compliance and costs the relationship.

What I'd check before calling it done: the other team can state back what they're doing and why it helps their metric, and there's a ticket with a date and an owner. Anything less is a hope.

Curated: · Written: · Reviewed:

Q&A-51How do you keep customer-facing teams informed without creating promises?

What a strong candidate says first

A promise is any statement a customer can hold you to: a date, a capability, or a "yes, we'll have that." The roadmap itself isn't the risk. The unlabeled date is. So I'd publish one internal roadmap where every item carries a confidence band and a matching external script, and make the safe sentence the easiest one for a rep to repeat.

Clarify first: is this about a specific deal, or the standing process? If a customer contract already commits a date, that's a commitment, not a roadmap item, and it doesn't belong in this system.

Mechanism: why promises form

Three forces push an internal "planned" toward an external "committed":

  • Sales comp closes on the deal in front of them, so ambiguity resolves toward optimism.
  • Any artifact that leaves the building is quoted verbatim, stripped of the caveat that was in the presenter's head.
  • Support inherits the promise with no context, so the first ticket is where the mismatch surfaces.

The fix is to remove the ambiguity at the source: each band gets exactly one approved sentence.

BandInternal meaningWhat may be said externallyOwner of a change
committedfunded, staffed, dated"Shipping <date>."PM
plannedfunded, not dated"We're working on <theme>; no date."PM
exploringnot funded"Nothing to share."PM

Trade-offs and when not to use it

Over-restrict and you get a shadow roadmap: reps keep dates in DMs and CRM notes, and you've lost the ability to correct them. Under-restrict and you get the leak. The table only works if the "planned" script is genuinely useful to a rep, if it's "no comment," they'll improvise. For competitive embargoes, hold the item out of the published roadmap entirely rather than marking it exploring; a band implies the thing exists.

Failure modes and detection

  • Band drift: an item moves exploring → committed without a re-brief. Detect by diffing this week's published bands against last week's and flagging upgrades.
  • Caveat stripping: "planned" repeated as a date. Detect by sampling CRM notes and call recordings for roadmap mentions and checking the wording.
  • Unknown blast radius: you can't correct what you can't find. Detect by asking each rep which items they've repeated to customers in the last month: that list is your correction queue.

Worked example (hypothetical)

A 40-item roadmap: 6 committed, 18 planned, 16 exploring. A planned item slips a quarter. If 9 of the 22 accounts that heard the "planned" script had it repeated with a date, the blast radius is 9 accounts, not 40: the band table is what keeps the correction list that small. Without bands, the same slip touches every account that saw the slide.

The one question I'd ask

Ask sales to name the roadmap items they've repeated to customers in the last month. If they can't, the process isn't running.

Curated: · Written: · Reviewed:

Q&A-88How do you keep an executive informed without an escalation every week?

What a strong candidate says first

Ask what decision the exec actually owns for your area (budget, headcount, launch date, vendor choice) and how much lead time they need before making it. Set the cadence to that lead time, not to the calendar. If they make calls monthly, a weekly update is noise they will filter out; if they are making calls weekly, a monthly update is negligence.

The default that works: a short written update on a fixed rhythm, sized to the decisions pending, plus a pre-agreed trigger that escalates without waiting for the rhythm.

Alternatives, and when each wins

  • Written async update, weekly or biweekly. Wins when scope is stable and the exec is in another timezone or building. Leaves a record you can point at later. Loses when the decision needs debate between two functions: a doc documents disagreement, it does not resolve it.
  • Live 15-minute sync. Wins when two execs disagree and you need them in the same room. Expensive, and it is the first thing cancelled when their week goes sideways, so never make it the only channel.
  • Shared dashboard. Wins for numbers that move continuously: burn, error budget, launch readiness. Loses for narrative risk: a risk that is not a metric yet has no tile, so nobody sees it.
  • Escalate immediately, every time. Correct when the decision deadline is shorter than your cadence, when the call crosses a budget, headcount, legal or security line, or when another function is about to make the decision for you.

Cadence is per-executive. One wants five lines by Friday; another wants a page before Monday staff. Ask once, then hold it.

A worked trace

Integration with a payments vendor, blocked on sandbox access. The update line is what the exec read that week.

WeekWhat you knewUpdate lineExec action
1contract signed"integration started, on track"none
3vendor docs changed, sandbox not provisioned"sandbox access pending, no impact yet"none
4sandbox blocked at least 2 weeks; building against a stub costs 3 engineer-days"risk: sandbox blocked. Recommend stub. Default: proceed Friday unless told otherwise"approves stub Thursday
5stub in place"on stub, beta date holds"none

Cost of naming it at week 4: 3 engineer-days. The counterfactual is the same block unnamed until the quarterly review at week 12: recovery then pulls 2 engineers off another team mid-quarter and slips the beta 4 weeks. Three days against four weeks and a re-plan is the whole argument for the cadence.

Failure modes, and how you catch them

  • Green-washing. A status carrying only good news trains the reader to discount all of it. Detect: ask the exec to name your top risk back to you. If they cannot, the update is not landing.
  • Surprise escalation. The risk reaches them from another function first. Detect: log the source of first mention for every risk; any "partner team" entry is a miss.
  • Escalation as a question. "We are blocked, what do you want?" hands the work back. Send a recommendation, its cost, and a default you will take if you hear nothing by a date.
  • Cadence drift. Updates slip exactly when things are bad, which is when they matter most. Detect: compare send dates to the agreed rhythm; two misses in a row means the format is too heavy.

If you are escalating the same issue weekly, the cadence is not the problem: either the decision is not being made or you are not the right owner for it.

The check that tells you whether any of this is working: in the last month, did any risk reach leadership from another function before it reached you?

Curated: · Written: · Reviewed:

Q&A-89Design and product disagree about a flow. How do you resolve it?

Most design-versus-product fights are not about taste. They are two people making different predictions about the same user, and neither has said the prediction out loud. My first move is to get both predictions on the table, because a stated prediction can be tested and a preference cannot.

Assume first that the flow is not blocked by a constraint. If it is (accessibility, a platform convention, a legal requirement, a hard latency budget) there is nothing to resolve. The constraint wins and the argument is over. Testing a constraint burns a week and insults the person who raised it.

For everything else, I ask each side to write down three things: what the user will do, how we will observe it, and what number would change their mind. Design says "users abandon at the address step because they can't tell which address is being validated." Product says "users complete that step; the drop-off is at payment." Those are different claims about different steps, and one five-user session separates them.

Then I pick the cheapest instrument that can actually discriminate between the two claims, and we agree the decision rule before it runs. The rule matters more than the instrument. "If 3 of 5 users hesitate at the address step, we ship design's version" is a decision. "Let's see what the test says" is not.

Rough costs, and what each instrument can and cannot settle:

InstrumentCostSettlesDoes not settle
Five-user moderated test1 dayWhere users hesitate, which label they misreadWhether it moves a conversion metric
Instrumented funnel on existing traffic2–3 daysWhich step loses the most usersWhy they left
A/B test on the flow2–4 weeksWhether the change moves the metricAnything on a flow too low-traffic to reach significance

The failure I have watched: a checkout flow disagreement went to the PM's preference because the PM was senior. It shipped. A usability round two weeks later found 7 of 8 users taking the path design had predicted, and the rework cost three weeks: more than the one-day test that would have settled it. The test is almost never the expensive part.

This process has two failure modes of its own. The first is a test too small to detect the effect you care about: five users will find a broken label, they will not find a 2% conversion difference, so don't run a five-user test on a metric question. The second is the goalposts moving after the result, when someone calls the test unrepresentative. That is why the decision rule is written down first, and why I would rather lose an argument to a written rule than win one by seniority.

If the two positions turn out to be equally supported and the cost of being wrong is small, ship one, instrument it, and revisit in a month. Reversible decisions do not deserve a two-week experiment.

Curated: · Written: · Reviewed:

Q&A-87Two teams are optimising metrics that pull against each other. What do you do?

What I'd say in the first two minutes

Two teams pulling against each other is usually an incentive problem, not a communication problem. Before proposing anything, I'd get both teams' numbers onto one page: what each is optimising, over what window, and what the joint outcome looks like. If growth is up 4% on signups and monetisation is up 2% on conversion while the combined funnel is down, the local metrics are lying, and no amount of joint standups fixes that.

Then I'd work out which of three situations I'm in, because the fix differs:

  • Genuine conflict with a single owner. Escalate to whoever owns both metrics and ask for a decision, not a negotiation between the teams.
  • Coordination failure. The teams aren't actually in conflict; they're measuring different things on the same surface. Fix the measurement, not the org.
  • Structural conflict. Both goals can't be maximised by the same team at the same time. Then you change the structure.

The options, and when each is wrong

Shared outcome metric. Replace both local metrics with one number both teams are graded on: revenue per session rather than signups and conversion separately. Works when the joint outcome is measurable within a quarter and both teams can actually move it. It's the wrong tool when the two metrics are in real tension at the product level: a blended metric hides the tradeoff, and both sides can degrade while the blend stays flat. Latency versus throughput is the classic case: there you want a guardrail, not an average.

Guardrail. Keep each team's local metric, add the other team's as a constraint that can't regress. Growth keeps optimising signups, but conversion can't drop more than an agreed bound. This is the right default when you can't agree on a single number, because it preserves the local signal while bounding the damage. It fails when the guardrail is soft: if it can be waived under deadline pressure, it will be.

Restructure. Give one team ownership of both metrics. Blunt, but it removes the negotiation entirely. You pay for it in lost specialisation and in the parallel throughput of two teams.

Sequence. Freeze one team's surface while the other ships, then swap. Cheap and slow, and it works when the changes are genuinely incompatible on the same screen.

Arbitrate with an experiment. If the teams disagree about which change is better, run both as an A/B test with the joint metric as the decision criterion and let the data pick. This is the fallback when nobody can agree on the metric itself.

Failure modes to watch

A shared metric that neither team can move alone, so both stop trying. A guardrail that gets waived. A blended metric that lets both sides degrade. All three look like alignment on a dashboard and behave like drift.

Illustrative case (numbers are a worked example, not a measured result)

A growth team optimising signups and a monetisation team optimising conversion ran opposing prompts on the same screen for seven weeks.

MeasureGrowth viewMonetisation viewCombined
signups+4%not measured−3%
conversionnot measured+2%−1%

Both teams hit their local target; the joint funnel went backwards. The tell was that neither team's dashboard could see the loss.

Decision rule

If the joint outcome is measurable this quarter and both teams can move it, use the shared metric. If it isn't, use a hard guardrail with a named owner who can refuse the waiver. If the conflict is structural, change who owns what rather than asking the teams to be nicer to each other.

Curated: · Written: · Reviewed:

Q&A-86A senior leader wants a direction you believe is wrong. How do you handle it?

The 90-second answer

Ask one question before you argue: what would have to be true for you to change your mind? If the leader can name evidence, the disagreement becomes an experiment with a decision procedure. If they can't, you're not disagreeing about facts, and another round of argument won't move it: you decide whether this is worth spending capital on.

Then state the specific claim you disagree with, the evidence behind your position, and the cheapest test that would resolve it. Offer to run the test yourself. If the decision goes against you, execute it fully and make sure the measurement you argued for is in place before execution starts, not bolted on after.

Assumptions that matter: the decision is reversible and the cost of being wrong is bounded. If it's a one-way door, or it's legal, safety or ethical, the calculus changes: you escalate, and you say so in the room rather than after.

What that sounds like

"I think moving everyone to the new pricing tier will cost more in churn than it gains in ARPU. The evidence is the cohort data from the last change. The cheapest test is a 5% holdout for two weeks: I can have it running Thursday. If you still want to go all-in, I'll run it, but I want the holdout in place so we know which of us was right."

Four sentences: claim, evidence, test, commitment. The holdout is the instrument.

Three ways to lose the argument, and the one that wins

ResponseWhat it looks likeDecision instrumentedWhat you learn
Quiet non-complianceagree in the room, don't do itnonothing, and trust is gone
Malicious complianceexecute the letter, skip the parts you argued againstnonothing; you look right by sabotage
Escalate past the decisiontake it to their boss after losingnonothing; capital spent
Execute and instrumentfull execution, agreed metric, review dateyeswho was right

Why quiet non-compliance is the worst of the three

It's the only option where you get the blame and no information. A PM I worked with disagreed with a pricing change, implemented it half-heartedly, and skipped the measurement they had argued for. Revenue fell 7% and nobody could tell whether the change or the implementation caused it. The leader concluded the PM had been wrong; the PM concluded the leader had been wrong; the organisation learned nothing and repeated the argument six months later.

When escalation is the right move

Escalation is not how you win an argument you lost. It's how you stop a decision that can't be undone or crosses a line. Two triggers: the decision is irreversible with a large downside (a data deletion, a public commitment, a migration you can't back out of), or it's legal, safety or ethics. In both cases, say it in the room: "I'll take this to X if we proceed" is a statement, not a threat, and it's fair. Escalating after the fact on a reversible decision spends capital you'll need for the one that matters.

Failure modes to watch

  • Agreeing without instrumenting. You comply, the number moves, and nobody can attribute it. The measurement is the whole point of complying well.
  • Winning the argument and losing the room. If you're right often enough that people stop bringing you decisions early, you've traded influence for accuracy.
  • Treating every disagreement as a hill. Escalation has a budget. Spend it on one-way doors, not on reversible calls you could have tested.

Confirm the metric and the review date before execution starts, whichever way the decision went. That's the only version where the organisation learns something either way.

Curated: · Written: · Reviewed:

Q&A-80What belongs in a launch plan besides the code being live?

Assume a customer-visible feature in a B2B SaaS product, priced or bundled, with a support org and a sales team. If it is internal-only, or already behind a flag you can flip off, most of this collapses to a changelog entry and a note to support.

The launch plan is the set of decisions that must be made before the date, each with an owner and a check that proves it is done. Beyond the deploy: the pricing and packaging decision, the support macro and help article, the docs page, sales and CS enablement, customer comms (in-app, email, changelog), a kill switch, and a success metric with a threshold and a review date. The deploy is the cheapest item on that list.

Two shapes for the plan, and the choice is the interesting part. A hard gate: every function signs off before the date, and the launch slips if any row is red. A soft launch: ship to a cohort or behind a flag, no gate, watch the metric, expand or roll back. Pick the gate when the change is irreversible or touches money (pricing, a data migration, anything you cannot turn off in one toggle) because a slip costs days and a bad pricing launch costs a quarter. Pick the soft launch when the change is reversible and the blast radius is small; a gate there burns calendar for a decision the data would make for you. The gate is the wrong tool when the flag is the rollback: if one toggle undoes it, the sign-off meeting is theatre.

Failure mode: pricing undecided at launch. 12,000 accounts, 3% ask support in week one whether they will be charged: 360 tickets at roughly 6 minutes each is 36 agent-hours, and if the decision slips nine days the queue keeps arriving at about 40 tickets a day while support improvises an answer that may contradict what finance later decides. Detect it before the date: the pricing readiness check is a recorded decision, not a meeting invite. The same shape applies to docs: tag tickets by feature so a spike is visible in the first 48 hours rather than at the retro.

The success metric needs a number and a date. "Activation of the new flow above 25% by day 14, else we roll back" is a launch plan; "monitor adoption" is not.

Readiness at go time: owner, artefact, evidence:

FunctionOwnerEvidence it is ready
supportsupport leadmacro published, article live
pricingPM + financedecision recorded in the pricing doc
salessales enablementsession delivered, deck in the enablement repo
docstech writerpage published, linked from the feature

A red row is a no-go for that function, not a note in the launch channel.

Curated: · Written: · Reviewed:

Q&A-81How do you roll out a risky change?

In the first two minutes: a staged rollout is a controlled experiment, not a slow deploy. Before stage 1 I fix four things: the metric that decides, the threshold that halts, the stage sizes, and the segment each stage must contain. If I can't name the metric and the threshold, I don't have a rollout: I have a deploy with extra steps.

The stop condition is agreed before stage 1, and it's a gate, not a dashboard:

def halt(canary, baseline, min_n=2000):
    # (errors, requests) over the stage window
    if canary[1] < min_n:
        return "hold"          # too little volume to decide
    c = canary[0] / canary[1]
    b = baseline[0] / baseline[1]
    return "halt" if c > b * 2 and c - b > 0.005 else "advance"

Traced: halt((30, 3000), (10, 3000)) → c = 0.010, b = 0.0033, both conditions true → "halt". halt((4, 3000), (10, 3000)) → c = 0.0013, not above 2b → "advance". halt((5, 500), (10, 3000)) → 500 < 2000 → "hold". The hold branch matters: a stage that hasn't seen enough traffic cannot clear itself.

Stage composition is the part people get wrong. Population here is 20,000 accounts, 12% of them large (>10,000 records).

StageAccountsLarge accountsHeld
1%2000: hand-picked low-risk4 h
50%10,0001,2000 h
revised 5%1,00012048 h

Credible alternatives, and when a staged rollout is the wrong tool.

  • Feature flag. Same code path, per-tenant targeting, off in seconds without a deploy. It's a kill switch, not evidence: it tells you nothing about whether the change is correct. Use it as the rollback mechanism inside a rollout, not instead of one.
  • Blue/green. Two full fleets, cut over at once, roll back by flipping the router. Clean for stateless services, but it gives no partial exposure: a bug that hits 12% of accounts hits all of them the moment you flip. Wrong tool when the failure is segment-specific.
  • Canary analysis. Automates the gate above: compare canary against baseline statistically and promote or halt without a human. Needs enough traffic to reach significance in minutes; at low volume you're back to holding stages open.
  • Shadow traffic. Mirror production requests to the new code, discard responses. Catches crashes, latency regressions and 500s with zero user impact. Doesn't validate writes or anything where the side effect is the point.

Staged rollout is the wrong tool for irreversible changes: a schema migration that can't be dual-written, a wire-protocol change where mixed versions break, a one-shot backfill. There, partial exposure is worse than none: you want expand/contract migrations, version negotiation, or a maintenance window with a tested restore.

Failure mode: segment-blind staging. A change went 1% → 50% in a day because stage 1 looked clean. The error only fired for accounts with >10,000 records: 0% of stage 1 (hand-picked small tenants), 12% of stage 2. The fleet average was fine; the segment wasn't. Detect it by alerting on per-segment error rate rather than the average, and by requiring every stage to contain the segment the risk lives in.

I'd call it done when the last stage has run past the effect's detection window and the rollback path has been exercised, not just documented.

Curated: · Written: · Reviewed:

Q&A-82A launch is causing problems but not an outage. How do you decide whether to roll back?

Not an outage means the question is not "is it broken?" but "is the damage rate higher than the cost of reverting?" The answer depends on three things: whether rollback is safe, whether the damage is still growing, and whether a cheaper control exists.

Assume the release is behind a deploy artifact and no destructive migration has run. If a destructive migration has run, full rollback is usually the wrong call. Agree the threshold before launch in customer terms: failed checkouts per hour, wrong-price orders per hour, p99 latency. During the incident, measure against it. If the threshold is exceeded and rollback is safe, roll back. If it is not exceeded, fix forward with a time box and a named owner.

The alternatives matter because rollback is not the only control:

OptionWhen it winsWhen it loses
Full rollbackDamage is broad, the artifact is self-contained, rollback is testedSchema migration is irreversible; rollback discards data
Feature-flag flipNew path is behind a flag; no state written yetFlag does not cover all paths; state already persisted
Partial traffic rollbackDamage is load-dependent or confined to a subset of usersMixed versions write incompatible data
Fix forwardRollback is unsafe or the fix is minutes awayDamage continues; fix estimate is optimistic

The common failure is debating the threshold live. A pricing display defect showed the wrong currency. The team debated for 6 hours while the damage accumulated:

HourAffected sessionsOrders at wrong priceCumulative honoured pricing
148041,100
31,700195,225
63,2004011,000

At hour 6, 3,200 sessions saw the wrong currency and 40 orders were placed at the displayed amount, costing 11,000 in honoured pricing. The cost of continuing was visible. The cost of rollback was a redeploy and a re-announcement. The team had no pre-agreed threshold, so the decision became a debate.

Rolling back is the wrong call when the release has already made an irreversible schema change. If the release dropped a column or backfilled data, rolling back application code can leave old code reading a schema it does not understand. In that case, fix forward or roll forward with a compatibility layer. Rollback is also wrong when the defect is in a shared dependency or upstream provider: reverting your release will not change it. And it is wrong when rollback would discard more user data than the defect is creating.

Detection is the same as the threshold: error budget burn rate, failed operations per hour, revenue per minute, wrong-price orders per hour. If you cannot measure the damage rate, you cannot compare it to the cost of reverting.

The decision rule: if damage rate exceeds the pre-agreed threshold and rollback is safe, roll back. If rollback is unsafe, fix forward with a named owner and a time box. If damage is confined to a flag, flip it. The threshold is agreed before launch, not during.

Curated: · Written: · Reviewed:

Q&A-83A committed date will be missed. What do you tell the customer?

The 90-second answer

Tell them four things in this order: the date is moving, the new date, what changed, and what is unaffected. Send it the day you know, not the day you're sure. A slip announced five weeks out is a planning problem the customer can absorb; the same slip announced on the original date is an incident in their calendar, and you have spent their slack for them.

Own the date without a blame narrative. "The vendor's connector isn't certified until late March" is a fact; "the vendor let us down" invites the customer to wonder who else on your side is unreliable. Give a date you would bet on, not the earliest date that is defensible in a status meeting: a second slip costs more trust than a conservative first one.

The message

Subject: <Project> go-live moves from 14 Mar to 4 Apr — what changes for you

Status: 14 Mar is no longer achievable. New committed date: 4 Apr.
What changed: <one factual sentence, no blame, no internal names>
What this means for you: <the two or three things they had scheduled>
What is unaffected: <already delivered / already frozen interfaces>
What we are doing: <the specific change, e.g. a second migration rehearsal>
Next update: 21 Mar, whether or not anything changes.

The "next update whether or not anything changes" line is the one people drop, and it is the one that stops the customer from chasing you.

IncludeOmit
New date and how confident you are in it"We're working hard to get back on track"
One sentence on what changedBlame, internal team names, ticket IDs
What is unaffected"Should be fine" hedges
A fixed next-update dateA date you would not bet on

Worked example (hypothetical figures)

A team is migrating a customer's nightly batch to a managed pipeline, committed cutover 14 Mar. On 7 Feb they learn the connector for the customer's ERP version will not be certified until late March. Slip to 4 Apr: 21 days.

The useful part is not the date, it is the downstream arithmetic. The customer has six reports built on the old batch window; two feed a monthly close on the third business day. 4 Apr lands after the March close, so March runs on the old system: fine. But the April close is the first on the new pipeline and falls three days after cutover. So the message says: "Your April close is the first on the new pipeline, three days after cutover; we will run both in parallel for that close." That sentence is what decides whether the slip is a nuisance or a renegotiation.

Failure modes and how you catch them

  • Rolling slip. The new date moves again within two weeks. You shipped a date you had not de-risked. Track the interval between successive dates; if the second one moves, escalate internally before you tell the customer a third time.
  • Holding pattern. "We'll know more Friday" twice in a row with no date change. The customer stops planning and starts escalating. Count updates that carry no date change; two is the limit.
  • Backchannel. The customer's exec hears it from your exec first. Ask the account owner who else is talking to them before you hit send.
  • Contractual clock. Many MSAs start credit or termination-for-convenience windows at notice, not at the missed date. Read the MSA before you send, not after.
  • Partial delivery you cannot staff. Offering half the migration often leaves the customer running two systems, which is worse than the slip. Only offer it if you can name the person who supports both.

What done looks like

Every affected customer has the new date in writing from you before the old date arrives, and has told you which of their own commitments move. If you cannot name those commitments, you have sent an announcement, not a notification.

Curated: · Written: · Reviewed:

Q&A-84How do you review a launch that went badly without blaming a person?

The 90-second answer

Default to blameless, but treat it as a default with an exception, not a rule. Blameless review exists because most launch failures are produced by the system (an ambiguous runbook, a deploy with no canary, a rollback nobody had rehearsed) and naming a person ends the inquiry exactly where the fixable defects begin. So the output of the review is a small set of changes with owners and dates, checked at the next launch, not a verdict.

The exception matters as much as the rule. Blameless is the wrong frame when the behavior was reckless or deliberate rather than a mistake. The just-culture framework (Marx) draws that line in three categories: human error gets consoled and the system fixed; at-risk behavior, knowingly cutting a corner because the process is slow, gets coached and the process fixed; reckless behavior, like shipping a known data-loss bug to hit a date or disabling the audit log, goes to a separate accountability track through security, legal, or the manager. That track does not run inside the postmortem document.

Two consequences follow. Blameless is not nameless: the timeline names who did what and when, because "an engineer deployed at 14:02" is useless for reconstructing a decision. What you leave out is the moral judgment attached to the name. And the postmortem is not the performance-management tool: repeated recklessness is a management conversation with its own evidence, not a line in a doc the whole org reads.

Running the review

Build the timeline from records (deploy logs, CI runs, the incident channel, dashboards) not from memory. At each decision point, record what the person knew at that moment, not what we know now. That is the whole trick: the question is "what made this look like the right call at 14:02," and the answer is usually a missing signal, not a missing person.

Then produce three to five changes, each with an owner and a date. A review with twelve action items and no owners changed nothing.

Where blameless review fails

  • It becomes theater. If the same class of failure recurs and no action item ever closes, engineers stop writing honest timelines and start writing defensible ones.
  • It launders accountability. "The system encouraged it" is a real explanation for an ambiguous runbook and a fake one for a knowingly unsafe change. An org that cannot tell those apart ends up using the blameless norm as cover.
  • It is the wrong venue for security incidents, insider threat, harassment, or fraud. Those need chain of custody and confidentiality. Run the systems review in parallel if you want, but do not publish it as the accountability record.

The failure mode I would watch for

A review that concluded one engineer missed a step, produced no process change, and saw the same failure at the next two launches because the step was in no checklist either time. Composite example, not measured data:

Review outputAction items closedRecurrence in next 2 launches
named an individual02
named contributing factors30

How I would know it worked

Check at the next launch whether the changes from the last review are actually in place: the canary exists, the rollback was rehearsed, the checklist item is in the template. A review that names a person leaves the cause in place. A review that names a person and is the only response leaves the org unable to act on the next reckless change, because it never built the separate track for it.

Curated: · Written: · Reviewed:

Q&A-85Your largest customer escalates to the executive team. What do you do?

An executive escalation buys the customer a fast, factual answer about their own problem. It does not buy them a roadmap slot. I run those as two separate tracks from day one, with two different owners.

On the customer track I take ownership as the single named engineer, get the actual reproduction and the account's business impact from the account team, and send the exec sponsor a dated plan within 24 hours: what I will fix, what I will not fix, and when they hear from me next. "Your nightly export works by Friday; the custom fields you asked for are a separate decision" ends the escalation faster than any promise to look into the roadmap.

On the product track, product answers whether the ask belongs on the roadmap for everyone, with evidence rather than volume: how many accounts have asked for the same thing, what exactly they asked for, and what shipping it displaces.

The split matters because conflating the two turns escalation into a procurement channel: the loudest account sets the roadmap, and the customer learns that escalating beats requesting. Other customers pay for that. Here is a composite example from a company I worked with:

Capability builtAccounts requestingAccounts using after two quarters
Custom export schema11
Bespoke approval flow11
Audit trail (broadly requested)47not built

Three quarters of bespoke work at 31 percent of one team's capacity is 0.93 team-quarters, essentially a full quarter of that team's output, spent on two capabilities with one user each while 47 accounts waited on the audit trail. The escalation was answered; the roadmap was hijacked.

Failure modes I watch for, and how I catch them:

  • Silent engineering. The execs hear nothing for a week and assume the fix stalled. I hold a fixed written cadence: an update every 48 hours, even when the update is "no change, still on track for Friday".
  • Dates promised to end the meeting. I agree the date with the account's own support engineer before it reaches the exec sponsor, so the date I give is one I can defend in writing.
  • Roadmap capture wearing a support costume. Signals: the fix only works behind a flag the account turns on, the request appears in exactly one account's tickets, or the "workaround" is a permanent bespoke branch. Any of those routes to product, not to the sprint.
  • A genuinely broad need dismissed as capture. If the same ask shows up in dozens of accounts, the escalation is legitimate prioritisation input. Pull the general capability forward and give a date; a bespoke bridge would be the wrong fix.

The one case where I do not fully separate the tracks is when the account's need is a general capability already near the top of the roadmap. Then the fastest honest answer is to pull it forward, give the date, and say so.

The exec follow-up names both dates, the one for the customer's problem and the one for the product decision, so nobody has to escalate a second time to find out what happened to the roadmap question.

Curated: · Written: · Reviewed:

Q&A-76Your flagship bet did not work. How do you report it?

The 90-second version. I report a failed flagship bet as a decision, not a confession: here is the hypothesis we pre-registered, here is the result against the threshold we wrote down before we started, here is what I recommend we do, and here is what it costs to keep going. I name the person who owns the next decision and the date they need it by. Then I write it down where the next person scoping something similar will find it.

The report itself. Four lines, in this order:

  1. Hypothesis as written before the bet, with the threshold and the date it was registered.
  2. Result against that threshold, with the measurement window and the sample size.
  3. Recommendation (stop, wind down in stages, or pivot) and the decision owner.
  4. What I would have to believe to continue, stated as a falsifiable claim with a date.

Line 4 is the one people skip. It is not a hedge; it converts the argument from "is this team good" into "is this claim true", which is a question the room can actually settle.

Alternatives, and when each is right.

  • Hard stop. Right when the threshold was pre-registered, the metric directly measures the hypothesis, and the remaining work has no value if the hypothesis is false. Cheapest in engineer-weeks, most expensive in morale and in the option you destroy.
  • Staged wind-down. Right when the bet has a working component other teams now depend on. Stop new investment, keep a maintenance budget, set a date to revisit. Costs a fraction of a team and keeps the asset alive.
  • Partial pivot / interim report. Right when the result is genuinely ambiguous: the headline metric missed but a segment moved. This is also the most dangerous option, because it is what a team in denial reaches for. The test: the pivot must be a new pre-registered hypothesis with its own threshold and its own kill date, not a continuation of the old one.

When the pre-registered-threshold approach is wrong. If the threshold measured a proxy that has since stopped tracking the hypothesis (a signup metric after the funnel changed, say) reporting against it is theatre. Say the threshold is invalid, say why, and re-derive one before you report a number. And if the bet is a platform capability whose value is optionality rather than a metric, a threshold is the wrong instrument; report the option value and the cost of holding it instead.

Failure modes, and how you'd catch them.

  • Threshold set loose enough to survive. Detect: the threshold was met and nobody changed behaviour.
  • Metric moves, user behaviour doesn't. Detect: instrument the outcome, not the proxy.
  • Report lands after the next planning cycle. Detect: the decision owner's calendar.

Worked example (illustrative figures). A team ran a flagship bet for three quarters with 4 engineers. Each quarterly review compared the same metric against the same pre-registered threshold.

ReviewResult vs thresholdDecision taken
Q131% of targetcontinue
Q229% of targetcontinue
Q330% of targetstop

Three quarters at 4 engineers is 156 engineer-weeks; the last quarter alone was 52. The data at Q3 was the data at Q1. The delay cost 104 engineer-weeks, two quarters of a team, and bought no new information.

The line I'd want remembered: the expensive part of a failed bet is the quarter you spend not admitting it.

Curated: · Written: · Reviewed:

Q&A-77What does a good specification contain, and what should it leave to the team?

The 90-second answer

A spec fixes the problem, the observable behaviour, the rules and the edge cases, and the acceptance criteria. It leaves data structures, libraries, module boundaries, sequencing and internal naming to the team. The test for which side a sentence falls on: can a user or another system observe it, or does a business rule decide it? Then it belongs in the spec. Is it a choice between two equivalent implementations? Then it does not.

Assumptions worth saying out loud: the team owns the code end to end, the spec is read by engineers and by whoever owns the business rule, and it will be revised.

What goes in:

  • The outcome and the metric that says it worked, plus explicit non-goals.
  • Rules, numbered, with precedence stated for when two of them conflict.
  • Edge cases: each input missing, malformed, duplicated, out of order, oversized; permission denied; partial failure; concurrent edits.
  • Acceptance criteria as input to observable output, not adjectives.
  • Contracts other teams depend on: API shape, event schema, error codes.
  • Non-functional constraints as budgets: p99 latency, retention, audit trail.
  • Open questions, each with an owner and a date.

What stays out: schema, library choice, queue versus cron, module layout, work breakdown, test framework, internal naming.

Artefact: the rule table

Import contacts
Outcome: a user's CSV contacts are searchable in the CRM within 60s of upload.
Non-goals: dedupe against existing CRM records (v2).

R1  Row with a valid email            -> imported.
R2  Row with missing/invalid email    -> rejected, reported with line number.
R3  Duplicate email within one file   -> first row wins, later rows rejected as duplicates.
R4  More than 20% of rows rejected    -> whole file rejected, nothing imported.
R5  Caller without contacts:write     -> 403, nothing imported.

Acceptance
3 valid rows          -> 3 contacts, 0 errors
3 rows, 1 bad email   -> 2 contacts, error [line 2]
25 rows, 6 bad emails -> 0 contacts, file rejected (24% > 20%)

Trace: precedence

A 25-row file where row 4 duplicates row 2 and row 9 has no email. R2 fires on row 9, R3 on row 4, so 2 rejections out of 25 is 8%, under the 20% threshold, so R4 does not fire: 23 contacts imported, errors reported for lines 4 and 9. Change the file to 6 bad rows and R4 fires instead, and the answer is 0 contacts. That second case is the one the spec exists to settle, because without R4 the API, the web client and the CLI each pick their own behaviour and you find out after launch.

Alternatives

  • Full PRD with screens and flows. Right when a contract or a regulator fixes the behaviour; wrong when the domain is still moving, because you pay to rewrite it.
  • One-pager spec plus a separate design doc. The spec holds rules and acceptance; the design doc holds the implementation argument. Best default for a team that already has engineering judgement.
  • Spec by example (example mapping, Gherkin). The rule table is the spec and the tests come out of it. Strong for rule-heavy domains like pricing or permissions; weak for exploratory work.
  • Ticket plus a conversation. Fine for a small, reversible change; the cost shows up when a second client implements the same rule.

Failure modes and how you catch them

  • Screens specified, rules not. Symptom: two clients and the API disagree on partial failure. Catch it with a contract test per client against the same rule table.
  • Solution specified. Symptom: the spec needs rewriting when a library changes, and engineers stop reading it. Catch it by asking whether a sentence would survive swapping the datastore.
  • No non-goals. Symptom: scope grows in review. Catch it by diffing what shipped against the non-goals list.
  • Untestable acceptance criteria. Symptom: "fast", "intuitive". Catch it by trying to write the test; if you cannot, rewrite the criterion.
  • Stale rules. Symptom: a rule nobody can name the owner of. Catch it by dating and owning every rule.

When not to write one

If the change is reversible in an afternoon and one person owns it, a spec is overhead. If the domain is genuinely unknown, write the outcome and the constraints, timebox a spike, and write the rules once you have seen the data.

Own the rules and the outcome; rent the implementation.

Curated: · Written: · Reviewed:

Q&A-19Describe a case where the right product decision was to build nothing.

Here's a composite case I use. A 300-person B2B SaaS company with 200 enterprise customers. Support asked for a bulk CSV import for user provisioning. They were manually creating users one by one in the admin UI: 15 minutes per user, so a 50-user onboarding took 12.5 hours. They wanted a CSV upload feature.

We decided not to build it. Instead we built a 3-day internal script that called the existing admin API, and we documented a white-glove process for the three customers who actually onboarded more than 20 users at once. The data: in the past year, only 3 of 200 customers onboarded more than 20 users. Median onboarding was 5 users. The feature would have been used by <2% of customers monthly.

The tradeoff was clear. Building a CSV import meant file parsing, validation, error reporting, idempotency, audit logging, and a security review for PII. Estimate: 6-8 weeks of one engineer plus design and QA, and ongoing maintenance for encoding and partial failures. Not building meant 3 days of script plus some support time. The risk of not building: support frustration and churn if a large customer demanded it. We tracked support tickets tagged 'bulk import' and churn reasons. If tickets exceeded 10 per month or a top-10 customer threatened churn, we would revisit.

When would building have been right? If bulk import usage grew to >20% of the customer base, or if a self-serve tier needed it to reduce onboarding friction, or if the internal script became a security risk (support handling PII in spreadsheets). Also if the admin API didn't exist, we'd have to build.

OptionEffortOngoing costCoverageRisk
Build CSV import UI6-8 weeks eng + design + QAEdge-case maintenance, security reviewAll customersOver-engineering for 3 customers
Internal script + API3 daysSupport runs it, no UI3 large customersScript breaks if API changes
Do nothing0Manual entryNoneSupport burnout, churn

We chose the script. The decision was recorded in a one-page doc with the revisit condition. We told the requester directly. Six months later, no top-10 customer requested it, tickets stayed at 2/month. The script saved 12 hours per large onboarding. We avoided 6-8 weeks of engineering. The right product decision was to build nothing customer-facing, and instead build a thin internal tool.

Curated: · Written: · Reviewed:

Q&A-22An interviewer asks how you would improve a product you do not work on. How do you structure the answer?

The first 90 seconds

"Which product, and what's the goal: growth, engagement, or revenue? I'll assume I have no internal data, so I'll reason from public signals and flag what I'd check on a dashboard if I had one. I'd pick [product], focus on [segment], and move [metric]."

Then say the skeleton out loud before any detail, so the interviewer can follow the path:

  1. Segment, who, and why them (size × pain × how reachable they are).
  2. Goal: one metric, with a direction and a rough baseline.
  3. Problems: two or three, each tied to evidence.
  4. Pick one: with the reason (where the funnel leaks most, or impact vs. effort).
  5. Solution: two options, the trade-off between them, and which you'd ship.
  6. Success: the metric that moves, plus a guardrail.
  7. Risks: what would make you wrong.

The "you don't work on it" part is an evidence problem, not a creativity problem. Name your sources (app store reviews, your own usage, a competitor teardown, public pricing) and say what you'd validate internally. That's the difference between a guess and a hypothesis.

Worked trace (figures hypothetical)

Product: Strava. Segment: runners who log 1–2 activities in their first 14 days and stop. Call it 40% of signups. Goal: activation, 3+ activities in 14 days, baseline 35% → target 45%.

Problems, each with evidence: reviews say "nothing to do after the first run"; there's no social graph until you follow someone; clubs and segments are buried three taps deep.

Pick: the empty social graph. It compounds: every later feature depends on it.

Two solutions. A) Onboarding suggests three local clubs and three nearby segments at signup. B) Auto "you vs. your past self" comparison on every activity, no social needed. Trade-off: A builds network effects but needs local density, so it's thin in small towns; B works solo everywhere but creates no retention moat. Ship B first (cheaper, works everywhere) and layer A where density exists.

Metric: share of new users with 3+ activities in 14 days. Guardrail: notification opt-out rate and club-invite spam reports.

Risk: if the drop-off is actually crashes on older Android, none of this matters. Check crash-free sessions before proposing anything.

Failure modes, and how you'd catch them

  • Feature-first. Opens with "I'd add X." Catch: no segment named in the first minute.
  • No metric. Ends without a number. Catch: you can't say what "worked" means.
  • Boiling the ocean. "Improve the whole app." Catch: no single chosen problem.
  • Ignoring the constraint. Proposes work that needs internal data or a team you don't have.
  • No trade-off. One solution presented as obviously right.

Time budget for a 25-minute slot

SectionWhat it must containTime
Clarify + segmentproduct, goal, evidence base, who3 min
Problems2–3, each with evidence4 min
Pick onethe reason2 min
Solution + trade-offtwo options, why one7 min
Metric + guardrailnumber and counter-number4 min
Riskswhat makes you wrong3 min

That's 23 minutes, leaving two for their questions. The structure is the signal; the feature is the illustration.

Curated: · Written: · Reviewed:

Q&A-101A senior stakeholder insists on a roadmap item you believe is wrong. Walk me through how you handle the disagreement: what you ask, what data you bring, when you escalate, and when you just commit.

Three questions before I argue anything. What outcome does this item buy you, which number, by when? What would have to be true for it to work? And if we're wrong, what does it cost to undo: one-way door, or can we reverse it in a sprint?

Then I don't debate in the room. I say: give me a week and I'll come back with the strongest version of your case and what it displaces. That reframe matters. I'm not the person blocking the work; I'm the person pricing it.

1. Write the claim so it can be wrong. "Enterprise deals stall in security review" becomes "of the 14 deals over $50k that stalled last quarter, how many named SSO as the blocker?" If the claim can't be written that way, the disagreement is about values or about who owns the outcome, and no amount of data settles it: that's a commit or an escalation, not a research project. Naming that early saves two weeks.

2. Bring the cheapest evidence that exists. Not a survey. Closed-lost reasons in the CRM, the two churned accounts, support tickets, win/loss call notes. If nothing exists, propose the smallest test that produces a signal inside two weeks: five customer calls with the stakeholder on the line, a concierge version, a pricing page test. I'd rather spend five calls than six engineer-weeks.

3. Price the alternative by name. The item isn't free; it's the thing we don't do. "SSO is six engineer-weeks and it displaces the onboarding rewrite the retention target depends on." A cost with a name attached is arguable. "We don't have capacity" is not.

4. Decide one of three ways, and say which out loud.

SituationMove
Reversible, cheap, stakeholder owns the outcomeCommit, log the assumption, set a review date
Facts contested, test is cheapRun the test; agree in advance what result changes your mind
One-way door, or the cost lands on a team that can't absorb itEscalate with both cases written

Escalation is for irreversibility and cross-team cost, not for preference. And I escalate with the stakeholder, not about them: we walk into the VP together with a one-page memo. The decision, the two options, the cost of each, what we'd need to see to reverse. Escalating alone turns it into a loyalty contest and I lose the next three arguments.

Worked example (illustrative figures). A VP of Sales wants SSO for enterprise. I pull the CRM: 14 stalled deals over $50k, 3 name SSO in the closed-lost field, 11 name procurement or pricing. Cost: six engineer-weeks, displacing the onboarding rewrite. I bring that, plus a counter-proposal: SSO scoped to the two accounts that actually asked, two weeks, behind a flag. He takes the counter. If he hadn't, I'd have committed with a tripwire: "if SSO-attached accounts don't show a security-review pass rate above 60% by week six, we stop and reallocate." Written down before we start, not after.

Two failure modes I watch for in myself. Escalating on a reversible decision: that spends the capital I need for the one-way doors. And committing without a tripwire: the item ships, the number doesn't move, and nobody notices for two quarters because nobody wrote down what it was supposed to do. Losing an argument is fine. Losing it silently is how a roadmap fills with work no one can defend.

Curated: · Written: · Reviewed: —

Q&A-102You shipped a feature last month. How do you decide whether it succeeded: what metric, what does failure look like, and what do you do next either way?

The metric was chosen before we shipped, not after. If it wasn't, that's the first finding: a post-hoc metric is a story, not a measurement. So I start by restating the pre-registered criterion: one primary metric tied to the job the feature was hired for, one guardrail, and a decision rule with thresholds the team agreed before launch.

Worked example. Feature: saved searches in a B2B analytics tool, shipped to 12,000 weekly active users. Primary metric: share of WAU who create a saved search and return to it in a later session within 7 days. Target: 8%. Guardrail: p95 dashboard load time and support tickets tagged "search". Decision rule: ≥8% with guardrail flat → double down; 3–8% → iterate once with a named hypothesis; <3% → kill.

One month in, adoption is 3.1%. That's ~372 users; the 95% CI is roughly ±0.3pp, so this is not noise. It's a miss against the 8% target and it sits in the kill band.

Before I call it, three checks. Instrumentation: does the event fire on save, or on page render? A 3% number that is actually a logging bug is the most common false failure. Novelty decay: plot weekly cohorts. If week-1 adopters fell from 5% to 1% by week 4, the feature has a retention problem, not an awareness problem: different fix. Segment masking: aggregate 3% can hide 40% adoption in one segment. If the power-analyst segment is at 40% and everyone else is at 0.5%, that is not a kill, it is a positioning decision.

Failure takes four shapes, and they need different responses. Never adopted, low create rate, means the problem wasn't real or the entry point is invisible: kill or reposition. Adopted then abandoned (create high, return low) means it doesn't survive contact with the workflow: iterate on the return loop. Adopted and retained but guardrail damaged means success at a cost: fix the cost or roll back. Adopted by a segment you didn't target is not failure, but the roadmap assumed wrong: re-plan.

If it's a kill: sunset it, email the 372 users who used it, remove the entry point so it stops costing attention, and write down what the pre-launch evidence got wrong. The kill is only useful if the next discovery cycle inherits the lesson.

If it's iterate: one more cycle, one hypothesis, time-boxed to a quarter, same metric, raised bar. "Iterate" without a named hypothesis is how features live forever.

If it's double down: the constraint is now distribution, not the feature. Invest in onboarding and surfacing, and set the next target.

The line I'd say out loud: the decision rule has to be written before launch, because a month in, everyone has a reason the number is wrong. Pre-registration is what makes the kill possible.

Curated: · Written: · Reviewed: —

Q&A-104Pick a product you use regularly. For one specific type of user, what is broken or missing, what would you change first, and why?

I'd pick Notion, which my team uses as its internal wiki, and the user I'd focus on is the new hire in their first two weeks. The change I'd make first: give every page an owner and a review date, and surface both in search results. The wiki's failure isn't the editor: it's that a new hire can't tell which of three runbooks is current, so they ask in Slack instead.

Why that segment. Notion's users split roughly three ways: individual note-takers, small teams running projects, and companies using it as a wiki. The first two are well served; the editor is the product. The third is where the per-seat revenue is and where the product is weakest, because a wiki's job is retrieval and Notion's retrieval assumes you already know what you're looking for.

The pain, and how I'd confirm it. A new hire searches "how do we deploy". Results rank mostly on title match and recency, so they get the 2023 runbook, the 2024 runbook, and a meeting note mentioning both. Nothing says which is current or who owns it. Before building anything I'd sit with five hires from the last two quarters and have them screen-share the last three things they tried to find, plus what they did when search failed. If most of them hit Slack within two searches, the pain is real and the workaround is countable.

Candidate fixes.

FixEffectCostWhy not first
Embedding-based search rankingRight page surfaces more oftenHigh: infra plus an eval setDoesn't fix a stale page that ranks well
Owner + review date, staleness badgeMakes currency visibleLow: one property, one badgeDoesn't help if the right page never surfaces
Auto-generated "start here" index per teamCurated entry pointMedium: needs upkeepRots without owners, like everything else

Why ownership first. It's the cheapest, it attacks the root cause (nobody owns a page, so nobody updates it) and it makes the other two cheaper: ranking can down-weight stale pages, and an index can be generated from owned, current ones. Concretely: every page in a teamspace gets an owner (default the creator) and a review interval; pages past their interval show a badge and sort lower. Two weeks of work, no new infrastructure.

The trade-off I'm accepting is that this is a process change, and process changes get ignored. So the badge has to appear where the decision happens, in the result list and at the top of the page, not in a properties panel nobody opens.

Measurement. Primary: share of new-hire searches ending in a page view within 30 seconds, and time-to-first-useful-page. Baseline from those interviews; a plausible target is 12 minutes down to 4. Guardrail: pages edited per week per team, if that spikes, I've built review theatre, and I'd make the interval advisory or drop it.

What would change my mind. If the interviews show new hires find pages fine and the real gap is that the pages don't exist, the fix is templates and an onboarding checklist, not ownership.

Curated: · Written: · Reviewed: —

Q&A-95Tell me about a product decision you got wrong.

The answer

The story below is a worked example: swap in your own decision and your own figures. The structure and the reasoning are what's being scored.

"I shipped a change to our free tier's export limit, from unlimited to five exports a month, to push upgrades. I owned that call. It was wrong, and it cost us about a quarter of the upgrade volume we'd forecast.

Here's what I had. Three percent of free users exported more than five times a month. Forty percent of that group converted to paid within ninety days. I read that as: exporting drives conversion, so capping it pushes people over the line. I forecast 1,800 upgrades in the quarter from the change. We got 1,400, 22 percent below, and 90-day churn in the affected cohort went up, not down.

The cause I own is that I read correlation as causation and never tested it. The heavy exporters converted because they were already engaged; exporting was a symptom of engagement, not the cause of it. Capping the limit didn't create intent, it removed a reason to stay. I could have falsified that in a week with a holdout. I didn't, because the number was clean and I wanted the quarter.

What changed: any pricing or limit change now needs a written falsification condition, the result that would prove me wrong, and a 5 percent holdout before full rollout. I've applied it twice. On an API paywall test, the holdout showed a 0.4-point lift against the 3 points the model predicted, so we killed it before rollout and avoided the churn. On a seat-limit change, the holdout confirmed a 6-point lift and we shipped it.

The part I'd flag if you pushed: the outcome number is the least interesting thing here. What I got wrong was the inference, and that's the thing I can actually fix."

What the interviewer is probing next

Why self-assessment outweighs the outcome: outcomes are noisy, and a miss with a named process flaw tells them how you'll behave on their team. A miss blamed on circumstances tells them nothing they can use.

The three framings that lose: blame-the-org (nothing owned), the humblebrag. "I cared too much about quality" (evasion), and no number (they can't tell whether you noticed).

When a different emphasis wins: if the miss was a genuine judgement call with no process flaw, lead with the trade-off you knowingly took and how you'd re-price it now. If the miss was execution, lead with the detection gap: how long it took you to notice, and what would have caught it sooner.

FramingWhat it signalsWhen it wins
Blame the orgNothing ownedNever
HumblebragEvasionNever
No numberCan't tell if you noticedSmall scope, early career
Judgement call, no process flawYou price riskThe miss was genuinely a coin-flip
Process flaw, named and fixedYou can be trusted with the next oneDefault

The miss is the story; the change is the point.

Curated: · Written: · Reviewed:

Q&A-96Tell me about a time you disagreed with an engineer or a designer.

The short version: I was wrong on the metric that mattered and right about a cost nobody had measured, and what settled it was a decision rule we wrote down before we ran anything.

This was last year, on the checkout service I owned. The disagreement was with Priya, the designer on the flow, about whether to keep the separate "review your order" step before payment.

Her position, at its strongest: the review step was a four-second tax on every order, and 11% of sessions dropped between review and payment. She'd removed the same step on the mobile app two years earlier and completion went up. She also argued the step gave false confidence: people clicked through without reading, so it wasn't catching errors anyway.

My position: the review step was where we caught address errors before charging a card. About 6% of orders had an address correction at that step, and a wrong-address order costs a reship at roughly $14 plus a support contact.

We were both reading the same funnel and looking at different parts of it. What settled it was agreeing the decision rule before we ran anything: flag to 10% of traffic for two weeks, and call it a win only if payment completion rose at least 2 points and the wrong-address reship rate rose no more than 0.3 points. Both of us signed off on those thresholds, which mattered: neither of us could relitigate the result afterwards.

The arithmetic behind the thresholds. At 40,000 orders a week, 10% is 4,000 orders per arm. A 2-point lift is 80 extra orders; at a $52 average order value that's about $4,160 a week. The reship ceiling of 0.3 points is 12 orders, at $14 plus about $9 of support time, roughly $276. Worth doing if the lift was real and reships stayed under the ceiling.

MetricThresholdResultCall
Payment completion+2.0 pts+3.4 ptspass
Wrong-address reship≤ +0.3 pts+0.5 ptsfail

She won the main metric; I won the guardrail; the rule said we couldn't ship it as-is.

What we shipped was her flow with one change: address validation inline as you typed, using the same address service, so the correction happened before payment without a separate screen. That was her suggestion, made when we looked at the numbers together.

What I'd do differently: I argued from a reship cost I'd estimated in my head, and it took a week to get the real number. Pulling it on day one would have settled the guardrail faster and shipped two weeks earlier.

The takeaway I'd give: the disagreement wasn't about who understood users better. I was optimizing for a cost she couldn't see; she was optimizing for a drop-off I'd written off as acceptable. Writing the decision rule down first is what turned an argument into an experiment.

Curated: · Written: · Reviewed:

Q&A-97Tell me about a time you changed someone's mind.

A worked example: the figures are the ones from the case I tell; substitute your own.

The story

Last year I owned the checkout service. Fraud losses had roughly doubled quarter over quarter, and the staff engineer who owned checkout wanted a synchronous call to a third-party scoring vendor in the authorization path: score first, block anything above a threshold. His position was reasonable: the vendor's offline numbers looked strong, and the alternative was eating the losses for another quarter.

I thought the inline call would break our latency budget. Checkout ran a p99 of 180 ms against a 400 ms SLO, and the vendor quoted a 250 ms p95 on their scoring endpoint. Inline, that puts a large slice of traffic over the SLO.

I didn't argue it in the design review. I built a replay harness: one day of production authorization traffic, about 4.2 million requests, PII stripped, replayed against a stub of the vendor call with their published latency distribution injected. Then I replayed the same traffic with scoring moved to a pre-auth step that fires asynchronously 300 ms ahead of authorization.

Pathp50p99over 400 ms SLO
score inline210 ms640 ms11.3%
async pre-auth185 ms195 ms0.2%

The mechanism: request arrives → pre-auth scoring fires async → 300 ms later the score is in cache → authorization reads it in 4 ms → decision. The inline version holds the authorization thread for the whole vendor round trip, so the vendor's tail becomes our tail.

I took the harness and the table to him, not a slide. He changed the decision. We shipped the async pre-auth path; the design doc was updated that week with the replay results attached. Fraud losses fell about 40% in the first month and p99 stayed under 200 ms.

What the interviewer will probe

Why a replay harness rather than a prototype or customer sessions? Each lever proves a different claim. A replay harness proves a performance or cost claim against the real traffic mix: right here, because the disagreement was about latency. A prototype proves something can be built and that the API is workable; it would not have settled a latency argument. Recorded customer sessions prove desirability; they would have been the right lever if the fight had been about whether users would accept a step-up challenge. Picking the wrong lever is the most common way these stories fall apart: a prototype that answers a question nobody disputed.

Failure modes I'd name if asked. First, the artefact proves the wrong thing: my first replay used only card traffic, and 30% of our volume was wallet, which had a different latency profile; I caught it before the review. Second, the decision changes in the room and reverts in the next planning meeting, which is why I asked for the result to be written into the design doc with a date. Third, the person agrees but can't sell it upward: I gave him the table so he could, rather than making him re-derive it.

What I'd do differently: I ran the replay after the design review instead of before it, which cost about a week of the quarter. I'd have run it first and saved the week.

Curated: · Written: · Reviewed:

Q&A-99What would you have accomplished ninety days into this role?

By day 90 I would have accomplished three things: a written map of the system and its top constraints, one improvement shipped to production with before/after numbers, and a decision memo recommending the next direction with options and costs. I would not have reorganized the team or promised a replatform.

That assumes the team is stable enough to ship. If it is in firefighting, the first 90 days change.

Days 1–30: learn, but produce something checkable. I would trace one real request end to end, read the on-call handoff, and pull p99, error budget, and deploy frequency. The artefact is a one-page system map plus a ranked list of the top three constraints, each with a named owner. I would walk that map with two engineers and an SRE. If they correct it, that is the point: the map is a test of my understanding, not a report.

Days 31–60: ship the smallest change that moves a metric the team already watches. Hypothetical example: checkout p99 is 420 ms, and the trace shows a per-item query inside the cart loop. I would batch that call or add the composite index on orders(customer_id, created_at), then measure p99 and error rate. If it works, p99 might go 420→280 ms and DB calls per checkout 14→3. If it does not move, I roll back and write down why. The mechanism matters: a shipped change tests assumptions against production, reveals hidden dependencies, and gives the team a shared baseline. A findings document can be wrong because it is based on interviews, not traffic.

Days 61–90: propose direction with evidence. The artefact is a decision memo with two or three options, cost, risk, a leading indicator, and a recommendation. Example options: keep the current architecture and fix the top three reliability risks; extract the checkout service; or buy a vendor. I would include what would change my mind.

DaysWhat I would produceEvidence it is real
1–30System map + top 3 risks with ownersWalkthrough with 2 engineers and an SRE; p99, error budget, on-call load
31–60One shipped improvementBefore/after metric, rollback plan, demo
61–90Decision memo with 2–3 optionsCost, risk, leading indicator, recommendation, what would change my mind

This shape is wrong in some contexts. If the team is burning more than half its error budget or has weekly SEV1s, days 1–90 are stabilization, not a new direction. If I inherit a committed launch, the shipped improvement is the launch. If the team is three people and attrition is high, retention is the metric. The criterion for departing from the plan is simple: if the system is not stable enough to measure a change, stabilize first.

The failure I have seen: a PM committed to a new strategy and a replatform in the first quarter, delivered neither, and spent the second quarter rebuilding credibility after 5 weeks of planning churn. Detection is concrete: no production change by day 60, no metric moved, and the team cannot name the top risk.

If day 90 arrives and I only have documents, I have failed the plan.

Curated: · Written: · Reviewed:

Q&A-100What would you ask us at the end of the loop?

I ask three questions, one per interviewer, and I choose them so that a bad answer would change whether I sign.

To the hiring manager: "Walk me through the last three decisions this team made that mattered, who proposed, who decided, who could have vetoed." That tells me where authority actually sits, which no org chart does.

To a peer engineer: "What does the team disagree about right now?" A rehearsed answer is "we're pretty aligned." A real one names a live argument (build versus buy, splitting the monolith, the on-call rotation) and tells me whether disagreement is safe to voice.

To the skip-level or whoever owns the roadmap: "What would make the first year a success, and who judges it?" If the answer is a number with a name attached, the role is measured. If it's "we'll know it when we see it," the promotion case is political.

Then I follow up, because the first answer is the polished one. If the manager says "the PM decides," I ask: "Tell me about a time engineering pushed back and won." If they can't produce one, engineering is a service org.

QuestionWhat the answer decides
How were the last three decisions made, and by whom?Where authority sits
What does the team disagree about right now?Whether disagreement is safe
What makes year one a success, judged by whom?Whether the role is measured

Three strategies compete for those last ten minutes, and only one is worth the slot.

De-risking the offer (comp band, level, on-call load, remote policy) is legitimate, but the recruiter answers all of it before the loop. Spending panel time on it wastes the only part of the process where I set the agenda.

Signalling preparation (questions about the stack, the roadmap, the launch that just shipped) feels productive and scores nothing. The panel already has my résumé and my system design; a question they've answered in a blog post tells them I read the blog post.

Testing the job (decision rights, disagreement safety, success criteria) is the one that pays. Only insiders can answer, and the answers are the ones that move my decision. I cap it at three questions: the panel is still scoring me, and twenty minutes of interrogation reads as adversarial.

Two failure modes I've watched. A candidate spent all ten minutes on process and tooling, asked nothing about how decisions were made, and three of five panellists logged a doubt about whether they'd be effective in a company where that answer was contested. The other is the question that's really a pitch ("your CI looks slow, I'd fix that") which puts the interviewer in the position of defending their setup instead of telling me how the place works.

Ask the questions whose answers would make you decline. If none of them would, you're not evaluating the job, you're auditioning for it.

Curated: · Written: · Reviewed: