Top 100 Data Scientist Interview Questions and Answers

The first thing I would pin down about choosing the metric an experiment is powered on is which question the number is supposed to answer.

Name one primary metric before the experiment starts, chosen because it is the quantity the ship decision turns on, and treat every other metric as either a guardrail or an exploratory readout with no decision authority.

Concretely, write the decision rule first — ship if the primary metric moves by at least the minimum detectable effect and no guardrail regresses beyond its stated bound — then size the experiment for that primary metric alone and register the rest as secondary before any data arrives.

The reason for that specificity is a failure I have seen: A checkout test with fourteen equally weighted metrics shipped on the one that reached significance, add-to-cart rate, while revenue per session was down 1.8% and nobody had committed in advance to which number decided the launch.

The registration that has to exist before traffic is split.

Field	Committed value
Primary metric	Revenue per session
Minimum detectable effect	1.5% relative
Guardrails	Checkout error rate, p95 page latency
Decision rule	Ship if primary is positive at the stated bound and no guardrail regresses
Secondary	Add-to-cart rate, coupon usage, support contacts (no decision authority)

I would not consider it settled without evidence: Circulate the decision rule and the powering calculation before randomisation begins, and check afterwards that the readout answers with the metric that was registered rather than the one that moved.

An experiment with fourteen primary metrics has no primary metric and cannot be lost.

I would start by writing the estimand for sizing an experiment before it runs in a sentence before choosing any estimator.

Sample size follows from four numbers — baseline rate, the smallest effect worth shipping, the false positive rate you will tolerate, and the power you want — so the question cannot be answered until the stakeholder supplies the second one.

Concretely, ask what improvement would change the decision, convert it to an absolute effect against the baseline, compute the required sample per arm for that effect at the chosen error rates, then divide by traffic per day to get a duration and round up to whole weeks so every day of the week is represented.

The reason for that specificity is a failure I have seen: A test sized for a 5% relative lift was run to detect a 1% one, ended flat after ten days, and was reported as evidence that the feature does nothing — when the design could not have distinguished a 1% gain from zero.

Duration falls out of the effect size, not the other way round.

from math import ceil

# Two-proportion test, 80% power, 5% two-sided alpha: n per arm ~ 16 * p(1-p) / delta^2
def per_arm(baseline: float, relative_lift: float) -> int:
    delta = baseline * relative_lift
    return ceil(16 * baseline * (1 - baseline) / delta**2)

for lift in (0.05, 0.02, 0.01):
    n = per_arm(0.040, lift)
    print(f"{lift:.0%} lift -> {n:,} per arm -> {ceil(2 * n / 50_000)} days at 50k/day")

I would not consider it settled without evidence: State the minimum detectable effect alongside every flat result, so a null readout is reported as what it is: the experiment could not see an effect smaller than this.

A test that ends flat has said nothing until you say what it could have seen.

This is a place where the sample answer and the population answer for stopping an experiment early because it looks significant come apart.

A fixed-horizon p-value is only valid at the horizon it was designed for, so checking it repeatedly and stopping at the first crossing inflates the false positive rate far beyond the nominal level.

Concretely, either hold to the pre-registered end date, or switch the analysis to a method built for continuous monitoring — a group sequential design with alpha spending, or an always-valid confidence sequence — and state which one you are using before the test starts rather than after a favourable peek.

The reason for that specificity is a failure I have seen: A team that peeked daily on a two-week test and shipped on the first significant day was running an effective false positive rate near 20%, and three of the five features shipped that quarter failed to reproduce in a holdout.

Ten thousand simulated A/A tests, same data, two stopping rules.

Stopping rule	Times "significant"	Effective false positive rate
Fixed horizon, day 14 only	503 of 10,000	5.0%
Peek daily, stop at first p < 0.05	1,970 of 10,000	19.7%
Alpha-spending boundary, peek daily	486 of 10,000	4.9%

I would not consider it settled without evidence: Simulate the team's actual stopping behaviour under a true null effect and report how often it ships, which is the number that describes the process rather than the single test.

The p-value describes the procedure that produced it, and daily peeking is a different procedure.

My answer to sample ratio mismatch begins with how the data came to exist, because sampling contaminates everything after it.

A split that deviates from its intended ratio by more than chance allows means the randomisation or the logging is broken, and the treatment effect estimated from a broken assignment is not interpretable at any p-value.

Concretely, run a chi-square goodness-of-fit test against the intended ratio, treat a small p-value as a defect rather than a finding, and debug the pipeline — bot filtering applied after assignment, a redirect that drops slow clients, exposure logged on render rather than on assignment — before reading any metric.

The reason for that specificity is a failure I have seen: A mismatch of roughly 1% was dismissed as noise for three experiments before someone noticed the treatment bundle timed out on slow connections, which removed the least engaged users from treatment and manufactured a lift in every engagement metric.

The check is one chi-square line, and 1% is not noise at this scale.

from scipy.stats import chisquare

observed = [51_204, 49_102]
expected = [sum(observed) / 2] * 2
stat, p = chisquare(observed, expected)
print(f"chi2={stat:.1f} p={p:.2e}")   # chi2=44.0 p=3.2e-11

I would not consider it settled without evidence: Compute the split test automatically on every experiment and block the readout when it fails, rather than leaving it to whoever remembers to look.

A broken assignment does not produce a noisy estimate; it produces a confident wrong one.

I would treat choosing the unit of randomisation as a design question first and a computation question second.

Randomise at the level at which the treatment is delivered and at which the outcome is correlated, because units inside the same account see each other's treatment and their outcomes are not independent.

Concretely, pick the coarsest unit that still gives enough independent units for the required power, randomise on that unit's identifier, and compute standard errors clustered at the same level rather than at the row level.

The reason for that specificity is a failure I have seen: A workspace feature randomised per session gave both arms to the same team, so half the treated behaviour was actually control users reacting to a colleague's change, and the measured effect was roughly a third of what a team-level rerun found.

Same experiment, two units, and the interval that changes.

Randomisation unit	Independent units	Estimated lift	95% interval
Session	412,000	+0.9%	+0.6% to +1.2%
User	96,000	+1.4%	+0.7% to +2.1%
Account (correct)	3,100	+2.6%	+0.4% to +4.8%

I would not consider it settled without evidence: Compare the row-level standard error against the cluster-robust one on the same data; a large gap says the unit of analysis and the unit of randomisation disagree.

The unit of analysis has to match the unit of assignment, or the interval is fiction.

The useful framing for interference between treatment and control units is what decision changes if the estimate moves by its own uncertainty.

When treated units compete with control units for a shared finite resource, the treatment effect measured against those controls includes what treatment took from them, so the difference overstates what a full rollout would deliver.

Concretely, switch to a design that isolates the interference — randomise by market or region, run a switchback across time, or randomise the supply side — and accept the smaller number of independent units in exchange for an estimate that generalises to full traffic.

The reason for that specificity is a failure I have seen: A ranking change measured a 6% booking lift in a user-level test and delivered under 1% at full rollout, because most of the measured gain was demand redirected from control users onto the same limited inventory.

Three designs on the same ranking change.

Design	Independent units	Measured lift	Reproduced at rollout
User-level split	1,200,000	+6.0%	no
Market-level split	48 cities	+1.3%	yes
Switchback, hourly	720 hour-slots	+1.1%	yes

I would not consider it settled without evidence: Run the winning arm as a market-level or switchback test in a subset of regions and compare the effect to the user-level estimate before committing to the forecast.

In a marketplace, one arm's gain is often the other arm's loss, and the difference counts it twice.

I would settle novelty and primacy effects against the simplest defensible comparison, so any extra structure has to earn its place.

Report the effect on the population and horizon the decision applies to, which means a stabilised estimate from users who have lived with the change, not the first-exposure reaction that a permanent rollout will never repeat.

Concretely, split the readout by time since first exposure rather than by calendar week, plot the effect against exposure age, and take the decision number from the range where the curve has flattened; extend the test if it has not flattened.

The reason for that specificity is a failure I have seen: A navigation redesign shipped on a 9% first-week lift and settled at a 1.1% loss by week six, and because the rollout was global there was no untreated group left to detect it.

Effect by exposure age, which is not the same as calendar week.

Days since first exposure	Estimated lift	95% interval
0-2	+9.1%	+7.8% to +10.4%
3-7	+4.6%	+3.5% to +5.7%
8-21	+2.0%	+0.9% to +3.1%
22-56	-1.1%	-2.0% to -0.2%

I would not consider it settled without evidence: Hold back 5% of traffic from the rollout for eight weeks and compare it against the treated population once the novelty window has passed.

The first week measures how different the change is, not how good it is.

Most of the judgement in multiple comparisons across an experiment readout sits in what the comparison group is, not in the arithmetic.

Testing many metrics at a 5% level guarantees false positives when nothing is happening, so the count of significant metrics is uninformative until the error rate is controlled across the family being examined.

Concretely, declare the primary metric in advance and test it at the nominal level, then apply a false discovery rate correction such as Benjamini-Hochberg across the exploratory set and label surviving metrics as hypotheses for a follow-up rather than as findings.

The reason for that specificity is a failure I have seen: A team reported a "significant" 3.2% increase in a niche support metric from a 40-metric panel, staffed a project on it, and found nothing when the metric was tested on its own in a dedicated experiment.

Benjamini-Hochberg on the raw p-values, sorted ascending.

Rank i	Metric	p	BH threshold (i/40 x 0.05)	Survives
1	Support contacts	0.004	0.00125	no
2	Coupon views	0.031	0.00250	no
3	Session length	0.048	0.00375	no

I would not consider it settled without evidence: Run the same 40-metric panel on an A/A split and count how many cross the threshold, which shows the team what its dashboard produces under no effect at all.

Two hits in forty tests is what an inert change looks like.

Where analyses go wrong on guardrail metrics is usually one step upstream of the statistic being argued about.

A guardrail exists to detect harm the team is not looking for, so it is tested for the absence of a regression rather than the presence of a gain, and it keeps its decision authority even when it is not significant.

Concretely, define each guardrail with an explicit non-inferiority bound, block the ship when the confidence interval for the guardrail extends past that bound, and do not require significance to act — a wide interval that admits real harm is itself the finding.

The reason for that specificity is a failure I have seen: A recommendation change shipped on engagement while the page latency guardrail was up 84 ms with an interval spanning zero, and the latency regression cost more conversions across the quarter than the recommendation change gained.

Guardrails are read against a bound, not against zero.

Guardrail	Bound	Estimate	95% interval	Verdict
p95 latency	+50 ms	+84 ms	-12 to +180 ms	block, interval exceeds bound
Error rate	+0.1 pp	+0.02 pp	-0.03 to +0.07 pp	pass
Unsubscribes	+5%	+1%	-2% to +4%	pass, interval inside bound

I would not consider it settled without evidence: Report each guardrail as an interval against its bound rather than as a pass or fail, so a test too small to detect the harm is visibly too small.

Failing to reject harm is not the same as showing there is none.

I would answer heterogeneous treatment effects across segments by separating what the design identifies from what the data merely displays.

A segment effect discovered after seeing the data is a hypothesis, not a result, because the number of possible segmentations is large enough that one of them will look significant under any null.

Concretely, pre-register the small number of segments the decision could plausibly differ across, test the interaction rather than the subgroup in isolation, correct across the registered set, and confirm anything discovered post hoc in a dedicated experiment before it changes a rollout.

The reason for that specificity is a failure I have seen: A "new users on mobile" lift found in a flat test was shipped to that segment and did not reproduce; a later count showed the analysis had implicitly searched 96 segment combinations, where roughly five crossings are expected by chance.

Interaction test, not a subgroup test, and the search space that follows.

import statsmodels.formula.api as smf

# The claim is that the effect differs by segment, so that is the term to test.
model = smf.ols("revenue ~ treated * is_new + treated * is_mobile", data=df).fit()
print(model.pvalues[["treated:is_new[T.True]", "treated:is_mobile[T.True]"]])

segments = 4 * 3 * 2 * 4        # tenure x device x region x plan
print(segments, "cuts ->", round(segments * 0.05, 1), "expected false positives")

I would not consider it settled without evidence: Count how many segment cuts were examined before the interesting one was found, and compare that count against how many false positives the threshold implies.

The segment that stands out after the fact is usually the one that stood out by chance.

The substance of variance reduction with pre-experiment covariates is the size of the effect and its uncertainty, not whether a threshold was crossed.

Subtracting the part of the outcome that is predictable from pre-experiment behaviour removes variance without touching the treatment effect, because the covariate is fixed before randomisation and therefore cannot be affected by the treatment.

Concretely, regress the outcome on the same user's pre-period value, use the residual as the analysis metric, and confirm the adjustment is legitimate by checking the covariate is balanced across arms and measured strictly before assignment.

The reason for that specificity is a failure I have seen: A team applied the adjustment using a covariate window that overlapped the experiment by two days, which let treatment leak into the control variable and biased the estimate toward zero by roughly a fifth.

Same 60,000 users, same effect, half the interval width.

Estimator	Estimated lift	95% interval	Interval width
Difference in means	+1.9%	-0.4% to +4.2%	4.6 pp
Adjusted for pre-period spend	+1.9%	+0.7% to +3.1%	2.4 pp

I would not consider it settled without evidence: Run the adjusted estimator on an A/A split and confirm it returns an unbiased zero with a smaller interval, rather than assuming the reduction is free.

The covariate has to be untouchable by the treatment, and "before randomisation" is the only test of that.

Before running anything I would state what result would make me abandon the hypothesis behind outliers in a heavy-tailed experiment metric.

Decide the outlier rule before seeing which arm the outlier landed in, because any rule chosen afterwards is a decision about the result rather than about the data.

Concretely, pre-register a winsorisation percentile or a capped metric, report the result under the registered rule and unmodified as a sensitivity check, and if the two disagree say so rather than picking the one that supports the conclusion.

The reason for that specificity is a failure I have seen: A pricing test was reported as a $2.1M annualised gain that rested on a single enterprise renewal in the treatment arm; the same test winsorised at the 99th percentile was flat, and the renewal had been negotiated months before the test began.

Sensitivity to the cap, reported rather than chosen.

Winsorisation	Estimated lift	95% interval	Top row's share of the gap
None	+8.4%	+0.2% to +16.6%	71%
99.9th percentile	+3.1%	-0.6% to +6.8%	22%
99th percentile	+0.4%	-1.1% to +1.9%	3%

I would not consider it settled without evidence: Report the effect at several caps — none, 99.9th, 99th, 95th percentile — so the reader can see how much of the estimate rests on a handful of rows.

If one row can flip the decision, the experiment measured that row.

The first thing I would pin down about non-compliance and intention-to-treat is which question the number is supposed to answer.

Compare the groups as randomised, because the users who chose to engage differ from those who did not in ways the randomisation no longer controls, and the moment you filter on a post-assignment behaviour the comparison stops being an experiment.

Concretely, report the intention-to-treat effect as the primary result, and if the effect on the compliers is the quantity of interest estimate it with an instrumental-variables approach that uses assignment as the instrument rather than by filtering the data.

The reason for that specificity is a failure I have seen: An onboarding test analysed on "users who completed the tour" showed a 34% retention lift that was almost entirely selection: the users who finished a five-step tour were already the users who would have stayed.

Three estimates from one experiment, and only two are identified.

Analysis	Estimate	Identified by randomisation
Intention-to-treat (as assigned)	+3.2%	yes
Complier average effect (assignment as instrument)	+8.4%	yes
Completers vs everyone in control	+34%	no, selection

I would not consider it settled without evidence: Compare the pre-period behaviour of compliers and non-compliers within the treatment arm; a large gap shows what filtering on engagement would import into the estimate.

Filtering on something the treatment caused destroys the only thing randomisation bought you.

I would start by writing the estimand for switchback designs for time-varying systems in a sentence before choosing any estimator.

When units cannot be isolated from one another, randomise time instead of units, so that the whole system runs one policy in a slot and the comparison is between slots rather than between contaminated arms.

Concretely, choose a slot length long enough that carryover from the previous policy has decayed and short enough to give many slots, randomise the policy per slot, block on time of day and day of week, and analyse at the slot level with slot as the unit.

The reason for that specificity is a failure I have seen: A dispatch test with 15-minute slots measured a 4% efficiency gain that vanished at rollout, because trips assigned under the previous policy were still completing inside the following slot and their outcomes were credited to the wrong arm.

Carryover check: the first third of a slot is still running the old policy.

Slot length	Effect, first third	Effect, last third	Carryover
15 min	+0.4%	+5.8%	severe
60 min	+2.9%	+3.4%	mild
180 min	+3.1%	+3.2%	none detected

I would not consider it settled without evidence: Estimate the carryover directly by comparing the first and last thirds of each slot; if they differ, the slot is shorter than the system's memory.

The slot has to be longer than the system's memory, or the arms are measuring each other.

This is a place where the sample answer and the population answer for difference-in-differences come apart.

Differencing out each region's own level and the common time shock identifies the effect only if the two regions would have moved in parallel absent the rollout, which is an assumption about an unobserved counterfactual rather than a property of the data.

Concretely, plot both series for many periods before the change and check they move together, run the regression with unit and time fixed effects, cluster standard errors at the unit level, and test for a pre-trend divergence rather than asserting parallel trends.

The reason for that specificity is a failure I have seen: A pricing rollout in one region was credited with a 12% revenue gain that was mostly a currency move; the pre-period plot, produced afterwards, showed the two regions had already been diverging for five months.

The four cells, and the placebo that has to come back at zero.

	Before	After	Difference
Canada (treated)	48.2	55.9	+7.7
United States (control)	51.0	55.4	+4.4
Difference-in-differences			+3.3
Placebo on prior year	46.1 -> 49.0	48.8 -> 51.6	+0.1

I would not consider it settled without evidence: Estimate the same specification on the pre-period alone; a non-zero "effect" before the intervention says the parallel-trends assumption fails.

Parallel trends is an assumption you display, not one you state.

My answer to instrumental variables begins with how the data came to exist, because sampling contaminates everything after it.

An instrument identifies the effect only if it moves the treatment, affects the outcome through no other channel, and is unrelated to the unobserved determinants of the outcome — and the second of those cannot be tested from the data.

Concretely, show the first stage is strong rather than assuming it, argue the exclusion restriction from institutional knowledge of why the instrument varies, and report that the estimate applies to the units the instrument actually moves rather than to the whole population.

The reason for that specificity is a failure I have seen: A study instrumented adoption with a marketing email send, which also carried a discount code, so the instrument affected the outcome directly and the estimate absorbed the discount's effect entirely.

First stage, reduced form, and the ratio that is the estimate.

Quantity	Estimate	Standard error
First stage: instrument on uptake	+0.21	0.02 (F = 110)
Reduced form: instrument on outcome	+1.68	0.44
Instrumental-variables estimate (1.68 / 0.21)	+8.0	2.2

I would not consider it settled without evidence: Report the first-stage F statistic and the reduced form separately, so a weak instrument shows as a wide and unstable second stage rather than hiding inside a ratio.

A weak instrument gives an answer with more bias than the comparison it was meant to fix.

I would treat regression discontinuity as a design question first and a computation question second.

Units just above and just below a sharp cutoff are comparable in everything except the assignment, so the jump in the outcome at the cutoff estimates the effect for units at that margin and only for them.

Concretely, fit flexible functions of the running variable on each side within a bandwidth, read the effect as the discontinuity at the cutoff, and check that the density of the running variable and every pre-treatment covariate are continuous there.

The reason for that specificity is a failure I have seen: A loyalty analysis reported a large benefit at the threshold while the histogram of points showed a spike immediately above 1,000 — customers were topping up to qualify, so those just above were self-selected and not comparable to those just below.

Bandwidth sensitivity and the manipulation check.

Bandwidth (points)	Estimated jump	95% interval
+/- 50	+$14.10	+$3.20 to +$25.00
+/- 100	+$13.40	+$6.90 to +$19.90
+/- 400	+$27.80	+$22.10 to +$33.50
Density test at cutoff	p = 0.001	manipulation present

I would not consider it settled without evidence: Run a density test on the running variable at the cutoff and re-estimate at several bandwidths; an estimate that swings with the bandwidth is fitting curvature rather than a jump.

The design is only credible if units cannot precisely control which side of the line they land on.

The useful framing for propensity score matching and what it cannot fix is what decision changes if the estimate moves by its own uncertainty.

Matching balances what you measured, so it removes bias from observed confounders and leaves bias from unobserved ones exactly where it was; balance tables are evidence about the matching, not about the identification.

Concretely, check standardised mean differences after matching, report how many treated units failed to find support and what they look like, and state the unobserved confounder that would have to exist to explain the estimate away rather than claiming none does.

The reason for that specificity is a failure I have seen: A study of a premium feature matched on demographics and usage and reported a 22% retention gain; the entire effect was consistent with a single unmeasured variable, whether the user's employer was paying, which drove both adoption and renewal.

Post-matching balance is necessary and not sufficient.

Covariate	Standardised difference before	After matching
Tenure months	0.62	0.03
Sessions per week	0.48	0.02
Region mix	0.31	0.04
Employer-paid (not collected)	unknown	unknown

I would not consider it settled without evidence: Run a sensitivity analysis that reports how strong an unmeasured confounder would need to be to nullify the estimate, and judge whether such a variable is plausible in this system.

Balance on the covariates you have says nothing about the one you do not.

I would settle Simpson's paradox in a readout against the simplest defensible comparison, so any extra structure has to earn its place.

A pooled comparison weights each subgroup by its share of the arms, so when the arms have different subgroup mixes the aggregate can reverse every subgroup it is made of.

Concretely, check the composition of the arms before comparing them, and if it differs, either report the mix-adjusted effect using a common weighting or fix the assignment defect that produced the imbalance and rerun.

The reason for that specificity is a failure I have seen: A pooled readout showed a 1.4% loss while every country showed a gain, because a redirect bug pushed a disproportionate share of a low-converting market into treatment; the summary went to leadership before anyone compared the mixes.

Every stratum favours treatment; the pooled rate does not.

Country	Control	Treatment	Treatment share of arm
A (high converting)	30% of 8,000	32% of 2,000	20%
B (low converting)	5% of 2,000	6% of 8,000	80%
Pooled	25.0%	11.2%	—

I would not consider it settled without evidence: Report the arm composition alongside the aggregate, and standardise both arms to a common subgroup weighting before quoting a single number.

An aggregate over differently composed groups is a statement about the composition.

Most of the judgement in surrogate metrics for long-term outcomes sits in what the comparison group is, not in the arithmetic.

A short-run metric can stand in for a long-run one only if the historical relationship between them survives the kind of change being tested, which makes surrogacy an empirical claim to be validated rather than a convenience.

Concretely, fit the surrogate-to-outcome relationship on past experiments where both are observed, use it only for changes of the same kind, quantify the extra uncertainty the mapping introduces, and re-validate it as new experiments complete.

The reason for that specificity is a failure I have seen: A team used week-one sessions as a surrogate for annual retention and shipped a notification change that raised sessions 6% and lowered annual retention 3%, because the surrogate had been calibrated on content changes and not on interruption.

Back-test of the surrogate on past experiments with observed outcomes.

Change type	Experiments	Surrogate predicted sign correctly
Content ranking	22	20
Performance	14	13
Notification and interruption	9	4

I would not consider it settled without evidence: Back-test the surrogate against the completed experiments where the long-run outcome is now observed, and report how often it got the sign right.

A surrogate is only valid for the class of change it was calibrated on.

Where analyses go wrong on deciding what to do with a flat result is usually one step upstream of the statistic being argued about.

A flat result is a statement about the interval, not about the point estimate, so the recommendation follows from whether the interval excludes the effect that would have justified the work and its ongoing cost.

Concretely, compare the interval against the minimum effect worth shipping: if the interval lies entirely below it, decline and say so; if it straddles that bound, the test was too small to decide and the choice is to extend, to run a more powerful design, or to ship on cost grounds alone.

The reason for that specificity is a failure I have seen: A flat result was reported as "no effect" and the feature was removed; a later, larger test found a 0.9% gain that the original design could never have detected, and the removal itself cost a quarter of rework.

Three flat results that call for three different decisions.

Estimate	95% interval	Ship bound	Reading
+0.2%	-1.1% to +1.5%	+1.0%	undecided, extend
+0.1%	-0.2% to +0.4%	+1.0%	ruled out, decline
+0.2%	-4.0% to +4.4%	+1.0%	uninformative, redesign

I would not consider it settled without evidence: Report every null result as an interval against the minimum detectable effect, so "we saw nothing" and "we could not have seen it" stay distinguishable.

Absence of evidence is only evidence of absence when the design could have found it.

I would answer long-term holdouts by separating what the design identifies from what the data merely displays.

Individually positive experiments do not add up, because each was measured against a control that already contained the previous changes and because winner's curse inflates the effects that were selected for shipping.

Concretely, keep a persistent holdout that never receives new launches for a fixed period, compare the treated population against it at the end, and treat the gap as the accountable quarterly effect rather than the sum of individual readouts.

The reason for that specificity is a failure I have seen: A team's shipped experiments summed to a claimed 14% annual revenue gain while the year-long holdout showed 4%, with the difference traced to overlapping effects and to results selected for being large.

Summed experiment effects against the persistent holdout.

Quarter	Sum of shipped effects	Holdout-measured effect	Ratio
Q1	+3.8%	+1.2%	0.32
Q2	+4.1%	+1.5%	0.37
Q3	+3.2%	+0.9%	0.28

I would not consider it settled without evidence: Publish the holdout comparison next to the summed estimates every quarter, and use the ratio between them to discount future forecasts.

The sum of the wins is a forecast; the holdout is the measurement.

The substance of staged rollouts and the ramp plan is the size of the effect and its uncertainty, not whether a threshold was crossed.

A ramp is a separate measurement rather than a formality, because effects that depend on the treated share — capacity, marketplace balance, support load — only appear as the share grows.

Concretely, ramp in stated stages, hold each stage long enough to read the guardrails at that traffic level, keep a holdout at each stage so the comparison remains valid, and define the rollback trigger before starting rather than during an incident.

The reason for that specificity is a failure I have seen: A feature that passed at 50% saturated an internal service at 85% and caused a two-hour outage, because nothing between 50% and 100% had been measured and the ramp had no defined stopping rule.

A ramp with a stated hold and rollback trigger per stage.

Stage	Traffic	Hold	Rollback trigger
1	50%	3 days	error rate +0.1 pp
2	75%	2 days	p95 latency +50 ms
3	95%	2 days	queue depth above 1,000
4	100% minus 5% holdout	8 weeks	primary metric below -0.5%

I would not consider it settled without evidence: Read guardrails at each stage against the same bounds used in the test, and record the traffic share at which any of them starts to move.

The last 50% of traffic is the part you have not measured.

Before running anything I would state what result would make me abandon the hypothesis behind reconciling an experiment with an observational estimate.

When a randomised estimate and an observational one disagree, the randomised one is identified and the observational one is a description of who adopts, so the useful work is explaining the gap rather than averaging them.

Concretely, decompose the difference by re-running the observational estimator on the experiment's own data restricted to the control arm, which shows how much of the gap is selection, and only then look at differences in population, horizon, and outcome definition.

The reason for that specificity is a failure I have seen: A team averaged the two estimates into a 11% forecast, planned headcount against it, and missed by a factor of five when the experiment's number held.

The observational method run inside the experiment, where the truth is known.

Estimator	Data	Estimate
Observational (adopters vs non-adopters)	Full population	+20.4%
Observational, same method	Control arm only	+18.9%
Randomised difference in means	Both arms	+2.1%

I would not consider it settled without evidence: Apply the observational method to the experiment's control arm alone; if it recovers a large effect where randomisation found a small one, the gap is selection rather than context.

Two estimates that disagree do not average into a better one.

The first thing I would pin down about what a p-value states and what it does not is which question the number is supposed to answer.

A p-value is the probability of data at least this extreme if the null were true, which is a statement about data under an assumption rather than a probability that any hypothesis is true.

Concretely, answer with the effect size and its interval in the unit the executive cares about, state the p-value as the compatibility of the data with no effect, and if a probability of a hypothesis is genuinely wanted, compute a posterior with a stated prior rather than reinterpreting the p-value.

The reason for that specificity is a failure I have seen: A quarterly plan was built on "97% confident" readings of three p-values; one of them was a peeked result and another came from a 40-metric panel, and the plan's central assumption failed within a quarter.

The same result stated three ways, and only two are true.

Statement	Correct
"If the feature did nothing, data this extreme would appear 3% of the time."	yes
"The lift is +1.4%, 95% interval +0.1% to +2.7%, or +$41k to +$1.1M a year."	yes
"There is a 97% chance the feature works."	no

I would not consider it settled without evidence: Restate every readout as an effect with an interval in business units and check whether the recommendation still holds when the p-value is omitted entirely.

The p-value is conditional on the null, and the executive is asking about the alternative.

I would start by writing the estimand for interpreting a confidence interval in a sentence before choosing any estimator.

The 95% describes the procedure — intervals built this way cover the true value 95% of the time across repeated samples — rather than the probability that this particular interval contains it.

Concretely, report the interval in the decision's own units, describe it as the range of effects the data does not rule out, and check whether the decision changes at each end of the range rather than at the point estimate.

The reason for that specificity is a failure I have seen: A launch was approved on a point estimate of +$1.2M whose interval ran from -$0.3M to +$2.7M; the plan committed spend that was only justified near the upper end and had no contingency for the lower one.

The decision evaluated at both ends, not at the point estimate.

Scenario	Annual effect	Decision
Lower bound	-$0.3M	do not fund the expansion
Point estimate	+$1.2M	fund
Upper bound	+$2.7M	fund and accelerate

I would not consider it settled without evidence: State the decision at both endpoints of the interval, so the readout says what happens if the truth is at the pessimistic end.

Plan against the interval, because that is what the data actually supports.

This is a place where the sample answer and the population answer for the trade-off between the two kinds of error come apart.

The two error rates are set from the relative cost of shipping something inert and missing something valuable, which is a business judgement rather than a statistical convention.

Concretely, write the cost of each error in the same unit, choose alpha and power so the expected cost is minimised at the traffic you have, and record the choice with the reasoning so the threshold is not silently re-litigated after the result arrives.

The reason for that specificity is a failure I have seen: A safety-relevant change was tested at alpha 0.05 and 60% power, which meant a real regression had a 40% chance of passing unnoticed, and the team read the pass as evidence of safety.

Choosing the threshold from costs rather than from habit.

Setting	Cost of shipping nothing useful	Cost of missing a real gain	Sensible alpha
Cheap reversible UI change	Low	Moderate	0.10
Expensive irreversible migration	High	Moderate	0.01
Safety regression check	Moderate	Very high	0.10 with high power

I would not consider it settled without evidence: Tabulate expected cost under both thresholds at the achievable sample size and pick the one with the lower expected cost.

Point-oh-five is a convention, and conventions do not know what your errors cost.

My answer to power and the minimum detectable effect begins with how the data came to exist, because sampling contaminates everything after it.

Power is the chance the test declares an effect when an effect of a stated size is genuinely present, and in an underpowered design the estimates that happen to reach significance are systematically larger than the truth.

Concretely, compute the minimum detectable effect for the sample you can actually obtain, refuse to run when it exceeds any plausible true effect, and when a small study does reach significance, treat the magnitude as inflated and confirm it before planning against it.

The reason for that specificity is a failure I have seen: A 20%-powered study produced a significant +40% effect that was published internally as the planning assumption; the replication at adequate power found +6%, and the roadmap had been built on the inflated figure.

Winner's curse: significant estimates from a low-powered design.

Power	True effect	Mean of estimates that reach significance
20%	+6%	+19%
50%	+6%	+10%
80%	+6%	+7%

I would not consider it settled without evidence: Simulate the estimator at the study's actual power under the replication's effect size and show the distribution of significant estimates sits well above the truth.

An underpowered study that finds something has usually found an exaggeration.

I would treat one-sided versus two-sided tests as a design question first and a computation question second.

A one-sided test is defensible only when it is chosen before the data and when a negative effect would lead to the same action as no effect, and switching after seeing the direction halves the p-value without adding any information.

Concretely, fix the sidedness in the analysis plan, and where a regression genuinely matters — which is almost always, because shipping harm is not the same as shipping nothing — keep the two-sided test and put the harm question in a guardrail.

The reason for that specificity is a failure I have seen: A borderline result at two-sided p = 0.07 was re-reported as one-sided p = 0.035 and shipped, and the feature was rolled back six weeks later after a regression that the original test had already hinted at.

The same z statistic under both readings.

from scipy.stats import norm

z = 1.81
print(f"two-sided p = {2 * (1 - norm.cdf(abs(z))):.3f}")   # 0.070
print(f"one-sided p = {1 - norm.cdf(z):.3f}")              # 0.035

I would not consider it settled without evidence: Compare the analysis plan's registered sidedness against the readout; any mismatch means the threshold moved after the data arrived.

Halving a p-value after seeing the sign is not a test, it is a decision dressed as one.

The useful framing for choosing a test for a difference in proportions is what decision changes if the estimate moves by its own uncertainty.

For two independent binary outcomes at this scale a two-proportion z test or its chi-square equivalent is appropriate, because the normal approximation is accurate when the expected count in every cell is large.

Concretely, check that each arm has at least a few dozen successes and failures, use the pooled standard error under the null for the test and the unpooled one for the interval, and fall back to an exact test only when counts are small enough that the approximation fails.

The reason for that specificity is a failure I have seen: A team applied a t test to raw binary indicators on 40 conversions per arm and reported an interval that extended below zero conversions, which is not a possible value of the quantity being estimated.

Two proportions, pooled standard error under the null.

from statsmodels.stats.proportion import proportions_ztest, confint_proportions_2indep

count, nobs = [2_460, 2_640], [60_000, 60_000]
stat, p = proportions_ztest(count, nobs)
low, high = confint_proportions_2indep(2_640, 60_000, 2_460, 60_000, method="wald")
print(f"z={stat:.2f} p={p:.3f} diff 95% CI = ({low:.4f}, {high:.4f})")

I would not consider it settled without evidence: Compare the approximate interval against a bootstrap or exact interval on the same counts; agreement confirms the approximation is safe at this sample size.

The approximation is fine at 2,400 conversions and misleading at 24.

I would settle testing a skewed continuous metric against the simplest defensible comparison, so any extra structure has to earn its place.

The t test relies on the sampling distribution of the mean being approximately normal, not on the data being normal, so at large sample sizes skew is tolerable while extreme kurtosis and a few dominant observations are not.

Concretely, check the effective sample size that the heavy tail leaves you, compare the t interval against a bootstrap interval on the same data, and if they disagree report the bootstrap or switch to a metric — capped revenue, or a rank-based comparison — whose sampling distribution behaves.

The reason for that specificity is a failure I have seen: A revenue test with 30,000 users per arm produced a t interval that missed the bootstrap interval entirely, because 40% of the arm's revenue came from eleven users and the central limit theorem had not taken hold at that sample size.

The parametric and bootstrap intervals disagree when a few rows dominate.

Metric	t interval	Bootstrap interval	Top 11 users' share
Revenue per user	-$0.12 to +$0.94	-$0.41 to +$1.55	40%
Revenue capped at 99th pct	+$0.05 to +$0.31	+$0.05 to +$0.32	4%

I would not consider it settled without evidence: Bootstrap the difference in means 10,000 times and compare the resulting interval against the parametric one; a visible gap says the approximation has not converged.

Skew is a question about how many observations the mean is really made of.

Most of the judgement in paired versus unpaired comparisons sits in what the comparison group is, not in the arithmetic.

Pairing removes between-unit variation by analysing within-unit differences, which raises power sharply, but it also removes the randomisation, so a before-and-after comparison confounds the change with everything else that happened over the same period.

Concretely, use pairing when the pairs are randomised — the same unit under both conditions in a crossover, or matched pairs assigned at random — and when the comparison is only over time, add a concurrent control group so the time trend is differenced out.

The reason for that specificity is a failure I have seen: A before-and-after study credited a redesign with a 15% engagement gain that a concurrent untreated group would have shown was almost entirely seasonal, since the two periods straddled the start of a school term.

The same before-and-after window, treated and untouched.

Group	Before	After	Change
Treated	12.4	14.3	+15.3%
Untouched control	11.9	13.5	+13.4%
Difference			+1.9 pp

I would not consider it settled without evidence: Run the same before-and-after comparison on an untouched population over the same dates; whatever it shows is the part of the effect that is not the change.

Pairing buys precision, and only randomisation buys identification.

Where analyses go wrong on the bootstrap is usually one step upstream of the statistic being argued about.

Resampling the observed data with replacement approximates the sampling distribution of any statistic, which gives intervals for quantiles, ratios, and other estimators with no tractable analytic form.

Concretely, resample at the unit of independence rather than the row, compute the statistic on each resample, take the percentile interval from the resulting distribution, and use enough resamples that the interval endpoints are stable to the precision being reported.

The reason for that specificity is a failure I have seen: A bootstrap over event rows rather than over users produced an interval roughly a third as wide as the truth, because the resamples treated 40 events from one heavy user as 40 independent observations.

Cluster bootstrap: resample users, then take all of their rows.

import numpy as np

def p90_interval(df, users, draws=10_000, seed=0):
    rng, stats = np.random.default_rng(seed), []
    by_user = {u: g["seconds"].to_numpy() for u, g in df.groupby("user_id")}
    for _ in range(draws):
        picked = rng.choice(users, size=len(users), replace=True)
        stats.append(np.percentile(np.concatenate([by_user[u] for u in picked]), 90))
    return np.percentile(stats, [2.5, 97.5])

I would not consider it settled without evidence: Re-run the bootstrap resampling users rather than rows and compare interval widths; a large change confirms the unit was wrong.

The bootstrap resamples whatever you told it was independent, and it will not check.

I would answer permutation tests by separating what the design identifies from what the data merely displays.

Under the null that assignment is unrelated to outcome, relabelling the arms at random generates the distribution of the statistic you would see by chance, and the observed statistic's place in it is an exact p-value for that null.

Concretely, shuffle the assignment labels at the unit of randomisation, recompute the ratio on each shuffle, count how often the shuffled statistic is at least as extreme as the observed one, and use enough shuffles that the p-value's own resolution is finer than the threshold.

The reason for that specificity is a failure I have seen: A ratio metric tested with a naive delta-method standard error reported p = 0.02, while the permutation distribution put the observed value at the 11th percentile of chance outcomes; the analytic formula had ignored the correlation between numerator and denominator.

Permutation p-value for a ratio metric, shuffling at the user level.

import numpy as np

def permutation_p(num, den, treated, draws=20_000, seed=0):
    obs = num[treated].sum() / den[treated].sum() - num[~treated].sum() / den[~treated].sum()
    rng, extreme = np.random.default_rng(seed), 0
    for _ in range(draws):
        flag = rng.permutation(treated)
        diff = num[flag].sum() / den[flag].sum() - num[~flag].sum() / den[~flag].sum()
        extreme += abs(diff) >= abs(obs)
    return (extreme + 1) / (draws + 1)

I would not consider it settled without evidence: Run the permutation test alongside the analytic one and report both; agreement validates the formula, disagreement says the formula's assumptions do not hold here.

Shuffling the labels asks exactly the question the null poses, with no distributional assumption.

The substance of correcting for many tests is the size of the effect and its uncertainty, not whether a threshold was crossed.

Bonferroni controls the chance of even one false positive across the family, which is what a single ship decision needs, while a false discovery rate procedure controls the expected share of false positives among the discoveries, which is what a screening exercise needs.

Concretely, use the family-wise correction when one wrong call is costly and the family is small, use Benjamini-Hochberg when producing a ranked list of candidates for follow-up, and in both cases define the family before looking at the p-values.

The reason for that specificity is a failure I have seen: A screen of 2,000 features under Bonferroni returned nothing at all and the project was closed; the same p-values under a 10% false discovery rate returned 34 candidates, 29 of which replicated.

The same 2,000 p-values under both corrections.

Procedure	Threshold	Discoveries	Expected false among them
Uncorrected 0.05	0.0500	143	about 100
Bonferroni	0.000025	0	0
Benjamini-Hochberg at 10%	adaptive	34	about 3

I would not consider it settled without evidence: State the family and the procedure in the analysis plan, and report how many discoveries each procedure would return on the same p-values.

The two procedures answer different questions, so the choice follows from what a false positive costs.

Before running anything I would state what result would make me abandon the hypothesis behind Bayesian and frequentist readouts of the same experiment.

A posterior probability answers the question stakeholders actually ask, but it is a function of the prior as well as the data, so the prior becomes part of the result and has to be stated and defended.

Concretely, choose the prior from the historical distribution of effects in the same surface rather than from convenience, report the posterior probability alongside the posterior interval, and check the decision's sensitivity to a flatter and a tighter prior.

The reason for that specificity is a failure I have seen: A team quoted a 94% probability of improvement from a strongly optimistic prior fitted on shipped experiments only, which had already selected for wins, and the reported probability barely moved when the data were removed entirely.

Prior sensitivity on the same likelihood.

Prior on relative lift	Posterior mean	P(effect > 0)
Flat	+1.4%	0.86
Historical, centred at 0 with sd 2%	+0.9%	0.79
Optimistic, centred at +3%	+2.4%	0.94

I would not consider it settled without evidence: Report the posterior under the chosen prior and under a deliberately flat one; a large gap means the conclusion is being carried by the prior.

A posterior with an undisclosed prior is an opinion with an interval attached.

The first thing I would pin down about equivalence and non-inferiority testing is which question the number is supposed to answer.

Failing to reject a difference is compatible with a large difference the study could not detect, so demonstrating similarity requires testing against a stated margin rather than against zero.

Concretely, agree the largest regression that would still be acceptable, size the study to detect that margin, and conclude non-inferiority only when the confidence interval for the difference lies entirely on the acceptable side of it.

The reason for that specificity is a failure I have seen: A vendor swap was approved on p = 0.44 from a study whose interval ran from -8% to +3%, and the eventual regression was 5% — well inside what the study had never ruled out.

Three results against a -2% non-inferiority margin.

Study	Difference	95% interval	Non-inferior
A	-0.3%	-1.2% to +0.6%	yes
B	-0.3%	-8.0% to +7.4%	undetermined, too small
C	-3.1%	-4.0% to -2.2%	no

I would not consider it settled without evidence: Compare the confidence interval against the agreed margin explicitly in the readout, rather than reporting the difference's p-value against zero.

Proving similarity means excluding the difference that matters, not failing to find one.

I would start by writing the estimand for clustered standard errors in a sentence before choosing any estimator.

Standard errors computed as though correlated rows were independent understate uncertainty by roughly the square root of the design effect, which turns noise into significance at a rate that grows with the rows per cluster.

Concretely, cluster standard errors at the assignment unit, or aggregate to one row per unit before testing, and estimate the intra-cluster correlation so the design effect can be carried into future sample size calculations.

The reason for that specificity is a failure I have seen: A page-view-level analysis reported p = 0.001 on a change that a user-clustered analysis put at p = 0.31, and the design effect from 18 views per user explained the entire gap.

The same comparison at three units of analysis.

Analysis	Rows	Standard error	p
Page-view level, unclustered	180,000	0.0021	0.001
Page-view level, clustered by user	180,000	0.0068	0.31
Aggregated to one row per user	10,000	0.0069	0.32

I would not consider it settled without evidence: Report the intra-cluster correlation and the implied design effect alongside the result, so the inflation is visible rather than assumed away.

Ten thousand rows from five hundred users is five hundred observations wearing a costume.

This is a place where the sample answer and the population answer for uncertainty for a ratio of two random quantities come apart.

A ratio of two random sums has no simple variance, and treating the denominator as fixed understates uncertainty, so the interval must account for the variability of both terms and their correlation.

Concretely, use the delta method with the covariance between numerator and denominator, or bootstrap at the unit of randomisation, and prefer the bootstrap when the denominator is small enough that its variability is a meaningful share of its size.

The reason for that specificity is a failure I have seen: A revenue-per-session interval computed with the denominator held fixed was roughly 30% too narrow, and two experiments were declared significant that a correct interval left inconclusive.

Delta method with the covariance term retained.

import numpy as np

def ratio_se(num, den):
    n = len(num)
    r = num.sum() / den.sum()
    mu_d = den.mean()
    var = np.var(num - r * den, ddof=1) / (n * mu_d**2)   # keeps cov(num, den)
    return r, np.sqrt(var)

I would not consider it settled without evidence: Compare the delta-method interval against a cluster bootstrap; a systematic gap indicates the covariance term was dropped.

Both halves of the ratio are estimates, and the interval has to know that.

My answer to sequential testing with always-valid inference begins with how the data came to exist, because sampling contaminates everything after it.

An interval that is valid at every moment must be wider than a fixed-horizon one, because it has to hold under any stopping rule including one chosen adversarially after looking.

Concretely, publish a confidence sequence or a group sequential boundary rather than a fixed-horizon interval, tell readers the cost is a wider interval early on, and keep the fixed-horizon estimate as the final summary once the planned end is reached.

The reason for that specificity is a failure I have seen: A dashboard showing nominal 95% intervals updated hourly produced eleven "significant" launches in a quarter, four of which reversed in the holdout, because acting at any moment on a fixed-horizon interval is the peeking problem with a nicer interface.

Interval width, fixed-horizon versus always-valid, on the same data.

Day	Fixed-horizon 95% width	Always-valid 95% width
2	6.1 pp	14.8 pp
7	3.2 pp	5.6 pp
14	2.3 pp	3.4 pp
28	1.6 pp	2.2 pp

I would not consider it settled without evidence: Simulate the dashboard's own stopping behaviour under a null effect and report the realised error rate for that behaviour rather than the nominal one.

If people will look continuously, the interval has to be built for continuous looking.

I would treat base rates in a conditional probability question as a design question first and a computation question second.

The answer is dominated by the base rate rather than by the accuracy figures, because the far larger healthy population produces more false positives than the small affected population produces true ones.

Concretely, work in expected counts on a concrete population rather than in conditional probabilities, put true positives and false positives side by side, and read the answer as one share of the other.

The reason for that specificity is a failure I have seen: A fraud team set a review threshold from a 99% accuracy figure and buried its analysts, because at a 0.1% fraud rate roughly 91 of every 100 flagged cases were legitimate customers.

One million people, counted rather than reasoned about.

Group	People	Test positive
Has the condition	1,000	990
Does not	999,000	9,990
Total positive		10,980
Share truly affected		990 / 10,980 = 9.0%

I would not consider it settled without evidence: Restate any accuracy claim as expected counts on the actual population before it is used to set a threshold or staff a queue.

Accuracy figures say nothing useful until the base rate is in the room.

The useful framing for updating a belief with Bayes' theorem is what decision changes if the estimate moves by its own uncertainty.

The posterior is proportional to the prior times the likelihood, and for a binomial outcome with a beta prior the update is arithmetic on the prior's counts rather than an integral.

Concretely, express the prior as pseudo-counts of successes and failures, add the observed successes and failures, and read the posterior mean and interval from the resulting beta distribution rather than from the raw sample proportion.

The reason for that specificity is a failure I have seen: A team read a 4.0% rate from 300 visitors as a firm estimate and re-planned spend on it; the beta posterior's 95% interval ran from 2.1% to 6.7%, which spanned both the "increase spend" and "cut spend" decisions.

Beta-binomial update, prior counts plus observed counts.

from scipy.stats import beta

a0, b0 = 8, 192          # prior: 4% from 200 pseudo-observations
a, b = a0 + 12, b0 + 288 # observed: 12 conversions in 300 visitors
print(f"mean {a / (a + b):.4f}  95% CI {beta.ppf([0.025, 0.975], a, b).round(4)}")

I would not consider it settled without evidence: Report the posterior interval alongside the point estimate and check that the recommended action is the same at both ends.

A sample proportion from 300 visitors is a wide distribution, not a number.

I would settle expectation and variance of a sum against the simplest defensible comparison, so any extra structure has to earn its place.

Expectations always add, and variances add only when the terms are uncorrelated, so a positive correlation between components makes the sum more variable than the parts suggest.

Concretely, estimate the covariance rather than assuming it away, include the cross term when propagating uncertainty into a composite metric, and check the sign of the correlation because it decides whether combining components damps or amplifies noise.

The reason for that specificity is a failure I have seen: A composite health score treated its four components as independent and reported an interval roughly 25% too narrow, because three of the components moved together during incidents — exactly the periods the score existed to flag.

The cross term is the whole difference here.

Quantity	Value
Var(A)	4.0
Var(B)	9.0
Corr(A, B)	0.6
Var(A + B) assuming independence	13.0
Var(A + B) with 2 x 0.6 x 2 x 3	20.2

I would not consider it settled without evidence: Compare the analytically propagated interval against a bootstrap of the composite; a gap says a covariance term was dropped.

Independence is an assumption about the data, not a property of addition.

Most of the judgement in the law of large numbers against the central limit theorem sits in what the comparison group is, not in the arithmetic.

The law of large numbers says the sample mean converges to the population mean, while the central limit theorem says the sampling distribution of that mean approaches normality — neither says the data itself becomes normal.

Concretely, keep the two claims separate when justifying a method: use the central limit theorem to defend an interval on a mean, and check the tail behaviour of the data directly when the method depends on the data's own distribution, such as a quantile or a threshold rule.

The reason for that specificity is a failure I have seen: A capacity threshold set at "mean plus two standard deviations" on a log-normal latency distribution was exceeded on 6% of requests rather than the assumed 2.3%, because the rule assumed normality of the data rather than of its mean.

Normal rule against the empirical quantiles of a right-skewed metric.

Quantile	Normal rule (mean + k sd)	Empirical
95th	412 ms	508 ms
97.7th	486 ms	690 ms
99th	545 ms	1,120 ms

I would not consider it settled without evidence: Compare the empirical quantiles of the raw metric against the normal quantiles implied by its mean and standard deviation before any rule uses those parameters.

A large sample fixes the mean's uncertainty and does nothing to the shape of the data.

Where analyses go wrong on when a binomial model stops fitting is usually one step upstream of the statistic being argued about.

The binomial assumes a fixed number of independent trials with a constant success probability, and real user data usually violates the last two through clustering and day-to-day variation in the underlying rate.

Concretely, compare the observed variance against the binomial variance the model implies, and when the observed variance is larger, move to a model that carries the extra dispersion — beta-binomial, or a quasi-likelihood with an estimated dispersion parameter.

The reason for that specificity is a failure I have seen: Daily signup control limits set from binomial variance flagged alerts on 14 days of a 30-day month, because the true day-to-day rate varied with marketing spend and the model had no term for it.

Observed variance against what the binomial allows.

Period	Mean daily signups	Binomial variance	Observed variance	Ratio
Stable month	420	418	455	1.09
Campaign month	610	606	3,120	5.15

I would not consider it settled without evidence: Compute the ratio of observed to model variance; a ratio well above one is overdispersion and the intervals from the simple model are too narrow.

Overdispersion is the data telling you the success probability is not one number.

I would answer the Poisson model for counts and its failure mode by separating what the design identifies from what the data merely displays.

Poisson holds when events arrive independently at a constant rate, which fixes the variance equal to the mean, and it fails whenever arrivals cluster or the rate itself varies.

Concretely, check the variance-to-mean ratio on historical counts, use Poisson when it is near one, and switch to a negative binomial when it is materially above one so that intervals and alert thresholds carry the real dispersion.

The reason for that specificity is a failure I have seen: An alert threshold set from Poisson variance on a mean of 40 fired weekly, because incidents generate correlated bursts of tickets and the real variance was over seven times the mean.

Poisson and negative binomial thresholds on the same history.

Model	Fitted dispersion	99th percentile daily count
Poisson (mean 40)	1.0 by assumption	55
Negative binomial	7.5	90
Observed 99th percentile		89

I would not consider it settled without evidence: Plot the variance-to-mean ratio by week and check whether it is stable near one before committing a threshold to the Poisson assumption.

Variance equal to the mean is the assumption, and it is checkable.

The substance of when the normality assumption actually matters is the size of the effect and its uncertainty, not whether a threshold was crossed.

Methods that make claims about the sampling distribution of a mean rely on normality only asymptotically, while methods that make claims about individual observations — prediction intervals, quantile rules, control limits — depend on the data's own shape directly.

Concretely, classify each method by whether its output is a statement about an average or about a single unit, apply the central limit theorem's protection only to the first class, and check the empirical distribution before using anything in the second.

The reason for that specificity is a failure I have seen: A team justified a normal-based prediction interval for individual delivery times with "we have plenty of data", and the interval covered 71% of deliveries rather than the stated 95% because the distribution was right-skewed.

Nominal against realised coverage on held-out data.

Statement	Depends on data normality	Realised coverage
95% interval for the mean delivery time	no, asymptotically	94.6%
95% prediction interval for one delivery	yes	71.2%
95% empirical quantile interval for one delivery	no	94.9%

I would not consider it settled without evidence: Compute the empirical coverage of any interval that describes individual units on held-out data, rather than trusting its nominal level.

Large samples protect the mean, not the individual observation.

Before running anything I would state what result would make me abandon the hypothesis behind memorylessness and waiting times.

The exponential distribution is memoryless, so the remaining wait has the same distribution as the original one and the expected wait from any point is still the full mean.

Concretely, test memorylessness before relying on it by comparing the hazard rate across elapsed-time buckets, and when the hazard rises or falls with age, use a Weibull or an empirical hazard rather than assuming a constant rate.

The reason for that specificity is a failure I have seen: A retry policy built on memorylessness kept retrying a queue whose failures were bursty; the real hazard fell sharply with elapsed time, so the policy retried hardest exactly when success was least likely and tripled load during incidents.

Empirical hazard by elapsed time, which should be flat if exponential.

Elapsed	Hazard per minute (queue A)	Hazard per minute (queue B)
0-5 min	0.102	0.180
5-10 min	0.098	0.091
10-20 min	0.101	0.038
Verdict	exponential is reasonable	not memoryless

I would not consider it settled without evidence: Estimate the empirical hazard by elapsed-time bucket; a flat hazard supports the exponential model and a sloped one refutes it.

Memorylessness is a strong claim, and the hazard curve is where it is checked.

The first thing I would pin down about modelling revenue on a log scale is which question the number is supposed to answer.

A model on the log scale estimates multiplicative effects on the median-like centre of the distribution, which is a different quantity from the additive effect on the mean that the business usually books.

Concretely, decide which quantity the decision needs, and if it is total revenue, either model the untransformed outcome with a robust estimator or transform back with the correction the log-normal implies rather than exponentiating the coefficient alone.

The reason for that specificity is a failure I have seen: A team exponentiated a log-scale coefficient and reported an 8.0% revenue lift; the mean-scale effect was 2.7% because the residual variance differed between arms and the naive back-transformation has no term for that, and the revenue forecast was overstated by roughly $600,000.

Back-transforming a log-scale effect to the mean scale.

import numpy as np

beta = 0.0770                            # effect on the log scale
var_treated, var_control = 0.80, 0.90    # residual variance is not the same in both arms

# Exponentiating alone is the ratio of medians, and the mean-scale ratio only
# equals it when the two arms share a residual variance.
print(f"median ratio: {np.exp(beta) - 1:.1%}")                                    # 8.0%
print(f"mean ratio:   {np.exp(beta + (var_treated - var_control) / 2) - 1:.1%}")  # 2.7%

I would not consider it settled without evidence: Compare the model's implied total against the observed total on held-out data; a systematic shortfall reveals a missing back-transformation correction.

The average of the logs is not the log of the average, and finance books the second one.

I would start by writing the estimand for correlation against covariance and what neither shows in a sentence before choosing any estimator.

Correlation measures linear association only, so a strong non-monotonic relationship can sit at essentially zero correlation, and a scatter plot answers a question the coefficient cannot.

Concretely, plot the relationship before quoting a coefficient, use a rank correlation when the relationship is monotonic but curved, and use a mutual-information or binned-comparison measure when it may be non-monotonic.

The reason for that specificity is a failure I have seen: A pricing analysis dropped a feature at correlation 0.02 that had a clear inverted-U relationship with conversion, and the eventual model that included it as a quadratic term improved holdout accuracy by four points.

Outcome by decile of a feature with correlation 0.02.

Decile of discount depth	Conversion rate
1 (smallest)	2.1%
4	5.6%
6	6.0%
8	4.2%
10 (largest)	2.3%

I would not consider it settled without evidence: Bin the predictor into deciles and plot the outcome mean per decile, which shows shape that a single coefficient averages away.

Zero correlation rules out a line, not a relationship.

This is a place where the sample answer and the population answer for answering a probability question by simulation come apart.

A simulation that reproduces the generating process answers the question directly, and it also documents the assumptions in code where they can be inspected and argued with.

Concretely, state the process, sample it many times, compute the event's frequency, and report a Monte Carlo standard error so the reader knows how many digits of the answer are real.

The reason for that specificity is a failure I have seen: An answer quoted to four decimal places from 1,000 simulations was wrong in the second decimal, because nobody had computed the Monte Carlo error of roughly 1.5 percentage points at that draw count.

The estimate and the Monte Carlo error that decides its digits.

import numpy as np

rng = np.random.default_rng(0)
draws = 200_000
hits = (rng.random((draws, 3)).max(axis=1) > 0.9).mean()
se = np.sqrt(hits * (1 - hits) / draws)
print(f"P = {hits:.4f} +/- {1.96 * se:.4f}")

I would not consider it settled without evidence: Report the simulation's own standard error and increase draws until it is smaller than the precision being claimed.

A simulated answer needs its own error bar before it is quoted.

My answer to collision probabilities and why they surprise people begins with how the data came to exist, because sampling contaminates everything after it.

The chance of at least one collision grows with the number of pairs rather than with the number of items, so it becomes likely at roughly the square root of the space rather than near its size.

Concretely, compute the pair count, approximate the collision probability as one minus the exponential of the negative pair count divided by the space size, and size identifiers from the volume the system will actually reach rather than from its current volume.

The reason for that specificity is a failure I have seen: A 32-bit request identifier was chosen for a system doing 50,000 events a day, which passes the 77,000 identifiers that make a collision an even bet inside two days, and it merged two customers' traces in an audit export within the first week.

Collision probability against volume for a 32-bit space.

Identifiers issued	Approximate P(at least one collision)
10,000	1.2%
77,000	50%
200,000	99%
1,000,000	above 99.99%

I would not consider it settled without evidence: Compute the collision probability at the projected three-year volume before choosing an identifier width, and re-check it whenever volume forecasts change.

Collisions arrive at the square root of the space, which is much sooner than it feels.

I would treat interpreting a linear regression coefficient as a design question first and a computation question second.

The coefficient is the expected change in the outcome for a one-unit change in that predictor with the other included predictors held fixed, which is a within-model statement and not a claim about what happens if you change tenure.

Concretely, report the coefficient with its units, its interval, and the range of the predictor over which the data actually supports it, and describe it as an association unless the design or a stated identification argument makes it causal.

The reason for that specificity is a failure I have seen: A "tenure is worth $4.20 a month" claim drove a retention incentive that produced nothing, because the coefficient reflected who stays rather than what staying causes and the model contained no exogenous variation in tenure.

The same coefficient, three defensible readings and one that is not.

Statement	Supported
"Among customers on the same plan, each extra month is associated with $4.20 more spend."	yes
"The interval is $3.10 to $5.30 over tenures of 1 to 36 months."	yes
"Beyond 36 months the same slope applies."	no, out of support
"Extending a customer's tenure by a month yields $4.20."	no, not identified

I would not consider it settled without evidence: State the counterfactual the coefficient would need to support and check whether the data contains variation that could identify it.

Holding other variables fixed in a model is not the same as holding them fixed in the world.

The useful framing for omitted variable bias is what decision changes if the estimate moves by its own uncertainty.

A coefficient absorbs the effect of anything correlated with its predictor and with the outcome that the model leaves out, so the estimate changes when the omitted variable is included because the first estimate was answering a different question.

Concretely, decide the adjustment set from a stated causal diagram before fitting, include what the diagram says to include, and avoid conditioning on variables that lie on the causal path or that are common effects of the predictor and outcome.

The reason for that specificity is a failure I have seen: An analysis reported that customers using a support chat spend less, then flipped sign once account size was added; the original number was mostly the fact that small accounts use chat, and a plan to reduce chat availability had already been drafted.

The coefficient on chat usage across nested specifications.

Model	Chat coefficient	Interpretation
Spend ~ chat	-$38	mostly account size
+ account size	+$12	sign flips
+ account size + tenure	+$9	stable
+ account size + tenure + region	+$9	stable

I would not consider it settled without evidence: Show the coefficient across a stated sequence of nested models so the reader sees which control moves it and by how much.

A coefficient is defined by the model around it, so report the model, not the number.

I would settle multicollinearity against the simplest defensible comparison, so any extra structure has to earn its place.

Collinearity inflates the standard errors of the affected coefficients without biasing them or harming prediction, so it is a problem for interpreting individual coefficients and not for the fitted values.

Concretely, decide first whether the model is for prediction or for interpretation; leave collinear predictors alone when predicting, and when interpreting, combine them into one construct, drop one on subject-matter grounds, or use a penalised fit and report the pair jointly rather than separately.

The reason for that specificity is a failure I have seen: A team dropped a predictor because its variance inflation factor was 21 and reported the survivor's coefficient as the effect of the construct, which doubled the apparent effect because the dropped variable's contribution was now attributed to it.

Individually insignificant, jointly significant.

Term	Coefficient	Standard error	p
Sessions	0.42	0.31	0.18
Page views	0.19	0.14	0.17
Joint F test on both			0.002

I would not consider it settled without evidence: Report the joint test on the collinear set alongside the individual coefficients, since the pair is often significant when neither member is.

Collinearity makes coefficients uncertain, and dropping one makes the other wrong.

Most of the judgement in heteroskedasticity sits in what the comparison group is, not in the arithmetic.

Non-constant error variance leaves the coefficient estimates unbiased and makes the default standard errors wrong, so the point estimates survive and every interval and p-value built from them does not.

Concretely, use heteroskedasticity-robust standard errors as the default rather than as a remedy, and consider modelling the variance directly or transforming the outcome when the pattern is strong enough that prediction intervals also matter.

The reason for that specificity is a failure I have seen: A spend model with strongly fanning residuals reported a significant coefficient at p = 0.03 that moved to p = 0.21 under robust standard errors, and a budget had already been reallocated on the first number.

Classical against robust standard errors on the same fit.

Term	Coefficient	Classical SE	Robust SE	p (classical)	p (robust)
Discount depth	1.84	0.85	1.47	0.03	0.21
Tenure	0.42	0.09	0.10	0.000	0.000

I would not consider it settled without evidence: Report both the classical and the robust standard errors; a large difference is itself the diagnostic.

The estimates are fine and the inference is not, which is the easy half to miss.

Where analyses go wrong on reading logistic regression coefficients is usually one step upstream of the statistic being argued about.

A logistic coefficient is a change in log odds, so it converts to an odds ratio of about 2.0 and to a probability change that depends entirely on the baseline rate the unit starts from.

Concretely, exponentiate for the odds ratio, then convert to marginal effects at representative baseline rates, and report the probability change at the baselines that actually occur in the population rather than at a single average.

The reason for that specificity is a failure I have seen: An odds ratio of 2.0 was reported to a product team as "doubles conversion"; at the 40% baseline of the segment in question the actual increase was 17 percentage points, not 40, and the forecast was overstated by more than double.

One odds ratio, four baselines, four different product statements.

Baseline probability	Odds	Odds x 2.0	New probability	Change
1%	0.0101	0.0202	1.98%	+0.98 pp
10%	0.111	0.222	18.2%	+8.2 pp
40%	0.667	1.333	57.1%	+17.1 pp
80%	4.00	8.00	88.9%	+8.9 pp

I would not consider it settled without evidence: Report average marginal effects computed over the observed covariate distribution, alongside the odds ratio.

Doubling the odds doubles the probability only when the probability is small.

I would answer regularisation when the goal is interpretation by separating what the design identifies from what the data merely displays.

A penalty selects a set of predictors that predicts well together, and among correlated predictors the choice between them is close to arbitrary, so the survivors are a prediction basis rather than a ranking of importance.

Concretely, report selection stability by refitting across bootstrap resamples and quoting how often each predictor is retained, and when the question is genuinely about importance, use an approach that acknowledges correlated groups rather than a single penalised fit.

The reason for that specificity is a failure I have seen: A "top five drivers" list from one penalised fit was used to set team priorities; refitting on bootstrap resamples retained three of the five in fewer than half of the runs, and two dropped predictors were near-duplicates of retained ones.

Selection stability across 500 bootstrap refits.

Predictor	In the reported fit	Retained in resamples
Sessions per week	yes	98%
Support contacts	yes	91%
Feature A adoption	yes	46%
Feature B adoption	no	44%
Discount depth	yes	38%

I would not consider it settled without evidence: Report the selection frequency for every predictor across at least a few hundred resamples rather than the membership of one fit.

A penalty picks a basis, and a basis is not a ranking.

The substance of residual diagnostics is the size of the effect and its uncertainty, not whether a threshold was crossed.

Each diagnostic targets a specific assumption, so the value of looking is knowing which one a pattern refutes rather than forming a general impression that the fit looks reasonable.

Concretely, plot residuals against fitted values for functional form and variance, against each predictor for missing curvature, against time or cluster for dependence, and check leverage and influence to see whether a handful of rows are determining the fit.

The reason for that specificity is a failure I have seen: A model passed on its overall fit statistic while three of 4,000 rows carried enough influence to set the sign of the main coefficient, which only appeared when leverage was plotted after the result had already been circulated.

Each plot answers one question.

Diagnostic	Assumption it tests	Failure looks like
Residuals vs fitted	Functional form, constant variance	Curvature, fanning
Residuals vs each predictor	Missing non-linearity	Systematic bend
Residuals vs time or cluster	Independence	Runs, autocorrelation
Cook's distance	No single row dominating	One or two large spikes

I would not consider it settled without evidence: Refit with the highest-influence observations removed and report whether the conclusion changes, rather than reporting only the full-sample fit.

A fit statistic summarises agreement and hides which rows produced it.

Before running anything I would state what result would make me abandon the hypothesis behind interaction terms.

An interaction states that one predictor's slope depends on another, so it is the correct form when the question is about a difference in effects and it changes what every constituent coefficient means.

Concretely, include both main effects with any interaction, centre continuous predictors so the main effects remain interpretable at a meaningful reference point, test the interaction term itself rather than comparing two subgroup fits, and check the interaction is supported by data across the relevant range.

The reason for that specificity is a failure I have seen: A team reported an interaction as significant while the cell it depended on held 41 observations, and the effect did not reproduce; the readout had never shown the cell sizes.

The cell counts an interaction actually rests on.

	New users	Returning users
Control	18,400	42,900
Treatment	41	43,100

I would not consider it settled without evidence: Report the cell counts underlying an interaction alongside its coefficient, so a claim resting on a thin cell is visible.

An interaction is a claim about a difference, and differences need more data than levels.

The first thing I would pin down about fixed effects is which question the number is supposed to answer.

Fixed effects absorb every time-invariant difference between units, which removes confounding from unobserved stable characteristics and simultaneously removes the ability to estimate any predictor that does not vary within a unit.

Concretely, use fixed effects when identification comes from within-unit variation over time, check there is enough of that variation to estimate the coefficient at all, and cluster standard errors at the fixed-effect level.

The reason for that specificity is a failure I have seen: A model with store fixed effects reported a near-zero coefficient on store format, which was constant per store, so the term was absorbed entirely and the reported null was a mechanical artefact of the specification.

Within-unit variation decides what the specification can see.

Predictor	Between-store variance	Within-store variance	Estimable with store fixed effects
Store format	0.61	0.00	no
Weekly promotion depth	0.22	0.34	yes
Staffing hours	0.18	0.29	yes

I would not consider it settled without evidence: Report within-unit variation for each predictor of interest; a predictor with none cannot be identified in this specification.

Fixed effects trade unobserved confounders for the variables that do not move.

I would start by writing the estimand for autocorrelation in a time series in a sentence before choosing any estimator.

Correlated errors leave the coefficients roughly unbiased and make the standard errors far too small, so significance is manufactured by the fact that each day carries much less new information than a row count suggests.

Concretely, test the residuals for serial correlation, use Newey-West or other autocorrelation-robust standard errors, and model the dependence directly when forecasting rather than treating it as a nuisance.

The reason for that specificity is a failure I have seen: A weekly report claimed a significant upward trend from a regression on 400 days whose residuals had a first-order autocorrelation of 0.86; the effective sample was closer to 30 independent observations and the trend was not distinguishable from noise.

Effective sample size falls fast as autocorrelation rises.

Residual autocorrelation	Rows	Approximate effective sample
0.0	400	400
0.5	400	133
0.86	400	30

I would not consider it settled without evidence: Compute the effective sample size implied by the residual autocorrelation and compare it against the row count before quoting any interval.

Four hundred correlated days is not four hundred observations.

This is a place where the sample answer and the population answer for evaluating a forecast come apart.

A forecast is judged by backtesting against the decisions it feeds, using errors computed only from information available at the time of each forecast and compared against a naive baseline that costs nothing.

Concretely, roll the origin forward through history, forecast at each origin using only prior data, score with an error measure matched to the cost function, and report the ratio against a seasonal naive baseline rather than the absolute error alone.

The reason for that specificity is a failure I have seen: A forecast reported at 8% mean absolute percentage error was adopted over a seasonal naive baseline that scored 7.4% on the same backtest, and the added pipeline cost a quarter of engineering time for a worse forecast.

Rolling-origin backtest against the free baseline.

Origin	Seasonal naive MAPE	Model MAPE
2026-01	7.9%	8.4%
2026-03	7.1%	8.0%
2026-05	7.2%	7.6%
Mean	7.4%	8.0%

I would not consider it settled without evidence: Publish the baseline's score alongside the model's on identical folds, since an error figure without a baseline says nothing about value.

A forecast that cannot beat last year's same week has not earned its pipeline.

My answer to seasonality and calendar effects begins with how the data came to exist, because sampling contaminates everything after it.

A week-over-week comparison confounds the change with the calendar, so the meaningful comparison is against the same point in previous cycles with holidays and trading-day counts accounted for.

Concretely, compare against the same week in prior years adjusted for trend, hold out known calendar events explicitly rather than letting them contaminate the seasonal estimate, and count trading days when the metric depends on them.

The reason for that specificity is a failure I have seen: A 30% drop triggered an incident review that consumed two days; the same week had dropped 28% and 31% in the two prior years because it followed a public holiday, and the calendar had never been in the report.

The same week across years, which is the comparison that carries information.

Year	Week 27 vs week 26	Note
2024	-28%	follows public holiday
2025	-31%	follows public holiday
2026	-30%	follows public holiday

I would not consider it settled without evidence: Plot the metric against the same weeks of prior years before opening any investigation into a periodic movement.

Week over week compares two different weeks of the year, which is most of what it measures.

I would treat choosing a classification threshold as a design question first and a computation question second.

The threshold follows from the cost of each error type and the capacity of whatever acts on the positives, not from 0.5, which is only optimal when the two errors cost the same and the classes are balanced.

Concretely, write the cost of a false positive and a false negative in one unit, sweep the threshold over the validation set computing expected cost at each point, choose the minimum, and re-check the choice when either the base rate or the downstream capacity changes.

The reason for that specificity is a failure I have seen: A default cutoff of 0.5 on a queue with 40 analysts produced 900 daily alerts against a capacity of 300, and the effective policy became whichever third of the alerts happened to be looked at first.

Threshold sweep with a false negative priced at 12 times a false positive.

Threshold	Alerts per day	False negatives	Expected daily cost
0.20	1,410	12	$1,554
0.35	620	21	$872
0.50	300	44	$828
0.65	140	96	$1,292

I would not consider it settled without evidence: Report expected cost and alert volume across the threshold sweep, so the chosen point is visibly the minimum under stated capacity.

The cutoff is a business decision the model cannot make for you.

The useful framing for calibration of predicted probabilities is what decision changes if the estimate moves by its own uncertainty.

Ranking quality and calibration are different properties, so a model can order cases correctly while its scores systematically overstate or understate the true rate, which breaks any decision that multiplies the score by a value.

Concretely, bin predictions and plot observed frequency against predicted probability, fit a monotone recalibration such as isotonic or Platt scaling on held-out data, and re-check calibration by segment because a globally calibrated model can be badly miscalibrated within groups.

The reason for that specificity is a failure I have seen: An expected-value ranking multiplied uncalibrated scores by order size; scores near 0.9 corresponded to a true rate of 0.62, and the queue spent its capacity on cases whose expected value had been overstated by nearly half.

Reliability table on held-out data before recalibration.

Predicted band	Cases	Observed rate
0.0-0.2	41,200	0.04
0.4-0.6	6,800	0.37
0.8-1.0	2,100	0.62

I would not consider it settled without evidence: Report a reliability table on held-out data overall and per major segment, rather than only an area-under-curve figure.

Rank order is enough to sort and not enough to multiply.

I would settle class imbalance and resampling against the simplest defensible comparison, so any extra structure has to earn its place.

Resampling changes the base rate the model learns, which shifts its predicted probabilities away from the population rate, so it can help optimisation while breaking calibration unless the shift is corrected.

Concretely, prefer class weights or a threshold chosen on the true distribution, and if resampling is used for tractability, correct the intercept back to the population base rate and validate on data with the original class balance.

The reason for that specificity is a failure I have seen: A model trained on a rebalanced 50/50 sample was evaluated on the same rebalanced data and reported 0.94 precision; on live traffic at the true 0.5% rate, precision was 0.08 and the review queue was swamped.

The same model, two evaluation sets.

Evaluation set	Base rate	Precision at recall 0.7
Rebalanced sample	50%	0.94
Held-out production sample	0.5%	0.08

I would not consider it settled without evidence: Always evaluate on a sample with the production base rate, whatever the training sample looked like.

Rebalancing is a training convenience, and evaluation has to happen at the real rate.

Most of the judgement in joins that silently multiply rows sits in what the comparison group is, not in the arithmetic.

A join on a key that is not unique on the joined side produces one output row per match, so any aggregate computed after that join counts the same fact once per duplicate.

Concretely, assert the cardinality you expect before joining — verify the key is unique on the dimension side — aggregate to the grain you need before joining rather than after, and compare row counts before and after every join as a matter of routine.

The reason for that specificity is a failure I have seen: A revenue report joined orders to a customer table that held one row per customer per address change, which multiplied every order by that customer's address history and overstated the quarter by 34%.

The check that belongs before the join, not after the number is wrong.

-- Fails loudly if the dimension is not one row per key.
select customer_id, count(*) as rows
  from dim_customer
 group by customer_id
having count(*) > 1
 limit 10;

-- Or aggregate to the grain first, then join.
with orders_by_customer as (
  select customer_id, sum(amount) as revenue from orders group by customer_id
)
select c.region, sum(o.revenue)
  from orders_by_customer o join dim_customer_current c using (customer_id)
 group by c.region;

I would not consider it settled without evidence: Run a uniqueness check on the join key against the dimension table, and compare the fact table's row count before and after the join.

A join is a filter and a multiplier, and the second one is silent.

Where analyses go wrong on NULL semantics in filters and aggregates is usually one step upstream of the statistic being argued about.

A comparison against NULL evaluates to unknown rather than true, so rows with a NULL in the compared column fail the filter, while aggregate functions silently skip NULLs instead of failing.

Concretely, use IS DISTINCT FROM or an explicit COALESCE when a NULL should be treated as a value, count NULLs per column before writing any filter on it, and use COUNT(column) against COUNT(*) deliberately rather than interchangeably.

The reason for that specificity is a failure I have seen: A churn report excluded 4,100 accounts whose status was NULL because the filter was written as a not-equals comparison, and the reported churn rate was 1.8 points lower than the truth for two quarters.

Four predicates, three different row counts.

select
  count(*)                                                as all_rows,        -- 50,000
  count(*) filter (where status <> 'cancelled')           as not_equal,       -- 41,900
  count(*) filter (where status is distinct from 'cancelled') as distinct_from, -- 46,000
  count(status)                                           as non_null_status  -- 45,900
from accounts;

I would not consider it settled without evidence: Count rows before and after each predicate and reconcile the difference against an explicit NULL count on the columns involved.

NULL is not a value, and every comparison with it quietly disagrees.

I would answer window functions for running and ranked metrics by separating what the design identifies from what the data merely displays.

A window function computes over a set of rows related to the current row while keeping every row, which is exactly what a per-row rank or a running total needs and what GROUP BY cannot do.

Concretely, partition by the entity, order by the sequencing column with a deterministic tiebreak, choose the frame explicitly rather than relying on the default, and remember that ROW_NUMBER, RANK and DENSE_RANK differ precisely where ties occur.

The reason for that specificity is a failure I have seen: A running total using the default frame with a non-unique ORDER BY produced different values on each run, and two monthly reports built from the same table disagreed by roughly $9,000.

Explicit frame, deterministic ordering.

select
  customer_id,
  ordered_at,
  row_number() over w                                   as order_seq,
  sum(amount) over (w rows between unbounded preceding and current row) as running_total
from orders
window w as (partition by customer_id order by ordered_at, order_id);

I would not consider it settled without evidence: Add a unique tiebreak to every window ORDER BY and confirm the query returns identical output across repeated runs.

An ambiguous window ordering makes the result depend on the plan.

The substance of building a retention cohort table is the size of the effect and its uncertainty, not whether a threshold was crossed.

A retention table is defined by three choices — what the cohort is keyed on, what counts as returning, and whether week N means exactly that week or any time since — and different choices produce numbers that look comparable and are not.

Concretely, fix the cohort key to signup week, define the return event and its window explicitly, decide between bounded and cumulative retention and label the table with the choice, and exclude cohorts too recent to have completed the window rather than showing them as low.

The reason for that specificity is a failure I have seen: Two teams reported 41% and 63% week-four retention for the same product; one measured activity in week four exactly and the other any activity since signup, and both charts were titled "week 4 retention".

Bounded week-N retention, with incomplete cohorts excluded.

with cohorts as (
  select user_id, date_trunc('week', signed_up_at) as cohort_week from users
),
activity as (
  select c.cohort_week,
         floor(extract(epoch from e.occurred_at - c.cohort_week) / 604800)::int as week_n,
         e.user_id
    from cohorts c join events e using (user_id)
)
select cohort_week, week_n, count(distinct user_id) as retained
  from activity
 where week_n between 0 and 8
   and cohort_week <= current_date - interval '9 weeks'
 group by 1, 2;

I would not consider it settled without evidence: Publish the definition next to the table, and reproduce a competing team's number under their definition before disputing it.

Most retention arguments are definition arguments wearing a chart.

Before running anything I would state what result would make me abandon the hypothesis behind sessionising an event stream.

A session boundary is a gap larger than a stated threshold between consecutive events for the same user, which makes it a running comparison against the previous row rather than a grouping key that exists in the data.

Concretely, order events per user, compute the gap to the previous event with LAG, mark a new session where the gap exceeds the threshold, and take a running sum of those marks as the session identifier.

The reason for that specificity is a failure I have seen: A session definition applied per device rather than per user split one user's cross-device journey into three sessions, which inflated session counts by 22% and made average session length look like a regression.

Gap-based sessionisation in one pass.

select user_id, occurred_at,
       sum(is_new_session) over (partition by user_id order by occurred_at) as session_id
  from (
    select user_id, occurred_at,
           case when occurred_at - lag(occurred_at) over (partition by user_id order by occurred_at)
                     > interval '30 minutes'
                  or lag(occurred_at) over (partition by user_id order by occurred_at) is null
                then 1 else 0 end as is_new_session
      from events
  ) marked;

I would not consider it settled without evidence: Reconcile total events against the sum of events per session and check the distribution of session lengths against a known set of hand-traced journeys.

Sessions are constructed, so the construction has to be written down.

The first thing I would pin down about deduplicating an event stream is which question the number is supposed to answer.

Correct counting requires a key that identifies the same logical event across retries, so the deduplication has to happen against an idempotency key rather than against the whole row, which differs on timestamps and metadata.

Concretely, deduplicate on the producer-supplied event identifier within a window at least as long as the retry horizon, keep the first or last occurrence by a stated rule, and monitor the duplicate rate so a change in it is visible as a pipeline event.

The reason for that specificity is a failure I have seen: A report deduplicated on the full row and counted a retried purchase twice because the retry carried a new ingestion timestamp, which overstated a launch day's conversions by 8%.

Keep one row per event id, deterministically.

select *
  from (
    select *,
           row_number() over (partition by event_id order by ingested_at, source_offset) as rn
      from raw_events
     where ingested_at >= now() - interval '7 days'
  ) t
 where rn = 1;

I would not consider it settled without evidence: Track the duplicate rate per source per day and alert when it moves, rather than assuming the deduplication holds.

Deduplicate on what identifies the event, not on what happens to differ.

I would start by writing the estimand for WHERE, HAVING, and the order of evaluation in a sentence before choosing any estimator.

WHERE filters rows before grouping and HAVING filters groups after aggregation, so a predicate on a raw column belongs in WHERE where it also reduces the rows the aggregate has to read.

Concretely, put every predicate on a non-aggregated column in WHERE, reserve HAVING for conditions on aggregates, and read the query plan to confirm the filter is being applied before the aggregation rather than after.

The reason for that specificity is a failure I have seen: A report filtering the date range in HAVING scanned four years of a partitioned table on every run, took eleven minutes, and produced group totals computed over the full history before discarding them.

Same intent, two placements, two plans.

-- Scans everything, then discards groups.
select customer_id, sum(amount) from orders
 group by customer_id having min(ordered_at) >= date '2026-01-01';

-- Prunes partitions first.
select customer_id, sum(amount) from orders
 where ordered_at >= date '2026-01-01' group by customer_id;

I would not consider it settled without evidence: Compare the plan's estimated rows before and after the move; a correct WHERE placement shows partition pruning that HAVING does not.

HAVING is for aggregates, and everything else belongs earlier.

This is a place where the sample answer and the population answer for funnel analysis in SQL come apart.

Counting each step independently ignores ordering and produces conversion rates that can exceed one hundred percent between steps, because a user who did step three without step two is still counted at step three.

Concretely, enforce ordering per user with a self-join or a windowed first-occurrence timestamp per step, require each step's timestamp to exceed the previous step's, and apply a stated window within which the funnel must complete.

The reason for that specificity is a failure I have seen: A signup funnel reported 104% conversion from step two to step three, because a legacy deep link let some users reach step three directly and the query counted steps independently.

First occurrence per step, then ordering enforced.

with firsts as (
  select user_id,
         min(occurred_at) filter (where step = 'view')    as t1,
         min(occurred_at) filter (where step = 'cart')    as t2,
         min(occurred_at) filter (where step = 'pay')     as t3,
         min(occurred_at) filter (where step = 'confirm') as t4
    from funnel_events group by user_id
)
select count(t1) as s1,
       count(*) filter (where t2 > t1) as s2,
       count(*) filter (where t3 > t2 and t2 > t1) as s3,
       count(*) filter (where t4 > t3 and t3 > t2 and t2 > t1) as s4
  from firsts;

I would not consider it settled without evidence: Assert monotonically non-increasing counts across steps as a query test, so an ordering bug fails loudly.

A funnel is a sequence, and counting steps separately forgets the sequence.

My answer to point-in-time joins against a changing dimension begins with how the data came to exist, because sampling contaminates everything after it.

Joining a fact to the current state of a dimension attributes history to today's attributes, so any analysis by a mutable attribute needs the dimension's value as of the fact's own timestamp.

Concretely, keep the dimension as a slowly changing table with validity ranges, join on the fact timestamp falling inside the range, and treat the current-state table as a convenience for operational queries rather than for historical reporting.

The reason for that specificity is a failure I have seen: A plan-mix analysis joined to the current customer table and moved $1.4M of prior-year revenue into the enterprise tier, because those customers had upgraded since; the year-over-year growth by tier was wrong in both directions.

Validity-range join, so February is February's plan.

select date_trunc('month', o.ordered_at) as month, d.plan, sum(o.amount)
  from orders o
  join dim_customer_history d
    on d.customer_id = o.customer_id
   and o.ordered_at >= d.valid_from
   and o.ordered_at <  d.valid_to
 group by 1, 2;

I would not consider it settled without evidence: Reconcile the total by attribute against the ungrouped total in each historical period, and check that a customer who changed plan appears under both in the right months.

Reporting history against today's attributes rewrites the history.

I would treat making an analytical query fast enough to iterate on as a design question first and a computation question second.

Analytical query time is usually dominated by how much data is read, so the first lever is pruning — partitions, columns, and predicates that the engine can push down — rather than compute.

Concretely, read the plan for a full scan on a partitioned table, filter on the partition column directly rather than through a function, select only the columns needed, sample during exploration, and materialise a narrow intermediate table when the same scan is repeated.

The reason for that specificity is a failure I have seen: A query wrapping the partition column in a date function defeated pruning entirely and read 2.1 TB per run; rewriting the predicate against the raw column brought the same result to 34 GB and 22 seconds.

The predicate that prunes and the one that does not.

-- No pruning: the function hides the partition column from the planner.
where date_trunc('day', event_ts) between date '2026-08-01' and date '2026-08-07'

-- Prunes: the planner sees a range on the partition column itself.
where event_ts >= timestamp '2026-08-01' and event_ts < timestamp '2026-08-08'

I would not consider it settled without evidence: Compare bytes scanned before and after each change, which is the quantity that predicts both time and cost.

Rewriting the predicate beats renting more machine.

The useful framing for normalised sources against a denormalised analytical table is what decision changes if the estimate moves by its own uncertainty.

A normalised schema is optimised for consistent writes and a reporting layer is optimised for repeated reads at a fixed grain, so mirroring the source pushes the same join logic into every downstream query where it can be got wrong independently.

Concretely, model a small number of fact tables at explicit grains with conformed dimensions, resolve the joins once in the transformation layer, and document the grain of each table in the same place the table is defined.

The reason for that specificity is a failure I have seen: Six teams each wrote their own join from orders to customers to subscriptions; three of them fanned out on the subscription history and the company had four different revenue numbers in one board pack.

Grain stated on the table, not inferred by each reader.

Table	Grain	Joins resolved
fct_order	One row per order	customer, plan as of order date
fct_subscription_day	One row per subscription per day	plan, price, discount
dim_customer_history	One row per customer per attribute period	—

I would not consider it settled without evidence: Compare the headline metric computed from the modelled table against the same metric computed from the raw sources, and reconcile any difference before publishing either.

Resolve the join once, or resolve it differently in every notebook.

I would settle reading a live transactional table for analysis against the simplest defensible comparison, so any extra structure has to earn its place.

A long query under a weak isolation level can observe rows committed at different points during its own execution, so totals from different parts of the query need not correspond to any single state of the database.

Concretely, run reporting queries against a replica or a snapshot at a stated timestamp, use a repeatable-read or snapshot isolation level when reading the primary is unavoidable, and record the snapshot timestamp with the output.

The reason for that specificity is a failure I have seen: A reconciliation report read the orders table and the payments table twelve minutes apart within one query, and reported a $240,000 discrepancy that was entirely orders paid during the query's own run.

Pin the read to a single snapshot.

begin isolation level repeatable read;
select sum(amount) from orders   where created_at < now();
select sum(amount) from payments where created_at < now();
commit;   -- both reads see the same snapshot

I would not consider it settled without evidence: Rerun the report against a snapshot at a fixed timestamp and confirm the discrepancy disappears, which identifies it as a read-consistency artefact.

A report needs one moment in time, and a long query does not have one by default.

Most of the judgement in when a non-relational store fits an analysis better sits in what the comparison group is, not in the arithmetic.

The choice follows the shape of the traversal: relational engines are strong at set operations over rectangles, and a graph store earns its place when the query is a variable-length path whose depth is not known in advance.

Concretely, express the question as a query and count the joins; a fixed small number stays in SQL, an unbounded traversal such as connected components or shortest path goes to a graph engine or to an iterative job, and a wide sparse attribute set may justify a document store.

The reason for that specificity is a failure I have seen: A fraud ring analysis was written as seven self-joins to reach depth seven, ran for six hours, and missed rings at depth eight; the same question as a connected-components job over the same edges finished in four minutes.

The same question, two formulations.

Formulation	Depth reached	Runtime	Rings found
Seven self-joins in SQL	7	6h 10m	412
Connected components over the edge list	unbounded	4m	486

I would not consider it settled without evidence: Measure the runtime and the result completeness of both formulations on the same edge list before committing to a store.

Unbounded depth is where SQL stops being the right shape.

Where analyses go wrong on making a query result reproducible is usually one step upstream of the statistic being argued about.

A query result is a function of the query text, the data as of a timestamp, and the definitions in force, so reproducibility requires all three to be recorded rather than only the number.

Concretely, store the query alongside the result, pin the read to a snapshot or an as-of timestamp, version the metric definitions, and record the execution time and the code revision with every published figure.

The reason for that specificity is a failure I have seen: A board figure could not be reproduced because a late-arriving backfill changed six weeks of history and the deck recorded only the number, so nobody could tell whether the metric definition or the data had moved.

What travels with a published figure.

Field	Value
Metric	Weekly active accounts
Definition version	v4 (2026-07-02)
Data as of	2026-08-31 06:00 UTC
Query revision	a1f39c2
Value	184,402

I would not consider it settled without evidence: Re-run the stored query against the recorded snapshot and confirm it returns the published figure, as a routine check rather than an investigation.

A number without its query and its timestamp is an anecdote.

I would answer vectorised operations against row-wise iteration by separating what the design identifies from what the data merely displays.

Row-wise application executes Python once per row while vectorised operations execute a compiled loop over a contiguous buffer, so the difference is typically two orders of magnitude on numeric work.

Concretely, replace apply and iterrows with column expressions, use where or select for conditional logic, use groupby with built-in aggregations rather than a Python lambda, and reach for a compiled path only after the vectorised form is in place.

The reason for that specificity is a failure I have seen: A 40-minute nightly transformation used apply over a row-wise function to compute a tiered fee; the vectorised form with a piecewise expression ran in 11 seconds on the same data.

Same tiered fee, two implementations.

# 40 minutes: one Python call per row.
df["fee"] = df.apply(lambda r: r.amount * (0.03 if r.amount < 100 else 0.02), axis=1)

# 11 seconds: one compiled pass per column.
df["fee"] = df["amount"] * np.where(df["amount"] < 100, 0.03, 0.02)

I would not consider it settled without evidence: Time both forms on a fixed sample and check the outputs are identical before replacing the original.

The loop is the cost, and the vectorised form does not have one in Python.

The substance of validating a merge in pandas is the size of the effect and its uncertainty, not whether a threshold was crossed.

A merge on a non-unique key fans out, and pandas will do it without complaint, so the expectation about cardinality has to be asserted rather than assumed.

Concretely, pass the validate argument to declare the expected relationship, use the indicator argument to see unmatched rows on each side, and assert the resulting shape against what the operation was supposed to produce.

The reason for that specificity is a failure I have seen: A merge that should have been one-to-one duplicated 6% of rows against a dimension containing historical versions, and the resulting revenue total was reported before anyone compared row counts.

The merge fails loudly instead of fanning out.

before = len(orders)
merged = orders.merge(
    customers, on="customer_id", how="left",
    validate="many_to_one",       # raises if customers has duplicate keys
    indicator=True,
)
assert len(merged) == before, f"merge changed row count: {before} -> {len(merged)}"
print(merged["_merge"].value_counts())

I would not consider it settled without evidence: Assert the row count and the sum of a key measure before and after each merge in the transformation code itself.

Declaring the cardinality turns a silent bug into an exception.

Before running anything I would state what result would make me abandon the hypothesis behind copy semantics and chained assignment.

A chained indexing expression may return a copy rather than a view, so the assignment writes to a temporary object that is discarded, and whether it does depends on the memory layout rather than on the syntax.

Concretely, perform the selection and the assignment in a single indexing operation with loc, take an explicit copy when a subset is meant to be independent, and treat the chained-assignment warning as an error rather than as noise.

The reason for that specificity is a failure I have seen: A cleaning step that clipped outliers on a filtered subset appeared to work in a notebook and silently did nothing in the scheduled job, and three weeks of a downstream metric were computed on unclipped data.

The assignment that may vanish, and the one that does not.

# May write to a temporary copy.
df[df["amount"] > 1000]["amount"] = 1000

# Single indexing operation, always writes through.
df.loc[df["amount"] > 1000, "amount"] = 1000
assert df["amount"].max() <= 1000

I would not consider it settled without evidence: Assert the intended change is present after the assignment rather than trusting the statement to have applied.

One indexing operation assigns, and two indexing operations might not.

The first thing I would pin down about memory and dtypes on a large frame is which question the number is supposed to answer.

Default inferred types are far larger than most columns need, so declaring narrow dtypes and categoricals typically reduces the footprint several-fold before any change of tooling.

Concretely, read a sample to determine types, pass an explicit dtype mapping, convert low-cardinality strings to categorical, load only the required columns, and process in chunks or move to a columnar format when the data still does not fit.

The reason for that specificity is a failure I have seen: A team provisioned a 64 GB instance for a job whose frame fit in 3.8 GB once identifiers were read as integers rather than objects and three low-cardinality string columns were made categorical.

Per-column memory, before and after declaring types.

Column	Default dtype	Memory	After	Memory
user_id	object	4.1 GB	int64	0.4 GB
country	object	3.6 GB	category	0.05 GB
device	object	3.4 GB	category	0.05 GB
amount	float64	0.4 GB	float32	0.2 GB

I would not consider it settled without evidence: Measure memory usage per column with deep introspection before and after, so the reduction is attributable to specific columns.

Most of a frame's memory is usually four columns stored as objects.

I would start by writing the estimand for generators for data that does not fit in memory in a sentence before choosing any estimator.

A generator yields one item at a time and holds only the current one, so a streaming aggregation over an arbitrarily large source runs in constant memory as long as the aggregate itself is bounded.

Concretely, express the pipeline as a chain of generator expressions, keep only bounded accumulators such as counts, sums, and fixed-size sketches, and reach for an approximate structure when the exact answer would require unbounded state.

The reason for that specificity is a failure I have seen: A distinct-user count over 400 million events built a Python set that grew past available memory and killed the job at 80% completion each night for a week.

Bounded accumulators, including an approximate distinct count.

from datasketches import hll_sketch

def summarise(path):
    total, count, uniques = 0.0, 0, hll_sketch(14)   # ~1.5% error, fixed 16 KB
    with open(path) as fh:
        for line in fh:
            user, amount = line.rstrip("\n").split(",")
            total += float(amount)
            count += 1
            uniques.update(user)
    return total, count, uniques.get_estimate()

I would not consider it settled without evidence: Watch resident memory over a full run and confirm it is flat rather than growing with the input.

Streaming works while the accumulator is bounded, and a set of identifiers is not.

This is a place where the sample answer and the population answer for dataclasses for analysis configuration come apart.

A dictionary has no declared shape, so a misspelled key fails at the point of use rather than at construction and the set of parameters an analysis accepts is not written down anywhere.

Concretely, declare the configuration as a frozen dataclass with types and defaults, validate ranges in the post-initialisation hook, and serialise the instance into the run's output so the parameters that produced a result travel with it.

The reason for that specificity is a failure I have seen: A misspelled key silently fell back to a default significance level, and three weeks of readouts used 0.05 where the plan had specified 0.01 for a safety-relevant comparison.

Typed, frozen, validated, and serialisable with the result.

from dataclasses import dataclass, asdict

@dataclass(frozen=True, slots=True)
class ExperimentConfig:
    metric: str
    alpha: float = 0.05
    power: float = 0.80
    winsorise_at: float | None = 0.99

    def __post_init__(self):
        if not 0 < self.alpha < 0.5:
            raise ValueError(f"alpha out of range: {self.alpha}")

cfg = ExperimentConfig(metric="revenue_per_session", alpha=0.01)
result["config"] = asdict(cfg)

I would not consider it settled without evidence: Assert that reconstructing the configuration from a run's recorded output reproduces the same object, which fails when a parameter is not captured.

A typo should fail at construction, not change a threshold.

My answer to context managers for reproducible runs begins with how the data came to exist, because sampling contaminates everything after it.

A context manager binds setup and teardown to a lexical block, so the recording and the cleanup happen on every exit path including exceptions, which a pair of function calls does not guarantee.

Concretely, wrap the run in a context manager that captures the code revision, the configuration, and the data snapshot on entry and writes the manifest and releases resources on exit, and use the exception path to record failed runs rather than losing them.

The reason for that specificity is a failure I have seen: Runs that raised part-way left temporary tables behind and wrote no manifest, so a quarter's failed experiments were invisible in the audit and the warehouse accumulated 4 TB of orphaned scratch tables.

Manifest on success and on failure, cleanup either way.

from contextlib import contextmanager
import json, subprocess, time

@contextmanager
def analysis_run(name, config, snapshot_ts):
    started = time.time()
    rev = subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip()
    manifest = {"name": name, "rev": rev, "config": config, "as_of": snapshot_ts}
    try:
        yield manifest
        manifest["status"] = "ok"
    except Exception as exc:
        manifest["status"] = f"failed: {exc!r}"
        raise
    finally:
        manifest["seconds"] = round(time.time() - started, 1)
        json.dump(manifest, open(f"runs/{name}.json", "w"))

I would not consider it settled without evidence: Force an exception inside the block in a test and confirm the manifest is still written and the scratch resources are gone.

Cleanup that depends on reaching the last line does not run when it matters.

I would treat concurrency for I/O-bound data collection as a design question first and a computation question second.

The global interpreter lock blocks parallel Python bytecode but is released during I/O waits, so threads and async both help when the bottleneck is the network and neither helps when it is computation.

Concretely, use a thread pool or async client for network-bound collection with a bounded concurrency limit and retry policy, use processes for CPU-bound transformation, and measure where the time is actually going before choosing.

The reason for that specificity is a failure I have seen: A team moved a page-fetching job to multiprocessing, paid the serialisation cost on every response, and ran 40% slower than a thread pool at the same concurrency.

Bounded-concurrency fetch with retries.

from concurrent.futures import ThreadPoolExecutor, as_completed

def fetch_all(urls, workers=16):
    out = {}
    with ThreadPoolExecutor(max_workers=workers) as pool:
        futures = {pool.submit(fetch_with_retry, u): u for u in urls}
        for fut in as_completed(futures):
            out[futures[fut]] = fut.result()
    return out

I would not consider it settled without evidence: Profile one run to separate time spent waiting on sockets from time spent in Python, and pick the mechanism that targets the larger share.

The lock is released while you wait, which is what makes threads work here.

The useful framing for floating point in financial aggregates is what decision changes if the estimate moves by its own uncertainty.

Binary floating point cannot represent most decimal fractions exactly, so summing many values accumulates representation error, and the size of the error depends on the order of summation.

Concretely, store money as integer minor units or as a decimal type, perform the aggregation in that type, and round once at presentation with an explicitly stated rounding rule rather than at each intermediate step.

The reason for that specificity is a failure I have seen: A daily reconciliation drifted by a cent or two per invoice and by $91 over a year, because line items were rounded to cents individually in one pipeline and once at the total in the other, and two teams spent a week comparing queries before anyone compared the rounding step.

Three defensible-looking implementations, three different totals.

from decimal import Decimal, ROUND_HALF_UP

rows = ["19.995", "2.005", "8.615"]        # line totals before rounding to cents

print(sum(round(float(r), 2) for r in rows))   # 30.619999999999997, and 2.005 rounds down
print(round(sum(float(r) for r in rows), 2))   # 30.62

cents = Decimal("0.01")
print(sum(Decimal(r).quantize(cents, ROUND_HALF_UP) for r in rows))  # 30.63
print(sum(map(Decimal, rows)).quantize(cents, ROUND_HALF_UP))        # 30.62

I would not consider it settled without evidence: Recompute the aggregate in a decimal or integer type and compare against the floating-point total; a difference at the cent scale identifies the cause.

Money is a decimal quantity, and binary floats are not.

I would settle seeds and reproducibility in a stochastic analysis against the simplest defensible comparison, so any extra structure has to earn its place.

Stochastic methods vary by design, so the question is whether the variation is smaller than the precision being reported, and the fix is to control the seed and to report the method's own error rather than to chase an exact match.

Concretely, pass an explicit seeded generator rather than relying on global state, record the seed with the result, and increase the number of draws until the Monte Carlo error is below the reported precision.

The reason for that specificity is a failure I have seen: Two analysts reported intervals differing in the second decimal from the same data and spent a day looking for a data bug; the bootstrap had used 500 draws, whose Monte Carlo error was larger than the difference.

Interval endpoints stabilise as draws increase.

Draws	Lower bound across 5 seeds	Upper bound across 5 seeds
500	1.81 to 2.06	3.44 to 3.71
5,000	1.90 to 1.95	3.55 to 3.60
50,000	1.92 to 1.93	3.57 to 3.58

I would not consider it settled without evidence: Run the procedure at several draw counts and report where the endpoints stabilise to the precision being quoted.

Randomness is not a bug until it is larger than the digits you are printing.

Most of the judgement in hashing for counting and deduplication sits in what the comparison group is, not in the arithmetic.

Exact distinct counting needs state proportional to the number of distinct keys, so above a certain scale the choice is between paying that memory and accepting a bounded error from a sketch.

Concretely, use a hash set while the key space fits comfortably in memory, move to HyperLogLog for cardinality within a stated error, and use a Bloom filter for membership when false positives are acceptable and false negatives are not.

The reason for that specificity is a failure I have seen: A nightly distinct count built a set of 500 million identifiers, consumed 48 GB, and was replaced by a sketch using 16 KB with 1.6% error — well inside the tolerance of a figure quoted to two significant digits.

Cost against error for the same 500M identifiers.

Structure	Memory	Answer type	Error
Python set	48 GB	Exact distinct	0
HyperLogLog, p = 14	16 KB	Approximate distinct	~1.6%
Bloom filter, 1% target	600 MB	Membership	1% false positive

I would not consider it settled without evidence: Compare the sketch against an exact count on a day where the exact count is affordable, and confirm the error is inside the stated bound.

Exactness is worth paying for only when the decision can see the difference.

Where analyses go wrong on two pointers over sorted data is usually one step upstream of the statistic being argued about.

When both sequences are sorted on the same key, a single pass advancing two indices answers the matching in linear time, because neither pointer ever needs to move backwards.

Concretely, sort both inputs on the join key once, advance the quote pointer while the next quote is still at or before the transaction time, emit the match, and handle the boundary where no preceding quote exists as an explicit case rather than an off-by-one.

The reason for that specificity is a failure I have seen: A nested-loop implementation over 4 million transactions and 9 million quotes ran for over eleven hours; the two-pointer pass finished in under three minutes on the same machine.

As-of match in one pass over both sequences.

def asof_match(trades, quotes):
    """Both sorted by time. Returns the last quote at or before each trade."""
    out, j = [], 0
    for t in trades:
        while j + 1 < len(quotes) and quotes[j + 1].time <= t.time:
            j += 1
        out.append((t, quotes[j] if quotes and quotes[j].time <= t.time else None))
    return out

I would not consider it settled without evidence: Verify the linear implementation against the quadratic one on a small sample where both are affordable, then compare runtimes on the full input.

Sorted inputs turn a quadratic scan into one pass.

I would answer binary search for a threshold by separating what the design identifies from what the data merely displays.

When the quantity is monotone in the parameter, binary search finds the boundary in logarithmic evaluations rather than by scanning a grid, which matters when each evaluation is an expensive query.

Concretely, confirm monotonicity first, bracket the answer with a low and a high value known to fall on either side, halve the interval until the required precision is reached, and count the evaluations so the cost is explicit.

The reason for that specificity is a failure I have seen: A grid search over 1,000 candidate thresholds ran the scoring query 1,000 times at 40 seconds each, taking eleven hours for an answer that eleven evaluations would have found.

Eleven evaluations rather than a thousand.

def smallest_threshold(alerts_at, budget=300, lo=0.0, hi=1.0, tol=1e-3):
    while hi - lo > tol:
        mid = (lo + hi) / 2
        if alerts_at(mid) > budget:
            lo = mid          # too many alerts, raise the cutoff
        else:
            hi = mid
    return hi

I would not consider it settled without evidence: Confirm the alert count is monotone in the threshold on a sample before searching, since binary search on a non-monotone function returns an arbitrary point.

Monotone plus expensive evaluation is exactly where binary search pays.

The substance of heaps for a streaming top-k is the size of the effect and its uncertainty, not whether a threshold was crossed.

A bounded min-heap of size k keeps the current best k in memory and rejects each new item in logarithmic time, which answers top-k in a single pass without sorting the whole input.

Concretely, push until the heap holds k items, then compare each subsequent item against the heap's minimum and replace only when it is larger, and sort the heap once at the end for presentation.

The reason for that specificity is a failure I have seen: A top-100 report sorted 300 million rows to take the head, which spilled to disk and took 40 minutes; the heap version used 100 slots of memory and finished in the time it took to read the input.

One pass, k slots of memory.

import heapq

def top_k(stream, k=100):
    heap = []
    for account, spend in stream:
        if len(heap) < k:
            heapq.heappush(heap, (spend, account))
        elif spend > heap[0][0]:
            heapq.heapreplace(heap, (spend, account))
    return sorted(heap, reverse=True)

I would not consider it settled without evidence: Compare the heap result against a full sort on a sample to confirm correctness, then compare peak memory on the full input.

Top-k does not require ordering the part you are throwing away.

Before running anything I would state what result would make me abandon the hypothesis behind sliding windows over a metric stream.

A rolling aggregate recomputed from scratch at each position costs the window size per step, while maintaining the aggregate incrementally as the window advances costs a constant amount per step.

Concretely, add each entering observation and subtract each leaving one as the window slides, keep a deque or a counter for aggregates that cannot be subtracted such as a maximum or a distinct count, and state whether the window is inclusive of its endpoints.

The reason for that specificity is a failure I have seen: An hourly job recomputing a 7-day distinct count from raw events took 50 minutes and ran late into the next hour, until the window was maintained incrementally and the same output took under a minute.

Constant work per step for a subtractable aggregate.

from collections import deque

def rolling_sum(values, width):
    window, total, out = deque(), 0.0, []
    for value in values:
        window.append(value)
        total += value
        if len(window) > width:
            total -= window.popleft()
        out.append(total if len(window) == width else None)
    return out

I would not consider it settled without evidence: Compare the incremental output against a full recomputation on a bounded slice, then measure the runtime difference over the full history.

A window that slides should not be rebuilt at every step.

The first thing I would pin down about merging intervals is which question the number is supposed to answer.

Summing interval lengths double-counts every overlap, so the intervals must be merged into disjoint spans before their lengths are added.

Concretely, sort by start, walk the list extending the current span whenever the next start is at or before the current end, emit the span when it is not, and decide explicitly whether touching intervals count as contiguous.

The reason for that specificity is a failure I have seen: A tenure calculation summed raw subscription rows and credited customers with overlapping trial and paid periods twice, which overstated average tenure by 19% and fed a retention model as a feature.

Merge first, then sum.

def covered_days(periods):
    merged = []
    for start, end in sorted(periods):
        if merged and start <= merged[-1][1]:
            merged[-1] = (merged[-1][0], max(merged[-1][1], end))
        else:
            merged.append((start, end))
    return sum((end - start).days for start, end in merged)

I would not consider it settled without evidence: Assert that merged spans are disjoint and that their total never exceeds the calendar span they cover.

Overlapping intervals have to be merged before they can be added.

I would start by writing the estimand for connected components for entity resolution in a sentence before choosing any estimator.

Grouping by transitive closure of the match graph is a connected-components problem, and it is unforgiving of false positive edges because one wrong edge merges two entire components.

Concretely, build the edge list from pairwise matches above a threshold, run union-find or a graph traversal, and guard against runaway merges by capping component size and reviewing components that exceed it rather than accepting them.

The reason for that specificity is a failure I have seen: A single spurious edge through a shared corporate email domain merged 340,000 records into one entity, and the resulting customer table listed a single account with $40M in spend.

Union-find with a size cap that surfaces runaway merges.

def components(edges, cap=5_000):
    parent = {}
    def find(x):
        parent.setdefault(x, x)
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x
    for a, b in edges:
        ra, rb = find(a), find(b)
        if ra != rb:
            parent[ra] = rb
    groups = {}
    for node in parent:
        groups.setdefault(find(node), []).append(node)
    return groups, [g for g in groups.values() if len(g) > cap]

I would not consider it settled without evidence: Report the component size distribution after each run and inspect the largest components before publishing the grouping.

Transitivity turns one bad edge into one bad entity of unbounded size.

This is a place where the sample answer and the population answer for dynamic programming over sequences come apart.

Overlapping subproblems with optimal substructure make this a dynamic programming problem, where the cost of aligning two prefixes is computed once and reused rather than recomputed down every branch.

Concretely, define the state as the pair of prefix lengths, write the recurrence from the three edit operations, fill the table iteratively, and reduce memory to two rows when only the distance rather than the alignment is needed.

The reason for that specificity is a failure I have seen: A naive recursive implementation on 3,000-character strings did not terminate within an hour, while the tabulated version completed in under a second on the same input.

Two-row Levenshtein: O(mn) time, O(n) space.

def edit_distance(a: str, b: str) -> int:
    prev = list(range(len(b) + 1))
    for i, ca in enumerate(a, start=1):
        cur = [i] + [0] * len(b)
        for j, cb in enumerate(b, start=1):
            cur[j] = min(prev[j] + 1, cur[j - 1] + 1, prev[j - 1] + (ca != cb))
        prev = cur
    return prev[-1]

I would not consider it settled without evidence: Compare the tabulated result against a brute-force implementation on short strings, then measure runtime growth against input length.

The recursion is exponential and the table is a product of the lengths.

My answer to deciding what runs in the warehouse and what runs in Python begins with how the data came to exist, because sampling contaminates everything after it.

Moving data across the network is usually the dominant cost, so any operation that reduces cardinality — filtering, aggregation, joining — belongs where the data already lives.

Concretely, push filters, joins, and aggregations into SQL, pull the reduced result, and keep in Python the work the warehouse cannot express well: iterative fitting, simulation, and plotting.

The reason for that specificity is a failure I have seen: A daily notebook transferred 22 GB to compute forty rows of group means, took 35 minutes, and failed whenever the network hiccuped; the same aggregation in SQL returned in nine seconds.

Same forty numbers, two placements.

Placement	Rows transferred	Wall time
Pull rows, group in pandas	80,000,000	35m
Group in SQL, pull the result	40	9s

I would not consider it settled without evidence: Compare bytes transferred and wall time for both placements, which makes the division of labour a measurement rather than a preference.

Send the query to the data, not the data to the query.