Overview
Curated: · Written: · Reviewed:
Key takeaways
- Bias is systematic error from the learning assumptions and available representation; variance is sensitivity to which training sample is observed; noise is outcome variation not predictably recoverable from the available information.
- Underfitting usually shows poor training and validation performance. Overfitting shows much better training than deployment-like validation performance. Neither diagnosis is reliable when the split, metric, or pipeline leaks.
- Complexity is relative to data quantity, signal, representation, regularization, and objective—not a permanent property of an algorithm name or parameter count.
- Learning curves vary training-set size; validation curves vary a hyperparameter. Read both training and validation distributions, not one score or one fold.
- More representative data often reduces variance, while a richer representation or model can reduce bias. More data does not repair the wrong target, leakage, corrupted labels, or a deployment shift.
- Regularization and ensembling change the bias–variance balance, but hyperparameters must be selected on validation evidence and evaluated once on an untouched test set.
1. Generalization, bias, variance, and noise
The goal is performance on new examples from the intended deployment process, not memorization of the training sample. Conceptually, expected squared prediction error can be decomposed into squared bias, variance, and irreducible noise under specific assumptions. Bias describes how the average fitted model misses the underlying relationship. Variance describes how fitted predictions change across plausible training samples. Noise belongs to the data-generating process and label measurement, although better features or target definitions can move what previously looked irreducible into predictable signal.
This decomposition is a mental model, not a license to label any train–validation gap “variance.” A gap can come from leakage, distribution shift, group overlap, metric variance, preprocessing mismatch, unstable labels, or selection on the validation set. Start with a deployment-like split, one end-to-end pipeline, simple baselines, confidence across folds/time windows, and slice-level error analysis before changing complexity.
2. Underfitting and overfitting signals
An underfit model cannot capture important structure: training and validation performance are both poor relative to an attainable baseline, often with a small gap. Remedies can include useful features, a richer hypothesis class, weaker regularization, longer optimization, or fixing an objective/implementation bug. Adding more rows usually offers limited help when both curves have already converged to poor performance.
An overfit model learns sample-specific noise or unstable patterns: training performance is strong while validation performance is materially worse. Remedies include more representative data, stronger or better-targeted regularization, simpler features/model, early stopping, bagging, data augmentation, leakage removal, and more robust validation. A small train–validation gap is not proof of success if both scores are poor or both datasets share the same leakage.
3. Learning and validation curves
A learning curve refits the full pipeline on increasing training sizes and plots training and validation performance with dispersion. If validation improves and the gap narrows as data grows, additional representative data may help. If both curves plateau poorly, investigate bias, features, target, objective, and optimization. Subsampling must preserve time, groups, class prevalence, and pipeline fitting boundaries; otherwise the curve answers a different question.
A validation curve varies one complexity-controlling hyperparameter—tree depth, regularization strength, neighborhood size, polynomial degree—while plotting training and validation performance. Select the region using cross-validation, not the final test. One-dimensional curves can hide interactions, and the best validation value is optimistically biased after extensive search. Nested cross-validation or a separate test estimates the whole tuning procedure.
4. Levers and their tradeoffs
Increasing model or feature capacity can reduce approximation bias but increase sensitivity. Stronger L1/L2 penalties, shallower trees, pruning, dropout, augmentation, or early stopping can reduce effective capacity and variance but may increase bias. The regularization value is a tuned hyperparameter; regularization is not automatically beneficial and excessive strength can underfit.
Bagging averages models fit on varied samples or features and often reduces variance when errors are not perfectly correlated. Boosting sequentially focuses on residual errors and can reduce bias, but deep learners, noisy labels, or too many rounds can still overfit. Stacking needs out-of-fold base predictions so the meta-model does not train on in-sample predictions. Ensemble benefit, latency, calibration, failure correlation, and explainability must be evaluated end to end.
5. Data quality, shift, and production diagnosis
More data helps only when it is representative, correctly labeled, available before prediction, and adds independent information. Near duplicates can make a dataset large without reducing uncertainty. Historical volume from an obsolete policy can worsen deployment fit. Diagnose by time, entity, geography, source, cohort, label version, and acquisition process; fix split and provenance before tuning.
In production, a rising error rate can reflect covariate shift, prevalence change, concept drift, instrumentation, label delay, threshold policy, or selective feedback rather than the training-time bias–variance balance. Track model/data versions, score/calibration and outcome metrics, training–serving feature parity, cohort maturity, and the incumbent/candidate comparison. Retraining or increasing complexity is a hypothesis tested through a governed release, not an automatic response to drift.
6. Worked example: reading a learning curve before changing capacity
A fraud model scores 0.94 AUC on training folds and 0.81 on a time-blocked validation window. The 0.13 gap invites "reduce capacity", but a learning curve answers whether that is the right lever. Refit the whole pipeline at increasing training sizes and read both curves with their spread across 5 folds:
| Train rows | Train AUC | Validation AUC | Gap | Validation spread |
|---|---|---|---|---|
| 25,000 | 0.981 | 0.742 | 0.239 | +/- 0.031 |
| 50,000 | 0.968 | 0.779 | 0.189 | +/- 0.024 |
| 100,000 | 0.951 | 0.801 | 0.150 | +/- 0.018 |
| 200,000 | 0.940 | 0.810 | 0.130 | +/- 0.015 |
The gap narrows monotonically from 0.239 to 0.130, which is decisive and says the model is variance-limited: halving tree depth here would trade that gap for a lower ceiling on both curves.
Whether more data still helps is a different question, and this table does not answer it. Validation rises +0.037, then +0.022, then +0.009 across the doublings — and that last increment is smaller than the fold spread of +/- 0.015 beside it. A step inside the noise is not evidence of a climb. The honest read is that the curve is at its knee: it was clearly rising, it may now be flattening, and the cheap next move is more folds or one more doubling to resolve the ambiguity, not a purchase order for 200,000 rows. Reading +0.009 against +/- 0.015 as "still climbing" is how a plateau gets funded.
The subsampling has to preserve what the split protects, or the curve answers a different question:
# Wrong: random subsample leaks later fraud patterns into smaller "earlier" training sets.
subset = df.sample(n=size, random_state=0)
# Right: take the earliest size rows by event time, then make the split entity-safe.
dev = df.sort_values("opened_at")
subset = dev.iloc[:size]
# A time prefix does not give entity isolation on its own: accounts opened before the
# cutoff keep generating rows after it, so some appear on both sides. Drop them rather
# than asserting they cannot happen -- on panel data that assert simply fires.
subset = subset[~subset["account_id"].isin(holdout["account_id"])]
Read the same table for an underfit model and it looks different: both curves converge to 0.72 by 50,000 rows and stay flat to 200,000. That is a bias signal, and 200,000 more rows buys nothing.