Browse
Model Monitoring & Drift Detection
Detecting when a production model's performance degrades because the world changed since it was trained.
What it is
Model monitoring tracks a deployed model's ongoing performance and the statistical properties of its inputs, to catch degradation from drift — a model quietly getting worse because the real world has shifted since training, not because of a code bug.
Key points
- Data drift: the distribution of input features shifts over time (e.g. user behavior changes) — the model wasn't trained on this new distribution, so predictions degrade even though the model itself hasn't changed.
- Concept drift: the relationship between inputs and the target changes (what used to predict churn no longer does) — a more fundamental shift than data drift, and it usually requires retraining on new labeled data, not just monitoring.
- Monitoring without ground truth: in production, true labels are often delayed or unavailable — monitoring typically relies on proxy signals (input distribution shifts, prediction distribution shifts) rather than directly measured accuracy.
- Model monitoring is what turns "the model silently got worse three months ago" into "the model started degrading on Tuesday, here's why" — the entire value is in catching it early.