Technical interview guide

Model Monitoring & Drift Detection

Detecting when a production model's performance degrades because the world changed since it was trained.

Read: 45 min
Practice MCQs: 25
Interview QA: 25
Edition: v4
Editorial status: Reviewed
Relevant for: AI EngineerMLOps Engineer

Scope: Azure Machine Learning, Vertex AI, AWS SageMaker AI, NIST AI RMF, and Google SRE guidance current 2026-09-01; AWS states Model Monitor is unavailable to new customers.

Interview QA

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: 2026-10-05 · Written: 2026-10-05 · Reviewed: 2026-10-03

QA-1

Distinguish data drift, concept drift, skew, and data quality.

QA-2

Design production monitoring for a high-impact model.

QA-3

How do you select statistical metrics for numerical versus categorical drift detection at scale?

QA-4

Choose reference data for drift monitoring.

QA-5

Design monitoring windows for delayed ground truth.

QA-6

Calibrate drift thresholds and alerts.

QA-7

Investigate an input-drift alert with stable reported accuracy.

QA-8

Investigate performance degradation without obvious input drift.

QA-9

Monitor fairness and bias drift responsibly.

QA-10

Use feature-attribution drift in production monitoring.

QA-11

Monitor a canary deployment with multiple model versions.

QA-12

Design privacy-preserving production data capture.

QA-13

Respond to monitoring capture failure.

QA-14

Build runbooks for model monitoring alerts.

QA-15

Integrate monitoring with automated retraining safely.

QA-16

Monitor batch inference jobs as rigorously as online endpoints.

QA-17

Monitor models with little or no ground truth.

QA-18

Handle expected seasonality without normalizing incidents.

QA-19

Test a model monitoring system before launch.

QA-20

Audit current use of SageMaker Model Monitor.

QA-21

Respond when a monitored model causes user harm before thresholds fire.

QA-22

Design monitor governance and change management.

QA-23

Measure whether model monitoring is effective.

QA-24

Retire or constrain a model whose risk cannot be monitored reliably.

QA-25