Skip to content
Tech Interview Prep
Technical interview guide

Model Monitoring & Drift Detection

Detecting when a production model's performance degrades because the world changed since it was trained.

Read
45 min
Practice MCQs
25
Interview Q&A
25
Edition
v4
Editorial status
Reviewed

Scope: Azure Machine Learning, Vertex AI, AWS SageMaker AI, NIST AI RMF, and Google SRE guidance current 2026-09-01; AWS states Model Monitor is unavailable to new customers.

Curated: · Written: · Reviewed:

25 questions

MCQ-6A fraud model scores transactions with a probability. Monitoring compares the last 7 days of scores with the training reference: PSI on the score distribution is 0.31, above the team's 0.25 alert threshold. PSI on each of the top 20 input features is below 0.05. Chargebacks arrive about 30 days late, so no accuracy figure exists for this window. What should the team do next?
MCQ-20A monitored numeric feature's distribution shifts sharply in one overnight window and its drift score crosses the alert threshold; the feature is produced by a scheduled batch job that transforms a raw events table. Which check best separates genuine population drift from a broken upstream source?
MCQ-7A team computes mean absolute SHAP values per feature daily for a deployed fraud model. Over two weeks, mean |SHAP| for `txn_amount_24h` falls from 0.31 to 0.09, while the feature's PSI stays at 0.02 and the model's AUC moves from 0.91 to 0.90. What does this attribution drift most directly indicate?
MCQ-1A fraud model's drift monitor has reported zero violations for three weeks, while the chargeback rate it is meant to track rose 0.4 points over the same window. Before reading the zero as a healthy model, which verification is required?
MCQ-8A feature `merchant_category` is monitored by PSI between the current hour and the previous hour, alerting above 0.1. The feature normally carries about 1% nulls. At 09:00 a deploy pushed that to 30% and it stayed there. PSI jumped to about 1.1 at 09:00, then read 0.02–0.03 every hour for the rest of the day, so no further alerts fired. Raw logs confirm 30% nulls from 09:00 onward. Why did the monitor go quiet while the bug persisted?
MCQ-15A ranking model's feature monitor runs a two-sample Kolmogorov–Smirnov test nightly, comparing the last 24 h of a feature against a 30-day reference window of 10 million rows, and pages when p < 0.001. Last night it paged: p = 0.0002, KS statistic D = 0.0012, and the feature mean moved from 0.5000 to 0.5004. The feature reaches the model only through a bucketized split at 0.5, and online CTR is flat at 4.1% ± 0.2%. What is the right read of this page?
MCQ-14A payments team monitors a feature's PSI daily, comparing a 7-day window against a 28-day reference. A 200-day backtest on known-stable traffic gives these false-alert rates: | threshold | false alerts per 100 days | | --- | --- | | 0.04 | 14 | | 0.07 | 6 | | 0.09 | 3 | | 0.12 | 1 | | 0.18 | 0 | The on-call rotation can absorb at most 3 false alerts per 100 days. A post-incident review of a real upstream schema change found PSI at 0.11 on day 1 and 0.17 by day 3, and the team's SLO is to page on the first day of real drift. Which threshold should they set?
MCQ-16A payments service pages on its 1-minute error rate. The rule fires when the rate is at or above 5% in three consecutive 1-minute windows, and clears when the rate is below 3% in two consecutive windows. Over the 15 minutes below, how many alerts does the rule fire, and in which minutes? | minute | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | error rate | 4.9% | 5.2% | 5.6% | 5.1% | 2.7% | 2.2% | 5.3% | 5.7% | 5.4% | 2.9% | 2.4% | 5.5% | 5.9% | 5.2% | 4.8% |
MCQ-22A team game-day-tests a service's paging path. The error-rate rule is `sum(rate(http_requests_total{code=~"5.."}[1m])) / sum(rate(http_requests_total[1m])) > 0.05` with `for: 5m`. They inject a 90-second burst of 100% 5xx responses. No page arrives. A separate synthetic alert posted to Alertmanager reaches the on-call receiver in 8 seconds; scrapes succeeded throughout the burst, and no silences or inhibitions are configured. What does this run establish?
MCQ-21Your checkout service emits one span per attempt: 30,000 per minute. The trace backend keeps spans for 90 days and all 1,400 engineers can query it. You need to filter the dashboard to a single customer's failed checkouts during an incident without exposing that customer's identity to everyone with access. What should the span attribute carry?
MCQ-9A daily accuracy monitor emits at 00:00 UTC for the previous 24 h, computing accuracy over labeled records only. Labels arrive with a median of 1 h and a p99 of 14 h, and the unlabeled records skew toward the model's errors. At 00:00 the dashboard reads 94.6%; by 14:00 the same window reads 87.2%. The team adds a 6-hour offset so the window closes at 18:00 and the metric still emits at 00:00. What happens to the 00:00 reading?
MCQ-10A payments team monitors a fraud-scoring model with a daily job that compares yesterday's predictions against chargeback labels. Chargebacks are confirmed 45 days after the transaction. For six months the job has reported AUC 0.91 ± 0.01 every day. Over the last three weeks the fraud loss rate on approved transactions has risen from 0.12% to 0.31%. What does this pattern indicate?
MCQ-17An unsecured lender deployed a new default model on a single vintage of loans approved over the last quarter, right after a pricing and underwriting overhaul. Defaults are only confirmed 12 months after origination, so no loan in the book carries a verified label yet, and the risk team needs a degradation signal well before the first labels arrive. Which monitoring setup fits that window?
MCQ-13A fraud model flags transactions for manual review. Labels come from chargebacks, which arrive within 45 days. The monitoring dashboard shows: new accounts: 4,000 flagged, 480 labeled (12%), precision 0.91; established accounts: 6,000 flagged, 4,080 labeled (68%), precision 0.88. The team wants to use the dashboard to decide whether to expand the model to new accounts. What is the most defensible conclusion?
MCQ-12A payments team evaluates a fraud model on 2,000 transactions sampled from the manual review queue built by an older rules engine. Analysts confirmed 1,600 of them as fraud. At the proposed threshold the model flags 1,360 transactions, 1,280 of which are confirmed fraud. Production fraud prevalence is 0.5%. Assuming the model's true-positive and false-positive rates carry over, what precision should the team expect in production?
MCQ-11A recommender team gates every rollout on a single blended daily click-through rate: promote the candidate if the day's blended CTR is within 0.10 percentage point of the 4.04% champion-only baseline. v42 is the 5% canary, taking 200,000 of the day's 4,000,000 impressions through a random hash of request ID, and v41 serves the rest. The day's dashboard reads 3.998% blended, inside the gate. Splitting the same day's logs by model_version gives v41 4.04% and v42 3.20%. What does that split show, and what should the team do?
MCQ-4A linear credit model scores applicants as log-odds = 0.9·z(credit_score) + 0.05·z(app_usage) + c, where z() standardizes each feature against the reference window; across the population the score has a standard deviation of 1.2. This month's monitoring shows z(app_usage) has shifted to a mean of +1.4 (PSI 0.31) while z(credit_score) is unchanged (PSI 0.02). Outcome labels arrive 90 days after each decision. Which conclusion about the model's current predictions is best supported?
MCQ-5A card-fraud model scores transactions in real time at a 0.5 operating threshold. Between January and March 2025, monitoring compared with the January training window shows a maximum PSI of 0.03 across the 40 input features (the team alerts at 0.10), a score-distribution PSI of 0.04, a flagged share steady at 1.9% in both months, an audited fraud rate across all transactions steady at 2.5%, and an unchanged fraud-review protocol. The one measurement that moved is precision inside the flagged population: 68% in January, 31% in March. Which diagnosis fits this evidence?
MCQ-2A real-time fraud model's precision@1% recall falls from 0.58 to 0.39 between 13:00 and 14:00 UTC. The on-call engineer pulls the monitors: the `user_risk_score` feature-writer job last succeeded at 13:12 UTC and runs every 15 minutes; Kafka consumer lag on `user_events` is 38k messages, inside its normal band; PSI of the `user_risk_score` distribution against the previous hour is 0.04; and the model binary and serving code have not changed in 9 days. Which root cause do these numbers support?
MCQ-18A card-fraud model scores 4,000 requests/s across 40 pods. Its feature txn_velocity_1h is a Redis lookup keyed by card_id over 16 hash slots; when the lookup returns nil the model substitutes the training median for that feature and still emits a score. At 01:39 UTC the share of requests with a nil lookup for this feature was 0.4%. At 01:40 it rose to 37.8% and has held there for 30 minutes. Nothing else changed: same model version, same 40 pods, same 4,000 requests/s, no deploy, and the training data for this feature was 0.5% nil. Grouped by hash slot over those 30 minutes, the nil rate is 100% on slots 10-15 and 0.4% on slots 0-9. Which diagnosis and first action fit this evidence?
MCQ-3A fraud model scores 0.94 AUC offline. In production it scores 0.71 AUC on the same traffic. A shadow job replays the last 24 hours of live requests through the offline feature pipeline and the same model artifact and scores 0.93 AUC. The feature `txn_count_1h` counts the card's transactions in the hour before the request: offline it counts warehouse rows with `event_time` in that hour; at serving a stream job maintains a 1-hour window over the event stream, whose watermark lags 45 minutes behind wall-clock at peak. Over the same 1,000 requests the offline value averages 3.4 and the serving value averages 2.2. What should the team do first?
MCQ-19An income-verification model in a loan-approval pipeline has a fixed 0.5 decision threshold. Over the last 24 hours, its AUC fell from 0.78 to 0.69, and the PSI for the debt_to_income feature rose from 0.03 to 0.41. The upstream data team deployed a change to the income-calculation job 26 hours ago. Loan outcomes take about 30 days to mature. What should the ML team do first?
MCQ-23A team monitors each of a model's 40 features, a mix of continuous and low-cardinality categorical, with a univariate drift test against a fixed reference window of about 1,500 rows, and separately watches the distribution of the model's output score. Nothing has alerted for three weeks, yet precision on the review queue has fallen from 0.81 to 0.62. Replaying the window, the loss is confined to rows where three of the features take a combination absent from the reference; each of those features on its own, and every pair of them, looks unchanged. Which monitor should the team add?
MCQ-25An abuse-detection model auto-bans accounts scoring at least 0.90. Nine days ago a device-data vendor changed its ID scheme, so the device-graph edge-count feature now resolves to 0 for about 30% of accounts instead of the true count; the model learned that 0 edges indicates inauthentic behavior, and those accounts now score 0.91-0.98 whether or not they are abusive. A labeled sample drawn from the affected 30% found 18 of 25 auto-bans wrong (precision 0.72); the other 70% is unchanged at 0.96, against a policy floor of 0.95. The model bans roughly 900 accounts a day. The human review queue has 700 seats a day and currently receives 300. The vendor's fix is due in two weeks. What should the team do with the model over those two weeks?
MCQ-24Your team monitors a payments fraud model. Last quarter: 1,800 alerts fired; on-call closed 1,746 as no action and escalated 54, all confirmed fraud; confirmed fraud losses were 480k USD. Postmortem found 312k USD of those losses came from 26 accounts that never triggered an alert; all 26 used a device type introduced five weeks before quarter end. Alert p95 latency was 28s, drift score 0.41, and there are 12 dashboards. With one engineer-week available, which change is most likely to reduce confirmed fraud loss next quarter?