Skip to content
Tech Interview Prep
Technical interview guide

Model Evaluation Metrics

Picking the right metric (accuracy, precision/recall, F1, ROC-AUC) for the problem and its class balance.

Read
26 min
Practice MCQs
25
Interview Q&A
25
Edition
v6
Editorial status
Reviewed

Scope: scikit-learn stable model-evaluation, calibration, and cross-validation guidance and Google ML Crash Course classification, fairness, and monitoring guidance accessed 2026-08-31..

Curated: · Written: · Reviewed:

25 questions

MCQ-3A fraud-detection model flags transactions as fraudulent. The team wants to report the fraction of flagged transactions that are actually fraudulent. Which metric should they compute?
MCQ-2A fraud model scores 10,000 held-out card transactions, 200 of which are fraudulent. At the current threshold it flags 240 transactions, and 180 of those are fraudulent. The team is weighing a lower threshold that would catch more fraud but generate more alerts, and first needs the fraction of actual frauds the current threshold catches. Which computation gives that number?
MCQ-1A payments team evaluates a fraud model on 200,000 transactions, 1,000 of which are fraudulent. The model flags 500 transactions as fraud, and 100 of those flags are genuine. The dashboard reports 99.35% accuracy, so the model ships. Which statement about this evaluation is correct?
MCQ-4An analyst scores 10 transactions with a fraud model and flags any transaction scoring at least 0.5. The scores and true labels are: | transaction | score | true label | | --- | --- | --- | | T1 | 0.95 | fraud | | T2 | 0.90 | fraud | | T3 | 0.85 | legit | | T4 | 0.80 | legit | | T5 | 0.75 | fraud | | T6 | 0.70 | legit | | T7 | 0.60 | fraud | | T8 | 0.55 | legit | | T9 | 0.40 | fraud | | T10 | 0.30 | legit | The team raises the flag threshold to 0.8. Which outcome follows?
MCQ-5A fraud model scores eight held-out transactions. The four fraudulent ones score 0.95, 0.85, 0.55, and 0.45; the four legitimate ones score 0.90, 0.70, 0.60, and 0.40. No two scores tie. A reviewer asks for the ROC AUC on this set. What should you report?
MCQ-7A fraud team evaluates a detector on a test set where 4% of transactions are fraudulent. They report average precision (AP) and need a random-score baseline to judge whether the model beats chance. What is that baseline approximately?
MCQ-11A three-class classifier is evaluated on a 300-example test set. The confusion matrix (rows are the actual class, columns the predicted class) is: | actual \ predicted | A | B | C | | --- | --- | --- | --- | | A | 175 | 25 | 0 | | B | 10 | 25 | 15 | | C | 0 | 5 | 45 | The team must report macro-averaged recall. Which value do they report?
MCQ-8A retention team scores 5,000 subscribers at 0.80 with a churn model and is deciding whether to act on that bin. Assuming the model is well calibrated, which expectation is correct?
MCQ-9A fraud model scores a transaction at 0.01 probability of fraud, and the transaction turns out to be fraudulent. Which metric penalizes that confident miss most heavily?
MCQ-19A team trains a gradient-boosted classifier on 200,000 labeled rows. On a held-out set of 20,000 rows the scores are badly calibrated: ECE with 10 equal-width bins is 0.11. They fit isotonic regression on those same 20,000 rows and re-measure ECE on them, getting 0.004. They ship the model and calibrator. Over the next week 20,000 fresh rows arrive from the same pipeline and population, and ECE there is 0.10. The ECE code and bin count are unchanged throughout. What explains the 0.004?
MCQ-10A team swaps a model and sees Brier loss fall from 0.26 to 0.09 on the same held-out set, but the new model's reliability diagram is visibly farther from the diagonal. Which explanation best accounts for the lower Brier loss despite worse reliability?
MCQ-18A fraud model outputs calibrated probabilities. On a 100,000-transaction validation set containing 100 frauds, the team sweeps five thresholds and records the counts below. A missed fraud costs $100 in chargebacks; a manual review of a flagged transaction costs $2. What should the team do? | threshold | frauds caught | false alarms | | --- | --- | --- | | 0.9 | 40 | 50 | | 0.8 | 55 | 150 | | 0.7 | 70 | 300 | | 0.5 | 85 | 1,200 | | 0.3 | 95 | 4,000 |
MCQ-12A payments team must choose one of two fraud models for production. The held-out test set has 100,000 transactions, 500 of them fraudulent. A missed fraud costs $500 in chargebacks; a false alarm costs $1 in manual review. Model A: 400 true positives, 1,000 false positives, 100 false negatives. Model B: 480 true positives, 5,000 false positives, 20 false negatives. Which model should the team ship, and why?
MCQ-25A payments team is choosing which of five fraud models to deploy. Each model was scored on the same 100,000-transaction holdout set, which contains 200 fraudulent transactions. The team's deployment rule is to block every transaction a model flags. A missed fraud costs $500 in chargebacks and investigation; a blocked legitimate transaction costs $8 in support and goodwill credit. The confusion matrices are: | Model | True positives | False positives | | --- | --- | --- | | A | 170 | 1,200 | | B | 150 | 500 | | C | 185 | 3,000 | | D | 100 | 100 | | E | 195 | 5,000 | Which model should they deploy?
MCQ-13A regressor is scored on 8,000 held-out rows. Its sum of squared errors is 5.2×10^6, and the sum of squared deviations of the held-out targets from their average is 4.1×10^6. The reported R² is −0.27. Which statement about that score is correct?
MCQ-15A demand planner compares two forecasters on two SKUs. SKU A actual = 0.1 units; SKU B actual = 100 units. Model X forecasts 1.1 and 100. Model Y forecasts 0.1 and 150. MAPE is 500% for X and 25% for Y; MAE is 0.5 for X and 25 for Y. Which conclusion is warranted?
MCQ-14A courier team predicts delivery ETA in minutes. Ops bills overtime at a flat rate per minute late, so a 10-minute miss costs exactly twice a 5-minute miss, and the residual distribution has a heavy right tail from a handful of storm days. Which training loss should the model optimize?
MCQ-16A demand-forecasting team fits a quantile regression at τ = 0.9 to size safety stock. On one holdout day the model predicts 120 units and actual demand is 150 units. What is the pinball loss for this observation?
MCQ-23A search team is evaluating a new ranker on a judged query set. For one query the graded relevance labels are A=3, B=2, C=0, D=1, E=0, and the ranker returns the documents in the order D, A, C, B, E. The team reports NDCG@5 using gain 2^rel − 1 and discount 1/log2(rank + 1). Which value should they report?
MCQ-21A classifier clears its launch gate at 94% accuracy on a 2M-row evaluation set, where one enterprise cohort supplies 4% of the rows. The team is deciding whether to add per-cohort accuracy to the launch report. Which failure does the aggregate number alone leave undetected?
MCQ-22A team evaluates a summarization model on a 400-example eval set. Overall quality-pass rate is 78% (312/400). On the 25 examples that contain a table, 16 pass (64%). A 10,000-resample bootstrap over the eval set gives a 95% interval of 45–80% for the table slice. The launch report needs one line about table handling. Which line is defensible?
MCQ-20A team trains 50 model variants on the same training set and scores each once on a held-out test set of 10,000 rows. The variants differ only in hyperparameters, they have essentially the same accuracy on unseen data, and each scores close to 90%. For this question, treat the 50 test accuracies as independent draws around that common accuracy. The best of them is 91.2%, and the team reports 91.2% as the model's accuracy on new data. Which statement about the reported 91.2% is correct?
MCQ-17A team predicts 30-day readmission from 4,000 encounters belonging to 1,200 patients; 380 of those patients contribute three or more encounters. They evaluate with 5-fold cross-validation that shuffles encounters, reporting mean AUC 0.93 (folds 0.92–0.94). A holdout drawn at random from the same period, 500 encounters from 500 patients absent from training, scores AUC 0.68. Which conclusion does the split's construction support?
MCQ-6A payments team ships a fraud model with ROC AUC 0.98 on a held-out set of 10,000,000 transactions taken from a recent production week; 10,000 of them are fraudulent, matching the 0.1% rate production sees. The split is chronological, and the operating threshold was fixed at 90% recall, where the measured false-positive rate on that set is 0.1%. The next morning the review queue holds about 19,000 alerts and analysts can clear 500 a day. Which explanation accounts for the queue?
MCQ-24A fraud model ships with unchanged weights, the same 0.5 threshold, and the same feature pipeline. Offline, on a temporal holdout drawn from last quarter's traffic, precision is 0.62. In production, precision over a fully labeled week is 0.35. AUC is unchanged at 0.88 and the score histogram is unchanged, but the fraud base rate has fallen from 1.2% to 0.4%. Which explanation accounts for the gap?