Skip to content
Tech Interview Prep
Technical interview guide

Automated Retraining Pipelines

Automatically retraining models as new data arrives, with validation gates before a new version replaces the current one.

Read
45 min
Practice MCQs
25
Interview Q&A
25
Edition
v4
Editorial status
Reviewed

Scope: TensorFlow TFX, SageMaker AI Pipelines, Azure Machine Learning, Vertex AI Pipelines, MLflow, and NIST AI RMF guidance current 2026-09-01.

Curated: · Written: · Reviewed:

25 questions

MCQ-21A team is backfilling 90 days of training data for a fraud model, covering examples dated January 1 through March 31, replaying the event log through the same versioned feature pipeline used for live scoring. Each example's features are computed over the days immediately before its date: a 7-day transaction count and a 30-day chargeback rate. The backfill reads the event log once, in date order, and joins it to labels. What is the earliest event-log date the backfill must read so that every example's features match what serving would compute?
MCQ-4A fraud model trains on labels that are individual card transactions. The feature table `user_txn_count_7d` holds one row per user per day; each row is written at 00:00 UTC on its `feature_date` and counts the user's transactions over the 7 days ending the day before. For user u1 the table holds 2024-03-08 → 12, 2024-03-09 → 15, 2024-03-10 → 9, 2024-03-11 → 20. A training label is u1's transaction at 2024-03-10T14:00Z. A point-in-time-correct join attaches which value?
MCQ-3A fraud team builds a training set from the last 12 days of card transactions; the oldest transaction is 11 days old at training time. Each transaction is joined to whatever confirmed-fraud label exists then, and transactions with no label yet are kept as negatives. For this example, confirmed-fraud labels arrive with a fixed delay: half are confirmed 6 days after the transaction and half 12 days after. The true fraud rate is 2.0% and stable, with roughly equal transaction volume each day. What fraud rate will the training set show?
MCQ-2A fraud model is retrained nightly from `events` (append-only, keyed by `event_ts`) joined to `labels_current`, a table that is upserted in place whenever an analyst corrects a label. Monday's 02:00 run produced 4.2M rows. A re-run of the same pipeline commit on Wednesday at 02:00 produced 4.6M rows, and 3.1% of the rows present in both runs carried a different label. The pipeline selects `WHERE event_ts <= now()` and joins the current label row. Which change makes Wednesday's re-run reproduce Monday's 4.2M rows and their labels exactly?
MCQ-5A nightly training pipeline validates each incoming batch against a frozen schema before training starts. The gate blocks the run when more than 1% of rows violate the schema, and warns but proceeds when the violation rate is between 0.1% and 1%. Last night's batch had 500,000 rows. The upstream service changed `device_type` from an enum of four known values to free-text strings; 7,500 rows carry a value outside the enum, and no other column changed. What does the gate do, and what is the correct immediate action?
MCQ-12A pipeline trains a candidate fraud model and logs AUC 0.838 and precision@0.5 0.68 to the tracking server. Before registration, a conditional evaluation step reads two thresholds from the pipeline config: candidate AUC must be at least the production model's 0.84 minus 0.005, and precision@0.5 must be at least 0.70. The registered production version has precision@0.5 0.66. What does the pipeline do?
MCQ-13An Evaluator in a TFX pipeline runs TFMA with two thresholds: `auc >= baseline_auc` and an absolute `precision >= 0.80`. Its `blessing` output is wired to the Pusher's `model_blessing` input. Today's candidate scores auc 0.79 and precision 0.84 on the eval split; the baseline model currently serving was evaluated on that same split and scored auc 0.80 and precision 0.86. The Evaluator run finishes without error. What is the outcome for the candidate?
MCQ-10A fraud model is evaluated on 10,000 transactions, 1,000 of them fraudulent. Overall recall rises from 78% in v1 to 80% in v2. The new-merchant slice is 500 transactions containing 200 frauds; its recall falls from 60% in v1 to 40% in v2. Which statement about the v2 evaluation is correct?
MCQ-9A team's candidate ranker scores NDCG@10 = 0.431 on 30 days of logged traffic. The champion it would replace scores 0.428 on the same logs, and a popularity baseline scores 0.430. Bootstrap resampling over days puts the 95% interval on any pairwise difference at ±0.004. The team plans to promote because 0.431 > 0.428. What does the evidence support?
MCQ-1A credit-default model has run unchanged for six months. Its drift monitor computes PSI against the training window and alerts above 0.2. This month PSI is 0.05 for income and 0.04 for loan_amount, but on loans that have now matured the default rate rose from 3.1% to 5.4% and the calibration slope fell from 1.00 to 0.72. What should the team do?
MCQ-16A team retrains its ranking model every week on click labels logged from the previous week's live traffic. Why does this retraining create feedback bias?
MCQ-17A fraud-detection model trains on a vendor-supplied labeled feed plus internal transaction logs. A monitoring job finds that 0.4% of the vendor's rows in the last two weekly drops carry labels flipped relative to internal corroborating signals. The vendor feed supplies both the training and the held-out evaluation rows, so the candidate's validation AUC is unchanged at 0.94. The candidate is scheduled to ship in three days, the previous version is still serving, and chargebacks take 60-90 days to resolve. What should the team do?
MCQ-18A fraud model retrains daily. A trigger fires when the raw label table holds ≥ 5,000 rows newer than the pipeline's watermark. Training eligibility requires label maturity ≥ 30 days (the chargeback window), and a run advances the watermark to the newest label timestamp it consumed. Today the trigger fires: 5,200 rows sit above the watermark, all with label timestamps in the last 9 days. What should the run do?
MCQ-24An hourly drift monitor launches a retraining run whenever drift exceeds a statistical threshold. Each run costs about 40 GPU-hours, and over the past 24 hours the candidate has failed validation against the incumbent baseline on all six runs. The drifted windows pass data-quality checks, the shift is confirmed real, and the team wants retraining to stay event-driven so genuine drift is still acted on. Which control keeps the pipeline from re-triggering every hour while the incumbent keeps serving?
MCQ-11The first learned ranker for the search page is ready to ship. The registry holds no champion: the incumbent ordering is a hand-tuned heuristic that was never registered, last measured offline at recall@20 = 0.205 on the same frozen holdout. Release policy: a candidate must reach recall@20 >= 0.30 on the frozen holdout and p95 scoring latency <= 60 ms, then serve 5% of traffic for 14 days with a 48-hour metrics review before any ramp; the 2% relative-lift gate applies only when a registered champion exists. The ranker's holdout numbers: recall@20 = 0.286, p95 = 47 ms; its training-window recall@20 is 0.341. What does this first-run evaluation support?
MCQ-25A payments service is promoted to 100% of traffic by an automated canary controller that evaluates 5xx rate in 1-minute windows of at least 1,000 requests. Postmortems show every real regression in the past year held 5xx above 3% for at least two consecutive windows, while cache-warmup spikes after a deploy routinely push a single window above 3% and clear within 60 seconds. Normal traffic never exceeds 0.4%. Which condition should fire the automatic rollback?
MCQ-15A ranking model ships behind a canary at 5% of traffic: 20,000 of 400,000 daily sessions. Its release guardrail pages when canary-wide conversion, over a rolling 6-hour window, falls more than 0.25 pp below control. After 24 hours the alarm is still green, canary 3.26% against control 3.44%, but the slice table shows EU Android, 2,000 of the canary's 20,000 sessions, converting at 2.0% against 3.6% for control, a 1.6 pp gap. Assuming the slice keeps its 10% share of canary traffic, what is the smallest conversion drop confined to that slice that would make the fleet guardrail fire?
MCQ-7build_features runs in a pipeline that builds each step's cache key by hashing the component image digest, the step's parameter values, and each input artifact's URI; caching is switched on. Run 1 read 12,000,000 rows from gs://lake/events/part-000.parquet and wrote a 4.1 GB feature table. Ops then backfilled that same path in place, and it now holds 14,200,000 rows; image, parameters and step code are untouched. Run 2 reports build_features as cached, its output artifact digest equals run 1's, and the training step downstream logs rows=12000000. Which diagnosis of that hit is correct?
MCQ-6A nightly retraining DAG resolves the `latest` snapshot alias in its first task and writes the resolved snapshot id into the run's idempotency key; the remaining tasks each read the `latest` alias themselves when they start. The orchestrator retries a failed task up to three times under the same key. On 11 June the 02:00 run resolved `s-8841` at 02:00:03. Task 3 hit an OOM at 02:31; snapshot `s-8842` had been committed at 02:20. The 02:33 retry of task 3 read `latest`, trained on `s-8842`, and registered model version 47, which the promotion step then promoted. The team expected the retry to reproduce the 02:00 run on `s-8841`. Which change makes the retry reproduce the original run?
MCQ-14A recs service runs a 6-pod Deployment that loads a single object, s3://models/recs/current/model.onnx, at pod start, and every training run overwrites that key on completion and then triggers a rolling restart. On 4 May an overnight backfill run trained on 3 days of partial click data overwrote the key with no review step in between; all 6 pods restarted onto it within 90 seconds and top-10 CTR fell from 4.1% to 2.2%. The 3 May object had already been overwritten, so the team burned 7 hours retraining to recover. Audit wants the training commit, data snapshot and approver behind the model that served at 10:00 on 20 April, and the team wants to return to the last approved model in one operation without retraining. Which change to the pipeline meets all of that?
MCQ-19Your team's model registry accepts POST /v1/models/{name}/versions with an Idempotency-Key header, and its spec says: the first request with a new key inserts a row and returns 201 with {"version": N}; a later request with the same key inside 24 hours returns 200 with that stored body, inserts nothing and consumes no version number; a request with no key always inserts a new row; a key counts as in use only once its row commits. Run 4711 registers the recs-ranker candidate. The payload dict is built once, before the retry loop. The client (Python 3.12, requests) runs up to three attempts: r = requests.post(URL, json=payload, headers={"Idempotency-Key": "run-4711"}, timeout=1.0), then r.raise_for_status() and break on a returned response, otherwise sleep(5) and retry on requests.Timeout or requests.HTTPError. recs-ranker has 11 versions when the run starts, and the registry's database is failing over during the run: the first POST (t=0.0 s) reaches the insert path and commits v12 at t=3.1 s, after the client's 1 s read timeout fired at t=1.0 s and the client stopped waiting. The loop's second attempt sends the identical payload under the same key at t=6.0 s. When the pipeline exits, what does the registry hold and with what exit status?
MCQ-8promote(candidate, expected_current) in the fraud-model registry applies one write per family at a time: if the registry's current production version equals expected_current it writes the candidate with the next version number (v17 -> v18 -> v19), otherwise it returns 409 and writes nothing. Three promotion workers each read current production, v17, at 09:00, run their offline gate, then call promote with the value they read; on a 409 a worker re-reads the current version and calls promote again with that refreshed value, up to three times, without re-running the gate. Their gates finish in this order: C-B (data through Aug 12) at 09:12, C-C (data through Aug 19) at 09:20, C-A (data through Aug 5) at 09:34. When the last worker exits, which candidate is serving and at what version?
MCQ-20A nightly retraining pipeline fires 60 model jobs from a single cron entry at 02:00 UTC. Each job runs 20 minutes on a shared pool of 8 workers; the pool rejects any job beyond 8 concurrent with a 429, and a job that finishes writes a candidate version and calls the promotion API, which promotes whichever candidate for that model finished last. On the current schedule 52 jobs are rejected at 02:00 and retried at 02:05, and twice this month two candidates for the same model were promoted 3 seconds apart, the second silently replacing the first. Which change removes both the 02:00 rejection wave and the double promotion?
MCQ-23A payments team automates retraining for its fraud model: the pipeline promotes a candidate when a drift alert fires and offline AUC clears 0.91. Last quarter it launched 340 training runs (up from 55) and cut median time from drift alert to promotion decision from 6 days to 11 hours. It made 96 promotions with 5 rolled back within 72 hours, against 41 promotions with 3 rollbacks the quarter before. Split by merchant tier, top-tier promotions went 22 → 31 with rollbacks 1 → 4, and small-merchant promotions went 19 → 65 with rollbacks 2 → 1. The dashboard headline reads "340 runs: 6x throughput." Which reading of the quarter's numbers is correct?
MCQ-22A nightly Kubernetes training job reads a dataset from S3 and pushes a model image to Amazon ECR. It runs as service account `trainer` in namespace `ml`, mapped via OIDC to IAM role `TrainerRole`. The role trust policy condition is `sub: system:serviceaccount:ml:trainer`. The job reads S3 successfully. When it tries to push to ECR repository `models`, it fails with `AccessDenied` on `ecr:InitiateLayerUpload`. The ECR repository policy grants `TrainerRole` `ecr:PutImage` on the repository. What is the most likely cause?