Skip to content
Tech Interview Prep
Technical interview guide

Canary Releases & Rollback for Models

Rolling out a new model version safely: a small traffic slice first, with a fast path back to the previous version.

Read
45 min
Practice MCQs
25
Interview Q&A
25
Edition
v4
Editorial status
Reviewed

Scope: Amazon SageMaker deployment guardrails, Argo Rollouts, KServe, Kubernetes, and NIST AI RMF guidance current 2026-09-04.

Curated: · Written: · Reviewed:

25 questions

MCQ-2A payments team is replacing its pricing engine. Before any user sees the new engine, they mirror 5% of live requests to it, discard its responses, and diff them offline against the current engine's output. Which property does this setup establish?
MCQ-10A 40-instance service behind a load balancer must be able to revert to the previous version within seconds if error rates spike after a release. Which risk is specific to rolling replacement rather than blue/green?
MCQ-15A checkout service runs v1 and v2 side by side behind a load balancer. Rollback means shifting 100% of traffic back to v1; no other step runs. v2 is at 10% of traffic when its p99 reaches 900 ms, and the on-call rolls back. Which v2 change makes that traffic shift insufficient on its own?
MCQ-24A canary release of checkout-service v42 at 5% traffic raised p99 latency from 40 ms to 180 ms and 5xx from 0.2% to 7%. The team rolled back to v41; latency and errors returned to baseline. What should the team do next?
MCQ-22A progressive-delivery controller is rolling out a new version behind a canary. The canary is at 30% of production traffic, and its analysis step has a 15-minute deadline. The metrics backend has been returning 503s since the step began, so the analysis has collected no samples and returned no verdict. The team's release policy is that no version may serve production traffic without a passing analysis verdict. The deadline expires. What should the controller do?
MCQ-23A service runs 60 pods of v1 behind a Service. At 10:00 a canary rollout promotes v2 to 100% of traffic, and the controller is configured with a 300 s scale-down delay for the previous ReplicaSet. At 10:02 p99 latency jumps from 40 ms to 210 ms and the on-call rolls back to v1. What happens?
MCQ-9A model-serving release promotes the candidate to 100% of production traffic at 09:00 and declares a 30-minute bake before the baseline deployment is removed. To free capacity for the bake, the autoscaler drops the baseline from 12 replicas to 0 in a single step; each replica needs about 90 seconds of model load and connection warm-up before its readiness probe passes. At 09:09 the candidate's error rate climbs from 0.2% to 4.1%, and the on-call moves the traffic weight back to the baseline. How long after that weight change until baseline replicas serve production traffic?
MCQ-5A team canaries a build at 5% of traffic for 30 minutes and promotes when error rate and p99 match the baseline. Last week a build passed, then OOM'd the fleet two hours after full rollout: the canary had served 400 requests, nearly all synthetic health checks, and the leak needs roughly 10k real requests to surface. Which change to the bake period would have caught it?
MCQ-4A canary bake promotes only if the candidate's error rate is within 0.1 pp of the baseline's, measured over at least 10,000 requests per arm. The controller is configured to fall back to the edge proxy's access logs when the primary metrics agent is unavailable. In one bake the candidate arm's primary metrics agent crashed at minute 2 of the 30-minute window, so the primary source recorded 0 candidate requests. The fallback source recorded 40,000 candidate requests at a 0.45% error rate; the baseline arm recorded 240,000 requests at 0.4%. Which verdict should the controller emit?
MCQ-11A canary controller compares candidate against baseline on request error rate. Policy: promote when the 95% CI on the difference sits entirely below the +0.5 pp non-inferiority margin; roll back when the CI's lower bound exceeds +0.5 pp; the controller must promote or roll back by the 2-hour maximum analysis duration. After 30 minutes at 5% traffic: candidate 12,000 requests, 156 errors (1.30%); baseline 240,000 requests, 2,400 errors (1.00%). The 95% CI on the difference is [+0.09 pp, +0.51 pp]. What should the controller do now?
MCQ-20A payments team is about to canary a new build of its authorization service. Before the canary they run an A/A test: two cohorts of the current build, 200k requests each, repeated five times, and the p99 latency difference between the two identical cohorts ranges from −6 ms to +6 ms. The canary then measures candidate p99 52 ms against baseline p99 46 ms over the same 200k-request sample size. What does the A/A result imply for reading the canary?
MCQ-13A B2B SaaS team tests a new onboarding checklist against the current flow. The design is registered: one analysis at the end, alpha 0.05, 12,000 signups per arm, about 80 percent power against a 1.0 point lift from an 8.0 percent trial-to-paid baseline. On day 9 the analyst has 5,000 signups per arm and reports 9.2 percent trial-to-paid in treatment against 8.0 percent in control, two-sided p = 0.03; the same cumulative test has been run every morning for nine days. What is the most defensible reading of that day-9 result?
MCQ-12An A/B test assigns 50,000 users to each arm. Exposure logs show 45,000 control and 38,000 treatment users actually saw the variant. Among exposed users, control converts at 10.0% and treatment at 12.0%. Among all assigned users, control converts at 9.2% and treatment at 9.6%. What is the most likely explanation and correct next step?
MCQ-17An ads team is deciding whether to ship a new ranking model. Last week the old model v1 converted 3.00% on 1,000,000 sessions. This week the new model v2 converted 3.50% on 1,000,000 sessions, so the weekly dashboard shows +0.50 percentage points. Because a shared feature store was slow for part of this week, the team also ran a 50/50 split during a six-hour clean window: v1 converted 3.40% on 500,000 sessions, v2 converted 3.45% on 500,000 sessions. What should they report as the estimated release effect of v2 on conversion?
MCQ-3An experiment on a checkout redesign assigns variants by hashing the session ID, so each of a user's sessions is randomized independently. Over the two-week window users have 3 sessions on average, and the split is 50/50. The metric is whether the user purchases at least once during the window. A separate holdout study estimates the true per-user effect of the redesign at +2.0 pp (10% → 12%) for users who see only one variant; a user who sees both variants converts at the average of the two rates. What lift should the session-ID experiment measure between the treatment bucket (users whose first session was treatment) and the control bucket?
MCQ-1A team ships a new ranking model behind a canary: 5% of production hosts run the new build, the other 95% run the old one. The load balancer assigns each request independently across all hosts, so a signed-in user's page views land on whichever host the balancer picks. After 24 hours the team compares aggregate click-through on canary hosts (4.1%) against baseline hosts (4.0%) and plans to ship. What does this comparison support?
MCQ-6Model v42 has served 10% of checkout traffic for two hours, with v41 on the other 90% and volume flat at 200,000 requests/hour. Before the canary the fleet p99 was 200 ms; now the fleet p99 reads 400 ms. Per version, p99 is 200 ms for v41 and 600 ms for v42, and 8% of v42's requests and 0.2% of v41's are slower than 400 ms. Which reading of the 200-to-400 ms fleet move does the data support?
MCQ-16A payments team runs a 50/50 A/B test of a new fraud-scoring model on 1.2M users. Aggregate hard-decline rate is 0.90% in both arms. A pre-registered cohort of 2,400 users on a legacy card issuer splits evenly: 14 hard declines in treatment, 3 in control. The team's ship gate is 'no cohort with relative harm >2× and two-sided Fisher p<0.05.' What should the team do?
MCQ-21A team wants to promote a new ranking model. Two weeks before the experiment started they filed a review packet: primary metric = sessions with a purchase; four guardrails (p99 latency, crash-free sessions, revenue per session in the top-5 locales, complaint rate); a minimum of 200k sessions per arm; a rule that no protected slice may breach its bound; and version labels on every session. The experiment has now run. Which evidence package supports promotion most credibly?
MCQ-8A payments team runs a candidate release at 10% of traffic. Predefined stop rules: 1. Roll back immediately on any confirmed candidate-caused data corruption or cross-tenant disclosure. 2. Roll back if the candidate's p99 latency exceeds 1.5× the control's, evaluated once the candidate has served at least 5,000 requests. 3. Roll back if the candidate's 5xx rate exceeds the control's by 0.5 pp or more, evaluated once the candidate has served at least 10,000 requests. 4. Treat conversion changes as actionable only at p < 0.05. After 5,400 candidate requests (control: 54,000): candidate p99 = 232 ms, control p99 = 145 ms; candidate 5xx = 1.1%, control 5xx = 0.7%; conversion 3.2% vs 3.5%, p = 0.18; two duplicate-charge reports, both traced to a PSP outage that also affected the control. What do the rules require now?
MCQ-25A fraud team ships weekly and has one promotion gate: any candidate whose error rate on the replay set stays under 0.5% ships. Labels for the small-merchant cohort arrive about 30 days late. Last quarter a candidate cleared at 0.31% and shipped; chargebacks on merchants processing under 500 transactions/day then rose 4x within 48 hours, while the overall error rate stayed flat. The small-merchant cohort is about 4% of replay traffic, roughly 4,000 requests per run. Which single criterion is unsafe to promote on here?
MCQ-14A canary model redefines the aggregation window of an existing feature and adds two new ones. Traffic can be shifted back to the frozen baseline model in under a second, and that artifact cannot be rebuilt. Which feature-store schema change keeps the rollback path safe?
MCQ-19A payments service runs v2 behind a flag. v2 enqueues `charge` jobs to a shared queue; each job carries `schema_version: 2` and an `idempotency_key`. v1 workers reject any job whose `schema_version` is not 1 and route it to the dead-letter queue. Flipping the flag stops new v2 dispatch but does not interrupt jobs already running on v2 workers. At rollback time the queue holds 4,000 v2 jobs and 300 jobs are in flight on v2 workers. The team flips the flag to v1 and leaves the queue alone. What happens to the queued and in-flight jobs?
MCQ-7A fraud-scoring service ships v57 as one release: new weights, tokenizer v3 (50,304 tokens), a new transform config, runtime 2.4, a dependency lockfile, and a migration that adds a `device_risk` column already backfilled and read by the downstream case-management service. After deploy, p99 latency rises from 45 ms to 210 ms and the 5xx rate from 0.2% to 4%. On-call re-tags `model:prod` to the v56 weights; latency stays at 210 ms and 5xx stays at 4%. They then revert the runtime to 2.3; latency falls to 48 ms but 5xx climbs to 11%, with logs showing `unknown token id 50257`. The v56 weights were trained against tokenizer v2's 50,257-token vocabulary, and `device_risk` cannot be dropped without breaking case-management. What should the rollback have restored?
MCQ-18A flag service issues break-glass overrides as signed tokens carrying `scope: [flag]` and `expires_at`, and the rollout controller re-checks the token at every step. A 6-hour data backfill was authorized under a 4-hour token; at hour 4 the controller re-paused the rollout and killed the job mid-write, leaving 3 of 7 shards migrated. The team still wants every override bounded and audited. Which change prevents this class of failure?