Overview
Curated: · Written: · Reviewed:
Canary rollouts and safe model releases
Why interviewers ask this
At senior and staff level, the rollout question is rarely "what is a canary?" It is usually a scenario: "You have a new ranking model that looks 3% better offline. How do you ship it?" The interviewer is testing whether you think in terms of exposure, evidence, and reversibility — whether you treat a model release as a risk-management problem rather than a deploy step. A weak answer names the techniques (canary, A/B, shadow) like vocabulary flashcards and never says what decision each one buys. A strong answer picks a technique for a stated question, sizes the exposure against the risk, and knows exactly what rollback does and does not undo.
Common follow-ups to expect:
- "Your canary looks clean after 24 hours. Do you promote?" — probing whether you know about detection floors and delayed ground truth.
- "Rollback just means redeploying the old model, right?" — probing whether you know the model is a bundle, not a file.
- "What metric do you gate on?" — probing whether you separate guardrail metrics from quality metrics.
- "The canary regressed latency but improved conversion. Ship or not?" — probing whether you have pre-agreed triggers instead of mid-ramp judgment calls.
The mental model: a release is a bundle, and the ramp buys decisions
A model rollout changes more than weights. It can change preprocessing, feature definitions, tokenizers, dependencies, runtime images, hardware, thresholds, policy logic, and downstream actions. Package all of it as one immutable, versioned release, and keep the previous complete bundle deployable as a unit. A rollback that restores only the model file can leave the incident in place — the new feature transformation or prompt template stays live against old weights, a combination nobody evaluated in either direction.
A canary is operationally three things: a small live slice of traffic, a stepped ramp (1% → 5% → 25% → 100%, say), and a gate at each step. The gate is a decision: advance, pause, or roll back, judged against thresholds you set before traffic moved. What the ramp buys over a big-bang swap is staged evidence with bounded harm — each step either produces evidence that justifies wider exposure or stops before the harm grows. A big-bang swap gives you one observation, taken at maximum exposure, with no comparison cohort left running.
The ladder: which rung answers which question
Each technique answers a different question, and confusing them is the most common interview mistake.
| Rung | Traffic | Users affected | What it tells you | What it cannot tell you |
|---|---|---|---|---|
| Offline eval | None | None | Accuracy/quality on a fixed dataset | Serving behavior, latency, user response |
| Shadow / dark launch | Copied production requests | None | Compatibility, latency, disagreement rate, serving bugs | Any user response — no treatment effect exists |
| Canary | Small live slice | Real users, bounded | Guardrail failures, live quality signals, early product signal | Small or delayed quality effects (detection floor) |
| Progressive ramp | Growing share | Real users | Evidence at each exposure level | Nothing new — it is a canary with more steps |
| A/B experiment | Randomized, sticky assignment | Real users | Causal product effect with confidence intervals | Technical failures (that is not its job) |
Shadow mode replays representative production traffic to the candidate with outputs kept out of user decisions. It catches serving and skew bugs — schema mismatches, feature drift between training and serving, latency blowups — at zero user risk, but it tells you nothing about how users respond, because no user ever sees the output. The canary is where real users, real risk, and real feedback start. If an interviewer asks "shadow or canary first," the answer is shadow when you doubt the plumbing and canary when you doubt the model.
Blue/green (separate old and new fleets, controlled switch) and rolling replacement are service-deployment variants: blue/green keeps a clean concurrent baseline at capacity cost; rolling saves capacity but changes instances incrementally, which complicates both comparison and recovery.
What you watch: three metric families
Model rollouts differ from ordinary service rollouts because the failure you are guarding against splits into three families, and interviewers probe whether you keep them separate.
Service/guardrail metrics — error rate, p99 latency, saturation, timeouts, queue depth, cost. These fail the release regardless of any quality gain. A model that is 3% more accurate but doubles p99 is not shippable, and the gate should say so mechanically.
Model-quality signals — prediction distribution shift, score and calibration drift, feature drift, safety-filter violations, schema validity, proxy quality metrics (click-through, accept-rate of recommendations). True quality is often only knowable after ground truth arrives — a fraud model's labels take days, a loan model's defaults take months — so during the canary you gate on proxies and distributions, and you state explicitly that final judgment is deferred.
Business metrics — conversion, retention, revenue. Noisy, slow, and the reason a canary is usually underpowered for them (see the worked example below).
Define the promotion contract before traffic moves: thresholds per metric, error budgets (how much residual harm the release is allowed to consume before it stops, and what the budget spends down against), minimum sample or exposure, observation windows, missing-data behavior, multiple-comparison controls where you run many slices, and who may promote, pause, or roll back. Route cohorts intentionally: sticky assignment by a stable user-level unit for behavioral comparisons (per-request randomization lets one person see both variants and contaminates the comparison), deliberate exclusion of employees, bots, and duplicate retries, and assignment recorded separately from successful exposure. Keep tenant, region, language, device, and risk-tier slices visible so aggregate success cannot hide concentrated harm.
Compare candidate and baseline concurrently whenever possible. Shared incidents and demand shifts otherwise read as candidate regressions. Validate telemetry before rollout — missing metrics or stale ground truth should stop promotion, not count as success. And check the baseline is itself healthy: a fleet-wide blended alarm can trigger on an old problem or dilute a candidate-only failure. Alarm per version, with a concurrent baseline.
Sizing the canary and the detection floor
Canary size comes from risk, traffic, detection power, and operational capacity — not a universal percentage. A tiny cohort limits harm but may never observe rare failures; a large cohort detects faster but exposes more users. The baking period must cover enough representative eligible traffic, not wall-clock time: time passing with no eligible requests is not evidence, and one quiet interval cannot validate a seasonal system's peak behavior.
Worked example (hypothetical figures, but the arithmetic is the point). Suppose 200,000 requests/day and a 1% canary. A 24-hour bake observes about 2,000 requests. Baseline conversion is 4%, so the expected conversions under the baseline are 2,000 × 0.04 = 80. A collapse to 2% yields an expectation of 40 — a drop of 40 against a standard deviation of √(80 × 0.96) ≈ 8.8, i.e. about a 4.6-sigma drop, which you will see. A drop to 3.7% yields 74 expected conversions against 80 — a 6-conversion drop, well inside one sigma, invisible. And 3.7% is the size of regression a model change usually produces. Compute the detectable effect from cohort size and baseline variance before starting; if the answer is "only a catastrophe would show," say so out loud and use the canary for what it can catch — schema violations, error rates, tail latency, safety filters — while sizing a longer, larger experiment for the quality question. Reporting "no regression detected" without the accompanying floor turns an underpowered test into false assurance. This exact calculation is a strong answer to the "canary looks clean, promote?" follow-up.
Rollback: triggers, mechanics, and what it cannot undo
Triggers must be defined before the canary starts, not improvised mid-ramp. Split them by speed: fast, automatic triggers for severe safety, privacy, security, schema, corruption, and availability failures — these should fire without a human in the loop; statistically disciplined triggers for noisy product metrics, because an automatic rollback on a noisy conversion dip will roll back good releases and train everyone to ignore the alarms. Guardrail metrics fail the release regardless of quality gains; that rule has to be written down in advance, along with the error budget that says how much harm the canary is permitted to cause before the gate closes, because mid-ramp the temptation to trade latency for accuracy is enormous.
Mechanically, rollback is a pointer swap in an artifact/version registry back to the previous immutable release — not a rebuild. That is why you keep the previous bundle deployable as a unit, rehearse the reversal on the same automation and permissions you will use under pressure, and time it: a rollback path that takes 40 minutes because it rebuilds an image is not a mitigation for an incident measured in minutes. After routing traffic back: drain or reconcile queues and in-flight requests, restore caches and feature contracts where needed, verify the loaded release identity, and validate service and model behavior.
Then account for what rollback cannot undo. Predictions already served, records created under the candidate, actions dispatched to users — routing traffic back stops new harm without undoing old harm, and the compensating work (re-scoring, reprocessing, customer remediation) is usually the longer half of the recovery. Some changes are not reversible at all: database migrations, feature backfills, schema removal, irreversible external actions, privacy disclosures, client contracts. Use expand-and-contract compatibility, dual reads/writes when justified, versioned events, and compensating actions — and test the rollback before rollout, with the same automation an incident would actually have.
Promotion is a state machine, not a dashboard glance
Each step moves a declared share, observes evidence, and either advances, pauses, or rolls back. Make control operations idempotent and auditable. Prevent two operators or automations from advancing competing versions. Apply maximum rollout duration and safe timeouts, and require explicit recovery from an inconclusive state — "paused for 6 hours with no verdict" needs a decision rule, not silence.
After full cutover, retain the baseline through a final observation window sized to your delayed signals, then terminate. Keep monitoring after 100%: rare and delayed outcomes appear after promotion, and promotion means the evidence supports broader exposure, not permanent superiority. Record release identity, cohorts, metrics, decisions, exceptions, and approvers, and feed incidents and false alarms into the next rollout's thresholds.
Quick self-check
- Shadow vs. canary in one sentence each: shadow replays traffic with no user impact and catches serving bugs; canary serves real users and buys live evidence at bounded risk.
- The three metric families, and which one fails the release regardless of quality gains: guardrails.
- Before promoting a clean 24-hour canary, state the detection floor out loud.
- Rollback restores a release bundle, not a model file — and it does not un-serve predictions already made.
- Triggers, thresholds, error budgets, and who may roll back are decided before traffic moves, not during the ramp.