Skip to content
Tech Interview Prep
Technical interview guideNew

Decision Models (System One)

Typed decision models return calibrated probabilities over options you name, in one forward pass, so code can branch on a model's judgement without parsing generated text.

Read
9 min
Practice MCQs
24
Interview Q&A
25
Edition
v1
Editorial status
Reviewed
Relevant for
AI / ML Engineer

Scope: Current to Jev 1.14.0 (TypeSafe, System One API at POST /v1/systemone, docs dated 2026-11) and to the open Kev models (0.8B to 27B, served via Ollama) as published in late 2026. Vendor figures for price, latency and benchmark accuracy are vendor-reported claims, not independently verified. Question types differ by version: Jev 1.13 supports Choice, Score and Noul; Jev 1.14 adds Box..

Interview Q&A

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1

Explain what a decision model is and how it differs from calling a chat LLM with structured output. What does that difference mean for the code that calls it?

Answer

The 60-second answer. A decision model takes an input and a set of options that the caller supplies, and returns a probability for each option. The probabilities sum to one, the type is fixed by the contract, and the caller chooses what to do with them. It does not write a sentence that happens to name an option. A chat model called with structured output does generate tokens, one at a time, conditioned on a schema. The schema constrains the shape of the text, but the result is still a sampled sequence, so the caller parses it, validates it, and handles the cases where the model picks an option that was not in the list or where the JSON is malformed. With a decision model the option set is an input, not something the model invents, and the output is a distribution the code can read directly.

Mechanism and what it changes. The difference is in the output space. A structured-output call maps to an open space of strings that is then checked against a schema. A decision model maps to a closed space: for n caller options, the output is a vector of n numbers. That gives three consequences for calling code.

// Hypothetical shapes, for illustration only
type Decision = { options: string[]; probs: number[] }; // probs.length === options.length

function route(d: Decision, minConfidence: number) {
  const [best, p] = argmax(d.probs);
  if (p < minConfidence) return { action: "escalate" };
  return { action: d.options[best] };
}

First, an unknown option cannot appear, so the parse-and-retry branch disappears. Second, the threshold is a real decision the code owns, and it has to be tuned against calibration. A probability of 0.7 is only useful if about 70 percent of predictions in that band are right on held-out data. Third, the output is a score, so tests check the distribution: for example, that probabilities sum to one within a tolerance, and that the threshold sends 20 percent of a labelled set to escalation.

Failure modes. The main one is miscalibration. The top option can be wrong with confidence 0.9, and the code acts on it. Detect this with a reliability diagram on a labelled holdout, and track expected calibration error per release. A second is an option list that changes between calls: a probability vector is only comparable across calls if the options and their order are the same, so key by option ID rather than by index. A third is treating a decision model as a generator, asking it for an explanation it cannot give. Keep rationale generation in a separate call if the product needs it.

When not to use one. If the answer space is open, such as a free-text summary, use generation. If the options are few, stable and fully enumerable, a small classifier or a rule may be cheaper and easier to audit. The decision model earns its place when the options are supplied at call time and the judgement has to be a number the code can threshold.

Curated: · Written: · Reviewed:

Q&A-2

For a typed decision call, what does the response return for each named question you send, and how do you match each result back to its question?

Answer

Each named question comes back as one entry: a probability for every option in that question's option set, plus a confidence in the answer. The probabilities for a question sum to one over its own options. The caller matches each entry to the question it sent by the question's name, not by position.

Illustrative shape (field names are hypothetical; the System One API reference has the real schema):

{ "escalate": { "probabilities": { "yes": 0.81, "no": 0.19 }, "confidence": 0.74 } }

The name is the join key. The response can arrive in any order, and one call can carry several questions, so code that reads results by index will attach an answer to the wrong question. A caller should check that every name it sent appears in the response, and treat a missing name as an error rather than filling in a default.

Confidence is reported separately from the probabilities. The top probability is a property of the option distribution; confidence is the model's signal about the answer itself, so the two should be read as different fields.

Curated: · Written: · Reviewed:

Q&A-3

You have three decision questions to answer per ticket: which queue (billing, outage, account, other), whether it is a duplicate of an open ticket, and a severity from 1 to 4. Which question type fits each, what would you lose by forcing the severity into a yes/no or the queue into a score, and how would you check the calibration of each before trusting it in production?

Answer

Start with what each downstream step does with the answer. Routing sends the ticket to exactly one queue, so the output is a choice among named options: one mutually exclusive label with a probability over the set. Duplicate detection is a yes/no question with a threshold, because the branch is binary: merge or do not merge. Severity is an ordered score: 1 to 4 where 4 pages someone, so the levels carry an ordering that matters to the action taken.

The rule I would use: match the question type to the shape of the action, not to the shape of the model's output. A routing decision with four named queues should not be split into four yes/no questions, because the four probabilities are not forced to sum to one and the caller ends up choosing a winner with no defined tie-break. A severity rated as yes/no ('is this severe?') throws away the middle levels, and the threshold it needs is arbitrary. A severity forced into named options (low, medium, high, critical) loses the ordering: the caller cannot ask for the probability that severity is at least 3, which is exactly what an on-call rule needs.

What you lose by getting this wrong, concretely. Suppose the severity call returns the probability of each level as a categorical distribution. The paging rule becomes P(level >= 3) = P(3) + P(4), a single arithmetic step the caller can read. If you collapsed it to a yes/no 'urgent', the rule becomes P(urgent) > 0.5, and that threshold silently moves the page rate when the base rate of severe tickets changes. Hypothetical figures: if 2 percent of tickets are severity 4, a model that is 0.9 confident on everything it marks urgent pages far more often than the on-call budget allows.

Calibration checks differ by shape. For the routing choice, take a labelled sample of 500 tickets, bin predicted probability of the top queue into deciles, and compare the observed accuracy in each bin with the mean probability. Check the top-1 error on the 'other' queue separately, because it is where the model will hide uncertainty. For the duplicate yes/no, plot precision and recall at the threshold you intend to ship, not at 0.5, and measure the false-merge rate, since a wrong merge loses a customer message and is worse than a missed duplicate. For severity, check that predicted probabilities are monotone in the human label and measure the expected calibration error on the >= 3 aggregate, since that is what the caller uses.

Failure mode to name: a model can be well calibrated on the majority queues and badly calibrated on the rare ones. Overall ECE can look fine while the 'outage' bin is wrong by 0.2. Detect it by reporting calibration per class, not only in aggregate.

Finally, I would not trust any of this without a distribution shift check: retrain or re-measure calibration after the product launches a new category, because the probabilities were calibrated against the old label set.

Curated: · Written: · Reviewed:

Q&A-4

A team moves a classifier from autoregressive JSON generation to a single-forward-pass decision model. Explain what changes in latency, cost and type safety, and what rationale they lose.

Answer

The 60-second answer. An autoregressive model produces one token per forward pass, and each new token depends on every token before it. A JSON answer of 120 tokens therefore costs 120 sequential decode steps before the call returns, and the caller pays for all of them. A decision model with a typed output head runs one forward pass over the prompt and reads the answer from fixed output positions. Runtime then depends on prompt length and a constant number of output heads, not on how much the model would have written. The output is a set of typed fields, so there is no string to parse and no way for the reply to be malformed. What you give up is the chain of reasoning written out in tokens. The model gets no intermediate scratchpad, and the only explanation left is what the typed fields carry.

Mechanism. In generation, the cost of one call is roughly prefill plus output_tokens × per-step decode time. In a decision model, the output heads are linear layers over the final hidden states, so the decision cost is one prefill pass plus a few matrix multiplies. Parsing disappears because the decoder never produces characters: each field is a categorical or numeric head whose value is selected from a declared set. A malformed reply cannot exist, although a wrong one still can.

Arithmetic (hypothetical token counts). Assume a 800-token prompt, a 120-token JSON reply, and 25 ms per decode step on the serving hardware. Prefill for the prompt takes 60 ms.

generation:  prefill 60 ms + 120 steps x 25 ms = 3060 ms, billed 800 in + 120 out
decision:    prefill 60 ms + head readout ~2 ms        =   62 ms, billed 800 in + 0 out

At 10 calls per second the generation path keeps about 31 requests in flight on average (10 x 3.06 s), and the decision path keeps under 1. Output token billing drops to zero, so the cost difference scales with reply length, not prompt length. These figures are illustrative; the ratio, not the milliseconds, is the point.

Failure modes. First, the decision is only as calibrated as the training of the heads. A head that outputs a confident 0.97 on out-of-distribution input is a typed wrong answer, and nothing in the type system catches it, so you check calibration on a held-out set, binned by predicted probability. Second, you lose the rationale a human reviewer reads to debug a bad call. If the team needs an explanation for each decision, that requirement is a reason to keep generation, or to run it on a sample. Third, label drift: adding a category means retraining the head, while a prompt change to a generative model takes minutes. Compare the two on their main trade-off: generation buys flexibility and explanation at the price of runtime that grows with output, and the decision model buys bounded latency and typed output at the price of a fixed label space and no narrative.

Curated: · Written: · Reviewed:

Q&A-5

A decision model gives 0.8 to each of 10,000 'yes' calls and only 6,900 of those are yes. Is its 0.8 confidence calibrated, and how would you show it with a reliability diagram and expected calibration error?

Answer

The 90-second answer. Calibration is a property of the probabilities. A model is calibrated when the outcomes it assigns probability p occur about p of the time, so the 0.8 bucket should be right about 80% of the time. Here it is right 6,900 of 10,000 times, which is 69%. So the 0.8 confidence is overconfident by about 11 points in this bucket. Before I trust that, I would check that the 10,000 calls are a representative labelled sample and that the 0.8 predictions are the only ones in this bucket.

Showing it. Group predictions into bins by confidence. For each bin, compare the mean predicted probability with the observed fraction of 'yes'. A reliability diagram plots observed accuracy against mean confidence per bin, with the diagonal as perfect calibration. Points below the diagonal mean overconfidence.

Expected calibration error averages the per-bin gap, weighted by how many predictions fall in each bin:

ECE = sum over bins of (n_b / N) * |acc_b - conf_b|

Worked example (hypothetical figures, labelled as such). Three bins, N = 10,000:

binn_bconf_bacc_bgap
0.7-0.82,0000.750.660.09
0.8-0.93,0000.850.720.13
0.9-1.05,0000.950.910.04

The weighted sum is 0.20.09 + 0.30.13 + 0.5*0.04 = 0.018 + 0.039 + 0.020 = 0.077, so ECE is about 0.077 across these bins.

What the number depends on. ECE is an estimate, not a fixed fact about the model. Fewer, wider bins smooth the gap but can hide a range where the model is wrong. More bins expose that range but leave fewer predictions per bin, so each accuracy figure gets noisier. With about 1,000 predictions per bin, the standard error on an accuracy near 0.7 is roughly sqrt(0.7*0.3/1000), about 1.4 points. Report the bin count and per-bin n. A bin with five predictions can swing the total.

Keep the diagram and the number together. A diagram with a few well-populated bins near the diagonal can still sit beside a high ECE if a sparse bin is far off, so check n_b alongside the gap. Errors can also cancel: overconfidence in one range and underconfidence in another can give a middling line. The diagram shows where the error sits; ECE summarises how much of it there is. Read both.

What to change. If the model is overconfident across the range, fit a single correction such as temperature scaling or Platt scaling on a held-out split. Use isotonic regression only when there is enough data per bin. Fit the calibrator on data the evaluation does not reuse, or the measured ECE will look better than production delivers. Because the pipeline writes only above 0.9, check the 0.9-1.0 bins first, since that is where the threshold lives.

Curated: · Written: · Reviewed:

Q&A-6

What does it mean for a model's 0.9 score to be calibrated?

Answer

Opening answer. A probability is calibrated when its stated value matches observed frequency: of all the cases the model scores at 0.9, about 90 percent turn out to be positive. Calibration is a property of the score across many cases, not of any single prediction.

This matters for code because a threshold such as if p > 0.9: write() only controls an error rate when the score means what it says. If the 0.9 bin actually matches 60 percent of the time, the branch makes 40 percent wrong writes while the code reads as if it guarantees 10 percent. The threshold is then a false guarantee.

A human reading the same number reweighs it against other evidence, so a miscalibrated score does less harm in their hands. Code applies the number mechanically, so the calibration has to hold in production.

Assumptions: a binary decision and a validation sample drawn from the same distribution as production traffic.

# Reliability check: bin scores, compare mean score with observed rate
for bin in bins(scores, width=0.1):
    print(bin.mean_score, bin.observed_positive_rate)
# calibrated when the two columns roughly match in every bin

Calibration is checked with a reliability table on freshly labelled data, and a fitted correction should be validated on data it was not fitted on.

Curated: · Written: · Reviewed:

Q&A-7

A support assistant has a 200 ms p95 budget before it drafts a reply, and a wrongly approved refund costs about 20 times a manual review. For routing, intent classification, extraction checks, guardrails and output scoring, where do you place a decision model, and where do you still need a generative model?

Answer

Clarify first. Before placing anything I need three facts: whether the 200 ms covers the whole path or only the gating call, whether each decision has a closed label set, and whether anyone can later label outcomes so I can measure calibration. Without the labels I cannot trust a probability, whatever the model outputs.

Placement, stage by stage.

StageFitWhy
Intent routing (refund, shipping, account, other)Decision modelClosed label set, runs on every message, cost of a miss is a slower queue
Refund eligibility gateDecision model, with a thresholdBounded yes/no per policy rule, error cost is asymmetric
Extraction check (does the order id appear in the source text)Decision modelSpan-level match, one probability per field
Guardrail on outbound text (PII, policy terms)Decision model, plus a deterministic regex layerBounded categories, must be fast, must not be generated
Scoring a drafted replyDecision model, asynchronousScores a closed rubric; cannot verify facts absent from the source
Drafting the reply, explaining a refusal, free-text extractionGenerative modelOpen output space, the text itself is the product

The threshold comes from cost, not from 0.5. Auto-approve when the expected cost of approving is below the cost of sending the case to review. With a wrongful refund costing C_fp = 80 (hypothetical, in currency units) and manual review costing C_review = 4, approve only when (1 − p) × 80 < 4, that is p > 0.95. If the calibrated model returns 0.93 on the approval band, the gate routes to review, and the review queue absorbs the cost. Measure that calibration on labelled outcomes per band before the threshold ships.

Latency arithmetic. Suppose a decision model answers in 35 ms p95 and the generator needs 1.8 s p95 for a draft (hypothetical). Only the decision calls sit on the 200 ms path. The draft runs after the gate, asynchronously, and a refusal explanation can be generated after the decision has already been recorded.

Failure modes. A decision model stays confident on inputs unlike its training data, so monitor the distribution of its probabilities, not only accuracy. Rule-based guardrails miss paraphrases, and the decision model misses rare categories, so the two layers need separate recall numbers. A scorer of drafts can reward fluent text that cites a refund policy that does not exist, so give it the policy text as input and score claims against it.

Curated: · Written: · Reviewed:

Q&A-8

A workflow step should issue a refund only when a typed decision model returns a probability at or above a threshold. Explain how you would structure the branch so the decision and its threshold can be audited later.

Answer

The 90-second answer. The model call is a predicate with a typed return value. It returns a probability for one named question, such as whether this request is a valid refund under policy v7. The workflow compares that probability to a threshold and branches on the boolean. The branch never reads prose. Two things need a deliberate place: the threshold lives in versioned config keyed by policy version, and the audit record is written before the refund side effect runs.

Why the threshold is not in the prompt. A value written into the prompt is an instruction the model can ignore, and nobody can diff it against last quarter's policy. A value hard-coded in the branch turns a business change into a deploy. Versioned config lets the policy owner change the threshold and lets you replay a past decision under the threshold that applied then.

// Illustrative shape; check the System One docs for the real call.
const policy = config.get('refund.valid', { version: 'v7' }); // threshold 0.85
const d = await decide('refund.valid', { order, amount, ticketText, policyId: 'v7' });
const take = d.probability >= policy.threshold;
await audit.write({ run, step, decision: 'refund.valid', policy: 'v7',
  threshold: policy.threshold, probability: d.probability,
  inputHash: d.inputHash, modelVersion: d.modelVersion,
  branch: take ? 'refund' : 'human_review' });
if (take) await issueRefund(order, amount);
else await queueForReview(order);

Trace with hypothetical figures. A request returns 0.82. Under v7 (threshold 0.85) it goes to human_review. The same input under v6 (threshold 0.80) would have gone to refund. The log records v7, the threshold and 0.82, so the two outcomes are explained by config, not by rerunning the model.

Failure modes and detection.

  1. Side effect before audit. If the process dies between issueRefund and the audit write, money moved with no record. Write the audit row first as pending, or use an outbox so the side effect and the record commit together.
  2. Threshold set against an uncalibrated probability. A 0.85 means little if the model is overconfident. Check the score against reviewed outcomes in a reliability table before trusting the threshold.
  3. Silent model change. The same threshold now means something else. Pin modelVersion in the log and watch the probability distribution on a fixed canary set after each upgrade.

Alternatives. If the policy can be written as rules, use rules and skip the model. A chat call that returns a reason string cannot be thresholded or compared across versions, so it cannot be the branch condition.

Curated: · Written: · Reviewed:

Q&A-9

A fraud hold branch sends a payment to manual review when the model's fraud probability is above a threshold. A missed fraud costs about $180 in chargebacks and a wrongly held legitimate payment costs about $12 in reviewer time. How do you choose the threshold, and why is 0.5 the wrong starting point?

Answer

Start with the costs, because the threshold is a cost decision that the model only informs. A missed fraud (false negative) costs about $180 in chargeback. A legitimate payment held for review (false positive) costs about $12 of reviewer time. If the model is calibrated, so a score p means a p chance the payment is fraud, the expected cost of holding is (1 − p) × 12 and the expected cost of releasing is p × 180. Holding is cheaper when (1 − p) × 12 < p × 180, which gives p > 12 / 192 = 0.0625. In general the threshold is C_FP / (C_FP + C_FN). It equals 0.5 only when both errors cost the same, which is why 0.5 is a default and not a decision.

Hypothetical figures, labelled as such: 10,000 payments with 100 true frauds. At threshold 0.5 the model holds 60 payments, catching 40 frauds and missing 60. Cost is 60 × 180 + 20 × 12 = 10,800 + 240 = 11,040. At threshold 0.0625 it holds 300 payments, catching 90 and missing 10, with 210 legitimate holds. Cost is 10 × 180 + 210 × 12 = 1,800 + 2,520 = 4,320. The lower threshold cuts expected cost by about 60% on these numbers, but it sends five times as many payments to review, so the review queue is part of the decision.

Two conditions have to hold before that number means anything. First, the scores must be calibrated on data that looks like production. Fit a reliability diagram on a labelled holdout: bin the scores into deciles and compare mean predicted probability with observed fraud rate in each bin. If payments scored near 0.06 are 9% fraud, the formula is using the wrong p and the threshold moves. Fit Platt scaling or isotonic regression on a calibration split, measure expected calibration error there, and keep a separate test split for the final cost report. Second, the costs are estimates. Sweep thresholds against expected cost on the holdout and look at how flat the curve is near its minimum. A flat curve means a $30 error in the chargeback estimate barely changes the choice. A steep curve means the cost inputs need better measurement before you ship.

Failure modes: a campaign raises fraud prevalence, the same score now means something different, and the threshold is wrong until you recalibrate. Monitor the observed fraud rate per score bin weekly, not only the aggregate accuracy. Tuning the threshold on the test split inflates the reported saving, so the test split stays untouched during the sweep. Finally, a cost-optimal threshold can exceed reviewer capacity. Add a capacity cap, and when the queue overflows, decide explicitly whether to release, hold or escalate rather than letting latency absorb the overflow.

Curated: · Written: · Reviewed:

Q&A-10

A decision model returns accept, reject or defer. What does defer tell the caller to do, and what happens to the item?

Answer

The short answer. Defer means the model declined to decide. The caller must not treat it as accept or reject. It routes the item to a path that can decide: a human reviewer, or a stronger model. Until that path returns a decision, the item is not written to the ledger as accepted or rejected.

That is why the three outcomes are not a fancy label on a yes/no. Accept and reject are decisions the caller can act on directly. Defer is a control-flow outcome with an owner. A pipeline that handles defer like a rejection will silently block invoices it could have paid, and one that handles it like an acceptance will pay invoices nobody checked.

The deferral path has to exist before the model ships. If no reviewer or stronger model is wired to the defer branch, deferred items pile up or fall back to a default outcome. Detect this by watching the count and age of unresolved deferred items.

Hypothetical figures: 1,000 invoices, 150 deferred. Those 150 go to a review queue, and the ledger receives writes only for the 850 that were accepted or rejected.

Curated: · Written: · Reviewed:

Q&A-11

Your refund-eligibility classifier assigns each ticket one of seven reasons, and product adds a reason every few weeks. Compare a typed decision model with a chat model using structured output for this job. What does each cost you at a 150 ms p95 budget, and when would you keep the chat model?

Answer

The first 90 seconds. I would start by asking what has to be true at the decision point. Here the branch that matters is whether a ticket gets auto-approved, so the output needs a probability I can threshold, not just a label. A decision model fits that branch: one forward pass returns a typed, calibrated probability per question, and the caller can compare it to a threshold. A chat model with JSON output returns a label and possibly a verbalized confidence, and neither is a probability I can rely on without measurement. The label set changes every few weeks, though, and that cuts the other way: a decision model is trained against a fixed set of questions, so a new reason means new training data and a new version before it can be trusted. For a changing taxonomy I would keep a generative model for labelling and use the decision model only for the stable questions.

What changes, with hypothetical figures. Assume a 2 KB prompt and a 6-token structured reply from a chat model, and a decision model that scores the same input in one forward pass.

Decision modelChat model, JSON output
Calls per ticket1 forward pass1 generation, 15 to 40 output tokens
p95 latency (hypothetical)35 ms180 ms, dominated by decoding
Confidence signalcalibrated probability, if measurednone, or verbalized and uncalibrated
Label set changeretrain and versionedit the schema and prompt
Rationalenonecan be requested, not guaranteed faithful

The chat path fits the 150 ms budget only with a smaller model, a shorter reply or streaming the decision early, and each of those moves the numbers. I would measure, not assume.

Failure modes I would check. A decision model can look calibrated on the validation split and drift once ticket mix shifts. I would keep a reliability diagram per release and alert on expected calibration error. The chat model's JSON can parse and still choose an invalid reason from an outdated list, so I would validate the enum against the live taxonomy and count rejections. For audit, a refund approval needs a stated reason, so if the decision model is used I would store the feature inputs, model version and threshold with each decision, because the model itself will not explain itself.

When I keep the chat model. When the label set is open or changes weekly, when the case needs a written explanation for a customer or auditor, or when there is not enough labelled data to calibrate a decision model per question. The honest answer is a split: a decision model gates the auto-approve branch, and a generative model handles new categories and explanations off the hot path.

Curated: · Written: · Reviewed:

Q&A-12

A decision model returns a calibrated probability for each of your ticket questions. Why does a fine-tuned classifier you host yourself need its own held-out calibration check, and how does that differ from what you can rely on from a hosted decision model?

Answer

The first 90 seconds. A softmax output from a fine-tuned classifier is a score, not a probability in the sense a caller needs. Cross-entropy training pushes the model toward confidence on examples it has already seen, and on fresh tickets it is usually overconfident. Calibration means that among tickets scored 0.8, about 80% are actually positive, on data like production traffic. I host this classifier, so I can refit its output. That also means I have to measure the result on data the calibrator never touched, or I will be reporting a number I fitted myself.

Mechanism. Split the labelled tickets into train, validation and test. Fit any calibrator on validation only, then measure on test, which nothing in the pipeline has seen. Bin the test predictions into 15 equal-width bins, compare the mean predicted probability in each bin with the observed positive fraction, and plot the reliability diagram. The diagonal is perfect calibration. Expected calibration error (ECE) is the bin-weighted mean of the absolute gap:

ECE = sum over bins b of (n_b / N) x |acc_b - conf_b|

Temperature scaling divides the logits by one scalar T before the softmax. T is chosen by minimising negative log-likelihood on the validation split. Because T is a single positive scalar, the argmax never changes, so accuracy stays fixed while confidence moves. Hypothetical figures for illustration: raw test ECE is 0.11; after T is fitted on validation, test ECE is 0.04. The test figure is the one to report, because the validation split was used to fit T and its own ECE would look better than the model deserves.

Hosted decision model. You cannot refit a vendor's logits or see its training data, so you cannot calibrate it the way you calibrate your own model. What you can do is run the same procedure on your labelled tickets: hold out a test split, bin the predictions, and report ECE from there. If the curve is poor, you move the decision threshold using a validation split and confirm the result on test. You give up the ability to correct the curve itself.

Failure mode. After a taxonomy change, such as adding a severity level, the probabilities for the old classes can drift while accuracy looks flat, because the new class absorbs some of the probability mass. Detection: keep a frozen slice of old tickets that is never used for fitting, and track ECE on it at every release alongside aggregate accuracy. A steady aggregate can hide a class whose confidence has gone wrong.

Where the two differ. A self-hosted classifier lets you refit T, retrain and monitor the curve on your own labels. A hosted model gives you a pinned version and a contract, and you verify its calibration from outside. In both cases the branch threshold is chosen on validation data, and its quality is reported on test data.

Curated: · Written: · Reviewed:

Q&A-13

A decision model only returns one of the labels we pass it. For a ticket that fits none of them, what does it return, and how would you catch that before it routes the ticket?

Answer

The model returns one of the labels I passed. It has no path for 'none of these' unless I offer one, so for a ticket outside the set it still picks the nearest label and attaches a probability. That probability describes how the model scores the labels it was given, not whether the true answer is among them. So a confident wrong route is the expected failure, and the probability alone cannot reveal it.

The check has to come from my side of the contract. I would reserve a held-out set of tickets whose true category is absent from the offered labels, for example a 'refund dispute' ticket sent to a router that only knows billing, outage, account and other. Then I measure how often those tickets land in an existing label with probability above the routing threshold. If that rate is high, the threshold does not protect the branch, and I would either add an explicit abstain label or route anything below a higher bar to a human queue.

The trade-off is against a chat model with a free-text field. It can say the ticket fits no category, but that output needs parsing and its probability is weaker evidence. For a fixed, well-covered label set the decision model is the better fit. When new categories appear every few weeks, I would keep the set under review and not trust the model to notice a gap.

Curated: · Written: · Reviewed:

Q&A-14

A decision model returns 0.92 on a branch that later proves wrong. Is that one wrong answer evidence that the model is miscalibrated?

Answer

What I say first. No. A calibrated 0.92 means that across many calls scored near 0.92, about 92 percent turn out correct. Any one call scored 0.92 is right or wrong, so one failure is the expected outcome about one time in twelve. Calibration is a property of a set of predictions, and I cannot judge it from a single case.

What I would do next is measure the rate. I take the labelled outcomes for calls in the 0.9 to 0.95 band and compare the observed share correct with the stated score. If that share is close to 92 percent, the model is behaving as calibrated and this call was in the tail. If it is far below, the calibration has drifted and the score no longer means what it meant at launch.

So the one-line answer is that a single wrong branch at 0.92 is consistent with a well-calibrated model. The question I can answer is whether the rate across the band matches the score, and that is what decides whether the threshold or the model needs attention.

band 0.90-0.95: 400 calls, stated mean 0.92
observed correct: 368  -> 0.92  (calibrated)
observed correct: 290  -> 0.725 (drifted)

The first line is the check. The single wrong call is one row of that table, not the table.

Curated: · Written: · Reviewed:

Q&A-15

How would you evaluate a decision model on your own labelled data before letting a branch act on its probabilities? Walk me through the plan.

Answer

The 90-second answer. I would not judge the model on overall accuracy. I would build a held-out set drawn the way production traffic actually arrives, score each named question separately, check whether the stated probabilities match observed frequencies, and run the model in shadow mode before any branch acts on it. Accuracy alone hides the failure that matters here: a model that is right 90% of the time but reports 0.99 on the cases it gets wrong will push bad actions through a threshold that was set on its numbers.

Data first. The held-out set must match production in label mix, input length, language, channel and time window. A random split of last quarter's tickets usually misses the rare classes that the branch cares about most. I would stratify by the question's labels and keep the set frozen, so later changes to the prompt or model cannot leak into it. For each question, the sample size sets how precise the accuracy can be: at p = 0.95 with n = 400, the 95% interval is about ±2.1 points (1.96 × sqrt(0.95 × 0.05 / 400) ≈ 0.021). A rare label with 30 examples tells you almost nothing, so I would report its count next to its rate.

Calibration, measured three ways. A reliability diagram bins predictions, then plots mean predicted probability against observed frequency per bin. Expected calibration error is the weighted mean gap between those two values across bins. Brier score is the mean squared error between the probability and the 0/1 outcome, and it penalises confident mistakes. Hypothetical figures: 10,000 calls at 0.8, with 6,900 true. Observed rate is 0.69, so the gap is 0.11 and the model is overconfident. Brier for that block is 0.69 × 0.2² + 0.31 × 0.8² = 0.226. A model that predicts the base rate of 0.69 everywhere would score 0.214, so this one is worse than a constant. That comparison is worth running every time, because a low ECE can come from a model that barely discriminates.

Abstention curve. If the decision model can return defer, sort held-out items by confidence and plot accuracy against coverage. The curve tells you what the defer band buys. Tune the cut where accuracy on answered items meets the bar the branch needs, then measure how much volume lands in review. A defer rate of 35% may be cheap in one queue and ruinous in another.

Threshold from cost, not habit. For a refund branch where a wrong approval costs 20 manual reviews, approving is worth it only when (1 − p) × 20 < 1, so p > 0.95. Take the threshold from the costs and then check it on the held-out set with the calibration numbers, not the other way round.

Shadow mode. Run the model beside the current path for a fixed window, log its output and the eventual ground truth, and act on nothing. Compare the shadow decisions with the live ones on the same items, and check the calibration again on live traffic, because the held-out set is only a sample of it.

Failure modes to watch. Calibration drifts when the input mix shifts, so track ECE weekly per label. A prompt edit can reorder confidences without changing accuracy, which breaks any threshold set before it. Label noise in the held-out set caps measured accuracy, so audit a sample of disagreements before blaming the model.

Curated: · Written: · Reviewed:

Q&A-16

Explain the Brier score and when you would prefer it to accuracy or log loss for a decision model.

Answer

The 60-second answer. For a binary decision model that returns a probability p that the item is positive, the Brier score is the mean squared gap between p and the outcome o, where o is 1 or 0: BS = (1/N) × sum of (p_i − o_i)². It is the mean squared error of the probability itself. Lower is better. For a single item it is bounded between 0 and 1, so one bad call cannot dominate the average. It is a proper scoring rule: the expected score is lowest when the model reports its true belief, so optimising it pushes the model toward calibration rather than toward a threshold.

Accuracy throws the probability away. It thresholds at 0.5 (or at your decision threshold), counts matches, and ignores how far from the threshold each call was. A model that says 0.51 and one that says 0.99 for the same correct yes score identically. Two models that disagree on the same wrong answer, one at 0.6 and one at 0.95, also score identically on accuracy. Accuracy is the right metric when the decision is the only thing that matters and every call is made at the same fixed threshold. It is the wrong one for choosing among probabilities that feed a threshold you may move later.

Log loss is the other proper rule: LL = −(1/N) × sum of [o·ln p + (1−o)·ln(1−p)]. It rewards confident correct answers more and punishes confident wrong ones without limit. As p approaches 0 for a true positive, the penalty goes to infinity. That is what you want when a probability near 0 or 1 must be trusted and a single extreme miss is a real defect (a tail-risk model, or a fraud score that drives an automatic block). It is what you do not want when labels are noisy: one mislabelled item with p = 0.001 can dominate the whole average.

Worked example, five calls, outcomes 1, 1, 0, 0, 1 (hypothetical figures):

callpoutcomeBrier termlog-loss term
10.9010.01000.1054
20.8010.04000.2231
30.1000.01000.1054
40.9500.90252.9957
50.6010.16000.5108

Model A (as above): Brier mean 0.2245, log loss mean 0.7881, accuracy at 0.5 is 4/5 = 0.80. Model B is the same except call 4 says 0.60. Its Brier mean falls to 0.1160 and its log loss to 0.3722. Accuracy is still 0.80. Accuracy cannot tell A from B. Brier halves the gap; log loss roughly halves it too but the error sits in the tail, so the per-item log-loss term for call 4 alone is 3.0 of the 3.9 total.

Choosing. Prefer Brier when the probability is reported to people or downstream systems, when several classes must be summed (multi-class Brier is the sum over classes), when labels are noisy, or when you want a bounded number to compare across models. Prefer log loss when the tail matters and you can defend the extreme penalty. Keep accuracy, or threshold-based precision and recall, for the final gate: the business threshold is a separate decision and should be chosen against the cost of each error type, not inferred from a score.

Failure modes. A constant forecast equal to the base rate scores well on Brier and can look respectable on a skewed dataset, so compare against that baseline (the Brier score of always predicting the base rate is p(1−p)). Brier score can hide miscalibration in a narrow band of probabilities if most calls sit in the middle, so pair it with a reliability diagram. A model trained to minimise log loss on a training set that differs from production can still emit extreme confident probabilities on out-of-distribution inputs; the log-loss penalty then shows up in production as a spike, which is a monitoring signal, not a reason to pick Brier by default.

Curated: · Written: · Reviewed:

Q&A-17

A reliability diagram for a decision model's yes-probability has 6 calls in its 0.9-1.0 bin, far below the diagonal, and 300 calls in its 0.5-0.6 bin, close to it. What does each bin tell you, and why should you hold off on quoting ECE?

Answer

The 90-second answer. A reliability diagram groups calls by stated probability into bins, usually ten of width 0.1. For each bin it plots the mean stated probability against the observed rate of yes. A calibrated model lands on the diagonal. Points below it are over-confidence: the model said about 0.95 and fewer of those calls were yes. Points above it are under-confidence. Every point needs its count beside it. I read the counts first, then the gaps, then the summary number, because ECE is only a weighted average of the gaps and inherits whatever the counts make unreliable.

Worked trace (hypothetical figures, N = 1000 calls).

bin 0.9-1.0: n = 6, mean stated 0.96, observed yes 4/6 = 0.67, gap 0.29 bin 0.5-0.6: n = 300, mean stated 0.55, observed 0.57, gap 0.02

ECE = sum over bins of (n_b / N) x |observed_b - stated_b|

The top bin contributes 6/1000 x 0.29 = 0.0017. The middle bin contributes 300/1000 x 0.02 = 0.0060. The weighting makes the well-populated bin dominate the total, even though the top bin is the one a branch acting at 0.9 relies on. So the single number reports the calibration of the calls the system rarely makes.

The top bin is also unstable. If one of its six calls had been labelled the other way, observed becomes 5/6 = 0.83, the gap falls to 0.13, and that bin's contribution falls to about 0.0008. One label moved the total by about 0.001, and a 95% interval around 4/6 covers a wide range. At six calls the diagram cannot separate 'over-confident by 0.29' from 'roughly calibrated'. Equal-width bins make this worse at the extremes, because high-confidence calls are the sparse ones.

What I would do before quoting ECE. I would say the top bin is evidence of too few calls, not of miscalibration, and I would collect more calls in that range or pool it with the 0.8-0.9 bin before drawing a conclusion. I would also report the binning rule, because equal-mass bins give a different ECE on the same data. If the branch acts at 0.9, I would measure calibration on calls near that threshold directly, since the overall average can look fine while that band is wrong.

Curated: · Written: · Reviewed:

Q&A-18

A decision model returns accept, reject or defer. Once a deferred item has been sent to a person, what should the caller do with the decision the model returned for it?

Answer

A deferred item is not a verdict, so the caller should not act on the model's output for it at all. Defer means the model declined to commit. The workflow should route the item to a human review queue or to whatever fallback it defines, and every branch that would have issued the refund, posted the ledger entry or closed the ticket stays closed until a person or a later rule decides.

Assumptions: the decision result is a typed value with exactly three verdicts, and the caller can distinguish them. If the model's output is loosely typed, the first job is to make the verdict a closed union so an unexpected string fails loudly instead of falling through.

In code, defer gets its own branch and there is no default that quietly maps to accept or reject:

// TypeScript 5.x, typed decision result
type Verdict = 'accept' | 'reject' | 'defer';

function route(ticket: Ticket, result: { verdict: Verdict; probability: number; modelVersion: string }) {
  switch (result.verdict) {
    case 'accept': return issueRefund(ticket);
    case 'reject': return closeWithReason(ticket);
    case 'defer':  return enqueueForReview(ticket, {
      modelProbability: result.probability,
      modelVersion: result.modelVersion,
    });
  }
}

With strict enabled, the compiler flags a missing case when the union grows, which is the point: a new verdict should force a decision at every call site, not inherit whatever the fallback happens to be.

Two things matter for the audit trail. The deferred record should keep the probability and the model version, so a reviewer can see what the model leaned towards without that lean becoming the outcome. And the reviewer's eventual decision is what the system records as the final result, because that is the decision the business acts on. The model's verdict stays in the record as an input, not as the outcome.

The failure mode to guard against is mapping defer to reject because it looks like the safer default. A refund that should have gone to a person is then silently denied, the ticket is closed, and nothing in the model's output reveals it. Detection is the audit query: count items whose final outcome is reject but whose first verdict was defer. A nonzero count that nobody reviewed means the routing is wrong. A second failure is the reverse, where a reviewer's queue fills faster than it drains and deferred items age past the point where the refund matters. Track queue age per deferred item and alert on the oldest one, not the average.

Curated: · Written: · Reviewed:

Q&A-19

A pipeline uses an LLM to extract invoice fields from PDFs, and each record writes to the ledger. How would you use a decision model to check the extraction before the write, which questions would you ask per field, what gate would you put on confidence, and where do failed records go?

Answer

Start by refusing to ask the checker to re-extract. A decision model that is shown the extracted value and asked whether it is right will often agree with the extractor, because both read the same text and the value is anchored in its context. Ask instead a question about the source and the role of the field, one question per field, and compare the answer to the stored value outside the model.

For an invoice, the questions are shaped like: 'In this document, is 1,240.00 the amount due to be paid?' not 'Does the document contain 1,240.00?' The second passes when the number appears as a line item, a tax subtotal or a PO reference. That is the first failure mode to design out: a yes/no question that a substring can satisfy. Each field gets its own question, its own threshold, and a span the decision model must cite, so the audit trail shows what the model read.

The gate needs arithmetic, not a single global threshold. Suppose a record has 12 fields and each field check is 0.98 likely to be right on its own. If the errors were independent, the record passes all checks with probability 0.98^12 = 0.785, so roughly one record in five would carry at least one wrong field past a per-field 0.98 gate. Errors are not independent, though: a bad OCR line corrupts several fields at once. So measure the joint failure rate on labelled records, not the product of per-field rates.

Calibration is measured per field and per document type on your own labelled held-out set, and the threshold is chosen from that reliability curve against the cost of a bad write versus a review. A wrong ledger write costs more than a review, so the accept threshold sits high and the band between accept and reject is defer.

Routing: accepted records write in one transaction with an idempotency key derived from document id and extraction version. Rejected and deferred records go to a review queue with the question, cited span, probability and model version attached. Nothing partial is written; a record with any failed field is held as a whole.

What I would not do: treat a 0.92 on one field as proof the field is right. It is a model's stated probability and needs the calibration evidence behind it.

Curated: · Written: · Reviewed:

Q&A-20

A ticket router answers three decision questions per ticket inside a 300 ms p95 budget. Compare one model call per question with one batched call that carries all three. What is the main trade-off, and what decides which you choose?

Answer

I'd start by asking what the three decisions gate. If they feed one routing outcome that is only consumed together, I'd batch. If any one of them selects a different queue, SLA or audit path, I'd keep it separate. The main trade-off is shared cost and one round trip against isolation of input, output, failure and threshold per question.

The batch wins on cost whenever the three questions read the same ticket context. Assume a 4,000-token ticket context and an input rate of $0.50 per million tokens. These figures are hypothetical:

separate_calls   3 calls x 4,000 in = 12,000 input tokens   ~$0.0060   3 x 60 ms = 180 ms if sequential
batched_call     1 call  x 4,000 in =  4,000 input tokens   ~$0.0020   ~80 ms model time

The input saving is two thirds. Output tokens still bill, and a batched response carries three answers, so measure output cost on real traffic before quoting a total saving. Parallel separate calls can bring model time near 60 ms, but they triple concurrent requests to the provider and still pay for the context three times. Under a 300 ms p95 budget, the latency difference matters less than the cost and rate-limit difference, so latency alone rarely decides this.

Batching has a failure mode that separate calls avoid. One response has to be split by question. If one answer is missing or attached to the wrong label, the whole batch becomes unreliable. Validate the output against a schema that requires exactly one result per named question, and discard the entire batch on any violation rather than routing on the parts that parsed. Separate calls fail per question, so a timeout on the severity check does not stop the duplicate check. The cost is the extra calls.

Batching only pays when the questions share context. If the duplicate check needs the open-ticket list and the severity check needs only the latest message, the batch sends tokens that one question never reads. Scope each question's context first, then compare. If the scoped inputs are small and overlapping, batch them. If they diverge, separate calls with narrow context are cheaper.

Caching changes the arithmetic. A cached decision is valid only if its key includes the model version and a canonical form of the input. A key on ticket ID alone returns stale decisions after the customer replies. Measure the hit rate on a sample of real traffic before counting it as a saving.

Two failure modes need alerts. First, batch context grows silently as tickets accumulate messages, which raises input tokens and latency together. Record input tokens per call in the trace and alert when a batch exceeds the agreed budget. Second, a provider slowdown appears first in the p95 of the decision step, not in the whole request. Alert on the decision step separately, and also alert on the schema-validation failure rate per question, since a rising rate often precedes a latency problem.

My decision rule: batch when the questions share one context, feed one combined routing outcome, and the output schema is strict. Keep them separate when any question gates a different branch, needs its own threshold or audit record, or reads context the others do not.

Curated: · Written: · Reviewed:

Q&A-21

A decision model answers whether a refund is eligible, but the policy clause and the order fact it needs sit in different parts of one long input. Why might one call get it wrong, and how would you change the design?

Answer

Opening answer. I would first confirm the failure rather than assume it. A decision model reads its whole input inside one bounded state window and returns its typed probabilities in a single pass. It does not search the input, re-read a passage, or write out intermediate steps that I can inspect. So if the eligibility rule is on one page and the order date is on another, the model has to link them in the same pass. It can weight the wrong sentence, or give a confident probability for a question the two passages do not settle. The fix is to change what that single pass has to do.

Mechanism. Two designs are worth comparing. The first is to split the state: select the one policy clause that matches the refund category, and send it with the order record. The call then compares one clause to one set of facts. The risk is in the selection step. If the selector drops the clause that matters, the decision model never sees it, and its answer looks just as confident.

The second is to decompose the question. Ask two named questions instead of one: is the order inside the return window, and is the item category excluded. Each question reads only the input it needs and returns its own typed probability. The branch combines them with explicit logic, so a weak result on one question names its own cause.

Trade-off. Splitting the state is cheaper, with one call per decision, but it moves the risk into clause selection. Decomposition costs more calls and works best when the sub-questions are independent. If one answer depends on another, decomposition can give a false sense of coverage. For a refund branch I would decompose, because each sub-question is auditable and the combined result is easy to log.

Failure check. Build a labelled set of refunds where the deciding clause sits far from the order fact. Compare accuracy of the single call with the decomposed calls. If the single call drops when the input is long, the window is the problem, not the wording of the question.

Curated: · Written: · Reviewed:

Q&A-22

Why does a decision model not explain its answer, and what does it return instead?

Answer

Open by separating what the model computes from what a reason would need. A decision model is a classifier or scorer. A forward pass maps an input to one label from a closed set and a score for each label. Nothing in that computation produces a rationale, so there is no reason inside the output to report. What you can state exactly is which label won, how strongly the model scored it, and which input, artefact and preprocessing produced it.

A logged record should look like this (all values hypothetical):

{"input_id":"txn_8841","model_artifact":"fraud-clf-3.2.1","feature_schema":"v14","question":"fraud_risk","label":"review","scores":{"approve":0.12,"review":0.61,"decline":0.27},"policy":"review if top score >= 0.50"}

The scores are usually softmax outputs. They rank labels reliably but are not probabilities of being right unless calibrated, so 0.61 does not mean a 61% hit rate.

Failure modes and how to detect them:

  1. Miscalibrated scores. A model can be accurate and still overstate its confidence. Bin held-out predictions into ten buckets, compare mean top score with observed accuracy in each bin, and compute expected calibration error. If the top bin shows 0.9 scores at 0.7 accuracy, fit temperature scaling on a validation split and re-measure before relying on the thresholds.

  2. Out-of-distribution inputs. The model returns a confident label for inputs unlike its training data. Track per-feature ranges against training percentiles and the share of predictions with top score above 0.9 by segment. A sudden shift in label mix from last week's baseline is the signal to investigate.

  3. Threshold drift. A cutoff chosen at one base rate loses precision when prevalence changes. Compute precision and recall per threshold on a weekly sample of reviewed decisions, and alert when either moves past a level agreed before launch.

  4. Unreproducible replays. A stored label can fail to reproduce if the artefact, feature schema or preprocessing changed, or if batch size or hardware changes floating-point results near a tie. Replay a nightly sample of logged inputs against the pinned artefact, count mismatches, and flag any top-two gap under 1e-4 separately so ties are not blamed on the model.

Operational cost: logging full inputs creates storage and retention obligations, and replay requires the old artefact to stay deployable for as long as its decisions can be disputed. Sampling is affordable for approvals, but declines that carry a regulated consequence should be kept in full.

What the log buys you is the ability to replay the exact decision, check scores against outcomes, and trace a disputed label to its artefact, schema and threshold. Feature attribution methods such as SHAP can be added on top, but they describe the model's sensitivity to inputs, not the reason it decided, and should be presented as approximations under that name.

Curated: · Written: · Reviewed:

Q&A-23

A branch acts when a decision model's yes-probability is at or above 0.9. Before you ship, what decides whether you use those probabilities as they are or recalibrate them, and how do you show the choice still holds after the model version changes?

Answer

Start with the decision the probability drives, not the metric. A branch that acts at 0.9 cares most about calibration in the bins near 0.9, so I would measure there first and look at ECE second.

The plan I would defend:

  1. Split the labelled data three ways: a calibration split used only to fit any recalibration map, a threshold-and-report split used to pick the cutoff and quote ECE, and nothing else. If the same rows fit the map and report the ECE, the number is optimistic and the reviewer will catch it.

  2. Measure ECE with equal-width bins on the reporting split, and always print per-bin counts. With 15 bins and 2,000 rows, a bin holding 120 calls gives a standard error on its observed rate of about sqrt(0.8 x 0.2 / 120), roughly 0.04. A gap of 0.10 in that bin is weak evidence on its own. Contribution to ECE is (n_b / N) x |acc_b - conf_b|: 120 / 2000 x 0.10 = 0.006.

  3. Use the probabilities as-is only if the bins that the branch depends on are within tolerance and the score distribution has not shifted from the labelled data. A good global ECE can hide a bad 0.85 to 0.95 band.

  4. If recalibration is needed, pick the method by data size. Temperature scaling fits one parameter T on the logits, minimising negative log-likelihood on the calibration split. It preserves rank order, so AUC does not move and the threshold keeps its meaning in ranking terms. Isotonic regression is nonparametric and can fit a sharper curve, but it produces a step function with flat plateaus, so many distinct scores collapse onto one probability. It needs far more positives to avoid overfitting. For a few thousand rows I would start with temperature.

  5. After any version change, treat the old map as invalid. T was fitted to the old model's logit scale. Re-measure ECE and the reliability bins on a fresh labelled sample, then refit only if the new model fails, and re-run the threshold choice, because a shifted score distribution moves where 0.9 falls in the ranking.

The failure I watch for most is a model that passes ECE after version 2 while its scores have compressed toward 0.5. The metric looks fine and the branch now fires on different tickets. Detect it by comparing the score histogram and the fraction of calls above the threshold between versions on the same traffic sample.

Temperature fixes scale, not prior shift. If production has more outages than the labelled set, the calibrated probabilities are still wrong for production, and only fresh labels from production reveal it.

Curated: · Written: · Reviewed:

Q&A-24

A branch sends a refund to the ledger when a decision model's yes-probability is at or above 0.9. You are moving from v1 to v2 of the same model. What breaks in that branch if you only compare accuracy, and what do you check instead?

Answer

The 90-second answer. A threshold is a claim about calibration. The 0.9 cutoff on the refund branch means that calls scoring 0.9 or above are right about nine times in ten. v1 earned that claim on the team's labelled data; v2 has to earn it again. A version can be more accurate overall and still put its 0.9 calls at 0.75 correctness, so the branch pays out wrong refunds at a higher rate with no change in the code. Accuracy averages over all calls and hides exactly this. What I check instead is the reliability of the calls above the cutoff, the share of calls the branch now defers, and whether the question the branch asks is still served at all.

Mechanism. The branch reads one number and compares it to a constant. Its error rate above the cutoff is the calibration of the bin the cutoff falls in, weighted by how many calls land there. Hypothetical figures, labelled as such: on 2,000 labelled refund tickets, v1 puts 300 calls at 0.9 or above with 0.92 observed correctness. v2 is more accurate overall, but puts 280 calls at 0.9 or above with 0.78 observed correctness. The branch's wrong-refund count above the cutoff goes from 24 to 62, roughly 2.6 times as many, and the overall accuracy number barely moves.

bin        v1 calls  v1 obs  v2 calls  v2 obs
0.9-1.0       300     0.92      280     0.78
0.5-0.9       700     0.71      640     0.73
<0.5         1000     0.22     1080     0.20

What to check, in order.

  1. Question-type availability. Confirm the refund question is served by v2 with the same label set. A missing type becomes a fallback path, and fallback averages can make the other numbers look fine.
  2. The 0.9 bin on a frozen labelled set. Compare observed correctness per bin with counts beside each, using the same binning and scoring code for both versions. A bin with six calls cannot support a conclusion.
  3. Deferral rate at the branch's threshold. If v2 defers more calls, the manual-review queue grows by that share, which is a real cost even when accuracy is higher.
  4. Re-tune the threshold on v2's own curve. The v1 cutoff is not carried over. Choose the lowest cutoff that meets the wrong-refund budget on the frozen set, then confirm it on a fresh sample from recent traffic.

Trade-off. A stricter cutoff on v2 protects the ledger but defers more and costs reviewer time. A looser cutoff keeps throughput and accepts more wrong refunds. Price each wrong refund against a manual review, then pick the cutoff that minimises the total.

Failure modes. A threshold copied across versions passes review because the code did not change, so detect it with the per-bin check above. A vendor evaluation set that overlaps the new version's training data inflates the bin accuracy, so detect it by repeating the check on a sample drawn after the vendor's cutoff date. Keep the v1 identifier pinned so rollback is one config change.

Curated: · Written: · Reviewed:

Q&A-25

A decision model's guardrail blocks an agent reply when its 'leaks customer data' probability is at or above a threshold. A leak costs about 20 times a false block. How do you set the threshold, and what does the model's score need to be before you trust it?

Answer

Start by naming the two errors. A false block sends a safe reply to a retry or a human, costing about C. A leak sends a reply exposing another customer's data, costing about 20C, so the cost ratio is 20 to 1 (hypothetical figures, labelled as such). The guardrail should block when the expected cost of letting the reply through exceeds the expected cost of blocking it.

With p as the model's probability that the reply leaks:

expected cost of allow = p x 20C expected cost of block = (1 - p) x C

Block when p x 20C > (1 - p) x C, which gives p > 1 / 21, about 0.048. The threshold is near 0.05, not 0.5. A candidate who starts at 0.5 has ignored the cost ratio, and the guardrail will leak far more than it should.

That number is only as good as the score under it. A threshold at 0.05 means the model's 0.04 and 0.06 outputs must mean what they say. If the model's low scores are overconfident, the real leak rate in the 0.02 to 0.05 band could be several times what the score claims, and the cut lands in the wrong place. Check this on labelled replies before shipping: bin the scores, count the true leaks in each bin, and compare the rates to the bin's stated probability. Look hardest at the bins around the threshold. The calibration you need is at the cut, not across the whole range.

If the low bins are poorly calibrated, two responses are honest. Move the threshold down until the measured leak rate in the blocked region meets your target, which accepts more false blocks. Or add a cheap deterministic check, such as a pattern match for account identifiers, and require either signal to trigger a block. Do not treat one cut as settled. Re-measure after each model or prompt change, because the calibration at 0.05 can move even when overall accuracy does not.

The design rests on two things you measured and one you estimated. The decision rule is algebra. The 20 to 1 ratio is an estimate from incident cost and retry cost, and it should be reviewed when either changes. The calibration at the cut is an observation you must make on your own labelled data.

Curated: · Written: · Reviewed: