Skip to content
Tech Interview Prep
Technical interview guideNew

Decision Models (System One)

Typed decision models return calibrated probabilities over options you name, in one forward pass, so code can branch on a model's judgement without parsing generated text.

Read
9 min
Practice MCQs
24
Interview Q&A
25
Edition
v1
Editorial status
Reviewed
Relevant for
AI / ML Engineer

Scope: Current to Jev 1.14.0 (TypeSafe, System One API at POST /v1/systemone, docs dated 2026-11) and to the open Kev models (0.8B to 27B, served via Ollama) as published in late 2026. Vendor figures for price, latency and benchmark accuracy are vendor-reported claims, not independently verified. Question types differ by version: Jev 1.13 supports Choice, Score and Noul; Jev 1.14 adds Box..

Overview

Curated: · Written: · Reviewed:

Decision models (System One)

Most AI integrations today end with a model producing text and code scraping meaning out of it. That works until the branch matters: routing a support ticket, blocking a request, deciding whether a document needs human review. A decision model removes the scraping step. You name the options, ask a typed question about a state, and get back a choice or probability per option plus a confidence estimate, which code can branch on directly. The interview question underneath is usually: when do you want probabilities over named options instead of generated text, and how do you know those probabilities mean anything?

The mental model to carry: a decision model is a classifier you can query in natural language, whose output contract is a typed answer with a probability, produced in one forward pass rather than by sampling tokens.

Why this class exists: the gap between generated text and branchable decisions

A generative LLM returns a string. If you want refund, escalate, or reject, you either ask for JSON and parse it, or you hope the wording is stable enough to regex. Both fail in known ways. The output can be malformed, the model can wander into prose, and the string carries no statement of how sure the model is. A 0.51 and a 0.99 that both come back as escalate look identical to your code.

System One models were built to close that gap. Per the TypeSafe docs, they evaluate a state and return typed answers and probabilities, and they accept natural-language input the way an LLM does. The point is not better prose. The point is that the thing software consumes is a decision with a number attached.

The interface: state, typed questions, and answers with probabilities and confidence

The shape is small. You send a state (text and, for Jev, images) and one or more questions. Every request goes to POST /v1/systemone. Answers reuse the question names, so you match each result to its question by key rather than by position. The response also reports the model used and token usage.

curl --request POST \
  --url https://api.typesafe.ai/v1/systemone \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '{ "model": "jev-latest", "state": "...", "questions": [...] }'

Each answer carries a probability distribution over the options you named, and a confidence estimate. Read the probability as the model's estimate for that option, and use the confidence as a separate signal about how far to trust the whole decision. The exact response field names are in the API reference; check them against the version you deploy rather than copying from a blog.

What interviewers probe here: whether you know the key-matching contract, and whether you understand that the answer is a distribution, not a label. A weak answer says "it returns the category" and stops.

Question types: choice among named options, yes/no probability, and score over ordered levels

The docs list four question types for Jev 1.14: Choice, Score, Noul and Box. Choice picks among options you enumerate, which is the common case for routing. Score places the state on an ordered scale of levels you define, useful for severity or urgency. Noul and Box are less widely documented, so confirm their semantics in the docs before quoting them. Yes/no is a choice with two options; treat it as a probability on the positive option and branch on a threshold you chose deliberately.

You can mix types in one request. That is convenient, but a single question per decision keeps the probability tied to one judgement you can calibrate.

One forward pass versus token-by-token generation: latency, cost and type safety, with worked arithmetic

A generative model decodes autoregressively: each output token needs a forward pass over the context, so latency grows with answer length. A decision model emits its answer in one forward pass, so latency scales with input, not with output length. Type safety follows: there are no tokens to malform, so there is nothing to parse.

The cost arithmetic depends on the price basis, so compute it explicitly. The docs list Jev 1.14 at $42 per Btok, which is $0.042 per Mtok. Hypothetical example, labelled as such: a routing call with a 2,000-token state and a 10-token answer costs 2,000 × $0.042 / 1,000,000 ≈ $0.000084 for the input. Ten thousand such calls cost about $0.84. Compare an LLM that generates a 300-token JSON answer: you pay for those output tokens too, and at each one you wait for a decode step. The savings are real but the bigger win for production is that the branch is typed and the confidence is available.

Do not claim a specific latency figure without measuring it on your workload. The argument is structural: one pass versus N decode steps.

Calibration: what it means, reliability diagrams, ECE and Brier score, and why thresholds depend on it

A model is calibrated when its stated probability matches the observed frequency. Among all decisions where it said 0.8, about 80% should be correct. Calibration is what makes a threshold meaningful. If you branch at 0.9 on an uncalibrated model, you have no idea what fraction of those decisions are wrong.

Two measures matter. Expected calibration error (ECE) bins predictions by confidence and averages the gap between confidence and accuracy in each bin, weighted by bin size. A reliability diagram plots that gap visually: a calibrated model tracks the diagonal. The Brier score is the mean squared error between the predicted probability and the 0/1 outcome, so it penalises both miscalibration and poor discrimination. Lower is better for both ECE and Brier.

TypeSafe states that the models are trained with Reinforcement Learning for Calibrated Decisions (RLCD) and that every decision includes a confidence estimate. Those are vendor claims. Verify calibration on your own labelled data (section 10) before setting production thresholds from the vendor's numbers.

Where decision models fit: routing, classification, extraction checks, guardrails, and the semantic if-statement

The clearest fit is a branch with a cost attached to each outcome. Routing a ticket to the right queue, classifying intent before a tool call, checking whether an extracted field is actually present in the source document, flagging content for a guardrail, and deciding whether a record needs human review. The community calls this a semantic if: if model.ask(state, question).prob('refund') > 0.9 then refund().

The pattern is weakest where the answer needs to be explained to a person. Because there is no rationale, the branch is defensible only through its measured calibration and an audit trail of inputs and outputs.

Thresholds, abstention and the cost of wrong branches

Choose the threshold from the cost of each error, not from a round number. If a false refund costs $200 and a missed refund costs $5 of customer goodwill, you want a high threshold on refund and a route to a human for the middle band. That gives you three outcomes: act, abstain and escalate, reject. Abstention is a first-class branch here, and it is what calibration buys you.

Worked example, hypothetical: with a positive-class threshold t, expected cost per decision is FP_rate × cost_FP + FN_rate × cost_FN, measured on your labelled set at each t. Pick the t that minimises it, then check that the abstain band is small enough for your review capacity.

Comparing with a structured-output LLM and a fine-tuned classifier

A structured-output LLM can return the same enum, but its output is still sampled tokens. The enum is enforced by constraints or retries, and it usually carries no calibrated probability unless you ask for one and trust a self-report, which is not a calibration guarantee. You also pay decode latency on every call.

A fine-tuned classifier gives you calibrated probabilities over a fixed label set that you trained on, is cheap to run, and is fully under your control. Its weakness is the training cost and the fixed label set: a new option means new labelled data and a retrain. A decision model accepts options at call time and needs no training run, at the price of depending on a vendor and its calibration.

Choose the fine-tuned classifier when you have a stable label set, abundant labels, and tight latency or cost requirements. Choose a decision model when labels change often, you lack training data, or you need the branch and the explanation layer to stay simple.

Limits: no rationale, caller-bounded answers, weak cross-document reasoning, vendor-reported benchmarks

The output has no rationale, so you cannot ask why it chose refund. The answer is bounded by the options and question you supply, so a missing option silently forces the probability mass elsewhere. Cross-document reasoning is weak: if the decision depends on combining facts from several documents, put the relevant facts into one state, or use an LLM to build that state first. Benchmarks and calibration figures come from the vendor and from the launch material; treat them as claims to verify, not as facts about your data.

Evaluating a decision model on your own labelled data before trusting it

Label a few hundred real decisions from your traffic, including the hard cases. Run the model, record the probability for each option, and compute accuracy, ECE, Brier score, and the reliability diagram. Sweep the threshold and plot precision against coverage, since coverage is the fraction you act on rather than escalate. Check that calibration holds on the slices that matter, not only on the average. Re-run the evaluation when the model version changes, because the model ID is a pin you control (jev-1.14.0 versus jev-latest), and latest can move under you.

Likely interview follow-ups and what a weak answer sounds like

Interviewers ask how you would know the probabilities are calibrated. A strong answer names ECE, a reliability diagram, and a labelled holdout set. A weak answer says "the model gives a confidence score, so it's calibrated."

They ask what happens when a new category appears. Strong: add it as an option, re-evaluate, and check that the old options' calibration has not shifted. Weak: "the model will figure it out."

They ask why not just use GPT with JSON. Strong: the output is one pass and typed, and the confidence is an explicit field whose calibration you measure, with the trade-offs against a structured LLM stated. Weak: "JSON is slower."

They ask how you pick a threshold. Strong: from the cost of each error on your labelled data, with an abstain band sized to review capacity. Weak: "0.8, it's standard."

They ask what the model cannot do. Strong: no rationale, bounded to the options supplied, weak across documents, and vendor benchmarks need independent checking. Weak: a list of benefits with no limits.

Feedback on this overview