Q&A-1Explain what a decision model is and how it differs from calling a chat LLM with structured output. What does that difference mean for the code that calls it?
Answer
The 60-second answer. A decision model takes an input and a set of options that the caller supplies, and returns a probability for each option. The probabilities sum to one, the type is fixed by the contract, and the caller chooses what to do with them. It does not write a sentence that happens to name an option. A chat model called with structured output does generate tokens, one at a time, conditioned on a schema. The schema constrains the shape of the text, but the result is still a sampled sequence, so the caller parses it, validates it, and handles the cases where the model picks an option that was not in the list or where the JSON is malformed. With a decision model the option set is an input, not something the model invents, and the output is a distribution the code can read directly.
Mechanism and what it changes. The difference is in the output space. A structured-output call maps to an open space of strings that is then checked against a schema. A decision model maps to a closed space: for n caller options, the output is a vector of n numbers. That gives three consequences for calling code.
// Hypothetical shapes, for illustration only
type Decision = { options: string[]; probs: number[] }; // probs.length === options.length
function route(d: Decision, minConfidence: number) {
const [best, p] = argmax(d.probs);
if (p < minConfidence) return { action: "escalate" };
return { action: d.options[best] };
}
First, an unknown option cannot appear, so the parse-and-retry branch disappears. Second, the threshold is a real decision the code owns, and it has to be tuned against calibration. A probability of 0.7 is only useful if about 70 percent of predictions in that band are right on held-out data. Third, the output is a score, so tests check the distribution: for example, that probabilities sum to one within a tolerance, and that the threshold sends 20 percent of a labelled set to escalation.
Failure modes. The main one is miscalibration. The top option can be wrong with confidence 0.9, and the code acts on it. Detect this with a reliability diagram on a labelled holdout, and track expected calibration error per release. A second is an option list that changes between calls: a probability vector is only comparable across calls if the options and their order are the same, so key by option ID rather than by index. A third is treating a decision model as a generator, asking it for an explanation it cannot give. Keep rationale generation in a separate call if the product needs it.
When not to use one. If the answer space is open, such as a free-text summary, use generation. If the options are few, stable and fully enumerable, a small classifier or a rule may be cheaper and easier to audit. The decision model earns its place when the options are supplied at call time and the judgement has to be a number the code can threshold.
Curated: · Written: · Reviewed: