Skip to content
Tech Interview Prep
Technical interview guideNew

Decision Models (System One)

Typed decision models return calibrated probabilities over options you name, in one forward pass, so code can branch on a model's judgement without parsing generated text.

Read
9 min
Practice MCQs
24
Interview Q&A
25
Edition
v1
Editorial status
Reviewed
Relevant for
AI / ML Engineer

Scope: Current to Jev 1.14.0 (TypeSafe, System One API at POST /v1/systemone, docs dated 2026-11) and to the open Kev models (0.8B to 27B, served via Ollama) as published in late 2026. Vendor figures for price, latency and benchmark accuracy are vendor-reported claims, not independently verified. Question types differ by version: Jev 1.13 supports Choice, Score and Noul; Jev 1.14 adds Box..

Curated: · Written: · Reviewed:

24 questions · Flow follows Relevance until this set has a learning order.

MCQ-1A support-ticket router sends each ticket to one of four queues (billing, outage, account, other) by calling a chat model with JSON output and parsing the reply. About 1 in 200 calls fail to parse, and the router has no confidence signal for the calls that do parse. Which change best fits a typed decision model?
MCQ-2A pipeline must decide whether an invoice field matches its source PDF, and it writes to the ledger only when the match probability is above 0.9. Which output contract gives the caller what that branch needs?
MCQ-3A payments alert gate rates impact from 1 (cosmetic) to 5 (data loss) and must page on-call when impact is 4 or higher. A five-label choice question returned P(1)=0.03, P(2)=0.02, P(3)=0.40, P(4)=0.30 and P(5)=0.25 for one incident. Which page rule fits this output?
MCQ-4A nightly batch makes 40,000 decision calls. Each generated-text call emits 350 output tokens at 40 tokens per second. Assume a single-pass decision answer takes 0.1 s per call (hypothetical figure, vendor timing unverified). What is the best estimate of the nightly latency gain from switching the batch to decision calls?
MCQ-5A refund branch acts when a decision model's yes-probability is at or above 0.9. On 2,000 held-out calls, the 0.9 to 1.0 bin holds 500 calls with a mean predicted probability of 0.93 and an observed yes rate of 0.97. What does that bin show?
MCQ-6A retrained decision model's reliability diagram for its yes-probability has most calls in the 0.95 to 1.0 bin, with observed accuracy near 0.97 there and only a handful of samples in the middle bins. ECE moves noticeably from one retraining run to the next. What explains the instability?
MCQ-7A claims router flags a warranty claim for inspection when its calibrated fraud probability is at or above 0.5, the default. On 2,000 labelled claims from a held-out set, 0.5 gives 40 missed fraud and 300 false flags, 0.3 gives 22 missed and 380 false flags, and 0.2 gives 12 missed and 800 false flags. A missed claim costs 10 units and a false flag costs 1. Which change is most defensible?
MCQ-8A reply guardrail's leak probability is calibrated on a labelled set: replies scored 0.05 leaked in 5 of 100 cases, and replies scored 0.95 were safe in 4 of 100. In production, 1,000 replies score 0.05 and are accepted, 1,000 score 0.95 and are blocked, and the rest go to a stronger model. Over the accepted and blocked replies, how many leaks reach users and how many safe replies get blocked?
MCQ-9A team wants a decision model to write the root-cause section of an incident report for a 40-event timeline. The model would return a probability for each of five cause categories (config change, capacity, dependency, code defect, human error), and reviewers want the report to explain why each cause was chosen. Which design fits?
MCQ-10Two decision models are scored on the same labelled set: model A has accuracy 0.91 and Brier score 0.21, while model B has accuracy 0.88 and Brier score 0.09. A branch sends a case to manual review when the model's yes-probability is at or above 0.8. Which model's output is best supported for that branch?
MCQ-11A decision model scores 91% on 400 short support tickets but 58% on 200 refund-policy documents averaging about 12,000 tokens. In each document, the eligibility rule in section 2 cites a time window defined in section 7 and an exception in section 9. Errors concentrate on documents where the rule and its window and exception all have to be applied together. The full input fits the context limit, and two annotators agreed on 97% of labels. Which is the most likely cause?
MCQ-12A lender's approval model is a linear logistic score: log-odds = intercept + sum of (weight × input). The reason code for a rejection must be the single input whose term pushes this applicant's score furthest toward rejection. For one rejected applicant (hypothetical figures): debt-to-income excess, input 0.10, weight 2.0; revolving utilization, input 0.90, weight 0.5; income volatility, input 0.35, weight 1.2; thin-file flag, input 1, weight 0.4. Which reason code must be recorded so an auditor can reproduce it from the scoring computation?
MCQ-13A team's pipeline calls the `Box` question type. Its upgrade test used a floating alias that resolved to jev-1.14.0 and passed. Prod's alias still resolves to jev-1.13.0 until the next deploy, and the team has not yet checked when `Box` was added. Which check should come first?
MCQ-14A triage job sends 12 yes/no questions about one support ticket in a single decision request, with the 3,000-token ticket text included in every request. Last night 1 in 25 requests returned answers for only some questions while the rest came back empty, and the job re-ran each whole ticket at full token cost. Which change keeps the shared-context saving and fixes the wasted re-runs?
MCQ-15A refund branch auto-approves when a hosted decision model's yes-probability is at or above the cutoff. The team dropped its fine-tuning pipeline and now only sends questions. On 2,000 labelled held-out tickets, calls scored 0.90 to 0.95 were yes 74% of the time and calls scored 0.95 to 1.00 were yes 97% of the time. The branch needs at least 95% precision. Which cutoff should it use?
MCQ-16A team fits a temperature on the 2,000-call validation split of a decision model, then measures ECE on that same split and gets 0.084 falling to 0.011. A refund branch will act at 0.9 on these probabilities. What should the team do before trusting the 0.011?
MCQ-17A decision model's `abstain` rate rose from 4% to 22% the week after a source system changed its PDF layout. The model's version and thresholds are unchanged. What does this most directly indicate?
MCQ-18An invoice pipeline checks each extracted field against its source PDF with a decision model that accepts at 0.9 or above and rejects at 0.1 or below. A labelled audit of 1,000 invoices found fields scoring 0.4 to 0.6 matched the source 52% of the time, and fields scoring 0.9 or above matched 98%. Ledger writes are all or nothing per invoice, and a re-check of the same PDF returns the same score. One invoice has a total field at 0.55 and a line-item field at 0.93. What should the pipeline do?
MCQ-19A product manager wants to cut decision-model cost by sending the full 200-page policy manual, about 110,000 tokens (hypothetical), with each of 15 yes/no questions asked per ticket. The model accepts 32,000 input tokens per request (hypothetical). What should the design do instead?
MCQ-20An agent step has a 1.5 s budget and calls a tool with side effects, which takes about 700 ms at p50. A decision model must approve the tool arguments before the call, and its call takes about 400 ms at p50 with a long tail. Which placement fits this budget?
MCQ-21A vendor benchmark reports that a decision model is well calibrated, and the team has not yet scored its own labelled tickets. What does that vendor figure establish for the team's branch?
MCQ-22A decision model trained on support tickets is run unchanged on 1,500 medical device complaints. About 1,100 of them return a yes-probability between 0.4 and 0.6, and the rest are spread across the range. Annotators agreed on 96% of the labels. On a labelled sample of 200 complaints, accuracy is 0.66 and the scores barely differ between yes and no items. What does this most likely indicate?
MCQ-23A pipeline routes tickets with a Choice question and a Score question on jev-1.13.0, and a refund branch acts when the Score is at least 0.9. The team adds a third question, a refund-eligibility Score on the same model, and a new branch will read its output next week. Which step should come before deploying?
MCQ-24A guardrail asks one question, 'is this request harmful?', and blocks at 0.5. In shadow mode it blocks 6% of benign requests, mostly security-training questions scoring 0.5 to 0.8. Phishing-kit orders that slip through score 0.4 to 0.6, overlapping those training questions. Which change is most likely to cut false blocks without raising misses?