Skip to content
Tech Interview Prep
Technical interview guide

LLM Observability & Monitoring

Monitoring, logging, and tracing LLM and agent systems in production: the specific signals and tooling that apply on top of general observability practice.

Read
45 min
Practice MCQs
25
Interview Q&A
25
Edition
v4
Editorial status
Reviewed

Scope: OpenTelemetry specification and semantic conventions reviewed at 1.44.0 on 2026-09-04, W3C Trace Context 2021 Recommendation, OpenMetrics 1.0, Google SRE monitoring, RAGAS, Lost in the Middle, Model Cards, NIST GenAI, and OWASP Prompt Injection/Sensitive Disclosure references.

Curated: · Written: · Reviewed:

25 questions

MCQ-6A streaming chat endpoint draws a bimodal complaint: some sessions leave the panel blank for about 1.5 s before the first character appears, then stream at a normal cadence. Decode throughput, per-token cadence and request volume are all flat month over month. The team wants a single dashboard metric that moves when the delay before any output appears regresses, and stays flat when decode slows. Which metric should they chart?
MCQ-1An LLM assistant answers each user question with a loop of model calls (median 4). Last week it served 10,000 questions and made 40,000 model calls. The logs show 200 model calls returned an error and were retried successfully, 250 questions ended with the assistant refusing, and users marked 150 answers as wrong. The team's SLO is 99.0% of questions answered successfully, where a question counts as successful only if the assistant returned an answer that was neither a refusal nor marked wrong by the user. The dashboard's headline panel reports model-call success at 99.5% and is green. Which value should the on-call compare against the 99.0% SLO?
MCQ-17An LLM gateway holds a 99.9% success SLO over a 30-day window (error budget 43.2 minutes). Its paging rule is multi-window burn rate: page when the 1-hour burn rate is at least 14.4 and the 5-minute burn rate is also at least 14.4; every other signal opens a ticket. Which of these signals pages the on-call?
MCQ-3A payments API serves 3,000 req/s from 400,000 distinct users. Prometheus holds 12,000 active series at 8 GB RSS. An engineer adds a `user_id` label to the `http_request_duration_seconds` histogram (12 buckets) to break latency down by user; within a day active series reach 5.6 million, RSS passes 60 GB, and dashboard queries that returned in 200 ms now time out at 30 s. What is the correct fix?
MCQ-2A RAG pipeline runs as three services: a gateway, a retrieval worker, and a tool executor. All three use OpenTelemetry with W3C traceparent propagation and export to one collector. Requests served synchronously by the gateway's /chat endpoint appear as a single trace with the gateway, worker, and executor spans nested correctly. The 38% of traffic the gateway hands to a queue for async processing appears in the tracing UI as three separate root traces: one per service, each with the correct service name and duration, and no parent. The gateway sets traceparent on its HTTP call to the enqueue endpoint. What causes the split?
MCQ-13A payments team runs 3 services on an OpenTelemetry SDK release that emits the deprecated `http.status_code` span attribute. 40 Grafana panels and 12 alerts query that attribute. The team is upgrading to a release that emits the stable `http.response.status_code` instead. The shared collector's configuration is frozen for the quarter, so no processor can be added there. Which sequence keeps every panel and alert returning data through the cutover?
MCQ-11A payments service emits exactly one server span per request. Over a 5-minute window the app's own counter records 50,000 requests and 1,900 errors (3.8%). The trace backend computes error rate as errored spans ÷ spans received and reports 380 ÷ 38,000 = 1.0% for the same window. The collector's memory limiter refused 12,000 spans in that window. What does the refused-span counter establish about the 1.0%?
MCQ-12A payments API serves 40 endpoints and produces roughly 30,000 traces per minute across 200 pods. Errors and requests slower than 2 s occur on any endpoint, and the team must retain every such trace while keeping only about 1% of the rest. Which sampling strategy meets that requirement?
MCQ-7A support agent runs two candidate workflows on the same model, priced at $3.00 per million input tokens and $15.00 per million output tokens. Workflow A makes one call per attempt, using 2,000 input and 300 output tokens; 40% of its attempts produce a usable answer. Workflow B makes one call per attempt, using 4,000 input and 400 output tokens; 90% of its attempts produce a usable answer. Failed attempts are billed in full and are not retried. Which workflow is cheaper per successful task, and at what cost?
MCQ-9An agent's trace marks a refund run complete when the last tool call returns HTTP 200. For order #5512, POST /refunds returned 200 at t+0.2s with {"id":"rf_88","status":"pending"}; the payments service settles asynchronously and writes the refunds table row at t+91s. A retry at t+30s with a fresh idempotency key issued a second refund. Which completion predicate should the trace use?
MCQ-10A checkout service writes one access-log line per request to stdout, and its OTel logging setup ships each line as the log record body; the line contains `Authorization: Bearer <token>`. The Collector's logs pipeline runs `attributes/redact` with `action: delete` on `http.request.header.authorization`, then `batch`, then `otlphttp/vendor`. Over 24 hours the vendor's log store returns 1,240 records whose body contains `Bearer `, and none of those records carries an `http.request.header.authorization` attribute. The service's release is frozen for the quarter, so only Collector configuration can change. What removes the token from the exported logs?
MCQ-5An LLM gateway exports Prometheus metrics with labels `model`, `route`, `status`, and `prompt` (the raw user prompt, up to 2 KB). Since the `prompt` label was added four months ago, `prometheus_tsdb_head_series` has grown from 18k to 4.3M, about 35k new series a day, and scrape p99 has gone from 25 ms to 1.9 s. The prompts also contain customer email addresses, and the team still needs to go from a metric spike to the exact prompt that caused it. Which change meets both requirements?
MCQ-4A support-assistant service calls a hosted chat model through one client whose config pins the alias `gpt-4o` and sends max_tokens, temperature and streaming on every call. Last Tuesday, with no deploy on the team's side, monthly spend went from $4,100 to $9,800 and p95 latency from 900 ms to 2.4 s. Spans carry only the name `llm.call`, duration and status, so no one can say which calls changed. Prompts contain customer PII, and the team has ruled out storing prompt or completion text. What should each model-call span carry so the regression can be attributed?
MCQ-8A RAG service logs the final answer, the 20 reranked chunk IDs with their ranks and scores, and the exact context string sent to the generator. That context is the 20 chunks concatenated in rank order and truncated at 6,000 tokens. An engineer samples 100 wrong answers: the gold chunk is among the 20 logged candidates in 88, and appears in the context string in 61. Which trace addition would explain the other 27?
MCQ-23A payments API increments a Prometheus counter `charges_total` and commits the charge to the ledger (PostgreSQL 16) inside the same request handler. For the last hour the dashboard shows 12,000 charges, while `SELECT count(*) FROM payments WHERE status = 'captured' AND created_at > now() - interval '1 hour'` on the primary returns 11,940. The hour's logs contain exactly 60 `serialization_failure` errors from that handler, and none of those requests were retried. What does the gap most likely mean, and what should the on-call do?
MCQ-24A team ships v2 of its LLM tracing SDK to a 5% canary for 24 hours, with v1 on the other 95%. App p99 latency and error rate are unchanged. Comparing spans from the two slices: span count per request is 12.0 on both, export success is 99.99% on both, but the `gen_ai.usage.input_tokens` attribute is present on 100% of v1 spans and 70% of v2 spans. What should the team do?
MCQ-21An LLM gateway caches completions under a key that is the SHA-256 of the full request body: model, system prompt, messages, and temperature. The only cache metric is `llm_cache_hits_total{model}`. The only change is a canary deploy that sends 30% of requests to a new system prompt and leaves the other 70% on the old one. The hit rate was 70% before the deploy and reads 49% an hour after. What does the reading show?
MCQ-20A team calls the provider's `gpt-4o` alias and logs the full response JSON. Their nightly regression suite replays 400 frozen prompts at temperature 0 and compares outputs to stored baselines; it scored 95–97% for six weeks, then 74%, 73%, 75% and 74% on the last four nights. Nothing in their repo, prompt templates or eval data changed. Across those four nights the `model` field in every response still reads `gpt-4o`, p95 latency is 795 ms against an 810 ms baseline, and the `system_fingerprint` value changed from `fp_9a1c` to `fp_4e77` on the first bad night and has been stable since. Which reading of this evidence is best supported?
MCQ-15A team gates each release of an LLM support assistant on its in-product rating widget, which is shown only after a session completes normally. Over 30 days: 240,000 sessions started, 18,000 ended in an error or timeout, and the widget collected 2,880 ratings from the rest. 94% thumbs-up. Which reading of the 94% is correct?
MCQ-14A support org wants to gate releases on an LLM judge that scores replies 1–5. The judge runs a different model family from the one that drafts replies, its prompt and model version are pinned in the repo, and the harness caches judge scores per reply. The 4,000-item eval set is a stratified sample of production traffic: 60% English billing, 25% Spanish refunds, 15% German cancellations. On it the judge's mean sits within 0.1 of the expert panel's. Which check is most likely to change whether the metric can gate releases?
MCQ-18An autonomous customer-support agent enters an unrecoverable state where it repeatedly calls the same external search tool with identical parameters across 15 turns without returning an answer or raising an exception. How should distributed tracing and telemetry be structured under OpenTelemetry conventions to detect and isolate this loop failure?
MCQ-19An LLM support agent calls `issue_refund(order_id, cents)` and `send_receipt(email)`. On 2024-05-01, 340 of 1,200 sessions issued two refunds for the same order. The trace records the full prompt (longest 3,100 tokens), the model alias `gpt-4o` (undated), temperature 0, the per-session seed, the tool schemas, and every tool call with its arguments and results. Six months later an engineer replays all 1,200 traces in a sandbox that returns each recorded tool result in its original position and never calls the live tools. The duplicate refund reproduces in 90 of the 340 sessions. What explains the other 250?
MCQ-25A summarization service handles 40,000 requests/day. After last Tuesday's model upgrade, the weekly dashboard is flat: success rate 99.8%, mean latency 1.9 s, mean output tokens 210. Support tickets about omitted action items rose 12%, and 80% of them come from one tenant whose documents average 60 pages. Traces are sampled at 1% and carry only latency and token spans. Which telemetry change would let the team separate a model-version regression from a change in that tenant's documents?
MCQ-16A fraud model scores 4M card transactions a day. Over the last 30 days the team observes: merchant_category PSI 0.31, sustained since day 3 with no reversion; amount PSI 0.02; prediction-score PSI 0.05; overall precision@1% on 30-day-lagged labels 0.62→0.58, inside its normal 0.57–0.63 band; precision@1% within the crypto slice (20% of traffic, volume up 8×) 0.62→0.44; and a one-day embedding-drift spike on day 17 that reverted by day 18. Which signal is the strongest evidence of actionable drift?
MCQ-22A payments API writes three observability streams to object storage billed at $0.023/GB-month: traces at 10% sampling (40 GB/day), debug logs that embed raw request bodies containing PAN fragments (150 GB/day), and audit events for every admin and data-access action (50 MB/day). SOC 2 requires audit events retained 12 months, the incident-review window is 30 days, cardholder data must not persist beyond 7 days, and the storage budget is $60/month. Which retention and sampling policy satisfies all four constraints?