Skip to content
Tech Interview Prep
Technical interview guide

Model Serving & Inference Infrastructure

The infrastructure choices behind getting predictions out of a trained model at production latency and scale.

Read
45 min
Practice MCQs
25
Interview Q&A
25
Edition
v6
Editorial status
Reviewed

Scope: NVIDIA Triton, KServe, Kubernetes autoscaling, Amazon SageMaker inference, and NIST AI RMF guidance current 2026-09-04.

Curated: · Written: · Reviewed:

25 questions

MCQ-16A service completes 10,000 requests in a 10-minute window. The dashboard shows a mean response time of 119.6 ms and a p50 of 20 ms, both well under the 200 ms the team targets, so the panel is green. The raw distribution says why: 9,800 requests finished in 20 ms and 200 finished in 5,000 ms. The team's SLO allows no more than 1% of requests in a window to exceed 200 ms. What should the team conclude?
MCQ-6A model-serving Deployment pulls a 12 GB weight file from object storage and loads it into GPU memory before it can answer requests. During a rolling update, new pods report Running and the Service routes to them while the download is still in progress, so clients get 503s. What should the readiness probe mean for this pod?
MCQ-18A Triton model repository holds one serving model with three onnx versions: model_repository/ranker/config.pbtxt # platform and max_batch_size set, no version_policy model_repository/ranker/1/model.onnx model_repository/ranker/2/model.onnx model_repository/ranker/3/model.onnx The server starts cleanly - no error and no warning line for ranker. Clients that name version 2 in the request are rejected; clients that leave the version out get version 3's outputs. Nothing failed to load or initialize, so which mechanism explains the rejections?
MCQ-8A ranking service defines a release as a container image digest plus the model weights file hash. Release R42 goes live Tuesday 09:00. Overnight the feature service regenerates its per-feature normalization constants from the newest data; the service loads them from a config store at process start. On Thursday the team rolls back to R42's image digest and weights hash, and the top-10 list differs from what R42 returned on Tuesday for 12% of requests. What must be added to the release definition for a rollback to reproduce R42's predictions?
MCQ-17A prediction service caches scores under `sha256(entity_id + model_name)`. Entity IDs are per-tenant integers that start at 1, and all 12 tenants share one cache. Every tenant runs the same model, so tenant 3's entity 42 and tenant 7's entity 42 produce the same key. Writes are spread evenly across tenants. Roughly what fraction of cache hits return a score that was computed for a different tenant?
MCQ-11A synchronous inference API has a 2,000 ms end-to-end deadline. Each model-server attempt costs 600 ms at p99, and the client SDK waits 150 ms between attempts. The client SDK makes up to 3 attempts and the gateway makes up to 2, so one request can reach the model server 6 times. What is the largest total number of model-server attempts per request that keeps the worst case inside the deadline?
MCQ-10An LLM team quantizes a 7B FP16 model to W8A8 INT8 with per-tensor scales and a 512-example calibration set. On a 2,000-example replay of production traffic, overall exact-match accuracy moves 91.3% → 90.3%, p99 latency 180 ms → 112 ms, and throughput rises 1.6x. On the 600-example numeric-table slice of that replay, exact-match accuracy moves 94.2% → 88.7%. The serving SLO is overall accuracy ≥ 90% and no slice more than 2 points below its FP16 baseline. What should the team do?
MCQ-15A team serves a 13B model on one 80 GB GPU. Requests arrive concurrently with unknown output lengths, and per-token decode latency must not increase. With a contiguous KV cache reserved to the 4096-token context limit, they fit 9 concurrent requests and measure that 21% of reserved KV slots ever hold a real token. Model weights and the context limit are fixed. Which change removes that unused reservation without shortening context or adding per-step latency?
MCQ-12A continuous-batching LLM server caps each batch at 32 sequences. A decode step takes 40 ms whether the batch holds 1 sequence or 32: decode is memory-bandwidth bound. Traffic holds steady at 900 requests/s. Per-request GPU execution time stays flat at 40 ms, yet p99 end-to-end latency climbs from 150 ms to 4 s and is still rising after ten minutes. What does the arithmetic say is happening?
MCQ-3A service runs a dynamic batcher in front of one GPU: max_batch_size = 24, max_queue_delay = 8 ms. Profiling shows a step costs 12 ms of fixed launch and kernel time plus 1.5 ms for every request in the batch. Production traffic is a steady 250 requests/s, one step per request. Minutes after the ramp, GPU utilization reads 99%, queue depth grows without settling, and p99 passes 2 s and keeps climbing. Configuration is the only thing the team can change this quarter; no second GPU is funded. Which conclusion do the numbers support?
MCQ-7An inference service runs on one node with 16 GB of memory free for activations, and each in-flight request holds 40 MB. The node serves 2000 req/s while arrivals run at 2500 req/s. The p99 latency SLO is 500 ms. What is the largest FIFO queue depth that both fits in the free memory and keeps p99 within the SLO?
MCQ-2A ranking service calls a feature store, then a GPU model server, then a downstream re-ranker. Instrumentation shows GPU kernel time p99 = 45 ms, model-server queue wait p99 = 260 ms, end-to-end p99 = 310 ms, and 4.1% of requests exceeding the 200 ms deadline. GPU utilization averages 55% but reaches 98% in two-minute bursts each hour. The team's SLO reads 'p99 kernel time < 50 ms' and has never fired. What should they change?
MCQ-23A ranking service moves from 16 vCPU to a single T4 GPU, batch size 1, fixed input shape. Under a 40 req/s load test the CPU version held p50 45 ms / p99 90 ms; the GPU version reports p50 12 ms but p99 310 ms, with SM utilization at 22% and one host core pinned at 100% running tokenization and feature assembly. The p99 SLO is 150 ms. What should the team fix first?
MCQ-20A model server runs in Kubernetes behind a Service. Its Deployment readinessProbe opens a TCP connection to port 8080. The image is `ranker:v7`; the entrypoint reads weights from `s3://models/ranker/latest/`, and the release job repoints `latest/` to the v7 weights 4 minutes after the Deployment starts. The server loads weights lazily on the first `/predict` call. `/healthz` returns 200 and echoes the `MODEL_VERSION` env var, which the release job sets to `v7` at deploy time. Each `/predict` response includes a `model_version` field naming the weights that produced it. During the rollout, all 6 pods pass the TCP check within 15 seconds, take traffic, and for the next hour every `/predict` response reports `model_version: v6`. Which readiness check would have kept the pods out of the Service until they served v7?
MCQ-19An inference gateway batches 8 requests into one 1.2 s GPU step. Clients set a 2 s deadline, and roughly 30% disconnect before their result arrives; the worker still runs each batch to completion and writes the result to a cache nobody reads. GPU utilization holds at 98% while completed-request throughput has dropped 25%. Which change recovers the lost throughput?
MCQ-14A serving cluster runs prefill and decode in a single continuous-batching pool on the same GPUs. Under bursty long-prompt traffic, p99 inter-token latency (ITL) for streaming requests jumps from 45 ms to 380 ms while time-to-first-token (TTFT) stays inside its SLO. Which property of moving prefill and decode onto separate worker pools addresses the violation?
MCQ-21An LLM inference service autoscales on requests per second, targeting 8 requests/s per replica; each replica sustains about 2,000 output tokens/s before p99 time-to-first-token exceeds 500 ms. At 09:00 the fleet serves 400 requests/s averaging 120 output tokens each, holds 50 replicas, and reports p99 TTFT of 80 ms. At 14:00 it still serves 400 requests/s, now averaging 900 output tokens each, holds the same 50 replicas, and reports p99 TTFT of 3.1 s with a queue depth of 60. Which conclusion about the 14:00 fleet is correct?
MCQ-4A six-pod GPU inference service runs one model per pod; its HorizontalPodAutoscaler targets 70% CPU utilization and holds at 6 replicas throughout a weekday burst. Two measured points from that burst: at 20 req/s (1,200 req/min) CPU is 23%, GPU SM occupancy 96%, GPU memory 88%, in-flight requests (running plus queued) 4, p99 210 ms; at 50 req/s (3,000 req/min) CPU is 24%, occupancy 98%, memory 88%, in-flight 130, p99 2.6 s. The team wants replicas to grow before p99 crosses its 400 ms SLO and to shrink again after the burst. Which scaling change meets that requirement?
MCQ-24A model-serving deployment runs 8 replicas, each sustaining 50 rps at the autoscaler's CPU target. The autoscaler evaluates every 30 s, and the traffic jump occurs immediately after an evaluation, so the scale-up is triggered 30 s later. A new replica must download its artifact (90 s) and complete warmup (25 s) before it serves any traffic. A scheduled event jumps demand from 400 rps to 1200 rps at t=0. What is the earliest time the fleet can serve the full 1200 rps?
MCQ-5A platform defaults every service to scale-to-zero. Cold start for the model-backed service is 45 s at p50 and 90 s at p99 because weights load from object storage; a warm instance bills per minute. For which workload is scale-to-zero the wrong default?
MCQ-1An ML platform scores 48,000 store-SKU rows from a warehouse snapshot that lands once a day at 02:00. Store managers read the forecasts on a dashboard from 07:00, and no forecast may be more than 24 hours old. A GPU worker pays a 4-minute model load when it starts cold, then scores 900 rows a second; GPU time costs USD 0.90 an hour while a worker or endpoint is running, and the service has a USD 250 monthly budget. A dashboard page must render in under 2 seconds. Which serving plan fits?
MCQ-9A fleet serves five endpoints from 8 replicas of the same 70B model, and the router currently sends each request to whichever replica has the shortest queue. The team will add sequence-aware (session-sticky) routing, pinning a session to one replica, but only for an endpoint that shortest-queue routing answers wrongly, since stickiness also concentrates load on hot replicas. Which endpoint needs it?
MCQ-13A document-intelligence service calls a 70B model to extract fields from scanned PDFs; extraction takes 25–90 s per document. A nightly ETL job submits 40,000 documents, can poll a status endpoint, must write each result to a warehouse table as soon as it is ready, and must not lose the rest of the batch when one document fails. Its workers run in a subnet with no inbound HTTP listener, and the API gateway drops any connection held open longer than 30 s. Which serving design fits this workload?
MCQ-22A fraud-scoring service runs this fallback ladder: v7 primary (250 ms timeout) → v6 previous version → cached v7 score (1 h TTL) → rules engine (declines anything over $500) → fail closed. A feature-store change pushes v7's p99 from 180 ms to 3.1 s, and every request now falls to v6. On the last 7 days of labelled traffic, v6's recall on the card-testing attack class is 0.22 against v7's 0.91, while v6's overall AUC is 0.94 against v7's 0.95. Card-testing is 3% of traffic, moves funds in under 10 minutes, and averages $340 per approved transaction. The service is taking 4,000 req/s. Which action should the on-call take?
MCQ-25A fraud-scoring endpoint returns scores in 12 ms at p99 with a 0.05% error rate and 31% CPU. Its overall alert rate has held at 2.1% for six weeks. Over the last two weeks, chargebacks on transactions the model scored below 0.3 rose from 0.4% to 1.9%, while chargebacks on transactions scored above 0.7 stayed flat. Which conclusion do the numbers support?