Technical interview guide

Model Serving & Inference Infrastructure

The infrastructure choices behind getting predictions out of a trained model at production latency and scale.

Read: 45 min
Practice MCQs: 25
Interview QA: 25
Edition: v6
Editorial status: Reviewed
Relevant for: AI EngineerMLOps Engineer

Scope: NVIDIA Triton, KServe, Kubernetes autoscaling, Amazon SageMaker inference, and NIST AI RMF guidance current 2026-09-04.

Interview QA

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: 2026-10-05 · Written: 2026-10-05 · Reviewed: 2026-10-03

QA-1

Design a production real-time inference service.

QA-2

Choose among real-time, asynchronous, batch, and streaming inference.

QA-3

Create an end-to-end latency budget for inference.

QA-4

Tune dynamic batching for a latency-sensitive GPU model.

QA-5

Design autoscaling for a slow-loading accelerator model.

QA-6

Define the artifact and configuration lifecycle for serving.

QA-7

Prevent overload and retry storms in an inference API.

QA-8

Design health probes for a model server.

QA-9

Evaluate a CPU-to-GPU serving migration.

QA-10

Introduce quantization without silently degrading users.

QA-11

Design prediction caching safely.

QA-12

Operate a multi-model shared serving cluster.

QA-13

Serve a stateful sequence model correctly.

QA-14

How do you manage KV cache memory allocation and fragmentation using PagedAttention in high-throughput LLM serving?

QA-15

Why and how do modern LLM inference architectures disaggregate prefill and decode stages across distinct GPU clusters?

QA-16

Investigate rising p99 latency with unchanged mean execution time.

QA-17

Handle malformed and adversarial inference inputs.

QA-18

Measure cost efficiency for model serving.

QA-19

Migrate an inference API schema without breaking clients.

QA-20

Design observability for an inference endpoint.

QA-21

Load-test an inference service before launch.

QA-22

Choose a fallback for a high-stakes prediction service.

QA-23

Roll back a model-serving release safely.

QA-24

Respond to a wrong-model-version serving incident.

QA-25