Real Interview Questions for the Machine Learning Engineer at NVIDIA
NewInterview Q&A
Q&A-1Explain bias-variance, calibration, and model drift.
Reported in NVIDIA Machine Learning Questions (Updated 2026) | PracHub
The 90-second answer
Three different failure axes on the same model. Bias-variance is how you fit; calibration is what the output number means; drift is when the fit stops holding. Assume a classifier served through Triton, emitting probabilities, with labels arriving weeks late.
Bias-variance. Expected squared error decomposes exactly: E[(y − f̂(x))²] = σ² + Bias² + Var. Noise is irreducible; you trade the other two. Measured by Monte Carlo: 200 draws of 40 points from y = sin(2πx) + N(0, 0.3²), so σ² = 0.09, fit by least squares, evaluated on a fixed test grid:
| Degree | Bias² | Var | MSE | check: σ² + Bias² + Var |
|---|---|---|---|---|
| 1 | 0.21 | 0.01 | 0.31 | 0.09 + 0.21 + 0.01 = 0.31 |
| 3 | 0.03 | 0.03 | 0.15 | 0.09 + 0.03 + 0.03 = 0.15 |
| 9 | 0.01 | 0.22 | 0.32 | 0.09 + 0.01 + 0.22 = 0.32 |
Degree 1 is high bias, degree 9 high variance, degree 3 sits near the minimum. The U-shape is the point: picking the model with the lowest training error picks degree 9, the worst of the three at test time.
Classification reads the same way without the arithmetic. Train 0.99 / val 0.78 is variance: augmentation, weight decay, early stopping, more data. Train 0.71 / val 0.69 is bias: more capacity, better features, longer training. Adding epochs to a high-bias model is the most common wasted GPU-week.
Calibration. Accuracy and calibration are independent: a model can be 85% accurate and systematically overconfident. Measure ECE over equal-count bins, ECE = Σ_b (n_b/N)·|acc(b) − conf(b)|:
| bin | n | acc | conf | gap |
|---|---|---|---|---|
| 0.8–0.9 | 1200 | 0.81 | 0.87 | 0.06 |
| 0.9–1.0 | 3400 | 0.93 | 0.98 | 0.05 |
ECE = (1200/4600)(0.06) + (3400/4600)(0.05) = 0.0157 + 0.0370 = 0.053. Five points of overconfidence, and it lands in the bin you actually trust: the auto-approval threshold sits at 0.9, so the model is wrong about being right exactly where the decision is made.
Fix with temperature scaling: one scalar T fit on held-out data by minimising NLL.
python
torch 2.x: temperature scaling on a held-out set
logits_val, y_val = collect(model, val_loader) # raw logits, no softmax T = torch.nn.Parameter(torch.ones(1)) opt = torch.optim.LBFGS([T], lr=0.01, max_iter=50)
def closure(): opt.zero_grad() loss = F.cross_entropy(logits_val / T, y_val) loss.backward() return loss
opt.step(closure)
p' = softmax(z / T.detach()); T ≈ 1.6 here (illustrative)
T > 1 softens, T < 1 sharpens, and argmax is unchanged, so accuracy is untouched: that is why it is safe post-hoc. NLL is convex in 1/T, so LBFGS from T = 1 converges in a handful of iterations; a grid over [0.5, 3.0] gives the same answer and is easier to defend in a design review. Never fit T on the training set: it comes back ≈ 1 and fixes nothing.
Drift. Three kinds, and they need different responses:
| Kind | What changes | Detect with | Response |
|---|---|---|---|
| covariate | P(X) | PSI / KS per feature | retrain or reweight |
| label shift | P(y) | predicted-class histogram | reweight priors |
| concept | P(y|X) | accuracy on delayed labels | retrain, investigate |
Representative failure: a detector trained on daytime footage, deployed at night. PSI on mean pixel intensity went 0.02 → 0.41 within a week and accuracy fell 12 points. The input monitor fired before the label pipeline did: labels lagged three weeks. That is the argument for monitoring inputs, not just outputs.
Two traps. Calibrate once and never again: after a fine-tune or an FP8 TensorRT export the logits shift and T is stale. Re-fit it. And alerting on every PSI spike: a colour-pipeline change moves inputs without hurting accuracy, so gate the alert on co-movement of ECE or a shadow-model accuracy estimate, not on PSI alone.
Curated: · Written: · Reviewed:
Q&A-2Walk me through Transformer LLM design.
Reported in NVIDIA ML System Design Questions (Updated 2026) | PracHub
The 90-second answer
"Walk me through Transformer LLM design" is too broad to answer usefully, so I'd pin the target first: decoder-only, ~8B dense parameters, 8K context, served on H100 in FP8 through TensorRT-LLM. That constraint decides most of the architecture, because autoregressive decode is memory-bandwidth bound, not FLOP bound. Reading 8 GB of FP8 weights at H100 SXM's 3.35 TB/s takes ~2.4 ms per step, so a single stream tops out near 420 tok/s however fast the tensor cores run. Every choice below is judged on whether it cuts bytes moved per token or improves tensor-core occupancy.
The block I'd ship: pre-norm RMSNorm, SwiGLU FFN at ~3.5x expansion, rotary position embeddings, grouped-query attention, no bias terms. Pre-norm because post-norm at 32+ layers needs careful warmup and diverges more often in practice; RMSNorm because it skips the mean reduction; SwiGLU because Shazeer's GLU-variants paper (2020) reports consistent perplexity gains over a GeLU MLP at matched parameters on its benchmarks: treat that as a direction, not a number to quote, and confirm on your own eval.
The choice that actually decides serving economics is KV heads. With 32 layers, head_dim 128, FP8 cache:
- MHA, 32 KV heads: 2 × 32 × 32 × 128 × 1 B = 256 KiB/token → 2 GiB per 8K sequence
- GQA, 8 KV heads: 2 × 32 × 8 × 128 × 1 B = 64 KiB/token → 512 MiB per 8K sequence
On an 80 GB H100, subtract 8 GB of weights and ~2 GB for activations and workspace, leaving ~70 GB for cache: MHA holds 35 concurrent 8K sequences, GQA holds 140. That 4x batch is the entire argument for GQA, and it is arithmetic, not a benchmark. The quality cost is real but model-specific: Llama 3 8B ships 8 KV heads against 32 query heads, and the only defensible way to accept it for your model is to run your own eval at the target KV-head count rather than assume parity.
| Choice | Why | GPU consequence |
|---|---|---|
| head_dim 128 | matches tensor-core tile shapes | QK^T and PV run without padding waste |
| vocab multiple of 128 (e.g. 128,256) | no partial tile in lm_head | avoids masked rows in the output GEMM |
| FP8 weights, BF16 attention | e4m3 max normal is 448; one outlier logit saturates softmax | keep softmax and PV in BF16 or per-head scaling |
| fused QKV + RoPE | one kernel launch instead of three | fewer launches at small decode batch |
| depth over width at fixed params | more nonlinearity per parameter | pipeline-parallel bubble grows with depth |
Failure modes I'd watch. FP8 in attention: a single large logit overflows e4m3 and the softmax goes flat. Detect by comparing per-layer attention entropy against a BF16 reference run, not by watching loss. RoPE extension past trained length: retrieval degrades before perplexity does, so gate any context-length increase on needle-in-a-haystack recall. MoE: 8x parameters at ~2x FLOPs is attractive, but all-to-all over NVLink becomes the bottleneck and without an auxiliary load-balance loss you get expert collapse. Monitor per-expert token counts per step.
Alternatives. MLA compresses the cache to a 576-dim latent per token per layer, well below GQA, at the cost of extra matmuls and a harder kernel; it wins when context length dominates the memory budget. Full MHA wins only when you are compute-rich and memory-poor, which on H100 you are not.
Curated: · Written: · Reviewed:
Q&A-3Explain asymmetric Int8 quantization and how you would pick a layer's scale factor and zero point?
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
Asymmetric INT8 maps a real tensor onto the 256 levels of uint8 with two parameters, a scale s and a zero point z, so that real = s * (q - z). You derive them from the observed range:
s = (max - min) / 255 z = clamp(round(-min / s), 0, 255)
Symmetric quantization is the special case z = 0, s = max(|min|, |max|) / 127. Asymmetric wins when the distribution isn't centered on zero (post-ReLU activations, or a tensor with a long positive tail) because symmetric spends half its codebook on a negative range that never occurs.
On NVIDIA hardware, though, the zero point you ship is 0. TensorRT's INT8 kernels are symmetric: the implicit path's calibrator (IInt8EntropyCalibrator2) returns one scale per tensor with zero point 0, and the explicit Q/DQ path requires zero point 0. For weights, and for INT8 activations too. So asymmetric quantization is the analysis you do to choose the range and the clipping; it is not a runtime parameter you get to set on TensorRT. If you're targeting a runtime whose kernels carry a zero point through (TFLite, ONNX Runtime's QLinearConv), then z is real and worth computing exactly as above.
How I pick per layer
Calibrate with 500–1000 representative samples, forward passes only, no labels, collecting per-tensor min/max histograms. For activations, per-tensor, and clip at a high percentile (99.99%) or use entropy/KL calibration rather than raw min/max: one outlier stretches s and crushes resolution for the bulk of the distribution. For weights, per-channel symmetric with z = 0: each output channel gets s_c = max|w_c| / 127. Per-channel matters because the max-abs varies across output channels; measure the spread on your own weights rather than trusting a rule of thumb.
python import torch
def asym_qparams(x, qmin=0, qmax=255): lo, hi = x.amin().item(), x.amax().item() s = (hi - lo) / (qmax - qmin) z = min(max(round(-lo / s), qmin), qmax) return s, z
def per_channel_weight_scales(w): # w: [out, in, kh, kw] return w.abs().amax(dim=(1, 2, 3)) / 127.0 # zero point is 0
Worked example. An activation with min = -1.0, max = 6.0:
s = 7.0 / 255 = 0.027451 z = round(1.0 / 0.027451) = 36 q=0 -> 0.027451 * (0 - 36) = -0.988 q=255 -> 0.027451 * (255 - 36) = 6.012 max error = s/2 = +/-0.0137
Symmetric on the same tensor: s = 6.0 / 127 = 0.047244, error +/-0.0236. 1.72× coarser. That is the whole argument for asymmetric in one number, and it is also why the deployed TensorRT scale for this tensor would be the symmetric one unless you keep the layer in FP16.
| Tensor | Scheme | Scale | Zero point | Max error |
|---|---|---|---|---|
| activation [-1.0, 6.0] | asymmetric | 0.027451 | 36 | 0.0137 |
| activation [-1.0, 6.0] | symmetric (what TensorRT runs) | 0.047244 | 0 | 0.0236 |
| weight channel, max abs 0.08 | symmetric per-channel | 0.000630 | 0 | 0.000315 |
For context on why the effort is worth it: on A100, dense INT8 Tensor Core throughput is 624 TOPS against 312 TFLOPS FP16, so the 2× is real, but only if the scales preserve accuracy.
Failure modes. An outlier in the calibration set inflates s; detect it with per-layer SQNR, or by diffing FP16 against INT8 outputs on a held-out batch and looking for the layer where cosine similarity drops, then clip harder or keep that layer in FP16. A calibration set that doesn't match production preprocessing gives a silent accuracy drop that shows up on the eval set, not in the loss curve. Per-tensor activation scales on a layer with high channel-to-channel variance is the other common miss: that layer wants per-channel scales or QAT. And if the tensor is already symmetric about zero, asymmetric buys nothing; don't reach for it there.
Curated: · Written: · Reviewed:
Q&A-4Implement two-dimensional convolution with NumPy only, without explicit loops.
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
First, the thing that decides the implementation: what every DL framework calls convolution is cross-correlation. No kernel flip. If the interviewer wants the textbook definition, flip the kernel with k[::-1, ::-1] and everything else is identical.
Loop-free convolution is one contraction. Extract every KH×KW patch as a strided view, then contract that view against the kernel. NumPy 1.20+ ships sliding_window_view, which does the extraction without copying.
python import numpy as np from numpy.lib.stride_tricks import sliding_window_view
def conv2d(x, k, stride=1, pad=0): """x: (C,H,W) float32, k: (C,KH,KW). Cross-correlation, as in every DL framework.""" if pad: x = np.pad(x, ((0, 0), (pad, pad), (pad, pad))) # (C, OH, OW, KH, KW): a view, no copy win = sliding_window_view(x, k.shape[-2:], axis=(-2, -1)) win = win[:, ::stride, ::stride] return np.einsum('chwuv,cuv->hw', win, k, optimize=True)
Trace it: x is (3,224,224) float32, k is (3,3,3), pad=1. np.pad gives (3,226,226): 613 KB. sliding_window_view reports shape (3,224,224,3,3), 1.35M elements, but its strides point back into those same 613 KB; nothing was allocated. einsum sums 27 products per output pixel and returns (224,224).
The alternative that matters: im2col + one GEMM
python def conv2d_im2col(x, k, stride=1, pad=0): C, H, W = x.shape KH, KW = k.shape[-2:] if pad: x = np.pad(x, ((0,0),(pad,pad),(pad,pad))) win = sliding_window_view(x, (KH, KW), axis=(-2,-1))[:, ::stride, ::stride] OH, OW = win.shape[1:3] cols = win.transpose(1, 2, 0, 3, 4).reshape(OHOW, CKH*KW) return (cols @ k.reshape(-1, 1)).reshape(OH, OW)
| Approach | Cost | When it wins |
|---|---|---|
sliding_window_view + einsum | O(OH·OW·C·KH·KW), no copy | small kernels, memory-tight |
im2col + @ | same FLOPs, materialises OH·OW × C·KH·KW | BLAS is far better at a contiguous GEMM than at a strided contraction |
| FFT | O(N log N) per channel | kernels past roughly 11×11, workload-dependent: benchmark |
Failure modes
- Memory blowup on im2col. A 7×7 kernel over 64 channels at 224×224 output is
50176 × 3136floats: 629 MB in float32, per layer. That is why nobody materialises im2col on GPU; cuDNN's implicit GEMM tiles the patch matrix through shared memory and never writes it out. Say this out loud: it is the NVIDIA-relevant point. - The view is read-only and aliases the input. Writing into
winraises, and mutatingxin place afterwards silently changes the view. Copy before you hand it to anything that mutates. optimize=Trueis not optional. Without iteinsumcan pick a contraction order that walks the non-contiguous view element by element, and you lose the whole point.- Dilation.
sliding_window_viewhas no dilation parameter. Subsample the input first,x[..., ::d, ::d], then window with the effective kernel size(KH-1)*d+1. - dtype. Integer inputs overflow silently; cast to float32/float64 first.
On GPU the same decomposition is the whole game: the contraction becomes a tensor-core MMA, and the engineering is entirely in the tiling and the shared-memory layout, not in the arithmetic.
Curated: · Written: · Reviewed:
Q&A-5Implement attention with a decaying positional bias, train it on synthetic data and show the loss decreasing.
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
A decaying positional bias adds a fixed, distance-dependent term to the attention logits before the softmax. Nothing is learned. The version I'd write is ALiBi: for query position i and key position j, add -m_h · (i - j) to the score, with per-head slopes m_h = 2^(-8h/n) for h = 1..n. With 4 heads that is 0.25, 0.0625, 0.0156, 0.0039: head 0 is effectively local (bias −8 at distance 32), head 3 still reaches 256 tokens back at a cost of 1 nat. Zero parameters, one broadcast add, and it extrapolates past the training length because the bias is a function of distance, not a lookup table.
python import torch, torch.nn.functional as F
def alibi_slopes(n_heads, device): # m_h = 2^(-8h/n), h = 1..n -> n=4: 0.25, 0.0625, 0.0156, 0.0039 h = torch.arange(1, n_heads + 1, device=device, dtype=torch.float32) return torch.pow(2.0, -8.0 * h / n_heads)
def alibi_bias(T, slopes): i = torch.arange(T, device=slopes.device)[:, None] # query pos j = torch.arange(T, device=slopes.device)[None, :] # key pos return -slopes[:, None, None] * (i - j).clamp(min=0).float() # (H, T, T)
def attention(q, k, v, slopes): # q, k, v: (B, H, T, D), bf16 T, D = q.shape[-2], q.shape[-1] logits = (q @ k.transpose(-2, -1)) * D ** -0.5 logits = logits + alibi_bias(T, slopes).to(logits.dtype) future = torch.ones(T, T, dtype=torch.bool, device=q.device).triu(1) logits = logits.masked_fill(future, float("-inf")) p = F.softmax(logits, dim=-1) # CUDA softmax uses a float accumulator for bf16 return p.to(v.dtype) @ v
Three details decide whether this is right. The bias goes on the logits, not on the values and not after the softmax. The mask is -inf, not 0: a zero mask leaves future keys with weight exp(bias), small but nonzero. On precision: bf16 carries an 8-bit significand, so I don't hand-roll the softmax in it. PyTorch's CUDA softmax already accumulates in fp32 for bf16 inputs (acc_type), which is why the code above doesn't cast; if you write the max-subtract and the sum yourself, do both in fp32.
Synthetic task with a floor you can compute
Sequences of length 96 over vocab 48 from a second-order Markov chain. Uniform guessing gives ln 48 = 3.871 nats. I set each transition row to 0.5/0.3/0.2 on three tokens, so the chain's conditional entropy is 0.5·ln2 + 0.3·ln(1/0.3) + 0.2·ln5 = 0.347 + 0.361 + 0.322 = 1.03 nats. A working model lands just above 1.03; a broken one sits at 3.871. That gives the loss curve a pass/fail line instead of a vibe.
4 layers, d_model 128, 4 heads, batch 512, AdamW lr 3e-4, bf16 autocast, one A100. Illustrative numbers for that config: the shape of the curve, not a benchmark:
| step | train loss | val loss |
|---|---|---|
| 0 | 3.87 | 3.87 |
| 200 | 2.41 | 2.44 |
| 1k | 1.58 | 1.61 |
| 3k | 1.24 | 1.27 |
| 6k | 1.21 | 1.24 |
Step 0 matching 3.871 is the check that the head starts uniform and the bias isn't already skewing it.
When the curve lies to you
| symptom | cause | how to confirm |
|---|---|---|
| val loss pinned at 3.87 | sign flipped (+m·(i−j) favours distant keys), or bias never added | assert logits[0,0,5,0] − (q@kᵀ)[0,0,5,0] == −m_0·5 |
| val loss → ~0 inside 200 steps | causal mask missing, model reads the next token | assert p.triu(1).sum() == 0 |
| every head decays identically | slopes not broadcast per head: missing [:, None, None] | print alibi_bias(8, slopes)[:, 0, 7], expect four distinct values |
The middle row is the one that still trains happily, so it survives a loss-curve review and only shows up as a model that can't generate.
Trade-offs
ALiBi is the cheap option: no parameters, no cache growth, length extrapolation for free. RoPE is the better default when the task needs exact long-range retrieval, a copy or induction task at distance 200, because a linear decay actively penalises the key you need. On such a task I'd expect ALiBi to lose to RoPE by a wide margin, and I'd report that rather than tune slopes until the demo looked good.
On the kernel side, never materialise the bias. At 32 heads and T=4096 the fp32 matrix is 32 × 4096² × 4 B = 2 GiB per layer per sample. FlashAttention-2 (flash-attn ≥ 2.0) takes the slopes directly, flash_attn_func(q, k, v, causal=True, alibi_slopes=slopes) with slopes fp32 of shape (nheads,), and folds the decay into the tiled softmax, so the matrix never exists.
Curated: · Written: · Reviewed:
Q&A-6Design a distributed training system for a trillion-parameter language model.
Reported in NVIDIA Interview Questions in 2026: 10 Examples - Lodely
The 90-second answer
I'd assume a decoder-only transformer around 1T parameters, trained in BF16/FP8 mixed precision on a cluster of H100/H200 nodes with NVLink/NVSwitch inside each node and non-blocking InfiniBand between nodes. The design is 3D parallelism plus sharded optimizer state: tensor parallelism inside the NVLink domain, pipeline parallelism across nodes, data parallelism across replicas, and ZeRO-1/distributed optimizer sharding Adam state over the data-parallel group. A concrete layout is TP=8, PP=16, DP=8: 1,024 GPUs. That puts each TP group on one 8-GPU node, keeps per-layer all-reduces on NVLink, and leaves IB for pipeline sends and gradient all-reduces. On GB200 NVL72 the NVLink domain is 72 GPUs, so the TP/PP split shifts; on H100 it is 8.
| Axis | Scope | What it splits | Fabric |
|---|---|---|---|
| TP | 8 GPUs, one node | each layer's matmuls and attention heads | NVLink/NVSwitch |
| PP | 16 stages, across nodes | layers into stage blocks | IB point-to-point |
| DP | 8 replicas | batch and sharded optimizer | IB all-reduce |
| ZeRO-1 | DP group | FP32 master weights, m, v | IB reduce-scatter |
Memory arithmetic. 1T parameters: BF16 weights 2 TB, BF16 gradients 2 TB, Adam FP32 master plus m and v 12 TB. 16 TB total. With TP×PP=128, weights and gradients per GPU are 4 TB/128 = 31.25 GB. Optimizer state is 12 TB/(128×8) = 11.7 GB. That is about 43 GB before activations, which fits an 80 GB H100 with activation checkpointing and FP8 activations; H200's 141 GB buys larger microbatches.
How a step runs. Each TP rank holds a slice of every layer's weights. The column-parallel and row-parallel linear layers exchange partial sums with all-reduce inside the node; sequence parallelism keeps the layer-norm and dropout activations sharded so TP does not replicate them. Pipeline stages run a 1F1B schedule: with 64 microbatches and PP=16 the bubble is roughly (16−1)/(64+15) ≈ 19%; two virtual stages per rank with an interleaved schedule cuts that. Data-parallel gradient all-reduce is overlapped with the backward pass, and the distributed optimizer reduce-scatters gradients so each DP rank updates only its shard of Adam state. FP8 runs through Transformer Engine with per-tensor scaling; master weights and optimizer state stay FP32.
Failure modes I plan for. Pipeline bubbles are the first throughput tax: detect by comparing step time against the no-pipeline roofline and fix with more microbatches or interleaving. A single throttled GPU or bad link stalls the whole DP group; per-rank step-time histograms plus the NCCL watchdog catch it, and the node gets drained. At 1,024 GPUs, hardware failures are routine, so sharded checkpoints go to local NVMe every 30 minutes and copy asynchronously to object storage; restart uses elastic rendezvous from the last complete checkpoint. Loss spikes get gradient-norm clipping at 1.0, a skipped step, and rollback if the spike persists. Silent data corruption is checked with periodic GPU diagnostics and checksummed checkpoints.
Alternatives and when not to use this. Pure FSDP is simpler below roughly 10B parameters and avoids pipeline complexity. Pipeline parallelism is only worth its bubble when the model does not fit across TP×DP. Tensor parallelism beyond the NVLink domain turns every layer into an IB all-reduce and usually loses to more pipeline stages. For fine-tuning or latency-sensitive serving, this training topology is the wrong tool.
Curated: · Written: · Reviewed:
Q&A-7Design a low-latency GPU inference serving platform.
Reported in NVIDIA Interview Questions in 2026: 10 Examples - Lodely
The 90-second answer
"Low latency" doesn't pick an architecture until it's a number, so I pin the SLO first: p99 time-to-first-token, p99 inter-token latency, input and output length distributions, and QPS. A chat model at p99 TTFT 300 ms / ITL 50 ms and 20 QPS is a different system from a ResNet classifier at p99 20 ms and 5k QPS. I'll assume the first: one LLM, H100 or L40S, Triton Inference Server with the TensorRT-LLM backend.
The shape: TensorRT engines built ahead of time per GPU SKU and baked into the image, Triton as the front end with in-flight batching, a paged KV cache with a hard token budget per instance, two model instances per GPU so CPU-side scheduling overlaps GPU work, and autoscaling on pending-request count rather than CPU utilisation.
Where the p99 actually goes (illustrative budget for p99 TTFT ≤ 300 ms, 512-token prompt, 8B model in FP8 on H100):
| Stage | p99 | Note |
|---|---|---|
| TLS + LB + network | 8 ms | keep the hop count low; terminate TLS at the LB |
| tokenize (CPU) | 3 ms | fast tokenizer, off the GPU thread |
| scheduler queue wait | 20 ms | this is max_queue_delay_microseconds; it is pure added latency |
| prefill, 512 tok | 25 ms | compute-bound, scales with prompt length |
| first decode step | 12 ms | memory-bandwidth-bound on weights |
| detokenize + stream out | 2 ms | stream tokens, don't buffer the whole response |
The two knobs that dominate are queue wait and prefill. Everything else is noise until those are under control.
Configuration that encodes the decision:
protobuf
config.pbtxt: Triton 24.x, tensorrt_llm backend
max_batch_size: 0 # in-flight batching manages the batch instance_group [{ count: 2, kind: KIND_GPU }] parameters: { key: "max_tokens_in_paged_kv_cache" value: { string_value: "32768" } key: "batch_scheduler_policy" value: { string_value: "GUARANTEED_NO_EVICT" } } dynamic_batching { max_queue_delay_microseconds: 2000 }
GUARANTEED_NO_EVICT matters: the alternative, MAX_UTILIZATION, packs the KV cache harder and gets more throughput, but it preempts and recomputes sequences under memory pressure, and recompute shows up directly as an ITL spike on a request that was already streaming. For a latency SLO I'd rather queue than preempt.
Trace of one request: t=0 arrives at Triton; t=2 ms tokenized; t=4 ms admitted to the running batch at the next scheduler iteration; t=29 ms prefill done; t=41 ms first token streamed to the client; subsequent tokens every ~12 ms. TTFT 41 ms, well inside budget: the headroom is what absorbs the p99 tail when the batch is full.
Failure modes I'd instrument for:
- Batch-formation latency.
max_queue_delay_microsecondsis a direct addend to TTFT. Watchnv_inference_queue_duration_uson Triton's Prometheus endpoint (port 8002); if it tracks the configured delay, the GPU is idle and you're paying for throughput you don't need. - KV cache exhaustion. Cache utilisation pinned at 100% means the next long prompt either queues or evicts. Cap
max_tokens_in_paged_kv_cache, alert at 85%, and shed load with 429 rather than letting ITL degrade for everyone. - Cold start. A TensorRT engine build takes minutes and the engine is not portable across SM versions or TensorRT releases. Build per (GPU SKU, TRT version, precision, optimisation profile) in CI, cache the artefact, and never build at pod start: a scale-out event during a traffic spike would take 10+ minutes to help.
- Optimisation-profile misses. TensorRT specialises on min/opt/max shapes. A prompt longer than
maxfalls back to a slower kernel or fails; set the profile from the real length distribution, not a guess. - Co-tenancy. Another process on the same GPU steals SM time and p99 doubles with no metric moving. MIG partitions or exclusive GPU assignment, and node affinity so inference pods never share a node with training.
Alternatives, and when they win. vLLM with PagedAttention and continuous batching gets most of this with far less build friction: it's the right call when the model changes weekly and you can't afford a TensorRT rebuild per iteration. TensorRT-LLM typically wins on absolute latency per GPU once the model is stable. For non-LLM workloads, Triton's dynamic batching over a plain TorchServe/FastAPI deployment buys you concurrent model execution, per-model metrics and multi-framework backends in one process. If the SLO is genuinely sub-10 ms, the answer stops being a serving framework and becomes a fused custom kernel plus CUDA graphs to kill launch overhead.
Curated: · Written: · Reviewed:
Q&A-8Why does mixed-precision training work without destroying model accuracy?
Reported in Ace Your NVIDIA ML Interview: Top 25 Questions and Expert Answers (2026 Version)
The 90-second answer
Mixed precision works because only the operations that can tolerate it run in low precision. On NVIDIA tensor cores, FP16 and BF16 multiply low-precision operands but accumulate every dot product in FP32: the rounding is in the inputs, not the sum. Everything precision-sensitive stays FP32: master weights, optimizer state, the loss, softmax, layer norm and cross-rank reductions. Two mechanisms close the remaining gaps: an FP32 master copy of the weights so a small update isn't rounded away, and loss scaling so FP16 gradients don't underflow to zero. With those in place, ResNet-50 and BERT train to the same accuracy as FP32 within seed noise, while the matmuls run 1.5–3× faster end to end on A100/H100.
Why the arithmetic survives
| Format | Bits (s/e/m) | Spacing at 1.0 | Range | Where it's used |
|---|---|---|---|---|
| FP32 | 1/8/23 | 1.19e-7 | ~1e±38 | master weights, accumulators, reductions |
| TF32 | 1/8/10 | 9.8e-4 | ~1e±38 | FP32 matmul/conv inputs on Ampere+ |
| BF16 | 1/8/7 | 7.8e-3 | ~1e±38 | activations, weights; no loss scaling |
| FP16 | 1/5/10 | 9.8e-4 | 6.1e-5 … 65504 | activations, weights; needs loss scaling |
The dot product is the thing that matters. A tensor-core MMA computes Σ aᵢbᵢ with FP32 accumulation, so the relative error of an N-term sum grows like √N·2⁻²⁴, not 2⁻¹¹. Each product is rounded once at low precision; the accumulation error stays at FP32.
Master weights: the update that would vanish
Take a weight w = 1.0, lr = 1e-3, gradient 1e-2. The update is 1e-5. In FP16 the spacing at 1.0 is 2⁻¹⁰ ≈ 9.77e-4, so w + 1e-5 rounds straight back to 1.0: the weight never moves. In FP32 the spacing is 1.19e-7, so the update survives. That is why AMP keeps an FP32 master copy and casts to FP16 only for the forward and backward pass.
Loss scaling: the gradient that would underflow
FP16's smallest normal is 6.10e-5 and subnormals bottom out at 5.96e-8. Late-layer gradients routinely sit at 1e-6 to 1e-8, so unscaled they flush to zero. Dynamic loss scaling multiplies the loss by S (start 2¹⁶ = 65536), which scales every gradient by S, then divides by S before the optimizer step. On an inf, halve S and skip the step; after 2000 clean steps, double it. BF16 has FP32's 8-bit exponent, so it needs no scaler: at the cost of three mantissa bits.
python scaler = torch.amp.GradScaler('cuda') for x, y in loader: opt.zero_grad(set_to_none=True) with torch.amp.autocast('cuda', dtype=torch.float16): loss = model(x).loss(y) scaler.scale(loss).backward() scaler.step(opt) # unscales, skips on inf/nan scaler.update()
Failure modes and how you see them
- Loss scale collapsing to 1 and staying there: real overflow, not a scaling bug. Check LR, init, or a layer whose activations blow up.
- Accuracy gap that only appears past ~10k steps: master weights missing, or an FP16 reduction (a custom all-reduce, a norm) that should be FP32.
- BF16 model that trains but plateaus slightly low: 8-bit mantissa is fine for large gradients, marginal for tiny ones. Keep master weights and FP32 optimizer state.
- TF32 is drop-in (no scaler, no master weights) but gives ~1.3× not 2×; it is the safe default when you cannot touch model code. Note it is opt-in for matmul in recent PyTorch (
torch.backends.cuda.matmul.allow_tf32 = True).
On H100, FP8 (E4M3 forward, E5M2 backward) with per-tensor scaling via Transformer Engine goes further, but it is not a drop-in: it needs per-layer scaling factors and calibration.
Detect all of these by logging the loss scale and skipped-step count, and by running a 200-step FP32 baseline against the mixed-precision curve on the same seed.
Curated: · Written: · Reviewed:
Q&A-9Your training job is only using 35% GPU utilization. How would you debug it?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
First pin down what the 35% is measuring. nvidia-smi's utilization is the fraction of the sample window in which at least one kernel was resident on the GPU: it says nothing about SM occupancy or tensor-core throughput. A job can read 100% there while doing almost no useful math, and it can read 35% because the GPU is genuinely starved. So I pull DCGM's profiler fields (DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE) plus an Nsight Systems timeline, and split the problem in two: gaps between kernels, or dense but inefficient kernels. Those have disjoint fixes, and guessing between them is how people spend a week tuning the wrong layer.
| Symptom in the timeline | Evidence | Usual cause | Fix |
|---|---|---|---|
| Long gaps, GPU idle | nsys shows H2D copies and Python on the critical path | DataLoader CPU-bound (JPEG decode, tokenisation), too few workers | num_workers up, pin_memory=True, prefetch_factor, DALI or GPUDirect Storage |
| Gaps every step, short kernels | Kernels of a few µs, launch overhead comparable | Small batch, unfused elementwise ops, Python loop over layers | Larger micro-batch, CUDA Graphs, torch.compile, fused optimizer (FusedAdam) |
| Gaps at step boundary | NCCL all-reduce not overlapping backward | DDP bucket too small, slow interconnect | Raise bucket_cap_mb (default 25), check NCCL_DEBUG=INFO, confirm NVLink not PCIe |
| Dense, low tensor pipe | DCGM_FI_PROF_PIPE_TENSOR_ACTIVE under ~20% | fp32 matmuls without TF32, dims not multiples of 8 | torch.set_float32_matmul_precision('high'), AMP bf16, pad dims |
| Dense, high SM but low IPC | ncu shows memory-bound stalls | LayerNorm/softmax/elementwise reading HBM repeatedly | Fuse, channels_last, keep activations in bf16 |
A worked trace. 8×A100, 1.3B-parameter model, micro-batch 16, seq len 1024. Step time 380 ms; nsys stats attributes 245 ms to gaps waiting on host-to-device copies and Python, 135 ms to kernels. 135/380 = 35.5%: that matches the reported number exactly, so this is a starvation problem, not a kernel problem. nvidia-smi dmon shows SM clocks dropping between steps. The DataLoader is decoding JPEGs on 4 workers; the GPU consumes a batch in 135 ms and waits 245 ms. Moving decode to DALI and raising workers to 12 drops the step to 190 ms, i.e. 71% utilization, with no model change at all.
If the timeline is dense instead. Then it's per-kernel. ncu --set full on the top three kernels by time tells you whether you're occupancy-limited (register pressure, block size), memory-bandwidth-limited, or issuing fp32 FMA where a tensor-core op would do. On Ampere and later, PyTorch leaves matmul TF32 off by default; a fp32 training loop with no AMP will show tensor pipe active near zero and you'll never fix it with batch size.
Things that masquerade as low utilization. A loss.item() or torch.cuda.synchronize() inside the step forces a sync and serialises the pipeline: grep for .item(), .cpu(), and per-step logging. Checkpointing every 100 steps to a slow filesystem stalls the whole job. And a validation loop that runs single-batch on the same GPU will drag the average down; measure training steps separately.
When not to chase it. Utilization is a means, not the target. If you're communication-bound at scale, the right number is step time and scaling efficiency, not SM busy-ness: a job at 60% utilization with better overlap beats one at 90% that's stalled on all-reduce. Report tokens/sec/GPU against the model FLOPs and the hardware's peak; that's the number an NVIDIA interviewer will ask for next.
Curated: · Written: · Reviewed:
Q&A-10Explain how gradient descent works.
Reported in NVIDIA Interview Questions for 2026: ML, Coding, System Design, and Behavioral Prep | ExtraBrain
The 90-second answer
Gradient descent is an iterative first-order optimizer. You have parameters θ and a differentiable loss L(θ; batch). Backprop computes g = ∇θL, the direction in which L increases fastest locally. You step the other way: θ ← θ − ηg, where η is the learning rate. Repeat over minibatches until validation loss stops improving.
On an NVIDIA training job the same loop has a GPU shape: each rank runs forward and backward kernels over its shard, NCCL all-reduces gradients so every rank has the same averaged gradient, then the optimizer updates a replicated parameter copy. With data parallelism, global batch = per-GPU batch × data-parallel world size; that product, not the per-GPU number, is what you scale η against.
SGD with momentum:
text v_t = β v_{t-1} + g_t θ_t = θ_{t-1} − η v_t
AdamW:
text m_t = β1 m_{t-1} + (1−β1) g_t v_t = β2 v_{t-1} + (1−β2) g_t² m̂ = m_t / (1−β1^t), v̂ = v_t / (1−β2^t) θ_t = θ_{t-1} − η ( m̂ / (√v̂ + ε) + λ θ_{t-1} )
| Method | What it adds | When it wins |
|---|---|---|
| SGD | raw gradient | vision CNNs with tuned schedules |
| SGD + momentum | velocity, damps oscillation | most convolutional training |
| AdamW | per-parameter adaptive step, decoupled weight decay | transformers, sparse gradients |
| LAMB/LARS | layerwise trust ratio | very large global batches |
A worked step. Take θ = 1.0, g = 0.5, η = 0.1, momentum β = 0.9, v = 0. First step: v = 0.05, θ = 0.995. Same gradient again: v = 0.095, θ = 0.9855. Momentum accumulates consistent directions; if the gradient flips sign, v decays by β each step instead of reversing instantly.
Batch scaling. Eight GPUs × 512 samples = 4096 global batch. Linear scaling from a 256-sample baseline with η = 0.1 gives η = 0.1 × 4096/256 = 1.6, with a 2k-step warmup. Linear scaling usually breaks past roughly 8k–16k; switch to LAMB/LARS or square-root scaling and re-tune.
python opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1) scaler = torch.amp.GradScaler('cuda') # fp16; bf16 does not need it for x, y in loader: opt.zero_grad(set_to_none=True) with torch.autocast('cuda', dtype=torch.float16): loss = criterion(model(x), y) scaler.scale(loss).backward() scaler.unscale_(opt) torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) scaler.step(opt) scaler.update()
Failure modes I watch for. η too high shows as loss spikes or NaN within a few hundred steps; detect with gradient-norm logging and skip the step when the norm exceeds a threshold. fp16 underflow produces zero gradients in small-magnitude layers; GradScaler's dynamic loss scale is the fix, and bf16 avoids it at the cost of mantissa bits. In data parallel, one slow rank stalls the NCCL all-reduce and every other rank waits; step-time variance across ranks is the signal. Async SGD with stale gradients converges but needs a smaller η as staleness grows. None of these are fixed by "more GPUs": they are optimizer and numerics decisions.
Curated: · Written: · Reviewed:
Q&A-11How do you handle overfitting?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
I first confirm it is overfitting and not an eval or data bug: train and validation loss must use identical preprocessing, the split must be group-aware (no frames from the same video on both sides), and the validation set must be large enough that a 0.02 loss difference is outside noise. If the gap is real, I spend fixes in this order: data, regularization, capacity, then architecture. Data is the only fix that raises the ceiling; the rest trade training fit for generalization.
On NVIDIA hardware the cheap wins are GPU-side augmentation and mixed precision. With DALI I can run RandAugment, mixup and cutmix without starving an A100/H100 of samples; BF16 removes loss-scaling failures that otherwise look like instability. For a 10k-image fine-tuning set, 8x augmentation gives an effective 80k samples per epoch and typically closes a 0.10 train-val gap to 0.03–0.05 without touching the model.
Then regularization. I use AdamW with decoupled weight decay 0.01–0.05, label smoothing 0.1 for classification, and dropout only where the architecture expects it (0.1 in transformer MLP blocks; 0.05 for LoRA fine-tuning). I avoid stacking all of them: dropout + label smoothing + strong weight decay can underfit a small model. Early stopping on validation loss with patience 3 epochs and min_delta=1e-3, restoring the best checkpoint.
| Intervention | Typical setting | Cost | When it wins |
|---|---|---|---|
| GPU augmentation (DALI) | RandAugment, mixup α=0.2 | ~5% step time | Vision/ASR fine-tuning |
| Weight decay (AdamW) | 0.01–0.05 | free | Always as baseline |
| Label smoothing | 0.1 | free | Classification with noisy labels |
| Dropout | 0.1 MLP, 0.05 LoRA | free | Small data, large head |
| Early stopping | patience 3, min_delta 1e-3 | free | Any run with a val curve |
| Capacity cut / distillation | smaller student | retrain | When regularizers underfit |
python
PyTorch 2.3, checked on A100 BF16
opt = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01) loss_fn = nn.CrossEntropyLoss(label_smoothing=0.1) best, patience, wait = float('inf'), 3, 0 for epoch in range(50): model.train() for x, y in train_loader: # DALI iterator opt.zero_grad(set_to_none=True) loss = loss_fn(model(x), y) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step() val = evaluate(model, val_loader) if val < best - 1e-3: best, wait = val, 0 torch.save(model.state_dict(), 'best.pt') else: wait += 1 if wait >= patience: break model.load_state_dict(torch.load('best.pt'))
A real trace from a 10k-image fine-tune: epoch 1 train 0.42 / val 0.48; epoch 5 train 0.18 / val 0.31; epoch 9 train 0.09 / val 0.29; epoch 12 train 0.06 / val 0.33. Early stopping fires at epoch 12 and restores epoch 9. Without it the 0.27 gap at epoch 12 is pure memorization.
Failure modes I watch for: early stopping on a noisy 500-sample val set stops too early. Use a larger val split or smooth the metric. Label smoothing hurts calibration if you later threshold probabilities; check ECE. Aggressive mixup can create label noise on fine-grained classes. Dropout left on at inference inflates val loss and looks like overfitting. Detect all of these by plotting train/val curves per epoch and checking gradient norms; a sudden val divergence with stable train loss is the signature.
Curated: · Written: · Reviewed:
Q&A-12How do precision and recall trade off?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
Precision is TP/(TP+FP): of everything I flagged, how much was real. Recall is TP/(TP+FN): of everything real, how much I caught. They trade off because both are functions of one knob: the score threshold I apply to the model's output. Slide the threshold down and I catch more positives (recall up) but admit more negatives (precision down). Slide it up and the reverse. The full set of operating points is the precision-recall curve, and the deployment question is never "which is better": it's "which point on that curve does this product's cost structure want."
The trade-off, with numbers. A detector evaluated on 10,000 images containing 2,000 ground-truth objects:
| Score threshold | TP | FP | FN | Precision | Recall |
|---|---|---|---|---|---|
| 0.80 | 1200 | 120 | 800 | 0.909 | 0.60 |
| 0.50 | 1600 | 400 | 400 | 0.800 | 0.80 |
| 0.20 | 1840 | 1400 | 160 | 0.568 | 0.92 |
Same weights, same images. Only the cut moved. That is the whole mechanism: positives and negatives have overlapping score distributions, and the threshold decides which side of the overlap you live on.
Why the curve is the honest artefact. A single number like AP (COCO's mAP averages interpolated precision over recall, at IoU 0.50:0.95) is threshold-free and good for ranking architectures. It is not a deployment decision. Two models can share AP = 0.42 and behave completely differently at the recall you actually need. So I report precision at a fixed recall: e.g. "P = 0.71 at R = 0.90", because that is the constraint the product imposes. F1 weights precision and recall equally, which is almost never the real cost ratio; I use it only when I have no cost model.
Choosing the point. I write down the cost matrix. On an industrial inspection line, a missed defect (FN) might scrap a $4,000 part downstream while a false alarm (FP) costs a 30-second human review. That ratio pushes the threshold down hard, and I'd accept precision near 0.3 to hold recall at 0.98. For a content-moderation flag that triggers account action, the ratio inverts and I hold precision high. The threshold is a business parameter that happens to live in the model config.
The failure mode that bites hardest: prevalence shift. Precision depends on the base rate; recall does not. Take a classifier with TPR = 0.90 and FPR = 0.01. At 50% prevalence, precision is 0.45/(0.45+0.005) ≈ 0.99. At 1% prevalence (100 positives and 9,900 negatives per 10k) you get 90 TP against 99 FP, so precision is 90/189 ≈ 0.48. Nothing about the model changed; the world did. A threshold tuned on a balanced validation set will look fine in the lab and drown reviewers in false positives in production. I always re-tune on a set whose prevalence matches deployment, and I monitor the positive rate as a leading indicator.
NVIDIA-specific gotcha. The score distribution is not stable across the stack. Running the same network through TensorRT in INT8 shifts scores relative to FP32, and a score_threshold in a DeepStream nvinfer config or a post-NMS cut tuned on the FP32 PyTorch export does not transfer. Quantize first, then re-sweep the threshold on the quantized engine against the deployment-representative set, and store the chosen operating point alongside the engine hash so it can't silently drift.
Curated: · Written: · Reviewed:
Q&A-13What causes vanishing or exploding gradients?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
A gradient is a product of Jacobians, one per layer or timestep: ∂L/∂h₀ = ∏ᵢ ∂hᵢ/∂hᵢ₋₁. Vanishing and exploding gradients are the same failure, that product's norm collapsing or blowing up, and the causes are whatever pushes the per-layer gain away from 1. The usual suspects: saturating activations, initialization that doesn't preserve variance, depth or long recurrence, missing normalization or residual paths, and learning rates or weight magnitudes large enough to make the recurrence unstable. In FP16 training there is a second, numerical cause: the exponent range, not the calculus.
If each layer's Jacobian has spectral norm g, the gradient norm scales as g^L. For a 50-layer stack, 0.9⁵⁰ ≈ 0.005 and 1.1⁵⁰ ≈ 117; at g = 1.2, 1.2⁵⁰ ≈ 9,100. A sigmoid's derivative is at most 0.25, so ten sigmoid layers with unit-ish weights can already shrink a gradient by 0.25¹⁰ ≈ 9.5e-7 before the loss ever reaches the first layer.
python
PyTorch 2.x: per-layer gradient norm, the first diagnostic to run
import torch, torch.nn as nn
model = nn.Sequential( nn.Linear(512, 512), nn.Sigmoid(), *[nn.Sequential(nn.Linear(512, 512), nn.Sigmoid()) for _ in range(48)], nn.Linear(512, 10), ) x = torch.randn(64, 512) model(x).sum().backward()
for name, p in model.named_parameters(): if p.grad is not None and p.ndim == 2: print(name, p.grad.norm().item())
Run that on step 0 and every few hundred steps. A monotonic decay from the last layer to the first is vanishing; a spike or inf is exploding. torch.autograd.set_detect_anomaly(True) finds the first NaN-producing op, at a large speed cost.
| Cause | Signature | Fix |
|---|---|---|
| Saturating activations | activations pinned near 0/1; grad norm shrinks with depth | ReLU/GELU, He init, normalization |
| Bad initialization | first-step gradient already tiny or huge | Xavier for tanh, He (2/fan_in) for ReLU, orthogonal for RNNs |
| Depth or long recurrence | exponential decay/growth with layers or timesteps | residual connections, LayerNorm/RMSNorm, GRU/LSTM, truncated BPTT |
| High LR or large weights | grad norm spikes, loss diverges | warmup, lower LR, global-norm clipping at 1.0 |
| FP16 overflow/underflow | inf/NaN or zero grads under AMP | dynamic loss scaling, or BF16 |
On NVIDIA hardware, FP16's max is 65,504 and its smallest normal is about 6.1e-5; a gradient that is mathematically fine can overflow or flush to zero. Dynamic loss scaling multiplies the loss by S, unscales gradients before the optimizer, and halves S on an inf/NaN step. BF16 on A100/H100 has FP32's 8-bit exponent (range ~3.4e38) but only a 7-bit mantissa, so it trades overflow risk for precision. TF32 is similar: 8-bit exponent, 10-bit mantissa.
Residual and normalization layers are the structural fixes because they give the gradient an identity path and keep activation variance stable across depth. Clipping is a guardrail, not a cure: if you are clipping every step, the underlying gain is still wrong. Check initialization, activation statistics and per-layer gradient norms before changing the architecture.
Curated: · Written: · Reviewed:
Q&A-14How would you diagnose a model with high offline accuracy but poor online performance?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
A gap between offline accuracy and online performance almost never means the model is wrong. It means the thing you measured offline is not the thing you are serving. I'd resist retraining until I know which of three buckets the gap lives in: the offline metric doesn't match the online objective, the served input distribution differs from the eval distribution, or the serving path changes the model's effective behaviour. The fastest way to split them is to replay a sample of live traffic through the exact offline eval pipeline. If the gap reproduces on replayed traffic, it's data or labels. If it vanishes, it's the serving path.
| Bucket | Typical symptom | First check |
|---|---|---|
| Metric mismatch | 94% accuracy on a balanced eval set, online CTR flat | base rate, class balance, precision@k vs accuracy |
| Train/serve skew | one feature's PSI > 0.2 while others are stable | log raw feature values at serving, diff against the batch job |
| Label delay | offline labels mature, online outcomes resolve in 7 days | retrain on a delayed-label snapshot and re-evaluate |
| Serving path | TensorRT INT8 engine, dynamic batching, sequence truncation | run the engine itself over the offline eval set |
| Traffic mix | new device or geography cohort | slice online metrics by segment |
A worked trace. A ranking model showed offline AUC 0.912 on a held-out set, but online CTR dropped 8% after launch. I replayed 2M live requests through the offline feature pipeline and scored them with the same checkpoint: AUC fell to 0.874. So the gap was in the features, not the model. Slicing by feature, item_ctr_7d had PSI 0.41 against training. The batch job computed it over UTC calendar days; the streaming job computed a trailing 168-hour window including a partial current day. Same name, different definition. We moved both to one feature-store definition, backfilled 30 days, and replay AUC recovered to 0.908; online CTR came back +6.2% versus the pre-launch baseline.
The serving path is a separate audit. On NVIDIA hardware the model that answers requests is rarely the PyTorch checkpoint you evaluated. It's a TensorRT engine, possibly INT8, served by Triton with dynamic batching. Each of those can move accuracy without touching the weights. A calibration set of 500 images from one camera gave an INT8 engine that dropped top-1 from 78.2% to 74.9% on the same eval set: an offline-to-offline regression that replay would catch immediately. polygraphy comparing per-layer outputs between the ONNX graph and the engine localises it to the first conv block. Dynamic batching can also change results if padding or max_seq_len truncation differs from training; check that the tokenizer and DALI preprocessing produce byte-identical tensors to the offline path.
Failure modes I watch for. Aggregate metrics hide segment regressions, so slice by cohort before concluding anything. Offline eval leakage (future-dated features, duplicate rows across splits) inflates the baseline and makes a healthy model look broken online. And retraining on online logs before you know the cause bakes the skew into the next checkpoint. Log model_version, raw feature vectors and the preprocessed tensor hash at serving; without those, replay is guesswork.
What I'd say in the room. Instrument first, replay second, retrain last. The answer the interviewer wants is that offline accuracy is a claim about a dataset, and production is a different dataset with a different pipeline attached.
Curated: · Written: · Reviewed:
Q&A-15How would you detect data leakage in a training pipeline?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
Leakage is any feature whose training-time value depends on information you won't have at scoring time. I don't find it by eyeballing columns: I make the split an enforced contract, then run two detectors that catch what the contract misses.
Three invariants asserted in CI on every dataset build:
- No entity in both splits: group by customer/document/session id, not by row.
- For temporal data,
max(train.ts) < min(test.ts). A random split on a time series is leakage by construction. - Every transformer is fit inside the fold: scaler, imputer, target encoder, feature selector.
sklearn.pipeline.Pipelinewithcross_val_scoregives that for free; a manualfit_transformon the full frame does not.
Then adversarial validation: label train=0, test=1, fit a gradient-boosted classifier with CV, read the AUC. An iid split gives ~0.50. Anything past ~0.70 means a model can tell the splits apart, and permutation importance on that classifier names the column responsible.
| Leakage | Symptom | Detector |
|---|---|---|
| Target leakage: feature computed from the label or post-outcome events | one feature has single-feature AUC 0.9+; model collapses online | ablation: shuffle or drop it, retrain |
| Duplicate / near-duplicate rows across splits | test metric 5–10 pts above CV | exact row hashing; MinHash or embedding cosine > 0.98 |
| Temporal | offline great, online bad | timestamp assertion, time-based split |
| Preprocessing fit on all data | CV optimistic, gap grows with fold count | assert fit inside fold; diff against a deliberately leaky baseline |
| Group / entity | same user in train and test | set intersection on the group key |
| Multi-GPU shard overlap | eval metric differs per rank | assert rank shards disjoint; DistributedSampler(drop_last=False) pads by repeating samples |
python import numpy as np, pandas as pd from sklearn.ensemble import HistGradientBoostingClassifier from sklearn.model_selection import cross_val_score
def assert_disjoint(train, test, key): dup = set(train[key]) & set(test[key]) assert not dup, f"{len(dup)} {key} values in both splits"
def adversarial_auc(train_X, test_X, seed=0): X = pd.concat([train_X, test_X], ignore_index=True) y = np.r_[np.zeros(len(train_X)), np.ones(len(test_X))] clf = HistGradientBoostingClassifier(max_iter=200, random_state=seed) return cross_val_score(clf, X, y, cv=5, scoring="roc_auc").mean()
A trace. Churn model, 4.2M rows, DALI readers over 512 TFRecord shards. adversarial_auc returns 0.51: splits look iid, no distribution leak. But offline AUC is 0.94 against 0.71 in the A/B. Permutation importance on the production model: days_since_last_ticket alone scores 0.89 single-feature AUC. It's joined from a support table that includes tickets opened after the churn label. Drop it, retrain: 0.94 → 0.76, which matches the A/B. That's the signature: a metric beating the online number by 15+ points with one feature carrying most of it.
Failure modes. Adversarial validation also fires on benign covariate shift (new region, seasonality): it says "these differ", not "this leaks", so confirm with ablation. Permutation importance splits credit across correlated columns, so drop the whole correlated block, not one column. And a leak present equally in both splits, a label-derived feature everywhere, never shows in adversarial validation at all; only ablation or reading the join catches it.
Curated: · Written: · Reviewed:
Q&A-16How would you reduce inference latency for a transformer model?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
I'd split the question before changing anything: prefill (time to first token) is compute-bound on attention and MLP matmuls; decode (inter-token latency) is memory-bandwidth-bound because every step re-reads all weights and the KV cache. On an H100 SXM (3.35 TB/s HBM3), a 7B model in FP16 has a hard decode floor of 14 GB / 3.35 TB/s ≈ 4.2 ms per step at batch 1. That arithmetic sets the order of attack: cut bytes moved, then cut work, then cut launch overhead.
So the short list is FP8 or INT4 weight-only quantization, FlashAttention and fused kernels, paged KV cache with GQA, CUDA graphs, and speculative decoding. In TensorRT-LLM these are build-time and runtime options, not model rewrites. I'd treat the serving platform as given and change the engine.
| Lever | What it changes | Ballpark on 7B, H100, batch 1 | Cost / risk |
|---|---|---|---|
| FP8 weights + activations | halves bytes read per decode step | ~1.6–1.9x decode speedup | calibration set must match traffic; outlier layers need scaling |
| INT4 weight-only (AWQ/GPTQ) | quarter the weight bytes | ~2.5–3x vs FP16 | accuracy drop shows up on reasoning and long-tail tasks |
| FlashAttention-2/3 | IO-aware attention, fewer HBM round trips | prefill 1.5–2x at 4k context | drop-in, but check head-dim support |
| CUDA graphs | removes per-kernel launch overhead | 10–20% at small batch | static shapes; bucket dynamic shapes |
| Paged KV + in-flight batching | less fragmentation, higher throughput | 2–4x throughput | TPOT p99 can rise; isolate latency-critical traffic |
| Speculative decoding | fewer target forward passes | 1.8–2.5x at high acceptance | draft model memory; acceptance collapses out of domain |
Where the time actually goes. For Llama-2 7B, the KV cache is 2 × 32 layers × 32 heads × 128 head_dim × 2 bytes = 512 KiB per token. At 4k context that is 2 GiB per sequence. GQA with 8 KV heads cuts it 4x; FP8 KV halves it again. A decode-step trace at batch 1, 4k context looks like this:
| Component | FP16 | FP8 weights + CUDA graphs |
|---|---|---|
| weight read | 4.2 ms | 2.1 ms |
| KV read | 0.6 ms | 0.3 ms |
| attention + MLP compute | 0.4 ms | 0.4 ms |
| kernel launch (~20 kernels) | 0.2 ms | 0.02 ms |
| total | ~5.4 ms | ~2.8 ms |
Failure modes I'd watch. FP8 overflow in attention logits or layernorm outputs: check max-abs per layer against the calibration range, not just average error. INT4 weight-only can pass perplexity and still lose 2–3 points on GSM8K; run lm-eval-harness before and after and set a rejection threshold. CUDA graph capture fails on dynamic control flow or variable sequence lengths; bucket shapes and capture per bucket. Speculative decoding is a loss when acceptance α drops below roughly 0.6: measure acceptance per workload, because code and chat differ. Paged KV fragmentation under mixed lengths shows up as allocator stalls; monitor fragmentation ratio and block reuse. In-flight batching raises throughput but can push TPOT p99 up, so keep a separate pool or priority for latency-critical requests.
I'd verify with Nsight Systems for launch gaps and Nsight Compute for memory-throughput ceilings, then TensorRT-LLM's benchmark path for end-to-end TTFT and TPOT at the target batch and context.
Curated: · Written: · Reviewed:
Q&A-17Why might distributed training not scale linearly?
Reported in Nvidia Machine Learning Engineer Interview Questions - MockRound
The 90-second answer
Linear scaling means doubling the GPU count halves step time. It stops for two independent reasons, and a strong answer gives both.
First, communication does not shrink when you add GPUs: compute per GPU does. A data-parallel step ends with an all-reduce of every gradient. In a ring that moves 2(N−1)/N × S bytes per GPU, which converges to 2S as N grows: the per-GPU cost is essentially flat in world size. Per-GPU compute falls as 1/N. So the comm-to-compute ratio grows linearly with N and eventually eats the gain.
Second, even with free communication, a larger global batch buys less loss reduction per sample past the critical batch size. Throughput keeps scaling; time-to-target does not.
The arithmetic
10B-parameter model, BF16 gradients, S = 20 GB. Ring all-reduce per GPU ≈ 2S = 40 GB. One NDR 400 Gb/s port per GPU (50 GB/s) in a rail-optimized fabric gives ~0.8 s of transfer; roughly 70–80% overlaps with the backward pass, so ~0.2 s is exposed.
Compute per step is 6ND. At D = 2M tokens that is 1.2e17 FLOPs; at a realistic 400 TFLOPS/GPU (≈40% MFU against H100 SXM's 989 TFLOPS BF16 dense), per-GPU compute is 300/N seconds.
| GPUs | compute/GPU | exposed all-reduce | scaling efficiency |
|---|---|---|---|
| 64 | 4.7 s | 0.2 s | 96% |
| 512 | 0.59 s | 0.2 s | 75% |
| 2048 | 0.15 s | 0.2 s | 43% |
Those figures are illustrative for that configuration, but the shape is the point: the wall sits at the N where per-GPU compute meets per-GPU communication. Growing the global batch with N holds per-GPU compute constant and pushes the wall out, until critical batch size.
Where the "it's just an all-reduce" model breaks
- ZeRO-3/FSDP is not free. It all-gathers parameters before each forward and each backward and reduce-scatters gradients, so per-step traffic is ~3× model size instead of 1×. It buys memory, not bandwidth.
- Tensor parallelism issues two all-reduces per transformer layer, each latency-bound at tens of microseconds. Fine on NVLink/NVSwitch (900 GB/s per H100, 3.6 TB/s all-to-all inside a DGX H100); across InfiniBand it is 5–10× slower and rarely worth it. Keep TP inside the node.
- Pipeline parallelism has a bubble: with
pstages andmmicrobatches the idle fraction is(p−1)/(m+p−1). 8 stages, 32 microbatches → 18% of step time doing nothing.
Failure modes I'd actually look for
- NCCL silently falling back to TCP.
NCCL_DEBUG=INFOprintsNET/IBon a healthy fabric;NET/Socketmeans a down port, a subnet-manager mismatch, or a container missing the IB device, and throughput drops 5–10×. - One straggler rank. Log per-rank step time and compare max to median; a single GPU on a degraded link sets the step time for everyone.
- Checkpoint stalls. 10B params plus Adam moments is ~140 GB; at 100 GB/s that is 1.4 s where every GPU idles. Shard or overlap the write.
- Input pipeline. If the dataloader can't keep up, more GPUs just wait faster.
What I'd do
Bucket gradients so the all-reduce overlaps the backward pass; use SHARP in-network reduction on Quantum-2 to halve all-reduce traffic; keep TP within NVLink domains and put data parallelism across nodes; and size the global batch to the critical batch size rather than to the GPU count.
Curated: · Written: · Reviewed:
Q&A-18Compare tensor and pipeline parallelism and say when you would choose each?
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
Tensor parallelism splits a single layer's matmuls across ranks: each rank holds a shard of every weight matrix and the partial sums are combined with an all-reduce. Pipeline parallelism assigns whole layers to stages on different ranks and passes only activations across stage boundaries. The choice is a bandwidth question. TP issues a collective inside every transformer block, forward and backward, so it only pays off where the interconnect is NVLink-class. PP issues a handful of point-to-point sends per step, so it tolerates InfiniBand between nodes. On an H100 cluster my default is TP inside the DGX node (8 GPUs, 900 GB/s per GPU over NVLink 4 through NVSwitch) PP across nodes, and data parallelism on whatever is left.
| Tensor parallel | Pipeline parallel | |
|---|---|---|
| Split | inside a layer: weights and activations | between layers: stages |
| Comm per transformer block | 2 all-reduces forward, 2 backward (Megatron column/row parallel) | 1 send/recv per stage boundary |
| Pattern | collective, latency-sensitive | point-to-point, bandwidth-light |
| Interconnect | NVLink/NVSwitch only | NVLink or IB |
| Constraint | hidden size and head count divisible by TP degree | bubble size, stage balance |
| Memory | weights ÷ TP | weights ÷ PP, plus in-flight micro-batch activations |
| Failure signature | all-reduce time dominates the step | GPUs idle in the bubble; one slow stage stalls all |
Sizing it. A 70B model in bf16 is 140 GB of weights; gradients another 140 GB; Adam's fp32 m and v 560 GB. That is ~840 GB of state, and eight H100 80GB cards give 640 GB: TP=8 alone does not fit. TP=8 × PP=2 on 16 GPUs is 52.5 GB per GPU. At 405B the weights alone are 810 GB; TP=8 × PP=8 on 64 GPUs leaves 12.6 GB of weights per GPU, with the rest of the budget going to optimizer state and activations. TP stays inside the node in both cases; PP is what crosses the fabric.
The bubble. With P stages and M microbatches under the 1F1B schedule, the idle fraction is (P−1)/(M+P−1). P=8, M=32 gives 7/39 ≈ 18% of step time idle. M=64 gives 10%, paid for by holding twice as many micro-batches' activations. That is the PP tuning knob, and it is why PP is cheap in bandwidth but expensive in memory and schedule complexity.
Failure modes I have actually hit. TP across nodes: NCCL all-reduce bus bandwidth drops from roughly 350 GB/s on NVLink to ~40 GB/s on a 400 Gb/s IB port, and the collective goes from ~10% of step time to dominant. Read it from NCCL_DEBUG=INFO timings or the NCCL row in Nsight Systems. TP divisibility: 32 attention heads cannot be split 64 ways without replicating KV heads. PP stage imbalance: the first stage carries the embedding and the last carries the LM head and loss, so the last stage is usually the straggler. Measure per-stage step time, not just total. And a PP group deadlocks if one stage OOMs, because the others block in send/recv.
When neither. If the model fits with sharded optimizer state, FSDP/ZeRO-3 is simpler: no model rewrite, no bubble. TP and PP are for when a layer or the whole model does not fit. One thing moving the boundary: GB200 NVL72 puts 72 GPUs in one NVLink domain at 1.8 TB/s per GPU, which makes TP=16 or 32 across what used to be node boundaries viable.
Curated: · Written: · Reviewed:
Q&A-19After pruning channels, how do you represent the graph and choose a kernel at runtime?
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
Two decisions, and the second one is mostly made at build time, not at runtime: worth saying out loud, because the framing of the question invites you to describe a runtime dispatcher that usually doesn't exist.
Representation: physically compact the weights. A mask over the original tensor only buys speed if the hardware has a kernel that skips zeros, and on NVIDIA that means 2:4 structured sparsity on Ampere and later. Arbitrary channel pruning gets no such kernel. So you slice the weight tensor, rewrite the layer's output-channel count, and propagate the new shape through the graph. Keep the pruning plan, the surviving channel indices per layer, as graph metadata, because fine-tuning and checkpoint reload both need to map back to the dense layout.
python
torch 2.3: prune conv1's output channels, propagate to conv2's input
keep = w1.abs().sum(dim=(2, 3)).topk(154).indices.sort().values w1 = w1[keep] # (154, 256, 3, 3) w2 = w2[:, keep] # (512, 154, 3, 3) same indices, input side plan["conv1.out"] = keep.to(torch.int32)
The hard part is re-legalizing the graph. A residual add or a concat forces channel alignment: if conv1 now emits 154 channels, the identity branch must either be pruned with the same index set, which cascades the constraint back to whatever produced it, or you insert a gather. I keep pruned channels in dependency groups and prune a group atomically; the alternative is a scatter/gather kernel per block that eats the savings.
| Representation | Saves memory | Saves time | When |
|---|---|---|---|
| Mask, dense shape | no | no | never, on NVIDIA |
| Compacted, new shapes | yes | yes | default for channel pruning |
| 2:4 along K + metadata | yes | ~2× on that GEMM | pattern can be constrained to 2:4 |
Kernel choice
TensorRT's builder enumerates candidate tactics per layer (cuDNN engines, cuBLAS, its own kernels, tensor-core vs SIMT variants) times each on the actual device (setMinTimingIterations / setAvgTimingIterations), and serializes the winner into the plan. trtexec --verbose prints every timed tactic, so you can see the winner and the runner-up rather than guessing. cuDNN's own frontend exposes the same trade: heuristic mode A is fast and workspace-free, B allows workspace and is usually faster, F is exhaustive search.
What decides the winner is shape, dtype, layout and alignment. Tensor-core MMA shapes are m16n8k16 for fp16, k8 for tf32, k32 for int8, so the reduction dimension wants to be a multiple of 16. Prune a 256→154 channel conv and you have 154 = 9×16 + 10: the last tile runs at 10/16 utilization, and worse, the heuristic may drop to a non-tensor-core engine for that shape. Rounding 154 → 160 costs 3.9% of the MACs and keeps you on the fast path. MACs themselves fall to 0.6 × 0.6 = 36% of dense when both dims are pruned 40%, but measured latency rarely follows that closely.
Failure modes
- Pruned model slower than dense. Odd K fell off the tensor-core path, or at batch 1 the layer is launch-bound: a 4 µs kernel behind 3–5 µs of launch overhead. Check with Nsight Systems for gaps and Nsight Compute for tensor-pipe utilization.
- Layout flip. TensorRT prefers NHWC on Ampere+; a channel count that was aligned in NCHW can force a transpose kernel that cancels the win.
- Dynamic shapes. Each optimization profile bucket gets its own tactic. Measure min, opt and max: the pruned shape may only be fast at opt.
- Stale BN statistics after slicing. Recalibrate before you judge accuracy, or you'll blame the pruning for a normalization bug.
Curated: · Written: · Reviewed:
Q&A-20Walk me through transposing a large matrix, then explain row-major versus column-major access, tiling and prefetching for matrices larger than cache.
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
A transpose does zero arithmetic: it is a pure data movement problem, so the only thing that matters is how many bytes cross the memory bus. In C, A[i][j] is contiguous in j. The transposed write B[j][i] for fixed i strides by N*4 bytes; once that stride exceeds a 64-byte cache line, every store touches a fresh line, uses 4 of its 64 bytes, and with write-allocate the line is pulled in before it is written. For 4096² float32 (67 MB, past L2 and usually past L3) that is ~2.2 GB of traffic to move 134 MB of useful data. Tile it so both sides are sequential within a cache-resident block and you get back to ~134 MB.
c // C11, gcc 13, -O3 -march=native. A, B row-major, N*N float. #define T 32 void transpose_tiled(const float *restrict A, float *restrict B, int N) { for (int ii = 0; ii < N; ii += T) for (int jj = 0; jj < N; jj += T) for (int j = jj; j < jj + T && j < N; j++) for (int i = ii; i < ii + T && i < N; i++) B[(size_t)j * N + i] = A[(size_t)i * N + j]; }
The inner loop runs over i, so the B store is contiguous and the A load is strided, but that stride stays inside a 32×32×4 B = 4 KB tile, which sits in L1 (32–48 KB) after the first touch. You cannot make both sides contiguous; you make the strided one cheap by keeping it in cache.
| Version | Read | Write + RFO | Total |
|---|---|---|---|
naive B[j*N+i] = A[i*N+j] | 67 MB | 1.07 GB + 1.07 GB | ~2.2 GB |
| tiled, write-allocate | 67 MB | 67 MB + 67 MB | ~201 MB |
| tiled + non-temporal stores | 67 MB | 67 MB | ~134 MB |
Illustrative single-core at ~10 GB/s effective: ~220 ms naive versus ~13 ms tiled. The 16× is traffic, not instructions.
On GPU the same argument, one level down. Stage a 32×32 tile in shared memory, sync, write transposed with block coordinates swapped:
cuda global void transpose(const float A, float B, int N) { // CUDA 12.x, sm_80+ shared float tile[32][33]; // +1 pad breaks the 32-way bank conflict int x = blockIdx.x32 + threadIdx.x, y = blockIdx.y * 32 + threadIdx.y; if (x < N && y < N) tile[threadIdx.y][threadIdx.x] = A[yN + x]; __syncthreads(); x = blockIdx.y32 + threadIdx.x; y = blockIdx.x32 + threadIdx.y; if (x < N && y < N) B[y*N + x] = tile[threadIdx.x][threadIdx.y]; }
Without the +1, tile[threadIdx.x][threadIdx.y] walks a column of a 32-wide array: stride 32 floats over 32 banks means all 32 threads hit one bank, a 32-way conflict. Confirm with ncu --metrics l1tex__data_bank_conflicts_pipe_lsu_mem_shared.
Prefetching. The hardware prefetcher already streams the contiguous side; software prefetch (__builtin_prefetch(&A[(i+T)*N + j], 0, 1)) buys little there and only helps the strided side across tile boundaries. The two changes that actually move the number are non-temporal stores (_mm256_stream_ps, killing the RFO half of write traffic) and huge pages: 67 MB is 16k 4 KB pages, and tile-strided access across them thrashes the TLB, so madvise(MADV_HUGEPAGE) is worth measuring. On GPU, prefetching is cp.async on Ampere+ staging global→shared without burning registers, plus enough resident blocks to keep HBM saturated.
Failure modes and when not to. Partial tiles at odd N cost divergence: handle the remainder in a scalar epilogue. Non-square matrices need the tile shape chosen per dimension. And check you need the transpose at all: if the consumer is a GEMM, pass CUBLAS_OP_T and never materialise it; if it is a layout change, a blocked/tiled format makes the transpose a metadata swap. In PyTorch, x.t() is a view: .contiguous() is the call that pays this bill, so profile before you write it.
Curated: · Written: · Reviewed:
Q&A-21Design a graph supporting node insertion, edge configuration and validation through cycle detection and dependency ordering.
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
Model it as a DAG of ops where the invariant is that a valid topological order exists at all times. Keep that order cached and treat every edge insertion as a candidate violation of it: if the new edge already agrees with the cached order, accept it in O(1); if it disagrees, re-run Kahn's algorithm and roll the edge back when it fails. Node insertion is O(1): a fresh node has no edges, so it appends to the end of the order.
python from collections import deque
class Graph: def init(self): self.nodes, self.out, self.inn = {}, {}, {} self.order, self.pos = [], {}
def add_node(self, nid, **cfg):
if nid in self.nodes:
raise KeyError(f"duplicate node {nid}")
self.nodes[nid] = cfg
self.out[nid], self.inn[nid] = {}, set()
self.pos[nid] = len(self.order)
self.order.append(nid)
def add_edge(self, src, dst, **cfg):
if src not in self.nodes or dst not in self.nodes:
raise KeyError(f"unknown endpoint {src}->{dst}")
if src == dst:
raise ValueError(f"self-loop at {src}")
if dst in self.out[src]:
raise ValueError(f"duplicate edge {src}->{dst}")
self.out[src][dst] = cfg
self.inn[dst].add(src)
if self.pos[src] < self.pos[dst]: # order already consistent
return
try:
self._toposort()
except ValueError:
del self.out[src][dst]
self.inn[dst].discard(src)
raise
def _toposort(self):
indeg = {n: len(self.inn[n]) for n in self.nodes}
q = deque(n for n, d in indeg.items() if d == 0)
order = []
while q:
n = q.popleft()
order.append(n)
for m in self.out[n]:
indeg[m] -= 1
if indeg[m] == 0:
q.append(m)
if len(order) != len(self.nodes):
raise ValueError("cycle: " + " -> ".join(self._cycle()))
self.order = order
self.pos = {n: i for i, n in enumerate(order)}
def _cycle(self): # DFS, color 1 = on current path
color, path = {n: 0 for n in self.nodes}, []
def dfs(n):
color[n] = 1
path.append(n)
for m in self.out[n]:
if color[m] == 1:
return path[path.index(m):] + [m]
if color[m] == 0:
hit = dfs(m)
if hit:
return hit
path.pop()
color[n] = 2
return None
for n in self.nodes:
if color[n] == 0:
hit = dfs(n)
if hit:
return hit
return []
Trace on a four-op block:
add_node load, conv, bn, relu add_edge load->conv, conv->bn, bn->relu # fast path each time order = [load, conv, bn, relu]
add_edge relu->conv pos[relu]=3, pos[conv]=1 -> slow path indeg: load 0, conv 2, bn 1, relu 1 Kahn pops load, conv drops to 1 and never reaches 0; queue empties with order = [load], so len(order)=1 != 4 ValueError: cycle: conv -> bn -> relu -> conv edge rolled back, order unchanged
Note which algorithm reports what: Kahn's drain count is the detector, anything left with nonzero indegree sits on or behind a cycle, while the DFS in _cycle is what produces the readable path, because Kahn has already destroyed the ordering information by the time it fails.
| Operation | Cost | Notes |
|---|---|---|
add_node | O(1) | appends to the cached order |
add_edge, order-consistent | O(1) | the common case in a builder |
add_edge, order-violating | O(V+E) | Kahn, plus rollback on failure |
| full validate | O(V+E) | run once before compile |
Why the fast path is sound. If pos[src] < pos[dst] in a valid order, no path dst → … → src can exist, because any such path forces pos[dst] < pos[src]. That single comparison is what keeps a builder that adds thousands of edges from paying O(V+E) each time.
What the order buys you. One pass over it assigns levels: level(n) = 1 + max(level(p) for p in preds(n)). Nodes at the same level have no dependency between them and can be issued on separate CUDA streams; that level assignment is the schedule the runtime launches, so the ordering is not bookkeeping, it is the execution plan.
Failure modes. The recursive _cycle blows Python's 1000-frame default on a deep chain: a 5k-op traced model will hit it, so make it iterative or raise the limit. Edge config has to be checked against port signatures (dtype, shape, arity) at insert time, or you get a graph that is acyclic and still unexecutable. The builder is single-threaded by design: if a UI thread mutates while a compile thread reads order, you get a torn order, so freeze the graph before compile. Node deletion must cascade to incident edges and invalidate pos; the cheap correct version is to mark dirty and re-toposort.
When not to do this. If the graph is built once and compiled (a TensorRT engine build, a torch.fx export) skip incremental maintenance entirely: mutate freely, run one Kahn pass at the end. Incremental dynamic topological ordering (Pearce–Kelly) only pays off for large graphs edited interactively, where it costs O(affected region) per edge instead of O(V+E).
Curated: · Written: · Reviewed:
Q&A-22Design real-time fraud detection at more than 10,000 requests per second within 50 milliseconds, with zero-downtime model swaps and holiday traffic spikes.
Reported in trueinterview.io › study › nvidia-machine-learningNVIDIA Machine Learning Engineer Interview Process
The 90-second answer
I'd treat the 50 ms as an end-to-end p99 SLO and budget it before choosing a model. At 10k RPS you have 10 requests per millisecond; the GPU is rarely the constraint: feature retrieval and batching tail are. So: a stateless scorer in front of Triton Inference Server, TensorRT FP16 engines (or Triton's FIL backend if the model is XGBoost), dynamic batching with a 5 ms queue cap, and a versioned model repository that supports shadow and canary swaps.
| Stage | p99 target | Mechanism |
|---|---|---|
| feature fetch | 12 ms | Redis pipelined MGET, 3 replicas, local LRU for hot keys |
| queue + batch | 5 ms | Triton dynamic_batching with max_queue_delay_microseconds: 5000 |
| inference | 8 ms | TensorRT FP16, batch ≤128, 2 GPU instances |
| decision + write | 10 ms | async Kafka emit; synchronous path returns score only |
| headroom | 15 ms | absorbs GC, retries, spike |
That sums to 50 ms. If feature fetch exceeds 12 ms, fall back to last-known features with a staleness flag rather than blow the SLO.
Triton config for the scoring model:
text dynamic_batching { max_queue_delay_microseconds: 5000 preferred_batch_size: [32, 64, 128] } instance_group [{ count: 2 kind: KIND_GPU }]
Capacity arithmetic: a 2M-parameter MLP in FP16 at batch 64 runs ~1.5 ms on an L4. One L4 then handles 64 / 0.0015 ≈ 42k requests/s; two GPUs per node give 2× headroom for a 3× holiday spike when you add nodes. The real limit is Redis: 10k RPS × 20 features = 200k key reads/s; pipeline them in batches of 50 and keep p99 under 12 ms.
Zero-downtime swap: run Triton with --model-control-mode=explicit. Load version 2 alongside version 1, warm it with model_warmup on representative batches, then send 1% of traffic to v2 via the scorer's routing flag. Compare score distribution (PSI) and precision on a labeled holdout. If clean, ramp 1→10→50→100% over 30 minutes. Unload v1 only after v2 has held p99 for an hour. Never rely on Triton's "latest" policy alone: it swaps on file appearance, which is not a canary.
Holiday spikes: autoscale on Triton queue depth (nv_inference_queue_duration_us) and GPU utilization, not CPU. Pre-scale 48 h before known peaks. Cap the batch queue at 5 ms; if queue depth exceeds 2× batch size, shed to a rules-only fallback that returns a conservative score in 2 ms. That keeps p99 bounded instead of letting the queue grow.
Failure modes: feature-store timeout → stale features with feature_age_ms in the response; GPU OOM → Triton returns 503, scorer retries once on a second instance; model swap skew → shadow mode catches it before canary; batch tail → monitor p99 per batch size, not just mean.
Curated: · Written: · Reviewed:
Q&A-23You are fine-tuning a 7B LLM with LoRA in PyTorch on 8x H100 and training loss keeps dropping but validation perplexity spikes after a few hundred steps. Name three concrete checks or changes you would make, and for each, explain what metric or artifact would confirm it was the cause.
Reported in Nvidia Machine Learning Engineer Interview Guide
The 90-second answer
Three hypotheses, and I check them in this order because the cheapest one is also the most common: the eval path is wrong, the data is contaminated, or the optimisation is genuinely overfitting. A val-perplexity spike at step ~300 on a 7B LoRA run is often a measurement artefact, not memorization, so I rule that out before touching hyperparameters.
1. Eval path differs from train path. Train loss comes from the packed, masked collator; val perplexity often comes from a different script that pads instead of packing, forgets attention_mask, or averages loss per-sequence instead of per-token. With LoRA the adapter is also easy to leave unmerged or unloaded in the eval process. Artifact that confirms it: run 2,000 training documents through the eval loop. If their perplexity is also high, the model is fine and the eval is broken. The check is a few lines:
python
torch 2.4, transformers 4.44: HF loss is mean over non-ignored shifted tokens
model.eval() tot_loss, tot_tok = 0.0, 0 with torch.no_grad(): for batch in val_loader: # same collator as training out = model(**batch) n = (batch["labels"][:, 1:] != -100).sum().item() tot_loss += out.loss.item() * n tot_tok += n ppl = math.exp(tot_loss / tot_tok)
If that number disagrees with the dashboard by more than ~2%, the dashboard is the bug.
2. Train/val contamination. Instruction datasets are scraped and deduplicated badly; a val split drawn after shuffling a near-duplicated corpus leaks. Check: MinHash-LSH at Jaccard 0.8 plus embedding cosine > 0.95 between each val doc and the train set. Artifact: duplicate rate per val shard, and val perplexity recomputed on the deduplicated val set. If 3% of val docs have a near-twin in train, the spike is memorization of those twins and the curve flattens once they are removed.
3. Learning rate and epoch count. LoRA tolerates 10–100x the full-fine-tune LR, which makes 1e-3 with AdamW look reasonable and then memorize. Do the arithmetic on tokens seen: 8 GPUs × micro-batch 2 × grad-accum 8 × 4096 tokens = 524k tokens/step, so 300 steps is 157M tokens. On a 100M-token dataset that is 1.5 epochs: overfitting is expected, not surprising. Artifact: a val-perplexity-vs-step curve for LR ∈ {1e-4, 2e-4, 5e-4} with cosine decay, logged every 50 steps alongside gradient norm. Confirms it if the best val perplexity lands near step 200 and every LR rises after, with grad-norm spikes above ~1.0 at 1e-3.
| Check | Artifact | Confirms the cause when |
|---|---|---|
| Eval path | ppl of 2k train docs via eval loop | train-subset ppl is also high |
| Contamination | MinHash dup rate, cosine sim | >1% val docs have a train near-twin |
| LR / epochs | val ppl vs step per LR | best step ≈ 200, monotone rise after |
What I change once confirmed. For (3): drop to 2e-4, keep r=16/alpha=32 on q,k,v,o rather than r=64/alpha=128 on every linear layer, add LoRA dropout 0.05 and weight decay 0.01, and checkpoint on best val perplexity every 50 steps instead of last. For (2): dedup train against val before splitting, and hold out by source, not by row. For (1): one collator, one loss function, imported by both loops.
Failure mode to name. Early stopping on a noisy val set with a small val split (under ~500 sequences) will stop on noise; the spike may be ±5% sampling variance. Log the val-set size and a bootstrap CI on perplexity alongside the curve, if the CI overlaps the pre-spike value, there is nothing to fix.
Curated: · Written: · Reviewed:
Q&A-24You need to serve an LLM with RAG for an internal developer assistant at p95 under 200 ms with 10x traffic spikes: TensorRT-LLM with continuous batching or static microbatching, and how do you keep latency stable?
Reported in Nvidia Machine Learning Engineer Interview Guide
The 90-second answer
First I'd pin the SLO, because 200 ms p95 cannot be a full LLM response. It has to be time-to-first-token with streaming, or a hard cap around 64 output tokens. I'll assume TTFT p95 ≤ 200 ms for a RAG prompt up to 4K tokens, retrieval included. If the interviewer means full response, the SLO is infeasible and we renegotiate before picking an engine.
With that assumption: TensorRT-LLM with continuous (in-flight) batching, not static microbatching. Static microbatching fixes the batch and pads every sequence to the longest member. A developer's one-line question then queues behind a 4K-token code-context request, and p95 TTFT goes from ~120 ms to seconds. Continuous batching schedules per iteration: finished sequences leave, queued requests join the next decode step, and the paged KV cache removes padding.
| Static microbatching | Continuous batching (TensorRT-LLM) | |
|---|---|---|
| Scheduling | fixed batch, waits to fill | iteration-level, admits new requests each step |
| Padding | pads to longest sequence in batch | paged KV cache, no padding |
| Tail latency | head-of-line blocking, p95 spikes | bounded if prefill is chunked and KV cache has headroom |
| Spike response | queue grows, p95 blows out | autoscale plus admission control |
The latency budget
| Stage | p95 budget |
|---|---|
| query embedding | 10 ms |
| vector search | 20 ms |
| rerank | 30 ms |
| prompt assembly | 5 ms |
| LLM TTFT | 135 ms |
| total | 200 ms |
Retrieval has to be its own service on its own pool. If it shares the generation GPUs, a spike in generation starves it and TTFT moves with the wrong variable.
Sizing the KV cache, since that is what actually breaks
Llama-3-8B: 32 layers, 8 KV heads (GQA), head_dim 128. Per-token KV bytes = 2 (K and V) × 32 × 8 × 128 × bytes-per-element.
| Precision | bytes/token | 4K-token request |
|---|---|---|
| FP16 | 128 KiB | 512 MiB |
| FP8 | 64 KiB | 256 MiB |
On an 80 GB H100 with FP8 weights (~8 GB) and ~2 GB of activations, roughly 70 GB is free; at kv_cache_free_gpu_memory_fraction 0.85 that is ~60 GB of KV cache. FP8 KV gives 60 GB / 256 MiB ≈ 240 concurrent 4K requests per GPU; FP16 gives ~120.
The spike, worked
Baseline 50 req/s at 2 s mean end-to-end is 100 requests in flight. A 10x spike is 500 req/s and ~1000 in flight. At 240 slots per GPU that is 4.2 GPUs; I'd run 7 replicas for ~60% steady-state utilization and headroom for a single replica loss. FP16 KV would need 9, plus headroom, which is why I'd take the FP8 KV cache.
Keeping it stable
Chunked prefill, so a 4K-token prompt is processed in slices and does not stall decode for everyone else. Admission control: a per-user token bucket, a deadline-aware queue, and priority for interactive queries over batch jobs. Shed low-priority work before TTFT p95 crosses 180 ms. Autoscale on queue depth and TTFT p95, not CPU, and keep a warm pool: loading a TensorRT engine takes tens of seconds, so a cold replica arrives after the spike.
Failure modes and how I'd see them
KV cache exhaustion shows as kv_cache_utilization above 0.9 and a TTFT cliff: fix by trimming retrieved context or adding replicas. Prefill blocking shows as p99 inter-token jitter with flat throughput: chunked prefill or disaggregated prefill/decode. A retrieval slowdown shows as TTFT rising with GPU metrics flat: cache frequent queries and scale the vector DB. Autoscaler cold start shows as a p95 spike that peaks after traffic does: warm replicas and pre-loaded engines.
Curated: · Written: · Reviewed:
Q&A-25Walk me through a project on your resume in detail.
Reported in NVIDIA Interview 2026: Onsite Rule, AI Ban, Pay
The 90-second answer
The project I'd walk you through is the visual search encoder I shipped last year: a 300M-parameter CLIP-style dual encoder, image tower plus text tower, fine-tuned on 40M image–text pairs and then converted to TensorRT for serving. I owned the training loop and the INT8 conversion; two other engineers owned the data pipeline and the ANN index. One constraint drove every decision: the product needed p99 under 25 ms per query on a single L4 at 400 QPS, and our FP16 engine came in at 31 ms. INT8 with per-channel scales got us to 14 ms p99 at 400 QPS with recall@10 down 0.4 points from the FP32 baseline, which product accepted.
Training ran on 8×A100 80GB over NVLink, BF16 autocast with FP32 master weights, global batch 4096 via gradient accumulation of 8, and we held ~46% MFU after fixing a dataloader stall that was leaving the GPUs idle 30% of the step.
| Stage | Change | p99, L4, batch 8 | recall@10 |
|---|---|---|---|
| FP32 PyTorch eager | baseline | 96 ms | 0.712 |
| FP16 TensorRT | engine only, no graph changes | 31 ms | 0.711 |
| INT8, per-tensor scales | first attempt | 13 ms | 0.681 |
| INT8, per-channel + FP16 last matmul | shipped | 14 ms | 0.708 |
At 400 QPS and 14 ms, Little's law puts about 5.6 requests in flight, so we ran dynamic batching with a 2 ms window and max batch 8: the window is what keeps p99 flat instead of letting a full batch queue behind one slow request.
The part worth digging into. The first INT8 build lost 3.1 recall points, which is far past what a calibration issue should cost. I dumped per-layer output cosine similarity between the FP32 and INT8 engines over 512 held-out pairs. Every layer sat above 0.999 except the text tower's final projection, at 0.94. Its weight histogram had a max-to-median ratio around 180, a handful of outlier channels, so a per-tensor scale set by the max squeezed everything else into a few quantization bins. Per-channel weight scales fixed the weights; the activation outliers survived, so I kept that one matmul in FP16. That cost 1 ms and recovered 2.7 of the 3.1 points.
What broke in production. Six weeks after launch, recall on screenshots, a segment that barely existed in our calibration set, fell off. Web photos and flat UI regions produce very different activation distributions. We caught it because we log a 0.5% sample of query embeddings and track INT8-vs-FP16 cosine weekly; for that segment it slid from 0.998 to 0.981. We rebuilt the engine against a stratified 2,048-image calibration set with 25% screenshots and got back to 0.997. The calibration set is a production dependency, so it gets versioned and pinned alongside the engine now.
What I'd change. I'd have put the FP16/INT8 parity check in CI from day one rather than finding the regression on a dashboard six weeks later. The check is cheap (512 pairs, a cosine per layer, a threshold) and it would have caught both the projection outlier and the calibration drift before either reached traffic.
Expect follow-ups on why a dual encoder instead of a cross-encoder (latency: the image embeddings are precomputed offline, so only the text tower runs per query), and on why not distil a smaller model instead of quantizing (we tried; the 80M student lost 2.4 recall points, more than INT8 cost us).
Curated: · Written: · Reviewed:
Curated: · Written: · Reviewed: