Real Interview Questions for the AI Engineer at Google
NewInterview Q&A
Q&A-1Implement a function to calculate the attention score between two sequences.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
"Attention score" names two different tensors (the raw compatibility scores QKᵀ/√D, or the post-softmax weights) so I'd say which one I'm returning and return both. Shapes: queries (B, H, Lq, D), keys (B, H, Lk, D), values (B, H, Lk, Dv). I keep Lq and Lk separate because cross-attention and KV-cache decoding both run with Lq ≠ Lk.
Three decisions decide whether it's correct: scale by 1/√D where D is the head dim, not d_model; apply the mask before the softmax as -inf; run the softmax in fp32.
python import math, torch, torch.nn.functional as F
def attention(q, k, v, *, mask=None, causal=False, dropout_p=0.0, training=False): # q: (B,H,Lq,D) k: (B,H,Lk,D) v: (B,H,Lk,Dv) mask: bool, True = attend D = q.size(-1) scores = (q @ k.transpose(-2, -1)) / math.sqrt(D) # (B,H,Lq,Lk) if causal: Lq, Lk = scores.shape[-2:] keep = torch.ones(Lq, Lk, dtype=torch.bool, device=q.device).tril(Lk - Lq) scores = scores.masked_fill(~keep, float("-inf")) if mask is not None: scores = scores.masked_fill(~mask, float("-inf")) weights = torch.softmax(scores.float(), dim=-1).to(q.dtype) # fp32 softmax if training and dropout_p > 0.0: weights = F.dropout(weights, p=dropout_p) return weights @ v, weights
The tril(Lk - Lq) offset is what makes the same function work for prefill (Lq = Lk) and for single-token decode (Lq = 1, Lk = N), where the one query may attend to every cached key.
Worked trace. B=H=1, Lq=2, Lk=3, D=2, no mask, q=[[1,0],[0,1]], k=v=[[1,0],[0,1],[1,1]].
| step | value |
|---|---|
QKᵀ | [[1,0,1],[0,1,1]] |
/√2 | [[0.707,0,0.707],[0,0.707,0.707]] |
| softmax rows | [[0.401,0.198,0.401],[0.198,0.401,0.401]] |
weights @ V | [[0.802,0.599],[0.599,0.802]] |
Drop the 1/√2 and the scores become [[1,0,1],…], weights [[0.422,0.155,0.422],…]: sharper. That is the whole point of the scale: for independent unit-variance q·k over D dims the dot product has variance D, so unscaled logits grow like √D, the softmax saturates onto one key, and gradients through it go to zero. At D=64 the logits are ~8× too large.
Cost and what breaks. Time is O(B·H·Lq·Lk·D); the weights tensor is O(B·H·Lq·Lk). At B=1, H=32, L=8192 in fp32 that is 32 × 8192² × 4 B ≈ 8.6 GB just for the score matrix: the reason FlashAttention exists.
| Failure | Symptom | Fix |
|---|---|---|
fp16 QKᵀ overflows before scaling | inf → NaN weights | matmul in fp32, or use SDPA |
| a row fully masked (padding-only sequence) | softmax(-inf,…) = NaN, propagates silently | assert ~torch.isnan(out).any(); never let a row be all-masked |
| mask polarity/dtype flipped (True = pad) | trains, just badly: inverted attention | fix the convention in one place, test with a hand-checked case |
dim=-2 in softmax | normalizes across queries; loss still falls slowly | unit-test that rows sum to 1 |
| dropout left on at eval | nondeterministic inference | use nn.Dropout so .eval() disables it |
In production I would not ship the naive version. F.scaled_dot_product_attention(q, k, v, attn_mask=..., is_causal=..., dropout_p=...) (PyTorch 2.0+) dispatches to a Flash or memory-efficient kernel that tiles the computation and never materializes the Lq×Lk matrix, cutting that 8.6 GB to O(L). Pass attn_mask or is_causal, not both. Writing the naive version by hand is still the right interview answer, because it is the thing the fused kernel is an optimization of, and the mask semantics, the √D and the fp32 softmax are exactly the details that survive the swap.
Curated: · Written: · Reviewed:
Q&A-2How would you find the top-K nearest neighbors given a set of embeddings, using an efficient approach?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
Before picking a structure I pin four numbers: N (how many vectors), d (dimensions), K, and the recall the product actually needs. Those decide everything. Exact search is O(N·d) per query and is the right answer surprisingly often: up to a few million vectors with a GPU or a BLAS-backed batch job it beats maintaining an index. Past that, or when p99 latency has a budget in single-digit milliseconds, I move to approximate nearest neighbour and I choose the index by what binds first: memory, latency, or recall.
My default order: HNSW in memory when recall must be ≥0.95 and the corpus fits in RAM; IVF-PQ or ScaNN when it doesn't; brute force when N is small or exactness is contractual.
Exact, done properly
python import numpy as np
def top_k_exact(Q, X, k): # Q: (nq, d), X: (N, d), float32, L2-normalised -> inner product == cosine scores = Q @ X.T # (nq, N) idx = np.argpartition(-scores, k, axis=1)[:, :k] # O(N) per query, not O(N log N) part = np.take_along_axis(scores, idx, axis=1) order = np.argsort(-part, axis=1) return np.take_along_axis(idx, order, axis=1)
argpartition is the whole trick: a full sort of 10^8 scores costs roughly 27× the comparisons of a partition when K=10.
The arithmetic that forces the decision
N = 10^8, d = 768, float32: 10^8 × 768 × 4 B = 307 GB of raw vectors. That does not fit one machine's RAM, so exact search is out on memory grounds alone, before latency. Product quantisation with m = 96 subquantisers at 8 bits stores 96 B per vector: 9.6 GB, a 32x compression, at the cost of approximate distances. HNSW adds roughly M=64 links per node at layer 0: 10^8 × 64 × 4 B ≈ 25.6 GB on top of the vectors.
| Approach | Query cost | Memory @ 10^8 x 768 | Recall@10 |
|---|---|---|---|
| Exact exact (BLAS) | O(N·d) ≈ 7.7e10 FLOPs | 307 GB | 1.00 |
| HNSW, M=32, efSearch=64 | ~O(log N), visits a few thousand nodes | ~333 GB | 0.95–0.99 |
| IVF-PQ, nlist=2^16, nprobe=32 | scans ~0.05% of lists | ~10 GB + centroids | 0.80–0.95 |
| ScaNN (anisotropic PQ) | same shape as IVF | ~10 GB | higher than IVF-PQ at equal bytes |
python import hnswlib index = hnswlib.Index(space="ip", dim=768) # inner product on normalised vectors index.init_index(max_elements=N, M=32, ef_construction=200) index.add_items(X, ids, num_threads=-1) index.set_ef(64) # efSearch: the recall/latency dial labels, dists = index.knn_query(Q, k=10)
Tuning is a recall-latency curve, not a setting. Illustrative sweep on a 10^7 corpus, 768-d, single 16-core box: efSearch=16 gives recall@10 0.82 at p99 0.4 ms; ef=64 gives 0.96 at 1.1 ms; ef=256 gives 0.995 at 3.8 ms. I measure recall against exact ground truth on a held-out query sample and pick the knee, then keep the sweep as a regression test: an index that silently loses recall after a data refresh is the classic production incident here.
Failure modes I check for
- Model change invalidates the index. Vectors from two embedding model versions are not comparable; a partial re-embed gives garbage neighbours with no error. Version the index by model hash and rebuild atomically.
- Metric mismatch. Cosine on unnormalised vectors indexed with L2 returns the wrong neighbours. Normalise at write and at query, or use
space="cosine"and accept the extra normalisation cost. - Filtered search. Post-filtering the top-K by tenant or date can return fewer than K; pre-filtering shatters HNSW graph connectivity. Partition the index per tenant, or use a filtered-HNSW implementation.
- PQ trained on a biased sample. A cluster absent from the training sample gets large quantisation error. Train on a stratified sample and check per-cluster recall.
- Deletions. hnswlib tombstones deleted ids and does not reclaim memory; schedule periodic rebuilds.
If N crosses 10^9, shard by IVF coarse centroid, scatter the query to each shard, and merge the per-shard top-K: the merge is exact even though each shard is approximate.
Curated: · Written: · Reviewed:
Q&A-3Explain the vanishing gradient problem and how residual connections solve it?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
Backprop through a stack is a product of Jacobians: ∂L/∂h₁ = ∂L/∂h_L · Π ∂h_{l+1}/∂h_l. Each factor is a weight matrix times an activation derivative. Sigmoid's derivative peaks at 0.25, and a weight matrix whose spectral norm sits below 1 shrinks the vector further. Fifty plain sigmoid layers at 0.25 per layer gives 0.25⁵⁰ ≈ 8×10⁻³¹: multiply by a learning rate of 1e-3 and the update to layer 1 is zero in fp32. The bottom of the network never moves while the head fits the training set, which is why deep plain nets plateau rather than overfit.
Residual connections turn that product into a sum. With h_{l+1} = h_l + F(h_l), the block Jacobian is I + J_l where J_l = ∂F/∂h_l, so
∂L/∂h_l = ∂L/∂h_L · Π_{i=l}^{L-1} (I + J_i)
Expanding that product, one term is the identity: the gradient reaches layer l at full magnitude at any depth. The other terms are sums of products of the J_i, which is the learned path. The failure mode changes from "the gradient is zero" to "the residual branch contributes little", which is far softer: the network still trains, and the branch grows into usefulness as the identity path carries the signal.
What residuals don't fix
- Explosion. If ‖J_i‖ > 1 the identity term doesn't rescue you; the sum still blows up. Residuals bound the vanishing direction, not the growth direction. You still need normalization and a bounded init.
- Where the norm sits. In the original post-LN transformer, h_{l+1} = LN(h_l + F(h_l)), the normalization is on the residual path, so the clean identity route is gone and gradient norms at init grow with depth: Xiong et al. (ICML 2020) trace post-LN's need for learning-rate warmup to exactly this. Pre-LN, h_{l+1} = h_l + F(LN(h_l)), leaves the identity path untouched and trains without warmup, at some cost in final quality that people usually buy back with a final LN before the head.
- Branch scale at init. If F's output has the same variance as h, the residual stream's variance grows linearly with depth. GPT-2's fix, scale the residual-branch output projection by 1/√(2N) for N blocks, holds the stream near unit variance. On TPU in bf16 that matters twice over: the stream is where additions accumulate, so an unnormalised stream that grows with depth is also the first place you lose precision.
A trace you can do on a whiteboard
Four blocks, ‖J‖ = 0.5 per block, gradient at the top = 1. Figures are illustrative.
| Depth | Plain product | Residual, J all aligned | Residual, random J |
|---|---|---|---|
| 1 | 0.5 | 1.5 | ≈1.0 |
| 2 | 0.25 | 2.25 | ≈1.0 |
| 4 | 0.0625 | 5.06 | ≈1.0 |
The plain column is the vanishing gradient. The aligned column is the other property of the same structure: what keeps the gradient alive can amplify it, so residual + no normalization + large init is an unstable combination, not a free win.
What I'd monitor
Log per-layer gradient norm, or its ratio to the parameter norm, every few hundred steps: global norm clipping hides a dead bottom half. A healthy deep residual net keeps bottom-layer grad norms within an order of magnitude of the top. A ratio decaying geometrically with depth is the signature; the fix order is normalization placement, then residual-branch init scale, then check whether an activation is saturating, since a saturated softmax or tanh at the output starves everything upstream no matter how clean the residual path is.
Curated: · Written: · Reviewed:
Q&A-4Implement a simplified version of a k-means clustering algorithm from scratch.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
First I'd pin down the contract: "from scratch" means I write Lloyd's algorithm myself, but NumPy is fair game. No sklearn KMeans. I'd also ask whether they want k-means++ init and multiple restarts, because plain random init is a different (much worse) algorithm. I'll assume: Euclidean distance, k-means++ seeding, n_init=10 restarts, stop when the largest center shift drops below tol.
python import numpy as np
def kmeans(X, k, *, n_init=10, max_iter=300, tol=1e-4, seed=0): rng = np.random.default_rng(seed) n, d = X.shape best = None
for _ in range(n_init):
# --- k-means++ seeding: O(n*k*d) ---
centers = np.empty((k, d))
centers[0] = X[rng.integers(n)]
d2 = ((X - centers[0]) ** 2).sum(1)
for j in range(1, k):
total = d2.sum()
idx = rng.integers(n) if total == 0 else rng.choice(n, p=d2 / total)
centers[j] = X[idx]
d2 = np.minimum(d2, ((X - centers[j]) ** 2).sum(1))
labels = None
for _ in range(max_iter):
# --- assign: ||x||^2 - 2x.c + ||c||^2, avoids an n*k*d tensor ---
dist = (X ** 2).sum(1)[:, None] - 2 * X @ centers.T + (centers ** 2).sum(1)
new_labels = dist.argmin(1)
# --- update ---
new_centers = centers.copy()
for j in range(k):
members = X[new_labels == j]
if len(members):
new_centers[j] = members.mean(0)
else: # empty cluster: reseed to the worst-fit point
far = np.argmax(dist[np.arange(n), new_labels])
new_centers[j] = X[far]
shift = np.linalg.norm(new_centers - centers, axis=1).max()
centers, labels = new_centers, new_labels
if shift <= tol:
break
inertia = ((X - centers[labels]) ** 2).sum()
if best is None or inertia < best[2]:
best = (labels, centers, inertia)
return best
A trace, so the loop is checkable. Six 1-D points [1,2,3,10,11,12], k=2, seeding picks 1 and 10:
| iter | centers | assignment | new centers | max shift |
|---|---|---|---|---|
| 0 | 1, 10 | {1,2,3} {10,11,12} | 2, 11 | 1.0 |
| 1 | 2, 11 | {1,2,3} {10,11,12} | 2, 11 | 0.0 → stop |
Inertia = 1+0+1+1+0+1 = 4. Assignment never flips after iteration 0, which is the usual case: most of the cost is the first two passes.
Cost. Assignment is O(n·k·d) per iteration, update O(n·d), so O(n·k·d·T) total with T typically 10–50. The naive X[:,None,:] - centers[None,:,:] broadcast materialises an n×k×d array: at n=10⁶, k=100, d=768 float64 that is 614 GB, which is why the code uses the ‖x‖² − 2x·c + ‖c‖² expansion and only materialises n×k. Lloyd's algorithm is monotone in inertia and the partition space is finite, so it terminates, but only at a local optimum, and worst-case iteration counts are exponential in n (though real data converges in tens of passes).
What I'd flag as failure modes.
- Bad seeding. Single-run random init lands in a local optimum often enough that
n_init=10is the default for a reason. sklearn 1.4 changedn_initfrom10to'auto', which resolves to 1 fork-means++and 10 forrandom: worth knowing if they ask why the default moved. - Empty clusters. Rare with k-means++ but real with duplicate points. Reseeding to the farthest point is better than my random fallback; say so.
- Unscaled features. Euclidean distance is unit-dependent: a revenue column in the millions swamps a ratio column in [0,1]. Standardise first.
- Wrong cluster shape. k-means implicitly assumes isotropic, similar-mass Voronoi cells. Elongated or density-varying clusters get sliced wrong; that's a model-choice error, not a bug. Silhouette or an elbow plot picks k, and neither is decisive.
When I wouldn't use it. Above roughly 10⁶ points, mini-batch k-means (sklearn's MiniBatchKMeans) trades a little inertia for a large constant-factor win. If I need soft assignments or elliptical clusters, GMM/EM. If d is large and sparse (embeddings, text) normalise the rows and run spherical k-means, which is the same loop with cosine distance.
Curated: · Written: · Reviewed:
Q&A-5How does the Adam optimizer differ from SGD with momentum?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
Both keep a memory of past gradients. The difference is what they do with it. SGD with momentum accumulates one velocity buffer per parameter and applies a single global learning rate to it. Adam keeps two buffers, a first moment (the momentum term) and a second moment of squared gradients, and divides the step by the square root of that second moment, so every parameter gets its own effective learning rate. Bias correction on both moments makes the very first step exactly η in the direction of the gradient's sign.
The consequence that decides most design choices: Adam's step magnitude is roughly η per parameter regardless of how big the gradient is; SGD's step scales with gradient magnitude and, under a consistent gradient, grows to η/(1−μ) times it. Adam's 1e-3 and SGD's 0.1 are not the same knob at different settings: they measure different things.
| SGD + momentum | Adam | |
|---|---|---|
| state per parameter | 1 (velocity) | 2 (m, v) |
| step magnitude | η·‖v‖, → η/(1−μ)·‖g‖ when gradient is steady | ≈ η, independent of ‖g‖ |
| gradient rescaling g→cg | step scales by c | step unchanged |
| hyperparameters | η, μ | η, β₁, β₂, ε |
| typical LR | 0.1, batch-scaled, cosine decay | 1e-3 small models; 3e-4→1e-4 for LLMs |
| state memory, 7B params fp32 | +28 GB | +56 GB |
python
torch 2.4 semantics, written out
SGD(momentum=0.9): dampening=0, so no (1-mu) factor
buf = mu * buf + g p -= lr * buf
Adam
m = b1 * m + (1 - b1) * g v = b2 * v + (1 - b2) * g * g p -= lr * (m / (1 - b1t)) / ((v / (1 - b2t)).sqrt() + eps)
Why bias correction matters, in numbers. Take g₁ = 0.1, β₁ = 0.9, β₂ = 0.999, η = 1e-3. Then m₁ = 0.01 and v₁ = 1e-5. Corrected: m̂ = 0.01/0.1 = 0.1, v̂ = 1e-5/0.001 = 0.01, so the step is 1e-3 · 0.1/(0.1 + 1e-8) = 1e-3. Exactly η, pointing along sign(g). Uncorrected, the ratio m₁/√v₁ = 3.16, so the first step would be 3.16× too large: β₂ sits closer to 1 than β₁, so the second moment is under-estimated more severely than the first. Compare SGD with μ = 0.9 on a constant gradient of 0.1: the velocity converges to 0.1/(1−0.9) = 1.0, so the step is 0.1. Ten times the raw gradient step, and it keeps growing with gradient magnitude.
When to pick which. Adam/AdamW is the default for anything with heterogeneous gradient scales: transformers, embeddings next to LayerNorm, sparse features, and any model where you don't want to hand-tune per-layer LRs. SGD with momentum plus a tuned schedule still wins on convnets: published ResNet-50/ImageNet comparisons generally show it ahead on top-1 by a small margin (order 0.5–1 point) at matched budgets, and it costs half the optimizer state. At Google scale that state is the binding constraint: a 7B model in fp32 is 28 GB of parameters, so Adam adds 56 GB of moments on top of 28 GB of gradients, which is why optimizer state sharding (ZeRO/FSDP) or Adafactor's factorized second moment (O(n+m) instead of O(nm), the T5 choice) is standard.
Failure modes I'd watch for. Coupled L2 in Adam gets divided by √v̂, so parameters with small gradients get decayed disproportionately; that's the whole point of AdamW's decoupled decay, and you detect it by comparing weight norms across layers after training. β₂ = 0.999 is a ~1000-step averaging window, so when gradient scale shifts (a distribution change, the end of warmup) v̂ lags and the effective LR is briefly wrong; LLM training drops to β₂ = 0.95 for this reason. In fp16, v underflows and the loss goes NaN; bf16 or a larger ε (1e-6 rather than 1e-8) fixes it. And a useful live check either way: log ‖Δθ‖/‖θ‖ per layer. For Adam it should sit near η, and a layer far above it is where the run will diverge.
Curated: · Written: · Reviewed:
Q&A-6Walk me through using the central limit theorem and the binomial distribution to size a confidence interval.
Reported in Google AI Engineer Interview Guide 2026: Coding, RRK, Design
The 90-second answer
Set it up as a proportion. We run n independent trials with success probability p: a model passing an eval item, a classifier flagging a bad response. The count is X ~ Binomial(n, p) and the estimate is p̂ = X/n, with E[p̂] = p and Var(p̂) = p(1-p)/n. The CLT applies to the estimate, not to the individual Bernoulli draws: for large n, (p̂ - p) / sqrt(p̂(1-p̂)/n) is approximately N(0,1). Invert that and you get the Wald interval p̂ ± z·sqrt(p̂(1-p̂)/n).
Sizing is the same expression solved for n: n = z²·p(1-p)/p². With p here the margin of error, so I'll call the half-width ε to keep it distinct: n = z²·p(1-p)/ε². We don't know p before the experiment, so plan with the worst case p = 0.5, which maximises p(1-p) at 0.25. At 95% (z = 1.96):
| Half-width ε | n at p=0.5 | n at p=0.9 |
|---|---|---|
| 0.05 | 385 | 139 |
| 0.02 | 2401 | 865 |
| 0.01 | 9604 | 3458 |
So "give me a ±1 point CI on a pass rate" costs ~9,600 eval items. That number is the answer to the sizing question, and it's usually the moment the interviewer pushes on whether the assumptions hold.
Where the normal approximation breaks
The Wald interval is a first-order approximation and it fails exactly where eval numbers live: near 0 and 1. With n = 50 and 50 passes, p̂ = 1, p̂(1-p̂) = 0, and Wald returns [1.0, 1.0]: a zero-width interval claiming certainty from 50 samples. Use the Wilson score interval, which inverts the test without plugging p̂ into the variance:
python import math
def wilson(k, n, z=1.96): p = k / n d = 1 + zz/n c = p + zz/(2n) h = z * math.sqrt(p(1-p)/n + zz/(4n*n)) return ((c-h)/d, (c+h)/d)
wilson(50, 50) # (0.9287, 1.0) Wald gives (1.0, 1.0) wilson(180, 200) # (0.8506, 0.9343) Wald gives (0.8584, 0.9416)
Wilson is asymmetric and pulled toward 0.5, which is the honest shape near the boundary. The rule of thumb for when Wald is safe is np̂ ≥ 5 and n(1-p̂) ≥ 5; below that, switch.
The assumption that actually bites
Var(p̂) = p(1-p)/n assumes independent trials. Eval harnesses rarely give you that: you sample k generations per prompt and they correlate. With intra-cluster correlation ρ, the variance inflates by the design effect 1 + (k-1)ρ. At k = 5, ρ = 0.3, that's 2.2×: your 9,604 items behave like 4,365, and the CI is ~48% wider than advertised. The fix is to size on prompts, not on generations, and to use a cluster-robust variance.
Two more things I'd flag before quoting an interval. First, if the question is "is model B better than model A", the sizing changes: n per arm is 2(z_{α/2}+z_β)²·p(1-p)/δ². Detecting a 2-point difference at 80% power, 95% two-sided, p = 0.5, gives ~9,800 per arm: roughly 4× the single-arm CI cost, which is why most eval comparisons are underpowered. Second, don't peek: a 95% interval recomputed after every batch and stopped at the first significant result has a false-positive rate well above 5%. Fix the n from the sizing calculation up front, or use a sequential test with alpha spending.
Curated: · Written: · Reviewed:
Q&A-7Compare greedy decoding, beam search, temperature, top-k and nucleus sampling, and give a failure mode for each.
Reported in ai-interview-atlas/docs/companies/google.md at main · eiler2005/ai-interview-atlas
The 90-second answer
Assume a decoder-only model emitting one token at a time from a distribution over the vocabulary, and we're choosing the policy for a production endpoint. Two of these are search procedures, greedy and beam, and three reshape or truncate the distribution before you draw from it: temperature, top-k, top-p. They compose, and the order is fixed: divide logits by temperature, apply top-k, apply top-p, softmax, sample.
| Method | Mechanism | Failure mode | How you detect it |
|---|---|---|---|
| Greedy | argmax each step | Repetition loops and flat, low-entropy text; also not bit-identical across batch shapes, because FP reduction order changes which of two near-tied logits wins | distinct-3 collapses, repeated n-grams, output that never terminates |
| Beam search | Keep b sequences by cumulative log-prob | Beam collapse: all b beams converge to the same continuation, so you paid b× for one answer; plus length bias toward short sequences unless you normalise by length | pairwise beam overlap > 0.9; offline BLEU up while human win-rate drops |
| Temperature | Divides logits by T before softmax | T > 1 flattens the tail into contention → off-topic or fabricated tokens; T < 1 sharpens until it is greedy with extra steps | sweep T, plot output entropy against task accuracy; the curve is a cliff, not a slope |
| Top-k | Keep the k highest-probability tokens | Fixed k ignores distribution shape: k=50 on a flat step throws away real mass, k=50 on a peaked step admits junk | log retained probability mass per step; below ~0.9 you are truncating real mass |
| Top-p (nucleus) | Smallest set whose cumulative mass ≥ p | p=0.95 on a flat step keeps hundreds of tokens; p=0.9 on a peaked step keeps one, so you silently get greedy | distribution of kept-set size per step |
A worked step. Softmax at T=1 gives A .45, B .25, C .15, D .08, E .05, F .02.
- Greedy → A, every time.
- Beam b=2 keeps {A, B}; if both beams' argmax continuation is the same string, that is beam collapse.
- top-k=3 → {A, B, C}, renormalised to .529 / .294 / .176.
- top-p=0.9 → cumulative .45, .70, .85, .93, so the smallest set ≥ .9 is {A, B, C, D}, renormalised to .484 / .269 / .161 / .086.
- T=0.5 → p_i ∝ p_i², giving A .682, B .211, C .076, D .022, E .008, F .001.
That last line is the point people miss: at T=1, top-p=0.9 keeps four tokens; at T=0.5 the same p keeps three, because cumulative mass reaches .9 at C. Temperature and top-p are not independent knobs, which is why tuning them one at a time in an eval sweep gives you a configuration that does not transfer.
python def filter_logits(logits, temperature=1.0, top_k=0, top_p=1.0): logits = logits / temperature if top_k: kth = np.sort(logits)[-top_k] logits = np.where(logits < kth, -np.inf, logits) if top_p < 1.0: order = np.argsort(-logits) probs = softmax(logits[order]) cutoff = np.searchsorted(np.cumsum(probs), top_p) + 1 mask = np.full_like(logits, -np.inf) mask[order[:cutoff]] = logits[order[:cutoff]] logits = mask return logits
What I'd actually ship. For interactive chat, greedy or low-T sampling with top-p around 0.95: beam search multiplies KV-cache memory by b and breaks continuous batching, since beams finish at different steps and you pad the batch on TPU. Beam search earns its cost in offline translation and summarisation, where a length-normalised score correlates with quality. For reasoning and code, neither wins: sample n candidates and rank with a verifier or unit tests, which beats beam search because beam optimises likelihood, not correctness. And none of these guarantee valid JSON: that needs constrained decoding, not a temperature tweak.
Curated: · Written: · Reviewed:
Q&A-8How would you implement a sliding window for long-context LLM inference?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
First I'd pin down what 'sliding window' means here. If the model was trained with sliding-window attention, Mistral 7B's config uses a 4096-token window, the serving stack must reproduce that mask exactly. If it's a full-attention model and we're bounding the KV cache at inference, that's an approximation, and I'd say so before shipping.
The 90-second answer
Use a per-layer ring buffer for K/V with S sink slots that are never evicted plus W recent slots. Keep absolute position ids for RoPE; never renumber. Build the attention mask from stored positions: query t attends key j iff j <= t and (j >= t-W+1 or j < S). Prefill in chunks so the temporary attention matrix is O(chunk x (S+W)), not O(L^2). For a 7B GQA model (32 layers, 8 KV heads, d_head=128, fp16), KV is 128 KB/token; 128K tokens is 16 GB, while S=4, W=4096 is about 512 MB. That's the win.
python class RingKV: def init(self, layers, kv_heads, d, W, S=4): self.S, self.W = S, W self.k = torch.empty(layers, S+W, kv_heads, d, dtype=torch.float16) self.v = torch.empty_like(self.k) self.pos = torch.full((S+W,), -1, dtype=torch.long)
def append(self, layer, t, k, v):
slot = t if t < self.S else self.S + ((t - self.S) % self.W)
self.k[layer, slot] = k
self.v[layer, slot] = v
self.pos[slot] = t
def mask(self, q_pos):
j = self.pos
causal = j[None, :] <= q_pos[:, None]
window = j[None, :] >= q_pos[:, None] - self.W + 1
sink = j[None, :] < self.S
return causal & (window | sink)
The mask is the contract. If sequence length <= W, it must equal the full causal mask; I'd unit-test logits against a full-cache reference at lengths W-1, W and W+1. For a model trained with SWA, this is exact. For a full-attention model, it is lossy: early tokens outside the sinks vanish. StreamingLLM's result is that four sink tokens plus the recent window keeps perplexity stable, but it does not recover arbitrary mid-context facts. So I'd gate it behind an eval: needle-in-a-haystack at 32K and 128K, long-doc QA, and a regression set. If retrieval of middle facts matters, use chunked retrieval or a sparse/global-attention model instead.
Serving details: paged KV blocks so continuous batching doesn't copy; a per-sequence ring pointer; prefix cache keyed by token ids and window policy; chunked prefill with 2K-token chunks; TPU sharding over KV heads and layers. Failure modes: evicting a system prompt or tool schema. Keep those as sinks or a pinned prefix; resetting position ids per chunk, which breaks RoPE; and beam or speculative branches sharing a ring buffer without copy-on-write. Detect with exact-match tests, attention-entropy drift, and task eval.
Curated: · Written: · Reviewed:
Q&A-9Explain the difference between Prefix-Tuning and Prompt-Tuning.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
Both are parameter-efficient tuning: you freeze the base model and learn a few thousand continuous vectors instead of updating weights. The difference is where those vectors enter the network.
Prompt-tuning (Lester et al., 2021) learns a soft prompt, typically 20–100 embeddings, that is prepended at the embedding layer only. From there it flows through every layer like ordinary input. Prefix-tuning (Li & Liang, 2021) learns a separate set of key and value vectors injected into the attention of every transformer layer, so the prefix never occupies input positions; it lives in the KV cache.
That single design choice drives everything else: parameter count, expressiveness per token, inference memory, and how hard it is to implement.
| Prompt-tuning | Prefix-tuning | |
|---|---|---|
| Where parameters live | input embedding table | K and V at each layer |
| Typical size | 100 tokens | 10–20 tokens per layer |
| Params (7B, 32 layers, d=4096) | 100 × 4096 = 410K | 10 × 2 × 32 × 4096 = 2.6M |
| Expressiveness per token | one 4096-vector | 262K params |
| KV-cache cost | 100 positions × 32 layers ≈ 51 MB fp16 | 10 positions × 32 layers ≈ 5 MB |
| Implementation | prepend to inputs_embeds | per-layer attention hook |
| Works below ~1B params | poorly | better |
The mechanism, concretely. Prefix-tuning is a two-line change inside attention:
python
prefix_kv[l]: learned (2, n_prefix, d_model) for layer l
k = torch.cat([prefix_kv[l][0].expand(B, -1, -1), W_k(x)], dim=1) v = torch.cat([prefix_kv[l][1].expand(B, -1, -1), W_v(x)], dim=1)
mask: real tokens may attend to prefix; prefix attends to nothing
Prompt-tuning needs no such hook: inputs_embeds = cat(soft_prompt.expand(B,-1,-1), embed(tokens)) and you are done. That is why prompt-tuning ships in every PEFT library and prefix-tuning often needs a custom attention path, and why prefix-tuning breaks under fused kernels like FlashAttention unless the kernel accepts a pre-filled KV cache.
Why prefix-tuning is more expressive per token. A prompt token is one vector that must survive 32 layers of transformation to influence anything. A prefix token carries a distinct key and value at every layer, so it can steer each layer independently. Li & Liang showed this matters most in low-data regimes and small models; Lester et al. showed the opposite trend: prompt-tuning closes the gap to full fine-tuning as the model grows, matching it at 11B with 100 tokens, and trailing badly at a few hundred million parameters.
Failure modes worth naming. Prompt-tuning is initialization-sensitive: random init loses several points versus initializing from the vocabulary embeddings of the task words, and it underfits on small models. Prefix-tuning trains unstably without the reparameterization MLP (a small feed-forward that maps a low-rank vector to the prefix, discarded at inference), and a long prefix steals attention mass: on short inputs the model can spend most of its attention budget on the prefix and degrade. Neither method injects new facts; both steer behaviour, so a knowledge gap needs retrieval or full fine-tuning.
Serving. Prompt-tuning batches trivially: different examples can carry different soft prompts in one batch because the prompt is just input. Prefix-tuning needs per-example KV, which most serving stacks handle poorly, unless the prefix is fixed per task, in which case you precompute its KV once and share it across requests, the same machinery as context caching. At Google scale that amortization is the reason a per-layer prefix is affordable at all.
Curated: · Written: · Reviewed:
Q&A-10How do you mitigate hallucinations in a RAG-based system?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 60-second answer
Most RAG hallucinations are retrieval failures wearing a generation costume, so I measure the two separately before changing anything. Retrieval recall@k against a labelled set tells me whether the answer was even in the context; faithfulness scoring tells me whether the model stuck to what it was given. If recall@20 is 0.7, no prompt fixes the other 30%: the model has to guess or refuse, and it usually guesses.
Then three layers, in order of return on effort.
1. Retrieval quality. Hybrid BM25 plus dense embeddings, pull 50–100 candidates, rerank with a cross-encoder down to the top 5. Chunk at 300–500 tokens with parent-document expansion so the answer sentence isn't severed at a boundary. Index by content hash and re-embed on change: a stale chunk is indistinguishable from a hallucination to the user, and it's the most common cause of "the bot made something up" reports I've seen.
2. Constrain the generation contract. Ask for structured output where each claim carries a chunk id and a verbatim quote, then check the quote is a substring of that chunk after whitespace normalisation. That check is free and kills fabricated citations outright. Give the model an explicit refusal path, "the provided documents don't cover this", and put unanswerable questions in the eval set, because that is the only place refusal behaviour shows up.
3. Verify after generation. Vertex AI's grounding check scores each claim against the retrieved sources; below threshold, drop the sentence or escalate. An NLI entailment model run per sentence is the portable version if you're not on Vertex.
| Failure | Symptom | Fix |
|---|---|---|
| Answer never retrieved | fluent, wrong, cites nothing | reranker, query rewrite, recall@k alarm |
| Distractor chunk | cites a US policy for an EU question | metadata filter, date-aware ranking |
| Chunk boundary | half-answer, hedged | parent-document expansion |
| Multi-hop | correct pieces, wrong join | decompose query, iterate retrieval |
| Numeric aggregation | sums retrieved numbers wrong | extract in the model, compute in code |
A trace. Query: "refund window for EU enterprise plans". Hybrid returns 60 candidates; the top dense hit is the US policy page because the wording matches better. The cross-encoder reranks the EU page to position 1. The model emits {"claim": "30 days", "chunk": "eu-refund-3", "quote": "Enterprise customers in the EEA may request a refund within 30 days"}. The substring check passes and the grounding check returns 0.93 support. Had the EU chunk been absent, the model would have answered 30 days from the US page: right number, wrong jurisdiction, and only the retrieval metric catches that.
On an internal set (illustrative figures), adding the reranker moved recall@5 from 0.71 to 0.89 and cut unsupported claims from 12% to 4% of sentences. Temperature 0 reduces variance but doesn't fix grounding: treat it as hygiene, not mitigation. Keep the eval set versioned alongside the index: a prompt change that raises faithfulness while dropping recall is a regression, not a win.
Curated: · Written: · Reviewed:
Q&A-11What are the trade-offs between LoRA and full fine-tuning for a 70B model?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
Two numbers decide it. Full fine-tuning a 70B model with Adam in mixed precision needs about 16 bytes of persistent state per parameter (bf16 weights and gradients, fp32 master weights, two fp32 Adam moments) so roughly 1.1 TB before a single activation. LoRA at rank 16 on every linear layer trains about 207M parameters, 0.3% of the model, with ~3 GB of optimizer state. The frozen base still costs 140 GB in bf16, so you don't reach one device, but you drop from a 64-chip slice to two.
What that buys: LoRA changes behaviour (instruction format, tone, output schema, a narrow domain's vocabulary) without moving the base weights, so one base in HBM can serve many hot-swapped adapters and you can train twenty customer variants for twenty customers. Full fine-tuning changes what the model knows. If the task needs knowledge the base lacks, or a genuinely new capability, LoRA's loss plateaus above full FT and raising rank does not close it.
| Full fine-tuning | LoRA, r=16, all linear | |
|---|---|---|
| Trainable params | 70B | ~207M (0.3%) |
| Optimizer + grad state | ~1.12 TB | ~3.3 GB |
| Minimum hardware | ~64 TPU v5p chips (95 GB HBM each) | 2 chips, or 1 with a 4-bit base |
| Checkpoint per save | 140 GB | ~400 MB adapter |
| Serving | one model per deployment | base + swappable adapters |
| Quality ceiling | highest | matches on style/format, trails on new knowledge |
| Forgetting risk | high | low |
70B, mixed precision + Adam: bf16 weights 2 B/param -> 140 GB bf16 gradients 2 B/param -> 140 GB fp32 master weights 4 B/param -> 280 GB Adam m, v (fp32) 8 B/param -> 560 GB total ~1.12 TB
LoRA r=16 on q,k,v,o,gate,up,down (d=8192, ffn=28672, 80 layers): per layer: 16*(8192+8192)2 [q,o] = 524k + 16(8192+1024)2 [k,v] = 295k + 16(8192+28672)*3 [gate,up,down]= 1.77M ~ 2.59M x 80 layers ~ 207M trainable; at 16 B/param ~ 3.3 GB
Why it works, and where it stops. LoRA assumes the task update ΔW has low intrinsic rank, so W + BA with B initialised to zero starts training exactly at the base model. Two consequences an interviewer will probe: the update is rank-limited by construction, and the α/r scaling means raising r without raising α shrinks the effective step. Empirically, extending LoRA to all linear layers matters more than raising rank: rank 8 everywhere beats rank 64 on q,v only.
Failure modes. LoRA underfits on knowledge injection: you see held-out domain perplexity sit 0.1–0.2 nats above a full-FT run while training loss looks fine, so compare on held-out domain data, never on train loss. Silent under-training from a full-FT learning rate: LoRA wants ~1e-4 to 2e-4, full FT ~1e-5; using 1e-5 makes people conclude "LoRA doesn't work". Merging an adapter into a 4-bit base throws away the precision you trained in; merge into bf16. On the full-FT side the real risks are catastrophic drift on general evals and checkpoint economics: 140 GB per save to GCS, so save cadence becomes a budget line.
What I'd do. Start LoRA r=16 all-linear, α=32, lr 2e-4, one epoch, and measure held-out domain perplexity plus a general regression suite. Escalate to full FT only when that gap is real and the task is knowledge, not format.
Curated: · Written: · Reviewed:
Q&A-12Compare tensor, pipeline, data, sequence and expert parallelism for a large model deployment.
Reported in ai-interview-atlas/docs/companies/google.md at main · eiler2005/ai-interview-atlas
The 90-second answer
They aren't five alternatives: a real deployment stacks three or four of them. What picks the stack is which resource binds first: parameter memory, activation memory, or interconnect bandwidth, and how fast the link is between the devices you're about to split across. My default ordering is tensor parallelism inside one node's fast fabric (NVLink, TPU ICI), pipeline parallelism across nodes when the model is too deep to fit any other way, data parallelism for throughput once the model fits, sequence parallelism when the sequence dimension is what's eating memory, and expert parallelism only when the model is genuinely sparse.
| Axis | What's split | Communication per step | Right when | Failure mode |
|---|---|---|---|---|
| Data (DDP / ZeRO) | batch | one gradient all-reduce, ~2× model bytes | model + optimizer state fit after sharding | one straggler stalls every replica; ZeRO-3 all-gathers params per layer |
| Tensor | each matmul's weight | 2 all-reduces per block, size b·s·h | ≤8 devices on NVLink/ICI | latency-bound off-node; MFU collapses |
| Pipeline | layers into stages | activations at stage boundaries only | model too deep for TP+DP | bubble (p−1)/(m+p−1); uneven stage cost |
| Sequence | tokens within a sequence | KV blocks or activation all-gather | long context, activation-bound | causal load imbalance; KV traffic grows with s |
| Expert | MoE experts | 2 all-to-alls per MoE layer | sparse model, many experts | token drop at capacity factor; all-to-all congestion |
Three numbers decide the split.
Pipeline bubble is (p−1)/(m+p−1). With p=8 stages and m=32 microbatches that's 7/39 ≈ 18% idle; at m=128 it's 5.2%. Microbatches cost activation memory, so PP only becomes cheap after TP, sequence parallelism or checkpointing have freed memory.
Tensor parallelism does two all-reduces of size b·s·h·dtype per transformer block. Under a ring, bytes per device ≈ 2·(N−1)/N·size: essentially independent of N. Going TP 8→16 doesn't cut per-device traffic, it adds a hop over a slower link. That's why TP stops at the node boundary; on TPU you map the tp mesh axis onto the fastest torus dimension.
Data parallelism: 70B params in bf16 is 140 GB of gradients, and a ring all-reduce moves ~280 GB per device per step. At ~40 GB/s effective per NIC that's ~7 s of pure communication, so you bucket gradients and overlap them with the backward pass, and shard optimizer state (ZeRO-1/2) or parameters (FSDP/ZeRO-3) when memory is the binding constraint.
A worked plan. 70B dense on 512 devices of 80 GB: TP=8 × PP=8 gives 64-way model parallelism, DP=8. Per device that's 70B/64 = 1.09B params → 2.2 GB of bf16 weights and 13.1 GB of Adam fp32 moments plus master copy, duplicated 8× across DP. ZeRO-1 shards that to 1.6 GB. Roughly 4 GB of state leaves ~70 GB for activations, which is what buys m=32 microbatches and keeps the bubble at 18%.
Sequence parallelism is two different things and interviewers notice when you conflate them. Megatron-style SP shards the LayerNorm/dropout activations along tokens: cheap all-gather/reduce-scatter, pure memory saving. Context parallelism for long sequences shards the tokens themselves and passes KV blocks around a ring; causal masking makes device 0 attend to one block while device 7 attends to eight, so you pair devices to balance the ring.
Expert parallelism (GShard, Switch) splits MoE experts across devices and pays two all-to-alls per MoE layer, dispatch and combine. Capacity factor 1.25 gives 25% headroom; tokens past it are dropped and passed through the residual. Watch the dropped-token rate and per-expert token counts: routing collapse shows up there before it shows up in loss.
python import jax, numpy as np from jax.sharding import Mesh, PartitionSpec as P, NamedSharding
512 chips as 8 (tensor) x 8 (pipeline stages) x 8 (data)
mesh = Mesh(np.array(jax.devices()).reshape(8, 8, 8), ('tp', 'pp', 'dp')) w = jax.device_put(w, NamedSharding(mesh, P('tp', None))) # column-split kernel x = jax.device_put(x, NamedSharding(mesh, P('dp', None, None))) # batch on dp
GSPMD propagates those annotations through the graph, so the mesh is the whole deployment decision written down.
Detection. Per-stage step-time variance catches PP imbalance (embedding and LM-head stages cost differently from a transformer block). The all-reduce fraction of step time from the profiler tells you TP has outgrown its fabric: a drop from 45% to 28% MFU when TP crosses the node boundary is the usual signature. Don't reach for PP on a model that fits with TP+DP, and don't shard a dense model expert-style: you pay all-to-all for nothing.
Curated: · Written: · Reviewed:
Q&A-13A long pretraining run suddenly diverges. How do you investigate data, numerics and optimiser state before resuming from a checkpoint?
Reported in ai-interview-atlas/docs/companies/google.md at main · eiler2005/ai-interview-atlas
First 60 seconds. Don't restart. Freeze the run, copy the checkpoint from the last step before the spike and the one at the spike, and replay from the earlier one with the exact same data order. If the loss spikes at the same step, the cause is in the data or the math and you can bisect it. If it doesn't reproduce, you have non-determinism: async collective ordering, a non-deterministic reduction in the attention kernel, or a loader that ignores the seed, and you're chasing a ghost until that's fixed.
Then three probes, cheapest first.
| Probe | What I read | What indicts it |
|---|---|---|
| Data | per-token loss on the spike batch, sorted | one token at loss 25 while the median is 2.1 |
| Numerics | max |logit| pre-softmax, max activation, inf/nan count per layer | logits past ~30 in fp16, nan in RMSNorm |
| Optimiser | max second moment v, ratio |update|/√v per param group | one group's ratio 4× the rest |
Data. A single token with loss 25 against a median of 2.1 is a bad document, not a bad model. Look for the usual: a scraped shard with 10k repeats of one character, a tokenizer edge case, a dedup failure that let one document appear 40 times. Also confirm the mixture weights actually applied at that step: a config change mid-run is a classic.
Numerics. bf16 has an 8-bit mantissa, about 3 significant decimal digits; you won't overflow at 3.4e38 but you will lose precision in the softmax denominator and in RMSNorm when the variance is tiny. In fp16, check the dynamic loss scale: a scale that grew to 2^16 and then met a real gradient spike produces inf, a skipped step, and the skip hides the spike.
Optimiser state. This is where a spike does lasting damage. Adam's second moment absorbs it and the effective learning rate for that parameter collapses.
Arithmetic: with beta2=0.999 (the paper default), a gradient of magnitude 10 against a typical 0.01 contributes 0.001 × 100 = 0.1 to v, and v decays as 0.999^n. Returning to the pre-spike 1e-4 takes ln(2e-5)/ln(0.999) ≈ 10,800 steps: that parameter is effectively frozen for 10k steps. With beta2=0.95 (GPT-3 and most LLM pretraining) the same spike decays in ~210 steps. If the run uses 0.999, that alone explains a divergence that "recovers" into a dead model.
python
per-step probe; JAX + Optax, jax 0.4.x
tok_loss, grads = value_and_grad_fn(params, batch) # [B, T], unreduced gnorm = optax.global_norm(grads) v_max = max(jnp.max(v) for v in jax.tree_util.tree_leaves(opt_state.nu)) log(step, tok_loss.max(), tok_loss.mean(), gnorm, v_max)
Resuming. Roll back 500–2000 steps before the spike, not to the spike checkpoint: the optimiser state there already carries the poisoned v. Fix or skip the offending shard, keep the LR schedule where it was rather than restarting warmup, and watch the first 200 steps for grad norm returning to its pre-spike band.
Failure modes. Rolling back 50 steps and re-hitting the same shard. Zeroing v for all parameters, which produces a second, larger spike because the update/√v ratio jumps. Skipping the shard without finding out why it was bad, so the next one does the same thing three weeks later.
Curated: · Written: · Reviewed:
Q&A-14Design training for a model that cannot fit on one accelerator; explain how you partition state and keep communication affordable.
Reported in ai-interview-atlas/docs/companies/google.md at main · eiler2005/ai-interview-atlas
The 90-second answer
Pin the numbers first, because the partition falls out of them. Take 70B parameters, bf16 compute with fp32 Adam master weights, 80 GB H100s with NVLink inside a node and 400 Gb/s InfiniBand (50 GB/s) between nodes. Mixed-precision Adam costs 16 bytes per parameter (2 bf16 weight, 2 bf16 gradient, 4 fp32 master copy, 4 + 4 for the two moments) so model state alone is 70e9 × 16 = 1.12 TB. That is the thing that does not fit, and no single axis fixes it cheaply. I'd use three at once: tensor parallel inside a node, pipeline parallel across nodes, data parallel outermost, with the optimizer state sharded inside each data-parallel group (FSDP, or ZeRO stage 1/2/3. Same idea, different granularity).
| Axis | What it shards | Collective | Volume per step | Where it may live |
|---|---|---|---|---|
| Tensor | each layer's matmuls | 2 all-reduces fwd + 2 bwd per block | 4 · 2·(TP−1)/TP · B·S·H · 2 bytes per block | NVLink only |
| Pipeline | layers into stages | point-to-point activations | B·S·H · 2 bytes per boundary per microbatch | across nodes |
| Data / FSDP | params, grads, optimizer | reduce-scatter grads, all-gather params | ~2P and ~2P bytes per rank | outermost, overlapped |
Memory, concretely. On 512 H100s with TP=8, PP=8, DP=8, each GPU holds 70B/64 = 1.09B parameters: 2.2 GB bf16 weights, 2.2 GB gradients, 13.1 GB fp32 optimizer state. 17.5 GB, leaving ~50 GB of the 80 for activations and workspace. Without sharding the optimizer, that same GPU would need 12 bytes × 1.09B = 13 GB just for m, v and the master copy on top of replicated weights, which is why ZeRO-1 is the cheapest first win: it cuts the largest term by the DP degree for one reduce-scatter you were paying anyway.
Communication is where this design lives or dies. Tensor parallelism is the expensive axis. At B=4, S=4096, H=8192, one activation tensor is 134M elements = 268 MB in bf16. Per transformer block that is 4 all-reduces at (TP−1)/TP = 7/8, so ~1.9 GB; across 80 blocks, ~150 GB per step. On NVLink at a few hundred GB/s that is sub-second and overlappable. On InfiniBand at 50 GB/s it is ~3 s of pure latency-bound transfer per step, and the job never reaches decent MFU. That single number is why TP stops at the node boundary and pipeline parallelism, which moves only B·S·H·2 bytes per stage boundary per microbatch, carries the cross-node sharding instead.
Data parallelism is bandwidth-bound rather than latency-bound: a 140 GB bf16 gradient reduce-scattered over 8 ranks moves ~2 × 140 × 7/8 ≈ 245 GB per rank, ~5 s at 50 GB/s. You do not remove that, you hide it: bucket the gradient and start the reduce-scatter as each bucket's backward completes, so it runs under the ~6 s of backward compute. If the exposed fraction still exceeds ~30% of step time, shrink DP and grow PP rather than adding nodes.
The other levers. Activation checkpointing: storing every block's activations at 34·B·S·H bytes is ~4.6 GB per block, hopeless; keeping only block inputs and recomputing costs ~30% extra FLOPs and drops it to ~2.7 GB per GPU after TP sharding. Sequence parallelism shards the LayerNorm/dropout activations that TP leaves replicated. Gradient accumulation raises the effective batch without raising collective frequency. Pipeline schedules: with P=8 and M=32 microbatches the naive bubble is (P−1)/(M+P−1) ≈ 18%; interleaving halves it.
Failure modes I'd watch. A single straggler rank turns an all-reduce into a cluster-wide stall: detect with per-rank step-time histograms and NCCL watchdog timeouts, not with the loss curve. Too few microbatches gives a bubble that looks like slow hardware. Checkpointing 1.12 TB serially takes hours; write sharded checkpoints in parallel (Orbax or torch.distributed.checkpoint) so a save is minutes. And changing the partition changes the effective batch and learning-rate schedule: re-tune rather than reuse the single-node recipe.
Curated: · Written: · Reviewed:
Q&A-15Build the evaluation harness for a new model release, covering reproducibility, execution isolation and comparable results.
Reported in ai-interview-atlas/docs/companies/google.md at main · eiler2005/ai-interview-atlas
The 90-second answer
"Comparable" is the requirement that actually costs engineering; reproducibility and isolation are how you earn it. So I start by defining the comparison unit. A run is the tuple (eval-set hash, prompt template, decoding config, weights digest, judge version, harness version). Two runs are comparable only if five of those six are byte-identical and you are varying the sixth. Everything in the harness exists to enforce that.
Three layers:
- Manifest, not flags. Every run emits an immutable manifest of content hashes. Paths lie:
/eval/latest.jsonlis a different file every week. Hashes don't. - Hermetic per-example execution. Each example runs in its own sandbox: no network, fixed CPU/GPU, wall-clock timeout. The model is reached over an RPC, never imported into the harness process, so a model OOM can't kill the harness and a harness bug can't leak state between examples.
- Paired statistics. Score both models on the identical item set and report the paired difference with a bootstrap CI, not two independent accuracies subtracted.
What gets pinned, and what bites if you skip it
| Artefact | Pinned as | Failure if skipped |
|---|---|---|
| eval set | SHA-256 of the JSONL | silent edit makes last week's number incomparable |
| weights | checkpoint step + digest over shards | two exports of "the same model" are not the same |
| tokenizer | vocab/merges file hash | BPE drift changes token counts and therefore truncation |
| decoding | temperature, top_p, max_tokens, stop strings, seed | serving-stack defaults differ between releases |
| judge | model version + prompt hash + few-shot set | LLM-judge drift is the most common false regression |
| container | image digest, not tag | :latest moves under you |
| hardware | accelerator SKU and count | kernel nondeterminism differs by SKU |
The harness
python @dataclass(frozen=True) class RunManifest: eval_sha: str # sha256(eval.jsonl) weights_digest: str # sha256 over checkpoint shards tokenizer_sha: str decode: dict # temperature, top_p, max_tokens, stop, seed judge: str # "gemini-1.5-pro-002@sha256:..." image: str # ".../harness@sha256:..." harness_git: str
def run(manifest: RunManifest, shard: list[Example]) -> list[Record]: out = [] with sandbox(network="none", cpu=4, gpu=1, wall_s=120) as box: for ex in shard: key = sha256(f"{manifest.eval_sha}|{ex.id}|{manifest.weights_digest}") if (hit := store.get(key)) is not None: out.append(hit); continue try: resp = box.call(model_rpc, ex.prompt, **manifest.decode) rec = score(resp, ex, manifest.judge) # label: answer|refusal|error|timeout except (Timeout, RpcError) as e: rec = Record(ex.id, label="error", detail=repr(e)) store.put(key, rec); out.append(rec) return out
Two details in that loop matter more than they look. The cache key includes the weights digest, so a new checkpoint cannot silently reuse the incumbent's cached answers. And errors are a separate label, not a wrong answer: otherwise a flaky sandbox reads as a quality regression.
Reading the result
12,000 items, 4,000 disagreements. McNemar on the discordant pairs: b = 2,300 items only A got right, c = 1,700 only B. χ² = (|b−c|−1)²/(b+c) = 599²/4,000 ≈ 89.7, p well under 1e-20: a real win overall. Then slice it: on the code subset (n = 800) the gap is 3.1 pp with a 95% bootstrap CI of [−0.4, 6.6]. Not significant. The honest release note says "improved overall, no measured change on code," not "improved at code."
Failure modes and how you catch them
GPU kernels are not bitwise deterministic: cuBLAS atomics and split-K attention reduce in nondeterministic order. Don't pretend otherwise: run the same checkpoint twice through the harness and publish that delta as your noise floor. Any claimed gain inside it is not a gain. Contamination is caught by n-gram overlap against training data plus a canary set written after the training cutoff. Harness drift is caught by re-running the incumbent model through the current harness on the same day and hardware as the candidate, if the incumbent's score moved, you changed the ruler, not the model.
For scheduling, shard across Borg or XManager jobs if you're on internal infra, Vertex AI Pipelines or plain Kubernetes elsewhere; results land in BigQuery keyed by manifest hash so any number on a dashboard traces back to an exact run.
Curated: · Written: · Reviewed:
Q&A-16Design a content moderation system for YouTube that handles 1M+ videos per hour.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
Three things decide this design and I'd state them before drawing anything: which policy categories we're enforcing (their error costs differ by orders of magnitude), whether moderation gates publishing or runs after, and the cost ceiling per video. I'll assume pre-publish gating only for the highest-severity categories (CSAM, terrorist content) and post-publish with a fast action ladder for everything else. A false negative on CSAM is catastrophic; a false positive on a cooking video is a support ticket. Thresholds are per-category, never global.
The core answer: don't run one big model over every frame. Build a cascade where each tier costs 10-100× more than the one above it and sees 10-100× less traffic, and put humans at the bottom.
- Tier 0, no model. Exact and perceptual hash match against known-bad and known-good corpora, channel reputation, title/description text classifier. Most uploads are re-uploads or come from channels with history; this clears a large fraction at near-zero marginal cost.
- Tier 1, cheap triage. One frame every 10 s plus a log-mel audio head, small model, many policy heads in a single forward pass.
- Tier 2, dense analysis on the flagged slice. 1 fps, ASR transcript, larger multimodal model with cross-policy context.
- Tier 3, human review, choosing from an action ladder (allow, limit recommendations, age-restrict, demonetize, remove, escalate) not a binary.
The arithmetic that sizes it
1M videos/h ≈ 278/s. Assume mean duration 7 min = 420 s (the mix skews short but the tail is long). The conversion I use throughout: seconds of model compute consumed per hour of upload ÷ 3600 s per hour = chips needed at full utilization.
| Tier | Volume in | Work per video | Compute per hour of upload | Chips |
|---|---|---|---|---|
| 0 | 1.0M/h | hash + metadata + channel rep | ~0 | ~0 |
| 1 | 700k/h (30% resolved at T0) | 42 frames @ 5 ms | 29.4M × 5 ms = 147,000 s | 147,000/3600 ≈ 41 |
| 2 | 35k/h (5% of Tier 1) | 420 frames × 20 ms | 14.7M × 20 ms = 294,000 s | 294,000/3600 ≈ 82 |
| 3 | 14k/h (40% of Tier 2) | 30 s human | 420,000 s = 117 reviewer-hours | ~156 on shift |
The 5 ms and 20 ms per frame are placeholders, not measured facts: you'd replace them with a benchmark run on the actual model and hardware. The sizing method is what I'd defend, not the constant; if the real number is 15 ms, Tier 1 is 123 chips and the design still holds.
Human capacity: 117 reviewer-hours per hour of upload, at 75% utilization → ~156 reviewers on shift at any moment, ~470 headcount across three shifts.
That last number is the point. The human queue is the scarce resource. Every tier above it is optimized to shrink queue volume at fixed recall, not to maximize its own accuracy. A model change that moves 1% of traffic out of the human queue beats two points of AUC.
Operating points and the action ladder
Per-policy models, not one multi-label head, because the operating points are incompatible. CSAM: recall target 0.999, threshold set low enough that everything above 0.05 goes to a trained reviewer, and confirmed hits go to the hash database. Spam: precision 0.99, because auto-removal at scale is user-visible and generates appeals. Borderline violence: no auto-removal at all, only recommendation limiting. Reversible, caps distribution, doesn't touch the video.
Temporal aggregation matters more than the frame model. Use top-k mean over frame scores, not the mean: a 30 s violation inside a 10-minute video averages to nothing.
python
per-video score from per-frame scores; k=3 of ~420 frames
score = np.sort(frame_scores)[-3:].mean()
evasion signal: a video engineered to dodge the sampler
swing = frame_scores.max() - frame_scores.min() if swing > 0.7 and score < threshold: route_to_tier2(video_id) # flicker / overlay evasion
Failure modes and how I'd catch them
| Failure | Detection | Mitigation |
|---|---|---|
| Adversarial evasion: flicker, overlays, pitch-shifted audio, burned-in text | Frame-score variance spike; audio and video heads disagreeing | Route high-swing videos to Tier 2; OCR on sampled frames; audio augmentation in training |
| Distribution shift: a new spam template | Score histogram develops a bulge in the 0.4-0.6 band | Preferentially sample that band into human review; it's active learning for free |
| Feedback-loop bias: model flags, reviewer confirms, model trains on its own errors | Eval set drifts from production distribution | Hold out a random 0.1% of the allow bucket for human review as an unbiased eval set |
| Virality race: post-publish, a violating video gets 1M views in 20 minutes | Time-to-first-view on new channels | Hold recommendations, not the video, for the first N minutes on low-reputation channels |
| Reviewer harm: CSAM and violence review causes PTSD | Exposure hours per reviewer | Rotate, cap exposure, mandatory counselling. This caps throughput per reviewer and is a real capacity constraint, not an HR footnote |
What I'd push back on
Don't build a video decode pipeline: reuse the transcoder's existing frame extraction. Don't make moderation synchronous on the upload path except for the high-severity pre-publish gate, and even there only for channels without history; the pipeline is Pub/Sub into a streaming Beam job, and the upload response returns long before Tier 2 finishes. And don't let the tiers share a threshold: each tier's job is a different precision/recall trade, and collapsing them into one number is how you end up either removing cooking videos or missing the thing you cannot miss.
Curated: · Written: · Reviewed:
Q&A-17Design a system to serve a large-scale LLM with low latency to 100M users.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
"100M users" is not a load figure, so I pin three numbers before drawing anything. Assume 20M DAU × 6 requests = 120M requests/day, ~1.4k QPS average, sized for 3× peak ≈ 4.2k QPS. Assume 250 output tokens per request. SLO: TTFT p99 under 1 s, inter-token latency p99 under 50 ms.
Everything else follows from one property of decode: it is memory-bandwidth bound, not FLOP bound. Every generated token re-reads all the weights, so adding sequences to a batch is nearly free until you hit the compute roofline, and latency is set by how much KV cache you keep resident per replica. So the design is: disaggregated prefill and decode pools, continuous batching, paged KV cache, prefix caching for the shared system prompt, conversation-affinity routing, int8 weights, and admission control that sheds load before the queue destroys p99.
| Layer | Choice | Why |
|---|---|---|
| Partitioning | TP=8 within a replica, replicas independent | TP all-reduce is per-token latency; keep it inside one slice |
| Prefill | separate pool, chunked | prefill is compute bound and would stall decode batches |
| Batching | continuous, iteration-level | a finished sequence frees its slot on the next step, not the next request |
| KV cache | paged, 16-token blocks | eliminates fragmentation; 90%+ block utilisation vs ~60% contiguous |
| Prefix cache | per-replica, keyed on system prompt hash | 1,200-token system prompt prefilled once, not 4.2k times/s |
| Routing | consistent hash on conversation id | turn N+1 reuses turn N's KV instead of re-prefilling history |
| Precision | int8 weights, bf16 activations | ~2× decode throughput; gate on eval delta, not vibes |
The arithmetic that sizes the fleet
Llama-3-70B shape: 80 layers, 8 KV heads (GQA), head dim 128, bf16.
KV per token = 2 × 80 × 8 × 128 × 2 B = 320 KB 1,500-token average context → 480 MB per sequence H100 80 GB, TP=8, int8 weights 70 GB → 8.75 GB/chip, ~70 GB free for KV → ~145 concurrent sequences per chip, ~1,160 per 8-chip replica
Throughput: a well-batched 8×H100 replica of a 70B model measures ~2,000 output tok/s at this latency target. Peak demand is 4.2k QPS × 250 tokens = 1.05M tok/s, so ~525 replicas, ~4,200 chips, spread over regions sized for N-1. That number is why the levers matter: int8 halves it, prefix caching removes most prefill, and capping max_tokens per tier removes the tail.
One request, traced
Turn 4 of a conversation: 1,800 tokens of history, 40 new. Router hashes the conversation id to replica R in us-central1. R's prefix cache holds the 1,200-token system prompt, so only 640 tokens need prefill. Prefill pool at ~8k tok/s → 80 ms. KV written into paged blocks, handle passed to the decode pool. Decode at batch 96, 22 ms per token → 250 tokens in 5.5 s, streamed. TTFT = 60 ms network + 70 ms queue + 80 ms prefill ≈ 210 ms.
Failure modes I'd instrument for
Queue collapse is the real one: arrival rate crosses service rate and TTFT goes 300 ms → 30 s with no error anywhere. Watch queue depth and TTFT p99, keep 30% headroom, and shed by deadline, if a request cannot make TTFT, return 429 with Retry-After rather than degrading everyone.
Cache-locality loss on autoscale: new replicas start with cold prefix caches and no affinity, so they look healthy and serve slowly. Pre-warm by replaying the top system prompts at startup and steer new conversations to new replicas.
Long-context noisy neighbour: one 128k-token request holds a decode slot for minutes. Cap context per tier and use chunked prefill so it interleaves instead of blocking the batch.
TP group as failure domain: one chip loss kills the replica mid-stream. Drain, reroute, and let the client resume from the last token: the KV is gone, so re-prefill is the cost of that failure.
Curated: · Written: · Reviewed:
Q&A-18How would you build a real-time recommendation engine for Gemini-powered search?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
"Real-time" is two clocks here, and I'd separate them before drawing anything. The recommendation slate renders next to organic results, so it has roughly 150 ms of p99 budget before it starts blocking the page. The user's last action (a click, a 20-second dwell, a reformulation) should change the next slate within a couple of seconds. Those pull against each other, and the whole design is about keeping the expensive model off the hot path.
Four stages: stream events into an online feature store, generate candidates, rank them, apply policy.
| Stage | What runs | Budget (illustrative) |
|---|---|---|
| Intent | Gemini query rewrite + facets, cached by normalized query | 0 ms on 60% cache hit, else 180 ms |
| Features | online store read (Bigtable/Redis), last ~50 events per user | 8 ms |
| Candidates | ScaNN ANN over item embeddings + trending + fresh-content sources | 12 ms for top 500 |
| Rank | distilled cross-encoder over 500 → 20 | 15 ms |
| Policy | dedupe by domain/brand, safety filters, diversity | 5 ms |
| Rationale | Gemini one-liner for top 3, cached on (item, intent) | off critical path |
Why Gemini doesn't score the slate. A live Gemini call is hundreds of milliseconds and you'd need it 500 times per request. So Gemini does the jobs where its judgment is worth the latency and it can run once: it rewrites the query into intent facets, it embeds documents for the item tower, and offline it labels (query, item) pairs that train the distilled ranker. The ranker that actually runs at 15 ms is a small cross-encoder that has learned to imitate Gemini's ordering. I'd keep one live Gemini call, the "why this result" line, because that's user-visible text, cacheable, and can be generated after the slate ships.
A trace. Query: "best noise cancelling headphones for flights." Gemini rewrite returns facets {product_research, travel, over_ear} and a 768-d intent embedding; that exact normalized query was seen 4 minutes ago, so the facet cache hits. The user tower concatenates the intent embedding with the last 50 events (two headphone page views, one comparison click) and emits a 128-d vector. ScaNN returns 500 neighbours in 11 ms; the fresh-content source adds 40 items published in the last hour. Availability filter drops 120. The distilled ranker scores 420 in 14 ms and returns 20. Policy removes three near-duplicate SKUs from one brand and caps two per domain, leaving 10. Gemini generates rationale for the top 3 from a cached template: 40% of those (item, intent) pairs were already generated, so two come from cache and one costs 300 ms, after the response.
Failure modes I'd watch.
- Embedding drift. Swapping the Gemini embedding model invalidates 10^8 vectors and a re-embed takes days. Run the new index in shadow, compare recall@20 on a frozen golden set, and only then shift traffic. Version the index name in the feature key so a stale vector can never be scored against a new query encoder.
- Feedback loop. A ranker trained on clicks learns to serve what it already served. Hold out 5% of traffic with epsilon-random exploration and log propensities, or the catalogue ossifies within weeks.
- Feature staleness. The whole "real-time" claim rests on write-to-read lag in the online store. Alert at p99 > 2 s; a silently lagging stream looks like a working system with worse CTR.
- Tail latency on the LLM path. Gemini p99 is fat. Hedge the rationale call with a 120 ms timeout and fall back to a template: never let it gate the slate.
If the interviewer pushes on cold start: new users get the intent-facet cohort's popular items, and new documents get a Gemini embedding from title plus first chunk on ingest, so they're retrievable within minutes rather than after the next training run.
Curated: · Written: · Reviewed:
Q&A-19Design the end-to-end architecture for a GCP-hosted RAG assistant in Google Search that answers with citations and keeps p95 latency under 800 ms while serving 10k QPS.
Reported in Google DeepMind AI Engineer Interview Guide
The first 90 seconds
Two things to pin before drawing boxes. First, what the 800 ms measures: I'll assume p95 end-to-end to the last token of an ~80-token answer. If it's time-to-first-token, streaming changes the budget completely and I'd spend it elsewhere. Second, that query volume is head-heavy: a small set of queries is a large share of 10k QPS. Without that, 10k QPS of generation isn't affordable and the honest answer is that the SLO needs a different product shape.
client → Global LB + CDN (exact-match cache) → query service (GKE): normalize, rewrite, embed → hybrid retrieval, in parallel: Vertex AI Vector Search ∥ lexical index → cross-encoder rerank, 200 → 8 → Gemini on Vertex AI, schema-constrained generation → citation verifier (NLI) → stream to client
Budget. I allocate per-stage p95s that sum to 700, not 800. p95s don't add: if stages were independent the end-to-end p95 would land below the sum, but queueing correlates the tails, so I budget the sum to 700 and treat the remaining 100 ms as the correction I expect to pay. Then I measure it under load rather than trusting the table.
| Stage | p95 budget (ms) |
|---|---|
| edge, TLS, LB | 40 |
| rewrite + embed (batched) | 60 |
| hybrid retrieval, parallel | 80 |
| rerank 200 → 8 | 90 |
| prefill 60 + 80-token decode at ~270 tok/s | 360 |
| citation verification, parallel | 50 |
| serialization | 20 |
| sum | 700 |
The 270 tok/s decode figure is an assumption to validate against the model version you actually deploy: it is the single number that decides whether this design fits, and if the deployed model is slower the answer length cap has to come down.
Retrieval. Hybrid, not dense-only: dense retrieval drops exact identifiers, part numbers and rare entities that a lexical index catches. ANN recall@k is a knob you measure on your own golden set, not a constant: I'd size k so measured recall@k clears what the reranker needs, typically a few hundred candidates, because the cross-encoder is what buys precision@8. Vertex AI Vector Search is Google's managed vector service built on ScaNN, Google's open-source ANN library. The alternative I reject is stuffing 50 spans into a long context: prefill scales with token count and the budget above has no room for it.
Citations. Number the 8 surviving spans and constrain generation with responseSchema, where the citation field is an enum of exactly those 8 IDs: the model then cannot cite a span that was never retrieved. The Gemini API's controlled generation supports enum on string fields, but the supported schema subset varies by model, so I'd verify it against the deployed version before relying on it. Then a small NLI model scores each (sentence, cited span) pair; unsupported pairs drop the sentence or trigger one regeneration. That verifier is the difference between "has citations" and "citations are true."
Capacity. 10k QPS × 80 output tokens = 800k output tokens/s. No single project quota covers that, so the design has to cut model calls before it shards: CDN exact-match, a semantic cache in Memorystore keyed on (normalized query, locale, index version) with a 10-minute TTL, and a cascade where a cheaper Flash-tier model serves the simple tier. At a 55% hit rate the model sees 4.5k QPS, and the remainder shards across regions with provisioned throughput.
Failure modes.
- Stale citation: the doc changes between retrieval and render. Store span hash plus doc version; the verifier re-fetches on mismatch and drops the citation rather than serving a dead link.
- Cache poisoning: one bad answer replayed to thousands. Cache only verified answers, bound the TTL, and key on index version so a reindex invalidates everything.
- Decode tail: hedge to a second region at p95, plus an extractive fallback. Top 3 spans with best-matching sentences highlighted, ~200 ms, no model call.
- Hot shards: replicate the vector index per region and quantize to keep it resident.
Curated: · Written: · Reviewed:
Q&A-20Design an agent policy that is robust to tool failures and explain how you would test it before launch.
Reported in Google AI Engineer Guide (2026): Job, Salary & Interviews
The 90-second answer
Tool failures split into three classes that need different responses, and the whole policy falls out of separating them. Transient (429, 503, connect timeout) retry with jittered backoff under a hard attempt cap. Permanent (400, 403, schema violation) never retry; the arguments are wrong, so replan or escalate. Ambiguous: a read timeout or 5xx on a write, where the side effect may or may not have landed. Retrying an ambiguous failure is how agents double-book rooms and double-charge cards, so the policy must reconcile by idempotency key before it retries.
Concretely: every tool gets a declared contract (timeout, attempt cap, backoff schedule, idempotency, fallback source), a classifier maps the error to one of the three classes, and a turn-level budget bounds the total. When the budget is spent the agent escalates with its trace instead of looping.
| Failure | Signal | Policy action |
|---|---|---|
| transient | 429, 503, connect timeout | retry, jittered, ≤3 attempts |
| permanent | 400, 403, 422, schema mismatch | no retry; replan or escalate |
| ambiguous | read timeout on a write | reconcile by key, then retry or escalate |
| degraded | fallback source exists | answer with as_of and a confidence flag |
| exhausted | attempt cap or turn budget | escalate to human with the trace |
python @dataclass(frozen=True) class ToolPolicy: timeout_s: float max_attempts: int backoff: tuple[float, ...] idempotent: bool fallback: str | None
POLICY = { "search": ToolPolicy(2.0, 3, (0.2, 0.8), True, "cached_index"), "calendar": ToolPolicy(1.5, 2, (0.3,), False, None), # write "fx_rate": ToolPolicy(1.0, 2, (0.1,), True, "last_known"), }
def call(tool, args, ctx): p = POLICY[tool] key = ctx.idempotency_key(tool, args) if p.idempotent else None for attempt in range(p.max_attempts): try: return ok(transport(tool, args, timeout=p.timeout_s, key=key)) except Transient: if attempt + 1 == p.max_attempts: break sleep(jitter(p.backoff[min(attempt, len(p.backoff) - 1)])) except Ambiguous: if reconcile(tool, key): # did the write land? return ok_cached(key) raise Escalate(tool, "ambiguous_write") except Permanent as e: raise Replan(tool, e) if p.fallback: return degraded(p.fallback, tool) raise Escalate(tool, "retry_budget_exhausted")
Worst case for search is 3 × 2.0 s plus 0.2 + 0.8 s of backoff = 7.0 s. A turn that allows six tool calls could therefore burn 42 s, so I cap the turn at 20 s and let the policy escalate rather than let one slow dependency eat the user's patience.
A trace. User asks to book a room on the 14th. calendar.create times out at 1.5 s: ambiguous, not transient. The policy calls reconcile with the idempotency key, finds the event was created, and returns it. No duplicate. Then fx_rate 503s twice, the fallback returns last_known with as_of: 2026-02-11, and the final answer carries a low-confidence flag on the price. The user sees a correct booking and an honest caveat.
Testing before launch. I run every tool call through a fault-injection wrapper driven by a seeded schedule, {"search": ["503", "503", "ok"], "calendar": ["timeout_after_commit"]}, so a failure is reproducible in CI rather than a flaky surprise. The scenario matrix is failure class × tool × position in the trajectory (first call, mid-run, last call), which for 8 tools and 5 classes is 120 cases, each with a golden expected outcome.
The metrics that gate the launch:
- Task success under fault versus the no-fault baseline. A drop from 94% to 71% is a policy bug, not noise.
- False-success rate: the agent claims completion when the tool failed. This must be exactly zero on the write-duplication suite; it is the failure that reaches users as a double charge.
- Escalation precision: of the runs that escalated, how many genuinely needed a human. Escalating everything passes the safety bar and fails the product.
- Added p99 latency and cost under injected faults, since retries are the expensive path.
Beyond the harness: replay 500 recorded production trajectories with faults injected and diff against golden answers, then canary at 1% with a per-tool kill switch and a shadow mode where the policy runs but writes are suppressed. That last one catches the case the harness cannot: a tool that returns 200 with stale or malformed data, which no error classifier will ever see.
Curated: · Written: · Reviewed:
Q&A-21You are shipping an LLM-based Smart Reply for Gmail and see a 1.5% increase in reply rate but a spike in user reports of 'pushy' tone. What decoding and alignment knobs do you change first, and what offline and online metrics do you use to verify the fix?
Reported in Google AI Engineer Guide (2026): Job, Salary & Interviews
I'd treat this as a trust regression that happens to carry a small win. The lift is real but tiny; the "pushy" reports are a guardrail breach. So I'd fix decoding first, reversible in hours, and alignment second, which takes days, and I'd gate the whole thing on a hard guardrail: user reports per 10k suggestions must not exceed baseline + 1.
Sizing the experiment first. "1.5% increase" is ambiguous, so I'd state my reading: 1.5% relative, i.e. reply rate 10.0% → 10.15% (+0.15 pp). Two-proportion sample size per arm is n = (z_{α/2}+z_β)²·[p₁(1−p₁)+p₂(1−p₂)]/δ². With α=0.05 two-sided and 80% power, (1.96+0.8416)² = 7.85; p₁(1−p₁)=0.0900, p₂(1−p₂)=0.0912, δ=0.0015. That gives 7.85 × 0.1812 / 2.25e−6 ≈ 632k suggestions per arm. The familiar shortcut n ≈ 16·p(1−p)/δ² is the same calculation with the two variance terms pooled to 2p(1−p) and 7.85 doubled to ~16: it lands at 640k, close enough, but it is an approximation, not the definition. If the 1.5% were absolute (10% → 11.5%), δ=0.015 and the same formula gives ~6.7k per arm. Worth confirming with whoever owns the metric, because it changes the experiment by two orders of magnitude.
Decoding knobs, changed first.
| Knob | From | To | Why |
|---|---|---|---|
| temperature | 0.7 | 0.2 | kills sampled urgency and imperatives |
| top-p | 0.95 | 0.85 | trims the persuasive long tail |
| top-k | 40 | 20 | same |
| repetition penalty | 1.1 | 1.0 | stops forced variation into emphatic phrasing |
| reranking | none | 8 samples, pushiness classifier < 0.3, keep max reply-probability survivor | decouples tone from likelihood |
| constrained decoding | none | block "You should", "Act now", "!" | stopgap only |
Trace: at 0.7/0.95 the model samples "Absolutely! You should reply now.": pushiness classifier 0.82. At 0.2/0.85 it samples "Sounds good, thanks.": classifier 0.12. Reranking keeps the second even when the first scores higher on reply probability, which is the point: the reward model learned that urgency gets clicks, so likelihood and tone are now in conflict and you have to break the tie explicitly.
Alignment, second. Retrain the reward model on tone preference pairs (pushy vs neutral) and add a pushiness penalty to the reward; run DPO with β=0.1 on ~20k pairs; put an explicit tone rubric in the system prompt: no urgency, no imperatives, no exclamation marks. Rejection sampling with the classifier is the bridge while the retrain is in flight.
Verification. Offline: human tone win rate ≥60% against current, pushiness classifier precision/recall on a held-out labelled set, reply rate on logged-context replay within 0.1 pp of the 1.5% lift, distinct-2 for diversity. Online: A/B at the size computed above, primary metric reply rate, guardrail reports per 10k, plus dismiss/undo rate and 7-day retention; sequential testing so a guardrail breach rolls back early.
Failure modes I'd watch: over-correction to bland replies (reply rate falls below baseline. Visible in replay before launch), classifier false positives blocking good suggestions (check recall on neutral-but-enthusiastic replies), reward hacking where the model avoids "!" while staying pushy in substance (human eval catches it, the classifier won't), and distribution shift as the classifier ages against new phrasing.
Curated: · Written: · Reviewed:
Q&A-22You fine-tune a Transformer encoder for Google Search query classification and training loss keeps dropping, but offline AUC stalls and calibration worsens for rare intents; what representation and optimization diagnostics do you run, and what 2 targeted changes do you try first?
Reported in Google AI Engineer Guide (2026): Job, Salary & Interviews
The 90-second answer
Assume a 12-layer encoder (~110M params, 768-d), ~200 intents, the rarest under 0.5% of queries, and dev/test drawn from the same distribution as training. Loss falling while macro-AUC stalls means the model is buying head-class accuracy with rare-class ranking. Calibration worsening on rare intents means softmax confidence there is decoupled from accuracy. I split the diagnosis in two: is the representation separating rare intents at all, and is the optimiser overwriting pretrained features before the head can use them.
Representation diagnostics
| Diagnostic | What I measure | Signal |
|---|---|---|
| Class-conditional geometry | cosine between per-intent [CLS] centroids; silhouette score | rare centroids inside frequent clusters = collapse |
| Effective rank | `exp(H(σ/ | |
| Linear probe | frozen-encoder macro-AUC vs fine-tuned macro-AUC | gap under 1 point means fine-tuning adds no class-relevant structure |
| CKA drift | CKA between pretrained and fine-tuned layers | lower layers drifting far = forgetting; I read the curve, not a fixed cutoff |
| Subword fertility | tokens per word on rare-intent queries | rare intents fragmenting into many subwords weakens their signal |
Effective-rank and CKA numbers are heuristics. I compare against the pretrained checkpoint on identical data rather than against a published threshold, because the right value is corpus-specific.
Optimisation diagnostics
- Per-class loss curves and per-layer gradient norms. A rare class with flat loss but high gradient norm means the head is fighting the encoder.
- Update-to-weight ratio
||Δw||/||w||per layer. Encoder layers moving more than ~1e-3 relative early means the LR is overwriting pretrained features. - Batch class composition: fraction of batches containing any rare intent. At batch 64 with 200 intents, many batches contain none.
- Reliability diagrams and ECE per frequency bucket (15 bins), plus Brier decomposition. A rare bucket at confidence 0.6, accuracy 0.2 contributes
0.05 × 0.4 = 0.02to ECE by itself. - Early stopping on macro-AUC, never on loss.
Two changes I try first
-
Two-stage fine-tune with layer-wise LR decay and class-balanced batches. Freeze the encoder, train the head to convergence, then unfreeze with encoder LR
1e-5, head LR1e-4, weight decay0.01, and a batch sampler guaranteeing at least one rare-intent example per batch. Early-stop on macro-AUC. This stops the encoder overwriting rare-intent features before the head has learned to use them. -
Temperature scaling plus logit adjustment, with the convention fixed once and used in both train and eval. I use the Menon et al. (ICLR 2021) form: add
τ·log π_yto the target logit in the loss, subtract the same term at prediction.
python
PyTorch 2.3: logit-adjusted loss, Menon et al. ICLR 2021
adjusted = logits + tau * torch.log(priors) # [B, C], training loss = F.cross_entropy(adjusted, y)
pred = (logits - tau * torch.log(priors)).argmax(-1) # inference
Subtracting in training and using raw logits at inference is the same model reparametrised; either is fine, mixing them is not. Fit T on a stratified calibration split by minimising NLL, then tune τ on dev macro-AUC.
When not to, and what it costs
- Logit adjustment assumes serving priors match training priors. Under seasonal or new-intent shift the correction pushes the wrong way; estimate priors on a recent serving sample or skip it.
- Large
τtrades head-class precision for rare recall. Watch head-class per-class AUC, not just macro. - Temperature scaling is monotone, so it cannot move AUC at all: it only fixes ECE. If ranking is the problem, temperature alone is not the fix.
- Alternatives: class-balanced loss (Cui et al., CVPR 2019) with β, resampling, focal loss, a supervised-contrastive auxiliary loss on [CLS]. Resampling changes the effective prior, so it interacts with logit adjustment: do not stack them blindly.
Failure modes and detection
- Over-correction. Predicted rare-class rate exceeds the true prior by more than 2×, or macro-AUC rises while micro-AUC falls. Detect by tracking predicted-vs-actual class distribution daily and per-class precision/recall on a held-out slice.
- Temperature fit on a thin or miscalibrated split. Rare classes with under ~100 dev examples give unstable
T. Cross-fit it (fit on fold A, evaluate on B) and report a bootstrap CI; if the CI spans 1.0, do not apply it. - If change 1 closes the linear-probe gap but ECE stays high, change 2 is the fix. If both stall, the rare intents need more data or a contrastive auxiliary loss.
Curated: · Written: · Reviewed:
Q&A-23Tell me about a time you had to build an AI feature where requirements were vague.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The first 90 seconds
Vague requirements are a measurement problem before they're a modelling problem. When someone says "build an AI feature that makes support faster," asking for a spec gets you a worse spec two weeks later. What works is writing down the three or four decisions the vagueness is hiding, picking a defensible default for each, and getting the person who owns the outcome to agree to a number. Then build the smallest thing that moves that number.
Last year that was a draft-reply feature for support agents. The entire ask was one line: "help agents resolve tickets faster with Gemini." Before any model code I turned it into this:
| What was said | What I pinned down | How it was measured |
|---|---|---|
| "faster" | agent edits the draft instead of writing from scratch | accept / edit / reject rate on a labelled set; AHT delta in canary |
| "good drafts" | every policy claim traceable to a retrieved doc | claim-level precision on 300 tickets |
| "fast enough" | p95 ≤ 1.5 s to first token, ≤ 4 s full draft | server-side trace |
| "cheap" | ≤ $0.005 per draft | token counts × quoted per-token price |
Eval set before model. I pulled 300 real tickets stratified by intent and language, and had two experienced agents label each candidate draft accept / edit / reject. Inter-annotator kappa was 0.71: good enough to be a bar. That set was the actual deliverable that made the requirement concrete: once you can score, "vague" becomes "we need 60% accept."
The iteration. v1 was a single prompt, no retrieval, mid-size model: 38% accept. I read 50 failures and 31 of them cited the wrong policy: return window, warranty tier. That's a retrieval bug, not a model bug. v2 retrieved policy docs by predicted intent and forced structured output with a citations field: 61% accept.
python
gate that shipped: no draft when retrieval is weak
if top_doc_score < 0.62 or len(citations) == 0: return {"draft": None, "reason": "low_confidence"}
Canary. 5% of traffic for two weeks. Handle time down 9%, but escalation rate up 1.2 points on billing tickets. Traced it to confident tone on refunds where policy is genuinely ambiguous: the model was answering questions agents would have escalated. The abstain gate above dropped accept rate to 57% and put escalations back at baseline. We shipped that version, not the higher-scoring one.
The arithmetic that funded it past pilot: 2.1M drafts/month at ~1,400 prompt + 180 completion tokens each, at the per-token price we were quoted, came to roughly $0.004 per draft. About $8.4k/month against 9% of handle time across 1,800 agents.
What I'd flag as failure modes. The eval set goes stale the moment the product or the ticket mix shifts, so it needs a refresh cadence and a drift alarm on accept rate. Accept rate alone is gameable: bland drafts get accepted and save nobody time, which is why we tracked edit distance alongside it. And the vagueness doesn't disappear; it gets redefined by whoever complains loudest, so the one-pager I wrote listed explicitly what the feature would not do (no autonomous sends, no refunds above $200) and that doc is what I pointed at when the scope crept.
The general rule I'd carry into a Google loop: the vague ask is a request to choose the metric, not to guess the feature.
Curated: · Written: · Reviewed:
Q&A-24Tell me about a time you disagreed with a lead on a model architecture choice.
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 60-second version
My tech lead wanted to replace the bi-encoder behind our embedding retrieval service with a 7B decoder-based embedder. Offline it beat our model by 4 points of recall@100, and he was right that the ceiling mattered. I thought the gain came from his training recipe, not the architecture, and that the serving cost would blow the latency budget. I asked for two weeks to test the cheap hypothesis before committing to the expensive one. The cheap version recovered most of the gain at a fraction of the cost, we shipped that, and I was wrong about one part of it: the reranker I added to close the gap had a worse p99 tail than I predicted.
The setup
Retrieval stage of a ranking system: 100M passages, ~8k QPS, an 80 ms p99 budget for the retrieval stage alone. Baseline was a 110M-parameter bi-encoder, 768-dim, int8-quantized index. The proposal was a 7B decoder embedder with 4096-dim output.
The numbers I brought to the design review
| Baseline 110M / 768d | Proposed 7B / 4096d | |
|---|---|---|
| Index size, int8, 100M passages | 77 GB | 410 GB |
| Query encode, p99 (measured) | 9 ms | 120 ms |
| Recall@100 (offline) | 0.412 | 0.454 |
| Serving cost per query | 1× | ~14× |
Index arithmetic: 100M × 768 × 1 byte = 76.8 GB; 100M × 4096 × 1 byte = 409.6 GB. That is 5.3× the memory and roughly 5× the shards, before counting the encoder fleet. The 120 ms encode alone broke the 80 ms stage budget.
How I disagreed
Not in the review. I went to him first with one slide: "I think the 4 points are the hard-negative mining and the longer schedule, not the decoder. Give me two weeks to retrain the 110M model on your data pipeline. If it doesn't recover at least 3 of the 4 points, I'll drop it and we ship yours." He agreed because the test was cheap and falsifiable: that framing is what made it a disagreement about evidence rather than about taste.
What the experiment showed
Retraining the 110M bi-encoder on his hard-negative pipeline: recall@100 0.412 → 0.441. So 3.1 of the 4.2 points were the recipe. The remaining gap I closed with a cross-encoder reranking the top 100: 0.441 → 0.463, at +18 ms p99. Net: better than the 7B proposal on quality, inside the latency budget, at about 1.4× baseline cost instead of 14×.
Where I was wrong
The cross-encoder's p99 was 31 ms in my benchmark and 47 ms in production for two weeks, because my benchmark used fixed-length pairs and production had long documents hitting the truncation path. We fixed it with length bucketing. I should have benchmarked the tail on real length distributions, and I said so in the postmortem rather than letting the win stand unqualified.
What I'd carry into the next one
Separate the architecture claim from the training-recipe claim before arguing about either: most "the bigger model is better" results are recipe confounds. Pre-register the decision rule (here: best recall subject to p99 ≤ 80 ms and ≤ 2× cost) so the outcome isn't negotiated after the fact. And once the call is made, commit fully; I've seen engineers keep a lossy scoreboard on a decision that already went the other way, and it poisons the team.
Curated: · Written: · Reviewed:
Q&A-25How do you handle a situation where your model's performance drops in production?
Reported in ai-interview-guides/guides/google.md at main · landedjobs/ai-interview-guides
The 90-second answer
A drop is a claim about one of three things (the model, the inputs, or the measurement) and I find out which before I touch the model. Retraining first is the classic mistake: if a feature pipeline broke, you bake the bug into the next checkpoint and lose the clean baseline you'd have rolled back to.
My order of operations:
- Bound it. When did it start (step change or ramp?), which slice (locale, traffic source, device, query type), and at which stage: raw features, model inputs, model outputs, or labels. A step change at a deploy timestamp is a different animal from a two-week ramp.
- Roll back if it's user-facing and large. A model or serving-config push in the last 24 h is the cheapest hypothesis to kill. Roll back, then diagnose on the old version.
- Classify the failure before proposing a fix.
| Class | What you see | First move |
|---|---|---|
| Input drift | feature PSI up, prediction distribution moves, labels still consistent | inspect upstream pipeline; retrain only if the shift is genuine |
| Concept drift | inputs stable, calibration and residuals degrade | retrain on a recent window, add features |
| Serving skew | offline eval fine, online worse, same request gives different feature values | diff train vs serve vectors for a sampled request id |
| Upstream change | step change at a deploy time, often one slice | roll back the upstream job, pin its output version |
| Measurement bug | metric moves, predictions unchanged | audit the label join and logging schema |
| Traffic mix shift | every slice flat, aggregate down | reweight the metric; don't retrain |
A worked trace (figures hypothetical). A pCTR model in Search drops from AUC 0.81 to 0.74 over 36 hours. Aggregate-only, it looks like concept drift. Slicing by market shows every market flat except Indonesia, down 0.09, so it isn't the model learning something wrong about the world, it's one path. Feature PSI for num_query_tokens in that slice goes from 0.03 to 0.41 while all other features stay under 0.05. The query-understanding team shipped a tokenizer change 40 hours earlier that segments Indonesian differently, so the feature is now computed on a different scale than the model was trained on. Fix: pin the feature version, backfill the affected window, retrain once the feature is stable. No model architecture change, no hyperparameter search.
That PSI check is a few lines and worth having pre-written:
python
PSI per feature, bins fixed from the training reference distribution
import numpy as np
def psi(ref, cur, bins=10): edges = np.quantile(ref, np.linspace(0, 1, bins + 1)) edges[0], edges[-1] = -np.inf, np.inf p = np.histogram(ref, edges)[0] / len(ref) + 1e-6 q = np.histogram(cur, edges)[0] / len(cur) + 1e-6 return float(np.sum((q - p) * np.log(q / p)))
PSI above ~0.2 is the usual rule of thumb for a meaningful shift; treat it as a trigger to look, not a verdict.
What I'd have in place before this happens. Vertex AI Model Monitoring for training-serving skew and drift on feature distributions, plus a separate monitor on the prediction distribution and on delayed-label AUC: input drift and output drift fail differently and you want them on separate alerts. A permanent 1–5% holdback so you always have a clean comparison. Canary rollout with guardrail metrics that auto-abort, and a schema contract on every feature so an upstream type or unit change fails loudly at the boundary instead of silently inside the model.
The judgement call interviewers push on: how long you wait for labels. If labels lag by days, you can't wait: you gate on proxy signals (prediction distribution, feature drift, downstream engagement) and treat the labelled metric as confirmation, not detection.
Curated: · Written: · Reviewed:
Curated: · Written: · Reviewed: