Skip to content
Tech Interview Prep
Technical interview guideNew

PyTorch Fundamentals for Training and Debugging

How PyTorch tensors, autograd, modules and the training loop work, and how to diagnose shape, device, dtype and gradient failures in training code for ML engineering interviews.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: PyTorch 2.x stable releases. Behaviour of torch.compile, DDP, autocast and GradScaler is stated for 2.x; check the release notes for the exact version a team runs and note the version in the answer..

Interview Q&A

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1

A teammate passes a CUDA tensor into np.array() and gets a RuntimeError, then tries a float64 numpy array multiplied by a float32 CUDA tensor and gets a different result than expected. Explain what each failure is, why it happens, and how you would write the code so the same pipeline works on CPU and GPU.

Answer

Short answer. A numpy array lives in host memory, and numpy's ufuncs run on the CPU. A torch tensor carries a device and a dtype, and its storage can sit on a CUDA device. Numpy has no concept of a device, so it cannot read GPU memory. np.array(t) goes through the array protocol, which calls .numpy(), and that fails because a CUDA tensor has no host buffer to return. The fix is an explicit device-to-host copy, .cpu().numpy(). That copy synchronises with the GPU and pays for a transfer.

Mixed-type arithmetic is a separate failure. The result dtype and device come from each library's promotion and dispatch rules, and numpy and torch do not agree on them. Do not predict the result from memory. Print it.

# PyTorch 2.x, NumPy 1.26 or 2.x. Check the exact exception text in your environment.
import numpy as np
import torch

t = torch.ones(3, device='cuda', dtype=torch.float32)

try:
    np.array(t)                  # goes through the array protocol, which calls .numpy()
except Exception as e:
    print(type(e).__name__, e)   # the message names the cuda device

host = t.cpu().numpy()           # explicit device-to-host copy, then a host view

Dtype promotion. Torch promotes a Python scalar against a tensor without widening the tensor, and a 0-dim tensor does not upcast a dimensioned tensor of the same category. Those rules cover tensor-with-tensor and tensor-with-scalar operations. An ndarray operand has to be converted first, so the float64 array times float32 CUDA tensor case is the one to test, not to reason about:

a = np.array([1.1, 2.2])                      # float64, host
x = torch.tensor([1.1, 2.2], device='cuda')   # float32, device

r = x * a
print(type(r), r.dtype)                       # check both the type and the dtype

Make the conversion explicit so the result type is fixed by your code rather than by operator dispatch:

x * torch.as_tensor(a, dtype=x.dtype, device=x.device)   # float32 result on cuda

This converts the array once, to the tensor's dtype and device. The float64 to float32 rounding happens at that line, on purpose, and a reviewer can see it.

Same code on CPU and GPU. Keep data preparation in numpy on the host. Choose the device once, and convert at a single boundary:

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
batch = torch.as_tensor(np_batch, dtype=torch.float32, device=device)

From there, the training path stays in tensors and never calls back into numpy.

Failure modes and how to detect them.

  1. Silent host round-trips. float(t), t.item(), t.numpy() or .cpu() inside a training step forces a sync every iteration. Profile with torch.profiler or Nsight Systems. Step time rises, and the GPU timeline shows idle gaps around the call. Logging loss.item() every step is the usual cause.

  2. Shared memory on CPU. torch.from_numpy shares storage with the array, and torch.as_tensor does too when no conversion is needed. Writes through either side show up on the other, and an in-place op on the tensor changes the numpy array. Detect it by hashing the array before and after the call, or by checking np.shares_memory. Use torch.tensor(a) when you need an independent copy.

  3. Dtype drift in metrics. Accumulating a loss in float32 over millions of steps drops low-order bits. Run a short job with a float32 accumulator and a float64 one, then compare the relative difference before choosing the accumulator dtype.

Curated: · Written: · Reviewed:

Q&A-2

In PyTorch, what is the shape of a + b when a has shape (3, 1) and b has shape (1, 4), and why does it not raise an error?

Answer

The 60-second answer. PyTorch aligns shapes from the trailing dimension. For each pair of aligned sizes, the two must be equal or one must be 1, and the result takes the larger size. A missing leading dimension counts as 1. Here the trailing pair is 1 and 4, which gives 4. The leading pair is 3 and 1, which gives 3. So a + b has shape (3, 4). Each pair is legal, so no exception is raised.

# PyTorch 2.x, Python 3.11
import torch

a = torch.zeros(3, 1)
b = torch.zeros(1, 4)
print((a + b).shape)   # torch.Size([3, 4])

c = torch.zeros(3, 4)
d = torch.zeros(5)
# c + d would raise: sizes 4 and 5 differ and neither is 1

The same rule explains the trap that matters in training code. A prediction of shape (4,) minus a target of shape (4, 1) also broadcasts without error, because (4,) is read as (1, 4). The result is (4, 4). An MSE loss then averages 16 pairwise differences instead of 4 sample errors (hypothetical values: 16 entries rather than 4). Most of those pairs compare a prediction with a target from a different sample, and the training curve still falls, just towards the wrong function.

Failure modes and how you catch them.

  1. Loss computed over a broadcast grid. The loss is finite and the run looks healthy. Detect it by asserting shapes at the loss boundary, for example assert pred.shape == target.shape, and by keeping the warning PyTorch emits when the input and target sizes differ.

  2. Shapes that match by accident. A batch of 1, or a sequence length equal to the feature width, hides the bug in small tests. Use unequal sizes in tests so a wrong broadcast changes the output shape.

The fix belongs at the source. I make the target and prediction the same shape before the loss, and I keep the assertion there. Broadcasting is useful, but the framework will not tell me when I used it by accident.

Curated: · Written: · Reviewed:

Q&A-3

In this code, which statement raises at backward, and why does the error name a slice the code never wrote to? Give a fix that keeps the gradient correct without cloning the whole tensor.

Answer

The 90-second answer. Views of one storage share a single version counter, so the counter tracks the storage, not the slice. A write to b bumps the counter that a was saved against, even though a and b cover different elements. a.pow(2) saves its input a, so backward compares the recorded version with the current one and raises. The error names a or the pow node, which is why it looks like a write the code never made.

import torch
x = torch.randn(4, requires_grad=True)
h = x * 2
a = h[:2]           # view of h
b = h[2:]           # sibling view, disjoint elements
y = a.pow(2)        # pow saves its input a: version recorded here
b.mul_(3)           # bumps the shared counter of h's storage
y.sum().backward()  # RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Two checks on the trace. First, sigmoid would not fail here, because it saves its own output, which is a fresh tensor with its own counter. The failure depends on what the backward formula reads, not only on what was written. Second, the write is to b, so an error that names h or a is expected.

The fix. y = a.clone().pow(2) gives the op a tensor with its own storage and counter, so the write to b cannot touch it. Cloning a, not the whole h, is enough. Keep the in-place b.mul_(3) only if nothing downstream saved a view of h. Otherwise h = torch.cat([h[:2], h[2:] * 3]) is out of place and costs one allocation.

Failure modes, and how you catch each.

  1. Disjoint slices still raise. A candidate who reasons from element ranges will expect no error. Detect: call y.grad_fn.next_functions and compare _saved_self._version against h._version before and after the write.

  2. .data writes pass silently. b.data.mul_(3) skips the counter, so backward runs and uses the new values for a's derivative. Detect: torch.autograd.gradcheck on a float64 copy with a small input.

  3. Clone fixes that hide a leak. Cloning every view removes the error but duplicates memory in a training step. Detect: compare torch.cuda.max_memory_allocated() before and after the change.

torch.autograd.set_detect_anomaly(True) prints the forward line that created the failing node, which points at a.pow(2) rather than at the write.

Curated: · Written: · Reviewed:

Q&A-4

Your training step computes loss_a = head_a(encoder(x)) and loss_b = head_b(encoder(x)) from one shared encoder output, then calls loss_a.backward() followed by loss_b.backward(). What happens on the second call, why, and which fix would you pick?

Answer

The 90-second answer. Autograd records each differentiable op as a node in a graph while the forward pass runs. Each non-leaf tensor carries a grad_fn pointing at its node, and each node keeps the tensors it needs for its backward formula (the saved tensors). backward() walks the graph from the loss, accumulates gradients into the .grad of leaf tensors, and then frees those saved tensors. The second loss_b.backward() walks back through the encoder nodes, which have already released their buffers, so PyTorch raises RuntimeError: Trying to backward through the graph a second time. Only the shared part fails: the head_b nodes are fresh and have their own buffers, so the error comes from the encoder.

# PyTorch 2.x
feat = encoder(x)
loss_a = head_a(feat).mean()
loss_b = head_b(feat).mean()

loss_a.backward()                    # frees buffers of encoder and head_a nodes
loss_b.backward()                    # RuntimeError: graph freed

# Option 1: keep buffers alive across the first backward
loss_a.backward(retain_graph=True)   # encoder buffers survive
loss_b.backward()                    # frees everything

# Option 2: one graph, one backward pass
(loss_a + loss_b).backward()

What retain_graph changes. It keeps the saved tensors of every node reached by the backward pass, so the graph can be traversed again. It does not change the graph's structure, and it does not stop gradients accumulating: each call adds into .grad, so you still zero gradients once per step. Non-leaf .grad values are not kept either way unless you call retain_grad().

Trade-offs, with hypothetical figures. Say the encoder saves 2 GB of activations and the heads together save 0.2 GB. Freeing after loss_a.backward() would release about 0.2 GB. With retain_graph=True the 2 GB stays resident until loss_b.backward(), so the peak can rise by that amount if other work runs between the two calls. The shared encoder's backward also runs twice, which costs compute, not just memory. Option 2 runs the encoder backward once and holds the 2 GB only until that single pass, so it is usually the right choice when both losses are updated together with equal weight. Use the two-backward form when the losses must be stepped separately, for example with different optimizers or a GAN schedule, and accept the extra compute and retention. If one branch should not push gradients into the encoder, call feat.detach() in that branch instead; that is a change to the learning objective, not a memory fix.

Failure modes and detection. The silent one is the reverse mistake: calling retain_graph=True everywhere to silence the error. Peak memory climbs by the retained activations, and the run only fails on the largest batch. Detect it with torch.cuda.max_memory_allocated() before and after the change, on a real batch size. Another is forgetting that torch.no_grad() is needed at evaluation time: otherwise every eval forward builds a graph that is never freed, because nothing calls backward().

Curated: · Written: · Reviewed:

Q&A-5

Two forward passes compute loss = (w * x).sum() with w = torch.ones(2, requires_grad=True) and x = torch.tensor([1., 2.]), and loss.backward() runs after each without zero_grad() in between. What is w.grad afterward, why does it hold that value, and why does a tensor like w * 2 never get a .grad by default?

Answer

Opening. After the second call, w.grad is tensor([2., 4.]). Each backward() adds the gradient of the loss with respect to w into w.grad rather than replacing it. The first call gives [1., 2.] because d/dw of sum(w * x) is x. The second call adds another [1., 2.], so the total stays [2., 4.] until someone clears it. Accumulation is deliberate: it lets you sum gradients over micro-batches before a single optimizer step.

Propagation. requires_grad is set on tensors you create. An operation's output requires grad if any input does, and autograd then records a grad_fn on it. Here x has requires_grad=False, so w * x requires grad only because w does. No gradient is computed for x.

Leaf versus non-leaf. A leaf is a tensor the user created, or one with no grad_fn. w is a leaf. y = w * 2 is not: it has a grad_fn and is_leaf is False. By default only leaves get .grad populated. Reading y.grad returns None (with a warning in recent 2.x releases). Calling y.retain_grad() before backward fills it, but the value is still accumulated per backward call.

import torch
w = torch.ones(2, requires_grad=True)
x = torch.tensor([1., 2.])
for _ in range(2):
    (w * x).sum().backward()
print(w.grad)                  # tensor([2., 4.])
w.grad.zero_()
y = w * 2
print(y.is_leaf, y.grad_fn)    # False <MulBackward0 ...>

Failure modes and detection.

  1. Missing optimizer.zero_grad() (or model.zero_grad()) before the next backward. Each step then uses the sum of several batches' gradients, so updates are larger than the learning rate implies. Detect it by logging the gradient norm per step: it grows roughly linearly between clears.
  2. Calling backward() twice through a graph whose saved buffers were freed. You get a RuntimeError unless retain_graph=True was passed on the first call. Detect it from the traceback, and fix it by restructuring so each step builds a fresh graph.
  3. detach() or .data cuts the graph silently. The loss stops moving on some parameters and no error is raised. Detect it by checking which parameters have a non-None .grad after backward().

Since PyTorch 2.0, zero_grad() defaults to set_to_none=True, which sets .grad to None instead of filling zeros. Code that reads .grad after clearing needs to handle None.

When to avoid retain_grad(). It keeps the intermediate's gradient in memory for the life of the tensor. Use it while debugging a specific activation, not in production training loops.

Curated: · Written: · Reviewed:

Q&A-6

Inside a torch.no_grad() block, does a tensor produced by a forward operation record a grad_fn?

Answer

Short answer. No. Inside torch.no_grad(), autograd does not record operations. A tensor produced by a forward op there has requires_grad=False and grad_fn=None, even when its inputs require gradients. Existing tensors keep their own flag, so w stays trainable after the block.

I'd state the assumptions first: PyTorch 2.x, and the question is about tensors produced by ops, not about the flag on the input.

# PyTorch 2.x
import torch

w = torch.randn(3, requires_grad=True)

y = w * 2
print(y.requires_grad, type(y.grad_fn).__name__)   # True MulBackward0

with torch.no_grad():
    z = w * 2
print(z.requires_grad, z.grad_fn)                  # False None

print(w.requires_grad)                             # True

The rule applies to the block, not the tensor. Every op whose output is created inside the block is untracked, whatever its inputs are. That is why no graph is built and no activations are saved for backward, which lowers memory use and skips the bookkeeping overhead on pure evaluation.

Trade-offs. no_grad only turns off gradient tracking. It does not change model.train() or model.eval(), so dropout and batch norm behave the same inside it. Use torch.inference_mode() when the tensors will never re-enter autograd. It is stricter: tensors created there cannot be saved for backward later, so reusing them in a training graph fails.

Failure modes and how to detect them.

  1. Loss computed inside no_grad during training. loss.backward() raises RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn. The traceback points at backward(), and the cause is the forward pass sitting in the block. Check loss.requires_grad right after the forward pass.

  2. Assuming no_grad changes eval behaviour. A model run inside no_grad but still in training mode keeps dropout active and batch norm updating its running statistics. Detect it by checking model.training before the forward pass.

  3. In-place updates to a leaf outside the block. Modifying a leaf that requires grad in place raises a leaf Variable that requires grad is being used in an in-place operation. Parameter updates belong inside no_grad (optimizers do this for you). Doing them outside is an error, not a silent bug.

  4. Tensors from no_grad reused in a training graph. They carry no history, so gradients stop at that point. Detect it by walking grad_fn back from the loss and finding None where you expected a chain. Calling requires_grad_() on such a tensor re-enables tracking for it from that point but does not restore the lost upstream history.

The operational cost of getting this wrong is usually loud (an exception at backward()), but the wrong-mode eval case in item 2 and the cut-off graph in item 4 can run without error and only show up as odd metrics or flat training curves.

Curated: · Written: · Reviewed:

Q&A-7

You want an effective batch of 64 built from micro-batches of 8 on one GPU, but the model's BatchNorm layers and the loss reduction both matter. Write the training loop for one epoch, say where zero_grad, backward and step go, and state what you change if the last micro-batch of the epoch has only 5 samples.

Answer

Start by fixing what the optimizer should see. An effective batch of 64 means one step() per 64 samples, and the gradient in .grad should equal the gradient of the mean loss over those 64 samples. Two things break that by default: backward() adds into .grad rather than replacing it, and a per-micro-batch mean loss counts each micro-batch equally, so a 5-sample micro-batch gets the same weight as an 8-sample one.

# PyTorch 2.x, single device
opt.zero_grad(set_to_none=True)          # once per optimizer step, before the first backward
seen = 0
for i, (x, y) in enumerate(loader):      # loader yields micro-batches of 8, last one may be 5
    out = model(x)
    loss = criterion(out, y, reduction='sum')   # sum, so the scale does not depend on len(x)
    (loss / 64).backward()               # divide by the fixed effective batch size
    seen += len(x)
    if seen == 64 or i == last_index:
        opt.step()
        opt.zero_grad(set_to_none=True)
        seen = 0

The order inside each accumulation window is: zero the grads, run every forward and backward, then step once, then zero again. Zeroing after step() is what stops the next window inheriting the last one. Calling zero_grad() inside the inner loop throws away the earlier micro-batches, which is a silent failure: the loss curve looks fine with an effective batch of 8.

For the 5-sample tail, dividing by 64 makes that step's update about 5/64 of a full step in gradient magnitude. That is often the intended behaviour for a short epoch tail, but it is a choice you should name. If you want the tail weighted as the mean over its own 5 samples, divide by seen at the step instead of by 64, which requires holding the window's loss graph or re-scaling the grads just before step().

Batch statistics are the trap that accumulation does not fix. BatchNorm computes mean and variance over each forward call, so eight 8-sample forwards normalise over 8 samples each, not 64. Running statistics update eight times per step. A large-batch run and an accumulated run therefore differ even when the gradients match. Options: use GroupNorm or LayerNorm, freeze BN statistics, or accept the difference and say so.

Under DistributedDataParallel, wrap the non-final micro-batch backwards in model.no_sync(), otherwise every backward performs an all-reduce and the communication cost grows by the accumulation factor.

Two failure modes to detect: gradients that grow linearly with the number of micro-batches (the loss was summed or not divided), and a loss curve that differs between an accumulated run and a real batch-64 run that should match. Log the grad norm just before step() and compare it against a single batch-64 forward on the same data.

Curated: · Written: · Reviewed:

Q&A-8

A colleague adds self.scale = torch.ones(d) and self.mask = some_tensor to an nn.Module and then wonders why one shows up in model.parameters(), the other does not appear in state_dict() at all, and a third plain attribute is neither. Explain how each is registered, what state_dict contains and omits, and how you would make the mask persist with the model.

Answer

The 60-second answer. nn.Module keeps three separate registries, and __setattr__ decides which one a value goes into. Assigning an nn.Parameter (or a submodule) registers it in _parameters or _modules, so it appears in parameters() and is returned by state_dict(). Assigning a plain tensor to an attribute does nothing special: it is a normal Python attribute, invisible to parameters(), buffers() and state_dict(). A tensor that must belong to the module's state but must not be trained goes through register_buffer, which puts it in _buffers. Buffers move with .to()/.cuda() and appear in state_dict(), but the optimizer never sees them.

import torch
import torch.nn as nn

class Block(nn.Module):
    def __init__(self, d):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(d))   # _parameters
        self.register_buffer('scale', torch.ones(d))  # _buffers
        self.tag = torch.zeros(d)                    # plain attribute

m = Block(3)
print([n for n, _ in m.named_parameters()])  # ['weight']
print(list(m.state_dict().keys()))           # ['weight', 'scale']
print(hasattr(m, 'tag'), 'tag' in m.state_dict())  # True False

What state_dict omits. It walks _parameters and _buffers recursively and nothing else. A plain tensor attribute is absent, so after torch.save(m.state_dict()) and a load into a fresh instance, tag is whatever __init__ set it to. Optimizer state is also not in the module's state_dict(); it lives in optimizer.state_dict().

The failure mode. A plain tensor is not moved by model.to('cuda'). The forward pass then mixes devices and raises a RuntimeError, or worse, the attribute stays on CPU silently when the code path uses it in a way that tolerates it. Detection is the traceback on the first forward pass. Buffers avoid this because _apply walks _buffers.

Making the mask persist. Use self.register_buffer('mask', mask, persistent=False) if it should move with the device but not be saved, or the default persistent=True if it is part of the checkpoint. Use a plain attribute only for Python-side configuration that is rebuilt from constructor arguments. Assigning self.mask = mask after construction re-registers nothing: if mask was already a buffer, __setattr__ routes a tensor assignment to the existing buffer slot, so the registration choice is made once, at the first assignment.

Trade-off. A parameter is trained and saved. A buffer is saved and moved but not trained, which is the right home for running statistics like BatchNorm.running_mean. A plain attribute is cheapest and wrong for anything that touches the forward pass across devices or checkpoints.

Curated: · Written: · Reviewed:

Q&A-9

During evaluation, a colleague turns off the gradient tracking but leaves the model in training mode, then reports that validation accuracy is noisy and drops between runs on the same checkpoint. Explain what dropout and BatchNorm each do differently in train versus eval mode, why the missing model.eval() produces this symptom, and what would change if the code ran eval under grad instead.

Answer

The 90-second answer. model.train() and model.eval() set a boolean flag on every submodule (module.training). They do not turn gradients on or off. Two layers read the flag. nn.Dropout zeroes random activations in train mode and is the identity in eval mode, so with the flag left on, every validation forward pass drops a different random subset of units and the predictions change between runs on the same checkpoint. nn.BatchNorm normalises with the statistics of the current batch in train mode, and with its stored running_mean and running_var in eval mode. Leave it in train mode and each prediction depends on the other examples in the batch, so accuracy moves with batch size and composition, and the running statistics keep updating from validation data.

The two switches are independent. torch.no_grad() only stops graph construction and saves memory. It does nothing to dropout or BatchNorm, so the symptom above happens with no_grad in place. This is the usual mistake: the author wrapped evaluation in no_grad, saw the memory drop, and assumed the model was in eval mode.

# PyTorch 2.x
import torch, torch.nn as nn
torch.manual_seed(0)
m = nn.Sequential(nn.Linear(4, 4), nn.Dropout(0.5), nn.BatchNorm1d(4))
x = torch.randn(8, 4)

m.train()
with torch.no_grad():
    a = m(x); b = m(x)
print(torch.equal(a, b))   # False: dropout masks differ, BN uses batch stats

m.eval()
with torch.no_grad():
    a = m(x); b = m(x)
print(torch.equal(a, b))   # True: dropout is identity, BN uses running stats

Failure modes.

  1. Eval left in train mode, under no_grad. Validation metrics vary between identical runs; the spread shrinks when you raise the batch size. Detect it by evaluating the same batch twice and comparing outputs with torch.equal. Also check any(not m.training for m in model.modules()), which should be false everywhere during evaluation.

  2. Eval mode set, but the last batch is size 1 under train mode. BatchNorm1d in train mode raises on a single-sample batch with more than one feature value per channel. This is an error, not a silent failure, and it appears only at the epoch boundary.

  3. Eval under grad. Setting model.eval() without no_grad gives correct predictions but builds the autograd graph for every batch. Memory grows with activations kept alive until the tensors go out of scope, and if the metric is accumulated as a tensor rather than .item(), the whole graph is retained across batches. This is a memory and speed problem, not a correctness one. The fix is torch.inference_mode(), which also skips version-counter bookkeeping.

  4. Running statistics corrupted by evaluation. Running eval in train mode updates running_mean and running_var on validation data. The next training epoch starts from statistics that include the validation set. Nothing errors. You only see it as a drift in the first training losses after each validation pass.

Trade-off to name. Dropout and BatchNorm both change behaviour with the mode flag, so one flag guards both. Some architectures keep dropout active on purpose at inference (Monte Carlo dropout for uncertainty estimates). That is a deliberate choice that needs explicit sampling and an aggregation rule, not a default left on by accident.

Curated: · Written: · Reviewed:

Q&A-10

In PyTorch, should you pass softmax probabilities or raw logits into nn.CrossEntropyLoss, and why?

Answer

The 60-second answer. Pass raw logits. nn.CrossEntropyLoss takes unnormalised scores and applies log-softmax internally, so the final linear layer should output logits and nothing else. If you apply softmax first, the loss normalises a second time. The inputs to the loss are then compressed into a narrow range, the loss value is wrong, and gradients for the true class are weaker than they should be, so training is slower and tends to plateau higher.

# PyTorch 2.x
import torch
import torch.nn.functional as F

logits = torch.tensor([[4.0, 1.0, -2.0]])
y = torch.tensor([0])

print(F.cross_entropy(logits, y).item())                    # 0.0508
print(F.cross_entropy(torch.softmax(logits, 1), y).item())  # 0.5837

Trace of the first line: exp(4)+exp(1)+exp(-2) = 54.598+2.718+0.135 = 57.452, so logsumexp = ln(57.452) = 4.0508. The loss is 4.0508 - 4.0 = 0.0508.

Trace of the second line: softmax gives p = [0.9503, 0.0473, 0.0024]. Those become the new logits. exp of each is [2.5867, 1.0485, 1.0024], which sums to 4.6375. The probability of class 0 is 2.5867 / 4.6375 = 0.5577, and the loss is -ln(0.5577) = 0.5837.

The gap between the true-class score and the others is what the loss responds to. Softmax squeezes that gap before the loss sees it, which is why the value moves so much.

What to check. If a batch's model outputs are non-negative and sum to 1 along the class dimension, softmax has been applied upstream. Look at the final layer of the model definition and remove any Softmax or softmax call before the loss. Apply softmax only at inference, when you need probabilities.

Alternatives and when each wins.

  • F.log_softmax followed by nn.NLLLoss: numerically the same as CrossEntropyLoss on logits. Choose it when you need the log-probabilities for something else too, such as a KL term or a custom loss.
  • Softmax, then -log of the selected probability by hand: avoid it. Taking the log of a probability that has underflowed to zero gives inf or NaN, while log-softmax stays finite.
  • nn.BCEWithLogitsLoss with one sigmoid per class: use it when classes are not mutually exclusive (multi-label). A softmax loss forces the classes to compete for probability mass, which is wrong for multi-label targets.
  • label_smoothing argument on CrossEntropyLoss (available since PyTorch 1.10): useful when you want the model to avoid very confident outputs. It changes the target, so the loss value is no longer the plain negative log-likelihood and you should not compare it with unsmoothed runs.
  • Probability targets passed to CrossEntropyLoss (also since 1.10): use them for soft or mixed labels. Hard integer labels are the usual case.

When not to use this approach. Do not use CrossEntropyLoss on logits when you need the model to output independent per-class probabilities, and do not feed it probabilities from a frozen pretrained head unless you have checked that the head is a logit layer. If a checkpoint already has a softmax at the end and you cannot retrain it, NLLLoss on its log-probabilities is a workable fallback, with clamping to avoid log(0).

Failure mode. The code runs without error and the loss decreases, just more slowly and to a higher floor than expected. The silence is the danger, which is why the output check above matters. A quick test is to pass a batch with a known class and confirm the loss matches the hand calculation.

Curated: · Written: · Reviewed:

Q&A-11

Walk me through what Dataset.__getitem__, the sampler, collate_fn and num_workers each control in a PyTorch DataLoader, and why two runs with the same seed can still produce different batches?

Answer

Each piece owns one decision. Dataset.__getitem__(i) maps one index to one sample. The sampler decides which indices are visited and in what order. collate_fn takes a list of samples and builds the batch. num_workers and pin_memory decide where that work runs and how the result reaches the GPU. shuffle=True is only shorthand for RandomSampler; passing shuffle together with an explicit sampler raises ValueError.

Where randomness enters. The default sampler draws its permutation from a generator when the epoch's iterator is created, in the main process. If you pass generator=g to the DataLoader, that generator is used; otherwise the global torch RNG is. So epoch order depends on the RNG state at that moment, and anything that consumed random numbers earlier shifts it. Augmentation inside __getitem__ runs in worker processes. Each worker's torch RNG is seeded from a base_seed drawn from the main process plus the worker id, so a rerun reproduces the same streams only if num_workers and the order of RNG draws are unchanged. Torch does not seed NumPy or Python's random in workers. Under fork, every worker inherits the same NumPy state, so np.random augmentations repeat across workers. The fix is a worker_init_fn that seeds NumPy and random from torch.initial_seed().

# PyTorch 2.x
def seed_worker(worker_id):
    s = torch.initial_seed() % 2**32
    np.random.seed(s); random.seed(s)

g = torch.Generator().manual_seed(0)
loader = DataLoader(ds, batch_size=32, shuffle=True, generator=g,
                    num_workers=8, pin_memory=True,
                    worker_init_fn=seed_worker, collate_fn=pad_collate,
                    persistent_workers=True, prefetch_factor=2)

Same seed, different num_workers: batches are dealt to workers in turn, and each worker's stream depends on the worker count, so augmented values differ even though the sample order matches. Same seed and same worker count still need deterministic kernels to match bitwise on GPU. That is a separate source of divergence.

Throughput. With num_workers=0, tokenization runs in the training loop and stalls the GPU. With workers, prefetch_factor=2 keeps 16 batches in flight at 8 workers, so host memory cost is roughly 16 batches. pin_memory=True makes the pin thread copy batches into page-locked memory, which allows .to(device, non_blocking=True) to overlap with compute. It does not speed up __getitem__. Diagnose with timing: if time spent waiting on the loader dominates the step, pinning changes nothing. Move tokenization offline or add workers. persistent_workers=True avoids respawning workers each epoch, but worker RNG state then carries across epochs, which changes the reproducibility argument.

Failure modes. Tokenizers with their own thread pools can deadlock or warn after fork. Ragged samples fail in the default collate with a stack error, so padding belongs in collate_fn, not __getitem__. A DistributedSampler needs set_epoch(epoch) or every epoch repeats the same order.

Curated: · Written: · Reviewed:

Q&A-12

A training run shows about 35% GPU utilisation, and timing shows roughly 60% of each step is spent inside next(loader). What does that tell you about where the time goes, and what would you change first?

Answer

The first 90 seconds. Time spent inside next(loader) is host time spent blocked until a batch is ready. The device cannot hide that wait, so on those steps the GPU is idle because it has no work. The conclusion is that the run is input-bound, not compute-bound. Before changing anything I would confirm the timer really wraps only next(), since a timer around an asynchronous forward pass measures kernel launch cost, not compute.

Mechanism. DataLoader workers prepare batches in parallel and place them in a prefetch queue. next() blocks only when that queue is empty. In steady state the batch rate is set by the workers: batches per second equals workers divided by the seconds one worker needs per batch, as long as that stays below the rate the device can consume.

Applied to the numbers (hypothetical). Assume a 100 ms step where 60 ms is loader wait and 40 ms is device work. The device could run 25 steps per second, but the loader delivers one batch per 100 ms, so 10 batches per second, and utilisation is about 10 / 25 = 40%, close to the observed 35%. If one worker needs 0.8 s per batch, 8 workers give 8 / 0.8 = 10 batches per second, which matches what the loop is getting. Going to 16 workers gives 20 batches per second, so utilisation can rise to about 20 / 25 = 80%, provided the machine has the cores to run them.

loader = DataLoader(ds, batch_size=64, num_workers=16, pin_memory=True,
                    persistent_workers=True)

The arithmetic stops working when CPU cores run out. If the host has 8 physical cores, 16 workers mostly add contention, and the per-batch time rises. So the check after the change is whether next() wait falls and CPU stays below saturation.

Trade-off. More workers cost CPU and host RAM, since each worker holds its own copies of batches in flight. If decoding or augmentation is the real cost and the cores are already busy, the better change is to move that work to the GPU or to store preprocessed data, not to add workers. pin_memory=True speeds the host-to-device copy but does not reduce loader wait, so it is not the first fix here.

Failure modes and detection.

  1. Workers added but wait unchanged: the bottleneck is storage or a shared lock in the dataset, visible as I/O wait in system metrics.
  2. Wait falls but utilisation stays low: a host-side sync (a .item() or .cpu() call per step) is idling the device. A profiler trace shows gaps between kernels, not loader wait.
  3. Throughput drops past the core count: CPU saturation and context switching, visible in per-process CPU usage.

Curated: · Written: · Reviewed:

Q&A-13

nvidia-smi shows 15.9 GB used on a 16 GB GPU while torch.cuda.memory_allocated() reports 9 GB, and the job still hits CUDA out of memory. Explain the gap, what memory_reserved means, and how you read the OOM message.

Answer

The 90-second answer. Three numbers, three owners. torch.cuda.memory_allocated() is the bytes held by live tensors. torch.cuda.memory_reserved() is the bytes PyTorch has obtained from the CUDA driver and keeps in its caching allocator, whether or not a tensor is using them right now. nvidia-smi reports what the driver sees for the process: the reserved pool, plus the CUDA context and library memory PyTorch does not count (the context is typically a few hundred MB and varies with driver and library versions), plus any other process on the card. So nvidia-smi is at least memory_reserved, which is at least memory_allocated. Whether a new request fits depends on the cached blocks, not on the driver's free total.

Mechanism. When a tensor is freed, its block goes back into the cache rather than to cudaFree, because cudaFree synchronizes the device and is slow. The allocator rounds requests up to 512-byte multiples, serves requests up to 1 MiB from a small pool and larger ones from a large pool, and splits and merges cached blocks. So reserved-but-unallocated memory is normal, and usually wanted.

import torch
x = torch.empty(256 * 2**20, dtype=torch.uint8, device='cuda')  # 256 MiB
print(torch.cuda.memory_allocated() / 2**20)  # 256.0
del x
print(torch.cuda.memory_allocated() / 2**20)  # 0.0
print(torch.cuda.memory_reserved() / 2**20)   # still about 256: cached, not returned
torch.cuda.empty_cache()
print(torch.cuda.memory_reserved() / 2**20)   # about 0

(Expected output for one process, not a benchmark.)

Reading the OOM message. An abridged example, with hypothetical figures:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB. GPU 0 has a total capacity of 15.56 GiB of which 12.00 MiB is free. Of the allocated memory 9.10 GiB is allocated by PyTorch, and 5.90 GiB is reserved by PyTorch but unallocated.

Tried to allocate is the single failing request, not the total. is free is the driver's view, so it ignores the cache. Compare the two PyTorch figures. Here 5.90 GiB is cached but unusable for a 40 MiB request, which points to fragmentation: no cached block is large enough or splittable enough. The first fix is PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, which lets segments grow instead of stranding space, or changing the allocation pattern (for example, sequence lengths that vary every step). If reserved-but-unallocated is small and allocated sits near capacity, it is a real capacity problem: smaller batch, activation checkpointing, or lower precision for activations.

Trade-off. empty_cache() returns cached memory to the driver, which helps other processes on the card, but the next allocations pay for cudaMalloc again. Calling it each iteration slows training. Use torch.cuda.max_memory_allocated() for the peak that actually matters, and reset_peak_memory_stats() to measure one phase.

Failure modes. (1) Sizing the batch from nvidia-smi, which includes context and cache, and halving it for no reason. Detect: compare nvidia-smi with memory_reserved() and max_memory_allocated(). (2) Variable tensor shapes fragmenting the cache. Detect: reserved-but-unallocated grows across steps while allocated stays flat. (3) Keeping the loss graph or per-step tensors alive, which raises allocated every step. Detect: memory_allocated() logged each iteration climbs monotonically.

Curated: · Written: · Reviewed:

Q&A-14

Under torch.autocast on CUDA with default settings, what dtype does torch.matmul produce for float32 inputs?

Answer

Short answer. float16. Under torch.autocast('cuda') with default settings, torch.matmul is on autocast's lower-precision op list, so both float32 CUDA inputs are cast to float16 before the kernel runs, and the result is float16. Passing dtype=torch.bfloat16 to autocast changes the reduced type to bfloat16.

Assumptions: PyTorch 2.x, a CUDA device, float32 CUDA tensors, default cache_enabled, and no custom autocast policy. Confirm the op list for your release in the autocast documentation before relying on it.

# PyTorch 2.x, CUDA
import torch
a = torch.randn(4, 4, device='cuda')
b = torch.randn(4, 4, device='cuda')
with torch.autocast('cuda'):
    c = torch.matmul(a, b)
print(a.dtype, c.dtype)   # torch.float32 torch.float16

The inputs stay float32 outside the block. Autocast decides per operator call, so the same tensor can feed a float16 matmul inside the block and a float32 op outside it. For leaf tensors that require grad, autocast caches the cast within one autocast region (cache_enabled=True by default), so a weight used by several matmuls is converted once.

Why float16 and when to choose bfloat16. float16 has 5 exponent bits and 10 mantissa bits, so its largest finite value is 65504 and it carries roughly three decimal digits. bfloat16 has 8 exponent bits, the same range as float32, and 7 mantissa bits. Pick bfloat16 when range matters and the GPU supports it (Ampere and later). Using float16 for training needs a gradient scaler, because small gradients underflow to zero.

Failure modes and how to detect them.

  1. Overflow to inf. Hypothetical example: a = torch.full((1, 2), 300.) and b = torch.full((2, 1), 300.). The float32 result is 180000.0, but under autocast the float16 result is inf, because 180000 exceeds 65504. Detect it with torch.isfinite(c).all() at module boundaries or in a forward hook that fails loudly. Activations from normalised inputs rarely hit this, so it tends to appear only on unusual data.

  2. Gradient underflow during training. Gradients below about 6e-8 (the smallest float16 subnormal) flush to zero, and training stalls without any error. Detect it by logging gradient norms and the fraction of exact zeros per parameter. Mitigate with torch.amp.GradScaler, and watch for repeated skipped steps caused by inf or nan gradients.

  3. Accumulated rounding error. A matmul over a long reduction dimension can lose several digits relative to float32, and the error shows up as drift in loss or metrics, not as a crash. Detect it by running a float32 reference on a sampled batch and comparing with torch.testing.assert_close using tolerances chosen for float16, not the float32 defaults.

  4. Dtype leaking downstream. The float16 output flows into later ops. Code that assumes float32, such as a float32 accumulator or a hand-written numerical routine outside autocast's coverage, can lose precision silently. Detect it with dtype assertions on intermediate tensors in unit tests.

Operational cost. The mixed-precision path roughly halves activation memory for matmul-heavy layers and uses tensor cores, but the gains depend on shapes and hardware, so measure them on your model rather than quoting a number. The added casts also cost time on small matrices, where autocast can be slower than plain float32.

Curated: · Written: · Reviewed:

Q&A-15

A float16 loop calls clip_grad_norm_(params, 1.0) between scaler.scale(loss).backward() and scaler.step(optimizer). With a true gradient norm of 0.8 and the default scale, what update is applied, and what does the scaler do on a step where one gradient is inf?

Answer

Ask first what the clipping threshold is measured against. clip_grad_norm_ sees whatever sits in .grad at the moment it runs. If it runs right after backward, those gradients are still multiplied by the scale S. With the default init_scale of 2**16 = 65536, a true norm of 0.8 becomes a scaled norm of 0.8 * 65536 = 52428.8.

Exact working for the wrong order.

scaler = torch.amp.GradScaler('cuda')  # init_scale 2**16, growth_factor 2.0, backoff_factor 0.5, growth_interval 2000
with torch.autocast('cuda', dtype=torch.float16):
    loss = criterion(model(x), y)
scaler.scale(loss).backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # wrong: sees scaled grads
scaler.step(optimizer)
scaler.update()

clip_grad_norm_ computes coef = max_norm / (total_norm + 1e-6) = 1 / 52428.8 ≈ 1.91e-5 and multiplies every gradient by it. The scaled norm is now 1.0. scaler.step then divides by S = 65536, so the applied gradient norm is 1 / 65536 ≈ 1.53e-5 instead of the intended 0.8. The step is about 52,000 times smaller than the clipping rule specifies, and it stays that small for as long as S is large. Nothing raises, and the loss curve just looks slow.

Correct order. Call scaler.unscale_(optimizer) before clip_grad_norm_. unscale_ divides the gradients by S in place and records found_inf for that optimizer. The clip then sees true gradients. scaler.step must not unscale again, and it does not.

Overflow step, with the correct order. Suppose one parameter's gradient overflows to inf during backward. unscale_ marks found_inf. clip_grad_norm_ gets a non-finite total norm, which can turn finite grads into NaN, and that is harmless because scaler.step checks found_inf and skips optimizer.step, so the weights and optimizer moments are untouched. scaler.update then halves the scale: 65536 to 32768. The growth counter resets. If the next steps are finite, the scale stays at 32768 until 2000 clean steps have passed.

Failure modes.

  1. Clipping before unscale_ looks like a learning-rate problem: the loss falls very slowly and the grad norm log reads about 1.0 every step. Detect it by logging the norm after unscale_.
  2. Calling unscale_ and then scaler.step is fine, but calling unscale_ twice divides by S twice and shrinks every update.
  3. A skipped step still advanced BatchNorm running statistics in the forward pass, so the discarded batch is not fully discarded.
  4. Repeated overflow that keeps halving the scale toward 1 means the forward pass itself is non-finite. Raising init_scale hides the symptom.

Curated: · Written: · Reviewed:

Q&A-16

A training loop wraps its model in torch.compile and the batches have varying sequence lengths. What triggers a recompile in that setup, and when is dynamic=True a better choice than the default?

Answer

The 90-second answer. torch.compile captures the tensor operations a function runs into a graph and caches that graph behind guards: conditions on the inputs it was compiled for, such as shape, dtype, stride and the values of Python constants. A call that fails the guards triggers a new compile. With varying sequence lengths, the default compiles the first length as a constant. The first batch of a new length fails the shape guard, so Dynamo recompiles, and by default marks that dimension dynamic, so later lengths reuse one graph. The cost is one or two extra compiles and a slower first step each time. dynamic=True compiles a symbolic graph from the start, so there is one compile up front, but the kernels are less specialised. My default for varying lengths would be to pad to a small set of buckets, then use the default compile, and reach for dynamic=True only when the lengths are too many to bucket.

# PyTorch 2.x
model = torch.compile(model)                 # specialises, then goes dynamic on the changed dim
# or
model = torch.compile(model, dynamic=True)   # one symbolic graph from the start

Mechanism. A guard is a cheap check run on every call. If the input's sequence dimension is 128 and the graph was built for 128, the check passes. A batch of 96 fails it. Python ints passed in as arguments are also specialised, so a changing counter can recompile on every call. Dynamo caps the number of cached variants per code object (torch._dynamo.config.cache_size_limit; the default has been 8 in recent 2.x releases, so check your version). Past the cap it runs eagerly and warns, which looks like an unexplained slowdown.

Trade-off, with hypothetical numbers. Suppose one compile takes 40 s and the loader produces 12 distinct lengths. Specialised, that is about 480 s of compile time over the run, paid once per length. Dynamic, it is one compile, but each step may run slower because the generated kernels cannot assume a fixed size. If a step takes 50 ms specialised and 60 ms dynamic, dynamic pays off only after about 480 / 0.01 = 48,000 steps of spread-out compiles, which is a fair test for your run length. Bucketing to 12 lengths gives 12 compiles and static kernels, which often beats both.

Failure mode. Timing the first epoch includes compilation. Warm up the graph for every bucket you use, then measure steady state. Also check the recompile count with TORCH_LOGS=recompiles, because a silent hit on the cache limit looks like a regular slowdown.

Curated: · Written: · Reviewed:

Q&A-17

A run saved only model.state_dict() every 1,000 steps and crashed at step 40,000. After resuming from step 40,000 the loss rises for a few hundred steps before settling. Explain which parts of the training state the checkpoint was missing, and how each one produces that bump.

Answer

The 90-second answer. A model state_dict restores the function, not the run. The bump after resume has a specific cause: the first optimizer steps read state that differs from the uninterrupted run. Adam's moment estimates were not saved, so exp_avg and exp_avg_sq restart at zero and bias correction restarts with them. Early updates are then sized by the gradient alone rather than by the averaged history, which moves weights more than the reference run does. The scheduler, if rebuilt, restarts at its base learning rate. The AMP scaler, if rebuilt, restarts at its default scale and may overflow and skip steps before it grows back. The step counter, if reset to zero, replays warmup and checkpoint intervals on the wrong schedule.

Weights are only part of it. The checkpoint should hold the following, and resume should restore them in this order:

# PyTorch 2.x
state = {
    'model': model.state_dict(),      # parameters and registered buffers
    'optimizer': opt.state_dict(),    # exp_avg, exp_avg_sq, step per param; param_groups (lr, betas)
    'scheduler': sched.state_dict(),
    'scaler': scaler.state_dict(),    # scale and growth tracker
    'step': step,                     # optimizer steps, not micro-batches
    'rng': torch.get_rng_state(),
}
torch.save(state, tmp_path); os.replace(tmp_path, path)

Load the model first, because optimizer.load_state_dict matches saved state to parameters by position in model.parameters(). Then load the optimizer, scheduler, scaler and RNG. Plain tensor attributes such as self.mask = torch.ones(...) are absent from state_dict and reset to their constructor values, so register any tensor that is state as a buffer.

RNG state covers shuffling, dropout and augmentation. If it is not restored, the resumed run draws different dropout masks and a different sample order, so the two curves diverge even when the weights match.

Failure modes and how you catch them.

  1. Dropped optimizer moments. Loss jumps at the resume boundary and the update norm spikes for roughly the first 100 to 1,000 steps. Detect by logging the update norm on both sides of the boundary.
  2. Scheduler reset. The logged learning rate drops back to base at step 40,000. Detect by logging optimizer.param_groups[0]['lr'] across the boundary.
  3. Scaler reset. Several skipped steps appear in the first few iterations after resume. Detect by counting skipped steps when the scale is logged.
  4. Step counted in micro-batches. With 8 accumulation steps, a per-micro-batch counter runs the schedule 8 times fast and misaligns every checkpoint interval.
  5. Mid-epoch data position. Without the batch index or a deterministic sampler seed for (seed, epoch), the resumed run either replays the epoch from the start or draws a different order.

Test. On CPU with torch.use_deterministic_algorithms(True), train 200 steps, checkpoint at 100, resume in a fresh process, and compare losses for steps 101 to 200. The curves should match exactly. On GPU, compare within a tolerance, since some kernels are nondeterministic.

Trade-off. Checkpointing every epoch is cheap but loses up to an epoch of work on a crash. Checkpointing every N steps bounds the loss but adds I/O, and it makes the data-position bookkeeping mandatory.

Curated: · Written: · Reviewed:

Q&A-18

What does load_state_dict(strict=False) do with keys that are missing from or unexpected in the checkpoint, and what does it still refuse to do?

Answer

The 60-second answer. With the default strict=True, load_state_dict requires the checkpoint's keys and the module's keys to match exactly. Any difference raises RuntimeError and names the missing and unexpected keys. With strict=False, it loads every key that matches by name, skips the rest, and returns a named tuple with missing_keys and unexpected_keys so the caller can see what was skipped. It does not relax shape checking. A key with the same name but a different shape still raises RuntimeError.

# PyTorch 2.x
import torch
from torch import nn

src = nn.Linear(4, 2)
sd = src.state_dict()              # keys: weight, bias
sd['extra'] = torch.zeros(1)       # a key the target does not have

dst = nn.Linear(4, 2)
r = dst.load_state_dict(sd, strict=False)
print(r.missing_keys, r.unexpected_keys)   # [] ['extra']

r = dst.load_state_dict({'weight': torch.zeros(2, 4)}, strict=False)
print(r.missing_keys)                      # ['bias'], weight was loaded

dst.load_state_dict({'weight': torch.zeros(3, 4)}, strict=False)  # RuntimeError: size mismatch

The practical consequence is that strict=False makes a partial load silent unless you inspect the result. A fine-tuning run that renames a head will leave the new head at its random initialisation, and nothing fails. Check missing_keys against what you expected to be restored. Use strict=False for deliberate partial loads, such as loading an encoder into a model with a different head, and keep the default everywhere else, because it catches a wrong checkpoint before training starts.

Curated: · Written: · Reviewed:

Q&A-19

A DDP job runs 8 processes on one 8-GPU node with per-rank batch size 32, and its loss curve differs from a single-GPU run at batch 32. Explain what each process does, when the gradient all-reduce happens, why DistributedSampler is required, and what the effective batch size is?

Answer

The 90-second answer. Each DDP process owns one GPU, one full replica of the model, and one shard of the data. Forward runs locally on every rank. During backward, each parameter's gradient is averaged across ranks with an all-reduce, so after backward() every rank holds identical gradients, and the local optimizer step keeps the replicas identical. The data sharding is what makes this one larger batch rather than the same batch eight times. The effective batch is per-rank batch times world size, not per-rank batch alone, and I would confirm that before quoting any convergence or throughput number.

# PyTorch 2.x, launched with torchrun --nproc_per_node=8 train.py
import os, torch, torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, DistributedSampler

dist.init_process_group('nccl')
torch.cuda.set_device(int(os.environ['LOCAL_RANK']))
model = DDP(build_model().cuda(), device_ids=[torch.cuda.current_device()])
sampler = DistributedSampler(dataset, shuffle=True, seed=0)
loader = DataLoader(dataset, batch_size=32, sampler=sampler)  # per rank
for epoch in range(E):
    sampler.set_epoch(epoch)
    for x, y in loader:
        loss = criterion(model(x.cuda()), y.cuda())  # mean over local 32
        opt.zero_grad(set_to_none=True)
        loss.backward()  # gradient all-reduce overlaps with this
        opt.step()

Mechanism. DDP registers autograd hooks on the parameters. Gradients are grouped into buckets (default bucket_cap_mb of 25) in roughly reverse parameter order. When every gradient in a bucket is ready, DDP launches an asynchronous all-reduce for it while autograd keeps computing earlier layers, so communication overlaps backward. The reduction sums and then divides by world size, so it is an average. With a mean loss per rank and equal shard sizes, the averaged gradient equals the gradient of the mean loss over the global batch of 8 × 32 = 256 samples. If the loss is a sum, the gradient is scaled by world size, and a learning rate tuned on one GPU becomes 8× too large without any error.

Arithmetic. Take 100,000 samples on 8 ranks. A single GPU at batch 32 runs 3,125 steps per epoch. Under DDP each rank gets 12,500 indices, so it runs ceil(12,500 / 32) = 391 steps per epoch, and each step averages 256 samples. Same data, roughly 8× fewer optimizer steps, effective batch 256. If the hyperparameters were tuned at batch 32, whether the learning rate should scale is an empirical question. Nothing in the execution model answers it.

Why DistributedSampler. Without it, every rank runs a plain shuffled DataLoader. With the same seed, all eight ranks see the same batches, so the all-reduce averages eight copies of one gradient and you have paid for eight GPUs to do one GPU's work. With different seeds, some samples are seen by several ranks and others by none, with no coverage guarantee. DistributedSampler partitions one permutation across ranks (rank r takes indices r::world_size). It pads the index list to a multiple of world size, so 100,001 samples become 100,008 indices and 7 samples appear twice. Forgetting set_epoch(epoch) reuses the same order every epoch, because the shuffle seed is seed plus epoch.

Failure modes, and how you catch each.

  1. Divergent control flow or unused parameters. If one rank skips a backward or a parameter on some iterations, its bucket never fills and the all-reduce waits. The symptom is a hang until the NCCL watchdog times out, or an error when find_unused_parameters is False. Detect it from the watchdog message naming the stalled collective, and by logging per-rank step counts.
  2. BatchNorm sees 32 samples, not 256. Each rank computes its own batch statistics. The model trains on different normalisation than the single-GPU baseline. Detect it by comparing running_mean across ranks. The fix is SyncBatchNorm, which costs a synchronisation per BN layer; measure that before adopting it.
  3. Metrics from rank 0 only. Validation accuracy on one shard gets reported as the whole set. Sum correct counts and totals with all_reduce before dividing.
  4. Replicas drift after wrapping. DDP broadcasts rank 0's parameters at construction. Seeding or mutating models per rank afterwards breaks equality, and the all-reduce only keeps gradients equal. Checksum weights across ranks at checkpoint time.

DDP fits a model that fits on one GPU and needs more data throughput. Once model state itself stops fitting, sharded schemes change the memory picture, and I would not claim DDP addresses that.

What is guaranteed. Identical averaged gradients on every rank after backward, given equal shard sizes and a mean-reduced loss. Bucket size, overlap ratio and step time are observations about this hardware and this model.

Curated: · Written: · Reviewed:

Q&A-20

A ResNet trained with DDP on 8 GPUs, with 8 images per GPU, reaches lower validation accuracy than the single-GPU run at the same global batch of 64. Explain the mechanism and your fix.

Answer

Clarify first: the model has BatchNorm layers, the global batch is 64 in both runs, and the single-GPU run sees all 64 images in every BN layer. Under DDP each process is a full replica that only sees its own 8-image shard.

Mechanism. In training mode, BatchNorm normalizes each channel with the mean and variance of the current local batch. On one GPU that is 64 samples. On each DDP rank it is 8. Two things follow. The normalized activations, and therefore the forward pass, are computed from noisier statistics, and the gradient noise changes with them. Second, the running_mean and running_var buffers are updated from 8 images per rank. DDP's default broadcast_buffers=True then copies rank 0's buffers to every other rank at each forward. The running statistics that evaluation uses therefore describe rank 0's shard, not the dataset. Validation depends on which images landed on rank 0 in the last few steps, and it drifts when the sharding changes.

Fix 1: SyncBatchNorm. Convert the model so each BN layer all-gathers per-channel sums and sums of squares across ranks during forward. Every rank then normalizes with statistics over all 64 samples, and the running buffers are identical on every rank without any broadcast.

# PyTorch 2.x, launched with torchrun, NCCL backend
model = nn.SyncBatchNorm.convert_sync_batchnorm(model).to(local_rank)
model = DDP(model, device_ids=[local_rank])

The trade-off is cost and coupling. Each BN layer adds one collective per forward pass, so a network with many BN layers on a slow interconnect loses throughput, and the step time should be measured before and after. The architecture and any pretrained weights stay unchanged.

Fix 2: a batch-independent norm. GroupNorm or LayerNorm compute statistics per sample, so the number of GPUs no longer affects the forward pass and no collective is needed. The cost is that the model changes. Pretrained BatchNorm weights no longer load into the same layers, the network must be retrained, and BN's regularizing noise is gone, which can cost accuracy on some vision tasks.

For this job, SyncBatchNorm is the smaller change if the goal is to match the single-GPU run. GroupNorm is the better choice only when the model is being trained from scratch and the per-GPU batch will stay small.

Check. Compare running_mean for one BN layer across ranks after a few hundred steps. Without SyncBatchNorm the values differ between ranks. With it they match to floating-point tolerance.

Curated: · Written: · Reviewed:

Q&A-21

A forward pass through a Sequential of Conv2d, MaxPool2d and Linear layers raises a RuntimeError about mat1 and mat2 shapes not being multipliable at the first Linear. Walk me through how you find the first layer where the shape is wrong, and where you would print shapes.

Answer

The 60-second answer. Read the error's dimensions before anything else. A message like mat1 and mat2 shapes cannot be multiplied (32x1600 and 1024x128) (wording varies across PyTorch versions, so read the numbers) means the Linear's input has last dimension 1600 while its weight expects 1024. nn.Linear(in_features, out_features) stores a weight of shape (out, in) and computes x @ W.T, so mat2 here is the transposed weight, and 1024 is the in_features the layer was built with. The batch dimension (32) is fine. The bug is the feature count arriving from upstream. I would not open the Linear's code. I would find the last layer whose output does not match what the next layer expects, which is usually upstream of the one that raised.

Trace the outputs. Register a forward hook on every leaf module and record the output shape. The hook for the failing layer never fires, because forward raised before returning, so the last recorded line is the failing layer's input.

# PyTorch 2.x
import torch

def trace_shapes(model, x):
    rows = []
    def make_hook(name):
        def hook(mod, inp, out):
            rows.append((name, tuple(inp[0].shape), tuple(out.shape)))
        return hook
    handles = [m.register_forward_hook(make_hook(n))
               for n, m in model.named_modules() if not list(m.children())]
    try:
        with torch.no_grad():
            model(x)
    except RuntimeError as e:
        print('forward failed:', e)
    finally:
        for h in handles:
            h.remove()
    for r in rows:
        print(r)

For a 28x28 input with Conv2d(1, 32, 3), MaxPool2d(2), Conv2d(32, 64, 3), MaxPool2d(2), the spatial sizes go 28 to 26 to 13 to 11 to 5. Channels are 64, so the flattened size is 64 x 5 x 5 = 1600. A Linear(1024, 128) therefore fails, and the trace shows a (32, 64, 5, 5) tensor reaching a Flatten that yields (32, 1600). The hand arithmetic and the trace should agree. If they do not, the trace wins.

Failure modes. (1) Missing batch dimension: a single sample (28, 28) passed without unsqueeze(0) gives a 2D input where the first dim is read as the batch, so the error names the wrong axis. Check x.shape first. (2) view(-1) or reshape on the wrong dims, which silently reorders features rather than failing. Compare against nn.Flatten() and check the flatten start dim. (3) Hand-computed sizes drifting after a padding or stride change. Detect with a dummy forward at construction time, not by recalculating.

Trade-off. nn.LazyLinear infers in_features on the first call and removes this class of error at construction, but it also hides the mismatch you are looking for. I would keep explicit sizes in code under review.

Curated: · Written: · Reviewed:

Q&A-22

Your training loss turns NaN at step 300. What do you check first, and what does torch.autograd.set_detect_anomaly tell you that the loss value does not?

Answer

First I check the data, because it is the cheapest check and it rules out a whole class of causes. I scan one batch before the forward pass:

assert torch.isfinite(x).all() and torch.isfinite(y).all(), (x.isfinite().all(), y.isfinite().all())

A NaN in a feature or label appears every time that sample is drawn, and no learning rate fixes it.

If the data is clean, I check the learning rate. A rate that is too high makes the weights grow until a forward value overflows to inf. Then operations such as inf minus inf or 0 times inf produce NaN. The usual signature is a loss that climbs for some steps, or a gradient norm that grows, before it turns NaN.

Next I check the loss code for log of zero and division by zero. Typical causes are log of a probability that reached 0, a division by a sum that reached 0, or log(softmax(x)) computed by hand. I prefer nn.CrossEntropyLoss on raw logits, which applies log-softmax internally, over a manual softmax followed by log.

Anomaly detection finds the operation. With torch.autograd.set_detect_anomaly(True), or the context manager torch.autograd.detect_anomaly(), backward raises a RuntimeError at the first function that produces NaN. Its traceback points to the forward line that created that node. The loss value alone only says that something upstream went wrong. Anomaly mode records extra information per operation and slows training, so I enable it for a few steps that reproduce the failure, then turn it off.

Gradient clipping is a fix only when gradients are finite but too large. clip_grad_norm_ returns the total norm and rescales finite gradients down to max_norm. A NaN norm stays NaN after rescaling, so clipping does not repair it. I pass error_if_nonfinite=True so a non-finite norm stops the run at the step where it occurs, and I log the returned norm to tell the two cases apart.

Curated: · Written: · Reviewed:

Q&A-23

A 30-layer plain MLP with ReLU activations has a loss that barely moves from its starting value over the first thousand steps. How would you use per-layer gradient norms to tell vanishing from exploding gradients, what norm profile would you expect from a healthy network, and which change would you make for each pattern?

Answer

The 90-second answer. I would not start with one global gradient norm, because it hides where the signal dies. I would log the norm of each weight matrix's gradient after backward() and before zero_grad(), then read the profile from the output back to the input. Vanishing shows as norms that fall roughly geometrically toward the input. Exploding shows as norms that rise toward the input, or as a few layers sitting orders of magnitude above the rest. A healthy deep network keeps the per-layer norms within about one order of magnitude across depth, and the update-to-weight ratio in a plausible band (about 1e-3 is a common rule of thumb, not a law). The fix follows the pattern: initialisation and activation choice for decay, residual paths and normalisation for depth, a lower learning rate, warm-up or clipping for explosion.

def layer_profile(model):
    rows = []
    for name, p in model.named_parameters():
        if p.grad is not None and p.dim() > 1:  # weight matrices only
            g = p.grad.norm().item()
            w = p.data.norm().item()
            rows.append((name, g, w, g / w))
    return rows

loss.backward()
for name, g, w, ratio in layer_profile(model):
    print(f'{name:20s} grad={g:.2e} weight={w:.2e} ratio={ratio:.2e}')
optimizer.step()
optimizer.zero_grad()

Why the decay happens. Backprop multiplies the incoming gradient by each layer's Jacobian. For a ReLU layer, the second moment of the activation is halved, and nn.Linear defaults to a uniform draw with variance 1/(3·fan_in). The second-moment gain per layer is therefore fan_in × (1/(3·fan_in)) × ½ = 1/6, and the gradient norm scales by about √(1/6) ≈ 0.41 per layer. Over 30 layers that is 6^−15 ≈ 2e-12. This is a hand estimate that ignores biases and treats the layers as independent. With sigmoid the situation is worse: its derivative is at most 0.25, so 0.25^30 ≈ 9e-19 before the weights even enter.

Expected profile (hypothetical figures). Suppose the output-layer gradient norm is 1e-2. With default init and ReLU, the layer 15 norm from the output is about 1e-2 × 0.41^14 ≈ 4e-8, and the input layer is about 2e-14. With He init (nn.init.kaiming_normal_(w, nonlinearity='relu')), the per-layer gain is about 1, so norms stay within a small factor across depth. A deliberately bad initialisation with per-layer gain 1.5 instead gives about 1.5^29 ≈ 1.3e5 growth toward the input, which is the explosive signature.

Failure modes and how to detect them. A global norm near 1e-4 looks the same for a shallow network with a small learning rate and a deep network with a dead input. Dead ReLUs produce exact zeros that resemble vanishing, so I log the fraction of zero gradient entries per layer. A layer with large weights can show a large gradient yet a tiny relative update, which is why the ratio matters. Under torch.autocast or GradScaler, the norms carry the loss scale unless you unscale first.

Changes, matched to the pattern. For decay: He initialisation, then ReLU or GELU in place of sigmoid. For depth: residual connections, which give the gradient an identity path, and LayerNorm, which does not depend on batch statistics the way BatchNorm does. For explosion: a lower learning rate or a warm-up schedule first, with clip_grad_norm_ as a guard rather than the cure. Residuals and normalisation change the model's function, not just its optimisation, so I would rerun the profile after each change and compare the ratios against the baseline.

Curated: · Written: · Reviewed:

Q&A-24

Why does model.to('cuda') leave an optimizer built before the move usable, while a buffer you saved a reference to before the move can still sit on the CPU? Also, what error does a CPU tensor raise when it meets a CUDA weight?

Answer

The 60-second answer. Tensor.to(device) returns a new tensor when the device changes and leaves the original untouched, so the result must be assigned. Module.to(device) mutates the module in place and returns self. For parameters it keeps the same nn.Parameter object and replaces its storage, which is why an optimizer built before the move still works. Buffers behave differently: the module replaces the entry in its buffer dictionary, so a reference captured earlier still points at the old CPU tensor. A CPU operand meeting a CUDA weight fails in the dispatcher with 'Expected all tensors to be on the same device', before any kernel runs.

Mechanism. Module._apply walks the children, then for each parameter computes fn(param) and assigns the result to param.data, keeping the Parameter object. For each buffer it does self._buffers[key] = fn(buf), which creates a new object. torch.__future__.get_overwrite_module_params_on_conversion() defaults to False, which is why parameter identity survives.

import torch
lin = torch.nn.Linear(4, 2)
opt = torch.optim.SGD(lin.parameters(), lr=0.1)
bn = torch.nn.BatchNorm1d(2)
running_mean_ref = bn.running_mean  # captured before the move

lin.to('cuda'); bn.to('cuda')
lin.weight is opt.param_groups[0]['params'][0]  # True: same Parameter object
lin.weight.device                # cuda:0
running_mean_ref.device          # cpu: the buffer was replaced, this reference is stale
bn.running_mean.device           # cuda:0

t = torch.zeros(3)
t.to('cuda')                     # returns a new tensor; t is still on cpu
lin(torch.zeros(1, 4))           # RuntimeError: Expected all tensors to be on the same device

A plain attribute such as self.mask = torch.ones(d) is invisible to Module.to(). It stays on the CPU and fails only on the first forward. Registering it with self.register_buffer('mask', ...) puts it under _apply.

Failure modes, and how you catch each.

  1. Unregistered tensor attribute. The forward raises a device error on a line that never touches the module's parameters. Detect: list devices over named_parameters() and named_buffers(). A plain attribute appears in neither list.
  2. Stale buffer reference. A hook, logger or helper keeps bn.running_mean from before the move. Its updates land on a CPU copy that the model no longer uses, and nothing raises. Detect: re-read the buffer through the module on each use, or assert ref.device == next(model.parameters()).device at the top of the loop.
  3. Partial move. A new submodule is added after model.to(device). Its parameters start on the CPU, and the error appears at the first mixed op. Detect: print each parameter's device after construction.

Fix. Move the model once, before building the optimizer. Register buffers with register_buffer. Create per-step tensors with device= and move batches with x = x.to(device, non_blocking=True), reassigning the result.

Curated: · Written: · Reviewed:

Q&A-25

What state does Adam keep for each parameter, how does AdamW apply weight decay differently from Adam with weight_decay, and why would you put biases and norm parameters in a group with no decay?

Answer

The 60-second answer. Adam keeps two tensors per parameter, each the same shape as the parameter: exp_avg, the running mean of gradients (first moment m), and exp_avg_sq, the running mean of squared gradients (second moment v). It also keeps a step counter step for bias correction. For a parameter of n elements that is 3n floats of state, so Adam costs about three times the memory of plain SGD with no momentum. The state is created lazily on the first step() for each parameter that has a gradient. AdamW changes one line of the update: the decay term is applied directly to the weights instead of being added to the gradient. That makes the decay independent of the adaptive scaling.

# PyTorch 2.x, per parameter p with gradient g, step t
# m = b1*m + (1-b1)*g ; v = b2*v + (1-b2)*g*g
# mhat = m/(1-b1**t) ; vhat = v/(1-b2**t)
# Adam with weight_decay=wd:  g = g + wd*p          -> p -= lr*mhat/(sqrt(vhat)+eps)
# AdamW with weight_decay=wd: p = p - lr*wd*p       ; p -= lr*mhat/(sqrt(vhat)+eps)

With L2 penalty inside Adam, the decay term wd*p enters g, so it passes through the division by sqrt(vhat). A parameter with a large gradient history has a large vhat, so its effective decay shrinks. Parameters with small gradients get proportionally more decay. The decay you configured is not the decay you get. AdamW applies lr*wd*p outside the division, so every weight shrinks by the same fraction per step, scaled only by the learning rate. This is the Loshchilov and Hutter argument, and it is why AdamW's default is weight_decay=1e-2 while Adam's is 0.

Parameter groups let you give different settings to different tensors:

decay, no_decay = [], []
for name, p in model.named_parameters():
    if not p.requires_grad: continue
    (no_decay if p.ndim < 2 or 'norm' in name else decay).append(p)
opt = torch.optim.AdamW([
    {'params': decay, 'weight_decay': 0.01},
    {'params': no_decay, 'weight_decay': 0.0},
], lr=3e-4)

The ndim < 2 test is a heuristic: weight matrices and conv kernels are decayed, biases and norm gains and shifts are not. The reason is convention with some empirical support, not a theorem. Decaying a norm's gain pulls the scale of normalized activations toward zero, and decaying a bias pulls an offset that often has no regularizing role. Check the groups by printing names, because a missing 'norm' substring silently decays a LayerNorm. Frozen parameters are excluded by requires_grad, not by the optimizer.

Failure modes. (1) Passing model.parameters() as one group decays everything, and the loss curve shows no error, only worse norm behaviour. Detect it by printing the group sizes against your name filter. (2) Calling load_state_dict on an optimizer built with different group order maps the saved moments onto the wrong tensors, and the shapes may still match. Detect it by comparing the parameter count per group before and after loading.

Trade-off. Adam with L2 is still the right choice when you deliberately want the coupled behaviour. For most transformer and CNN training, AdamW plus groups is the default. Tell the interviewer which one you chose and why.

Curated: · Written: · Reviewed: