Overview
Curated: · Written: · Reviewed:
PyTorch fundamentals for training and debugging
ML engineering interviews rarely ask you to recite API names. They ask you to write a training loop from memory, read a traceback that says RuntimeError: expected device cuda:0 but got cpu, and explain why memory climbs every epoch or why the loss went NaN at step 300. Almost all of these reduce to four things: what a tensor is in memory, which operations are recorded for autograd, what nn.Module tracks, and what order the training loop runs in. This guide builds those four, then the debugging method that locates failures in each.
The mental model to carry: a tensor is a view onto a storage plus metadata, and each differentiable op records a node in a graph that backward walks once and then throws away. Shape, device and dtype bugs are mismatches in that metadata. Memory and gradient bugs are about what the graph and the loop keep alive.
Tensors, dtypes, devices and broadcasting
A tensor is (storage, shape, strides, offset, dtype, device). Views such as x.view, x.transpose, slicing and expand change the metadata and share storage, so writing through one view changes the others. reshape returns a view when the strides allow it and a copy otherwise. transpose usually leaves the tensor non-contiguous, and .view will then raise where .reshape succeeds. Calling .contiguous() forces a copy when a kernel needs dense memory.
import torch
x = torch.arange(6).reshape(2, 3) # int64, contiguous
y = x.t() # view, shape (3, 2), strides (1, 3)
y.is_contiguous() # False
y.view(-1) # RuntimeError: view size is not compatible
y.reshape(-1) # copies, because a view is impossible
Dtype promotion follows the operands. An integer tensor combined with a float tensor yields float. A Python scalar does not widen a tensor's dtype within its category, so int_tensor * 2.5 promotes to float but float16_tensor * 2.0 stays float16. torch.tensor([1.0, 2.0]) defaults to float32, and torch.get_default_dtype() reports the default. Half precision tensors on CPU have limited kernel coverage, which is a common source of surprise.
Broadcasting aligns shapes from the trailing dimension. A size-1 dimension stretches, and a missing leading dimension is treated as 1. It is silent, which is why it hides bugs:
pred = torch.randn(32, 1)
target = torch.randn(32)
loss = ((pred - target) ** 2).mean() # (32,1) - (32,) broadcasts to (32,32)
No error is raised. Every prediction is compared with every target, and the loss is wrong but finite. MSELoss warns about mismatched sizes for this reason. The check is to assert shapes at module boundaries (assert pred.shape == target.shape) rather than trust the loss to complain.
Device mismatch raises RuntimeError when a binary op receives tensors on different devices. The fix is to move inputs and the model consistently with .to(device), and to build tensors created inside the forward pass on the input's device (torch.zeros_like(x) or device=x.device), not hard-coded 'cuda'.
Autograd and the dynamic graph
PyTorch builds the graph as ops run. Each op whose inputs require grad stores a backward function that refers to its inputs and saved intermediates. requires_grad propagates forward: an output requires grad if any input does. Autograd supports only floating point and complex tensors, so an integer index tensor can never require grad.
loss.backward() walks the graph from the scalar loss, accumulates gradients into .grad of leaf tensors, and then frees the saved buffers. A second backward through the same graph raises RuntimeError: Trying to backward through the graph a second time. Pass retain_graph=True only when you truly need it, because it keeps memory alive.
Leaf tensors are those the user created (parameters, inputs with requires_grad=True). Non-leaf tensors are intermediates. Their .grad is None after backward unless you call retain_grad(). This is a frequent interview question: printing x.grad for an intermediate and getting None is correct behaviour.
w = torch.tensor(2.0, requires_grad=True)
h = w * 3 # non-leaf
h.retain_grad()
out = h ** 2
out.backward()
w.grad # tensor(36.) dout/dw = 2*h*3 with h=6
h.grad # tensor(12.) dout/dh = 2*h
Three ways to stop tracking, and they differ:
torch.no_grad()disables graph construction in its block. Outputs haverequires_grad=False. Use it for evaluation when you still need ordinary tensors.torch.inference_mode()is stricter and faster. Tensors created inside cannot later be used in autograd. Use it for pure inference.detach()returns a tensor sharing storage with no graph history. Use it to cut a path, for example when storing values for logging or feeding a target network. It does not change the global mode.
In-place operations break backward when a saved tensor is overwritten. Autograd tracks a version counter, and backward raises an error such as one of the variables needed for gradient computation has been modified by an inplace operation. The usual fix is an out-of-place op. Keep this in mind for ReLU with inplace=True after operations whose backward needs their input.
nn.Module, parameters, buffers and state_dict
A module registers a child or parameter when you assign it as an attribute in __init__ (self.fc = nn.Linear(...), self.w = nn.Parameter(...)). Plain tensors and plain Python lists are not registered. Storing layers in a regular list means the optimizer never sees their parameters and they are missing from state_dict. Use nn.ModuleList or nn.Sequential. The same applies to nn.ParameterList and nn.ModuleDict.
Buffers are tensors that are part of module state but not trained, such as BatchNorm running statistics. They are created with register_buffer(name, tensor, persistent=True). Setting persistent=False excludes them from state_dict.
state_dict() contains parameters and persistent buffers, keyed by dotted names. It does not contain optimizer state (momentum, Adam moments), the scheduler, the scaler or RNG state. load_state_dict is strict by default and raises on missing or unexpected keys. A frequent failure is saving from a DistributedDataParallel wrapper, which adds a module. prefix to every key.
model.train() and model.eval() flip a flag on every submodule. They change the behaviour of Dropout and BatchNorm (batch statistics versus running statistics). They do not disable gradients. Evaluating without no_grad or inference_mode silently builds graphs and holds memory.
Loss functions take raw logits. nn.CrossEntropyLoss applies log-softmax internally, so passing softmax probabilities gives a valid-looking but wrong loss that trains poorly. nn.BCEWithLogitsLoss fuses the sigmoid and is numerically safer than applying sigmoid and then BCELoss. Under autocast, the fused form is also the one to prefer.
The training loop, data pipeline and checkpointing
The order is fixed:
for x, y in loader:
x, y = x.to(device), y.to(device)
optimizer.zero_grad(set_to_none=True)
logits = model(x)
loss = loss_fn(logits, y)
loss.backward()
optimizer.step()
scheduler.step() # once per step or per epoch, as configured
zero_grad must happen before the backward pass, or gradients from earlier batches add in. Since PyTorch 2.0 the default is set_to_none=True, which frees the gradient tensors rather than zeroing them. Clipping, if used, goes after backward and before step: torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm).
Gradient accumulation simulates a larger batch. Divide the loss by the number of accumulation steps so the summed gradient matches the mean, and step only every k batches:
k = 4
for i, (x, y) in enumerate(loader):
loss = loss_fn(model(x), y) / k
loss.backward()
if (i + 1) % k == 0:
optimizer.step()
optimizer.zero_grad(set_to_none=True)
Forgetting the division makes the effective learning rate k times larger. With a loss that is a sum rather than a mean, the scale changes differently, so check which reduction you use.
DataLoader with num_workers > 0 runs Dataset.__getitem__ in worker processes. Each worker gets its own copy of the dataset object. Torch seeds each worker's RNG from a base seed, but NumPy and Python's random are not reseeded that way. Augmentation that uses NumPy therefore repeats across workers and epochs unless you seed them in worker_init_fn. Lists or caches built in __init__ are duplicated per worker, which multiplies memory. persistent_workers=True avoids restarting workers each epoch. pin_memory=True helps host-to-device copies with non_blocking=True.
With DistributedSampler, call sampler.set_epoch(epoch) each epoch or every rank shuffles identically.
A resumable checkpoint stores more than weights:
torch.save({
'model': model.state_dict(),
'optimizer': optimizer.state_dict(),
'scheduler': scheduler.state_dict(),
'step': step,
}, path)
Resuming without the optimizer state restarts Adam's moments and changes the trajectory. Resuming without the step or scheduler state restarts the learning rate schedule. Since PyTorch 2.6, torch.load defaults to weights_only=True, so checkpoints containing arbitrary Python objects need weights_only=False explicitly, and only from trusted sources.
Memory, mixed precision, compilation and distribution
The CUDA caching allocator keeps freed blocks reserved for reuse. nvidia-smi shows reserved memory, which is why it stays high after tensors are freed. torch.cuda.memory_allocated() reports live tensor bytes and torch.cuda.max_memory_allocated() reports the peak. Out-of-memory errors that happen in a loop usually come from keeping the graph alive: appending loss (not loss.item()) to a Python list retains every graph. Accumulating outputs without detach() does the same.
Mixed precision uses torch.autocast, which runs matmuls and convolutions in lower precision and keeps reductions and some losses in float32. float16 needs a GradScaler because small gradients underflow. bfloat16 has float32's exponent range and usually does not need one.
scaler = torch.amp.GradScaler('cuda')
with torch.autocast('cuda', dtype=torch.float16):
loss = loss_fn(model(x), y)
scaler.scale(loss).backward()
scaler.unscale_(optimizer) # before clipping
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
scaler.step(optimizer)
scaler.update()
Skipping unscale_ before clipping clips scaled gradients, which is a real and quiet bug.
torch.compile traces Python code into graphs. A graph break happens on constructs it cannot capture, such as data-dependent control flow (if x.sum() > 0), some print or .item() calls inside the model, and unsupported ops. Each break splits the graph and adds overhead. Recompilation happens when guards fail, typically when input shapes change. Varying batch sizes trigger recompiles; dynamic=True or padding to fixed shapes reduces them. Diagnose with TORCH_LOGS=graph_breaks,recompiles. The first compile is slow and should be excluded from benchmarks.
Distributed Data Parallel keeps a full replica per process (one per GPU, launched with torchrun). During backward, gradients are grouped into buckets, and each bucket is all-reduced as soon as its gradients are ready, overlapping communication with computation. Every rank therefore must produce gradients for the same parameters in the same order. A parameter that is unused in some forward pass stalls the reduction and can hang the job. find_unused_parameters=True fixes that at a cost in speed. Build replicas with the same initial weights (seed them, or broadcast from rank 0, which DDP does on construction). Save model.module.state_dict() from rank 0 only, and load it into the unwrapped module.
Debugging: shape, device, NaN and vanishing gradients
Work from the cheapest check to the most expensive, and change one thing at a time.
Shape and dtype failures: print shape, dtype and device at each module boundary on one batch, and assert them. Use torch.func-free debugging: a forward hook prints shapes without editing the model.
def show(name):
def hook(mod, inp, out):
print(name, tuple(out.shape), out.dtype, out.device)
return hook
for name, m in model.named_modules():
m.register_forward_hook(show(name))
Remove hooks after debugging, since they run on every forward.
Device failures: check every tensor the forward pass touches, including those created inside forward and the loss target. The error message names the two devices, which points at the source.
NaN and infinity: find the first step where the loss is non-finite, then the first op that produces NaN. torch.autograd.set_detect_anomaly(True) raises at the backward op that created NaN, with the forward traceback. It is slow, so enable it only for a reproducible failing step. Common causes are log(0) (use log-softmax or BCEWithLogitsLoss), division by a near-zero norm, a learning rate that is too high, and float16 overflow under autocast. Check inputs with torch.isfinite(x).all() before the model.
Gradient health: after backward() and before step(), compute the L2 norm of each parameter's gradient and print it per layer. Vanishing gradients show as norms that fall by orders of magnitude toward the input layers (for example from 1e-1 at the head to 1e-8 at the first block). Exploding gradients show as norms that grow rapidly or spike to inf. Remember that parameters with grad is None are absent from the graph, which is a bug to investigate, not a zero. The usual remedies are initialization, normalization layers, residual connections, gradient clipping and a learning rate that is too high or too low for the problem.
For each failure, state the check that would distinguish it from the nearest alternative. A NaN that appears only with autocast points at dtype range. A NaN that appears at a fixed step points at data. A loss that is finite but flat points at a broadcasting or label bug, the shape check and a single-batch overfit test come first.
What a weak answer sounds like: "I'd lower the learning rate and see." What a strong one sounds like: "The loss is finite but stuck at ln(10). I'd check that logits, not softmax outputs, reach CrossEntropyLoss, then overfit one batch to confirm the graph updates."