Skip to content
Tech Interview Prep
Technical interview guideNew

PyTorch Fundamentals for Training and Debugging

How PyTorch tensors, autograd, modules and the training loop work, and how to diagnose shape, device, dtype and gradient failures in training code for ML engineering interviews.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: PyTorch 2.x stable releases. Behaviour of torch.compile, DDP, autocast and GradScaler is stated for 2.x; check the release notes for the exact version a team runs and note the version in the answer..

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

MCQ-1A training loop runs one epoch of 1,000 batches on a freshly initialised linear model with `torch.optim.SGD(model.parameters(), lr=0.1, weight_decay=1e-4)` in PyTorch 2.x. Each iteration computes `loss = criterion(model(x), y)`, calls `loss.backward()`, then `optimizer.zero_grad()`, then `optimizer.step()`. After the epoch, what do the model's weights hold?
MCQ-2A training script runs `x = torch.randn(32, 128)`, then calls `x.to('cuda')` on its own line without assigning the result, then runs `out = model(x)` where `model` already lives on the GPU. What does the forward call do?
MCQ-3A regression head returns `pred` of shape (4, 1) and the batch targets `y` have shape (4,). The loop runs `loss = nn.MSELoss()(pred, y)` with default reduction, and for this batch `pred` holds [1, 2, 3, 4] and `y` holds [1, 2, 3, 4]. What loss value does the training log show?
MCQ-4A validation pass runs `with torch.no_grad():` over a model containing `nn.BatchNorm1d(64)` and `nn.Dropout(0.3)`, and `model.eval()` is never called. The script then saves `model.state_dict()` as the release checkpoint, and each validation batch has 128 samples. Which account of that checkpoint is right, and which fix keeps the trained statistics?
MCQ-5A detector fine-tuned from an ImageNet checkpoint trains on 4 GPUs with 2 images per GPU, so the global batch is 8, and its BatchNorm2d layers stay in training mode. Validation mAP swings by several points between epochs, and the target domain's image statistics differ from ImageNet's. The architecture must stay as pretrained. Which change addresses the batch-size noise?
MCQ-6A 10-class image classifier ends with `self.head = nn.Linear(256, 10)` followed by `torch.softmax(logits, dim=1)` inside `forward`. Training passes that output to `nn.CrossEntropyLoss()`, and the same `forward` is used for evaluation. Training loss stays at about 2.30 (ln 10) for the first thousand steps and barely moves. Which change fixes the training?
MCQ-7A 2024 checkpoint was saved with `torch.save(model)` from the old `train.py`, where `Encoder` was defined. The new code moved `Encoder` unchanged to `models/encoder.py`, and the serving team must load this file in the new codebase on PyTorch 2.x without retraining. Which way should it be loaded?
MCQ-8A 4-GPU DDP job trains a model whose forward pass runs an auxiliary head only when `batch.has_aux` is true. That flag is set per batch, so on some steps some ranks skip the head while others run it. At step 17, rank 2 never returns from its first backward while the other three wait in the gradient all-reduce. Which DDP setting addresses the cause?
MCQ-9A training job on one GPU shows about 30 percent utilisation. The DataLoader uses `num_workers=0`, and the per-sample transform decodes and resizes JPEGs on the CPU at about 5 ms per image. A batch of 64 takes roughly 320 ms to prepare, while the forward and backward pass takes about 130 ms. Which change should you make first?
MCQ-10A training job on a 16 GB GPU fails with CUDA out of memory on a 2 GB allocation, while torch.cuda.memory_allocated() reads 4 GB and torch.cuda.memory_reserved() reads 15.5 GB. What accounts for the roughly 11.5 GB difference between those two numbers?
MCQ-11A float16 training run uses `torch.amp.GradScaler` with default settings: init_scale 65536, growth_factor 2.0, backoff_factor 0.5, growth_interval 2000. The scaler has seen 1,500 consecutive iterations with finite gradients since its scale last changed, when one backward pass produces an inf gradient. What does the scaler do right after that iteration?
MCQ-12A training loop wraps its model in `torch.compile` with default settings. A custom batch sampler packs samples to a token budget, so batch sizes differ on nearly every step, between 9 and 40 samples, and the logs show a recompile each time a new size appears. The loss must average over real samples only, so padding is ruled out. Which change stops the per-batch recompiles?
MCQ-13A run trains with torch.optim.SGD(momentum=0.9) and a StepLR scheduler stepped once per epoch. The checkpoint saved at the end of epoch 30 holds only model.state_dict(). After resuming at epoch 31, the first loss is 2.9, against 1.4 just before the crash, and the learning rate is back at its base value. Which checkpoint change restores the exact training trajectory?
MCQ-14A ResNet trains with DistributedDataParallel on 8 GPUs with 8 images per GPU and `BatchNorm2d` layers. After an epoch, each rank's `running_mean` differs from the others. Which change makes each normalization use statistics from the full global batch of 64?
MCQ-15A float16 run with `torch.autocast` and `GradScaler` uses gradient accumulation over 4 micro-batches of 8 samples before each `scaler.step`. From step 340 the scale halves on every optimizer step, the reported gradient norm is inf, and the weights stay finite. Which check isolates the source before any change to the optimizer?
MCQ-16A 12-layer fully connected network with sigmoid activations and default PyTorch initialisation has per-layer gradient norms that fall from about 3e-2 at the last layer to about 4e-10 at the first, a ratio near 1e-8. The loss has barely moved after 2,000 steps. Which change addresses the cause?
MCQ-17A pre-norm transformer trains with `torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.1)` as a single group. After 10,000 steps the LayerNorm gains have fallen to about 0.4 from their initial 1.0, though their gradients stayed small and their loss contribution is flat. The attention and MLP weights look normal, and the gains still need to learn. Which change keeps the gains learning while the linear weights keep their decay?
MCQ-18A module stores two layers as `self.layers = [nn.Linear(4, 4), nn.Linear(4, 4)]` in `__init__`, and nothing else in the class registers them. What does `len(list(net.parameters()))` print for a freshly built `net`?
MCQ-19A module keeps a float32 running statistic `ema` of shape (d,) as a persistent buffer, updated in forward under `torch.no_grad()` during training. For serving, the team calls `model.to(dtype=torch.bfloat16)` so the Linear layers run in bf16, then reloads the checkpoint. Which change keeps `ema` at float32 after the cast and still round-trips it through `state_dict`?
MCQ-20A 4-GPU DDP job builds `DataLoader(ds, batch_size=8, shuffle=True, sampler=DistributedSampler(ds))` on every rank, and construction raises a ValueError before training starts. The loop runs 50 epochs, and each rank must see a disjoint shard that is reshuffled every epoch. Which change gives that?
MCQ-21A forward pass runs `x = torch.randn(4, requires_grad=True)`, then `h = x.exp()`, then `h.mul_(2)`, then `loss = h.sum()`. Calling `loss.backward()` raises a RuntimeError saying a variable needed for gradient computation was modified by an inplace operation. Which change removes the error and keeps the gradient of `x` correct?
MCQ-22A loop accumulates gradients over 4 micro-batches of 8 samples, calling `loss.backward()` on each micro-batch's mean loss with no division. It then runs `optimizer.step()` and `optimizer.zero_grad()` once after the fourth backward. What does `p.grad` hold at the step, relative to the gradient of the mean loss over all 32 samples?
MCQ-23A checkpoint was saved from a DistributedDataParallel model, so its keys start with `module.`. The new unwrapped model adds `self.adapter = nn.Linear(256, 256)` inside the encoder and widens `head` from 10 to 12 classes. Calling `load_state_dict(torch.load(path), strict=False)` raises a size mismatch on `head.weight`, and the encoder weights are not restored. You need the pretrained encoder loaded and the adapter and head left fresh. Which change does that?
MCQ-24A fine-tuning step computes `feats = encoder(x)` on a frozen backbone, then `loss = head(feats).sum()` where `head` is a trainable `nn.Linear`, and `loss.backward()` must update only the head. The team wraps the encoder call in `with torch.inference_mode():` to save memory, and the backward pass then raises a RuntimeError. Which change makes the backward pass run while the encoder still records no autograd graph?
MCQ-25A training script builds `h = w1 * x`, where `w1` is an `nn.Parameter` and `x` is a plain input tensor, then `out = h * w2` and `loss = out.sum()`. After `loss.backward()`, `w1.grad` and `w2.grad` are populated, but `print(h.grad)` shows `None` with a warning about a non-leaf tensor. The team needs dL/dh from this same run without changing the forward computation. What should they do?