Skip to content
Tech Interview Prep
Technical interview guideNew

Gradient Descent and Optimizers

How gradient descent minimises a loss by stepping against its gradient, and how learning rate, batch size, momentum, adaptive methods and schedules change how that descent behaves in practice. Built for ML engineers who must explain and compare optimizers in interviews.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: Current to PyTorch 2.x (torch.optim defaults: Adam lr 1e-3, betas (0.9, 0.999), eps 1e-8, weight_decay 0; AdamW weight_decay 0.01) and TensorFlow 2.x / Keras 3 optimizers. Convergence statements assume smooth losses unless stated; learning-rate limits quoted for quadratic losses with curvature lambda..

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

MCQ-1A model minimises `L(w) = (w - 3)^2` with plain gradient descent and learning rate 0.1. At the current weight `w = 1` the loss is 4 and the derivative `dL/dw` is -4. What is the weight after one update step?
MCQ-2A quadratic loss has curvature λ = 50 along its steepest direction, and plain gradient descent with learning rate 0.05 makes the loss grow on every step. Which learning rate range keeps the loss converging?
MCQ-3A dense model trains on 200,000 examples on one GPU with a per-step cost of 40 ms at batch 64 and 52 ms at batch 256. The run uses batch 64 with learning rate 0.01, and the per-weight gradient noise has standard deviation 4.0 at that batch size. The team wants the noise halved and the time per epoch cut by at least 20 percent, with each epoch keeping today's learning-rate-to-batch ratio. Which setting does this?
MCQ-4On L(w) = w^2 the true gradient is 2w, but each step sees 2w plus zero-mean noise with standard deviation 1. Run A uses a constant learning rate of 0.1 and run B uses a rate that decays as 1/t, both starting at w = 5. After 10,000 steps, which description of their long-run behaviour is right?
MCQ-5A two-parameter model has stalled at training loss 0.41 with gradient norm 3e-6. Its Hessian there has eigenvalues +4.2 and -0.6, and the team's plan is to halve the learning rate and run 20,000 more steps. What does the Hessian say will happen, and what should change?
MCQ-6A small MLP trained with plain SGD at learning rate 0.01 has held its training loss near 0.6 for 3,000 steps. The gradient norm is about 3e-5, and the loss is neither spiking nor turning into NaN. Which change matches this symptom?
MCQ-7A scalar weight trains with heavy-ball momentum, `v <- beta*v + g` and `w <- w - eta*v`, with beta = 0.9, eta = 0.01 and v = 0 before the first update. The gradient g is a constant 0.3 on every step. What is the total change in w after five updates?
MCQ-8A narrow valley loss L(w) = 50w^2 is minimised with Nesterov momentum, beta = 0.9 and learning rate 0.01. Mid-run the weight is w = 1.0 and the velocity is v = 2.0, where the update is v = beta*v + eta*gradL(w - beta*v) followed by w = w - v. Which point does this step evaluate the gradient at?
MCQ-9An AdaGrad run uses learning rate 0.1 and a negligible epsilon. One scalar weight receives a gradient of exactly 1.0 on every step, and the training loss stops improving after step 100 while that weight is still far from its minimum. What is the size of that weight's update on step 101?
MCQ-10A non-convex network trained with AdaGrad for 200,000 steps has per-parameter steps that have shrunk to almost nothing, even though the loss is still falling. The team switches to RMSProp and keeps the same per-parameter scaling. What does RMSProp change so the effective step size stops shrinking?
MCQ-11A dense PyTorch `torch.optim.Adam` run (lr 0.001, betas 0.9 and 0.999, eps 1e-8, moments starting at zero) has one scalar weight whose gradient is exactly 0 on steps 1 and 2 and 0.2 on step 3. How far does that weight move on step 3?
MCQ-12A network trained with Adam carries an L2 penalty of 1e-4 on every weight. Over the same run, a weight whose loss gradients sit near 10 barely shrinks, while one whose gradients sit near 0.01 shrinks visibly. The team wants both weights to decay by the same fraction per step. What should they change?
MCQ-13A team trains a ResNet-50 image classifier on a 1.2M-image dataset and has budget for a 200-trial sweep per optimizer. Their goal is the lowest held-out top-1 error, not the fastest fall in training loss, and the schedule is cosine decay. Which configuration should they commit to?
MCQ-14A 350M-parameter transformer is trained from random initialisation with Adam at a peak learning rate of 1e-3 and batch size 16,384, with no warmup. The loss diverges within the first 50 steps. Why does adding a linear warmup over 2,000 steps stop the divergence?
MCQ-15A fine-tune runs for exactly 4,000 optimizer steps: 400 steps of linear warmup to a peak learning rate of 2e-4, then cosine decay to the final step. The team keeps the checkpoint with the lowest held-out loss. A previous run on the same data ended at a final rate of 1e-6, and its held-out loss rose over the last 800 steps as the weights drifted. Which final learning rate should this cosine schedule reach at step 4,000?
MCQ-16A least-squares fit uses two centred features: income in dollars with standard deviation 20,000, and age in years with standard deviation 10. Their correlation is 0.5. The Hessian of the mean squared loss is X^T X divided by n, and gradient descent uses a step near 1/λmax. After you standardise both features to unit variance, what is the condition number of this Hessian?
MCQ-17A four-layer scalar chain uses a sigmoid after every layer, with each weight equal to 1, and the loss gradient at the output is 0.8. At the current inputs the local sigmoid derivatives are 0.25, 0.1, 0.25 and 0.1, counting from the output layer back toward the input. What gradient reaches the first layer's weight?
MCQ-18A recurrent model's gradient at one step is the vector (300, -4, 2), and the first component is an exploding spike. The update must be capped at a threshold of 10 without changing which way the step points. Which action does that?
MCQ-19A dense network does 12 GFLOP per example in its forward pass and feeds one scalar loss through about 50 million parameters. One training step runs on a batch of 32 examples, and the backward pass for dense layers costs about twice the forward pass. Counting forward and backward together, how many GFLOP does one training step take, excluding the optimizer update?
MCQ-20A fine-tuning run has held its training loss flat for 3,000 steps, and the training-batch gradient norm is about 4e-4. Validation loss has still dropped from 0.512 to 0.498 over the last 2,000 steps, and the stopping rule ends the run after 1,000 steps without a new validation low. What should the run do next?
MCQ-21A linear model `ŷ = w*x` is fit with mean squared error on a batch of four inputs, x = 10, 20, 30 and 40. Raising the learning rate from 0.0005 to 0.002 makes the loss go to NaN within a few steps. Which of these learning rates is the largest one guaranteed not to diverge on this batch?
MCQ-22A two-hidden-layer ReLU network is initialised so every pre-activation in the first hidden layer is negative, which makes every hidden activation exactly 0 and the loss stop moving. What value does the gradient of the loss with respect to a first-layer weight take at this step?
MCQ-23A model `ŷ = w·x` (no bias) is trained on a batch of two points, (x=1, y=2) and (x=2, y=5), with `L = (1/n)Σ(ŷ − y)²` and learning rate `η = 0.1`. The batch is then split into two micro-batches of one point each, and gradients are accumulated over both before a single update, with the loss still divided by the full batch size `n = 2`. Starting from `w = 0`, what is `w` after one such update?
MCQ-24A hand-written backward pass for a float64 network is checked with central differences at h = 1e-5, using relative error |ga - gn| / max(|ga|, |gn|) per block. Measured errors: softmax head 3e-10, convolution 4e-3, output projection 2e-1, ReLU-adjacent layer 3e-5, embedding 1e-2. Which block's analytic gradient do you accept as correct?
MCQ-25A team fine-tunes a pretrained 350M-parameter encoder for binary classification with batch 32, AdamW, a peak learning rate of 1e-3, no warmup and linear decay. In the first 40 steps the training loss rises from 0.69 to 3.1 and the gradient norm grows about 50-fold, then the loss sits at 0.69 for the rest of the run. What should they change first?