Overview
Curated: · Written: · Reviewed:
Gradient descent and optimizers
Most interview questions on this topic come down to one question: given the loss surface and the update rule, what does each knob do to the trajectory? Learning rate, batch size, momentum, adaptive scaling and schedules all modify the same step, θ ← θ − η · g, applied to different estimates of g. Learn the step first and the optimizer zoo becomes a list of changes to two things: the direction and the size.
Why the update rule is the whole story
A loss L(θ) returns one number for how wrong the model is. Training changes the parameters θ to make that number small. Gradient descent uses only local information. At the current θ, it computes ∇L(θ), which points in the direction the loss rises fastest, and steps the other way.
For one weight w the rule is w_new = w_old − η · dL/dw. If the slope is positive, the loss grows as w grows, so w should shrink. If the slope is negative, w should grow. The gradient carries both direction and magnitude, so steep regions move faster than flat ones.
To see the rule run, take L(w) = w²/2, so g = w, and start at w = 4 with η = 0.5. Each step gives w ← w − 0.5·w = 0.5·w. The following snippet is Python 3.11 and was traced by hand:
def gd(w, eta, steps):
for _ in range(steps):
g = w
w = w - eta * g
return w
print(gd(4.0, 0.5, 3))
Trace: step 1 gives w = 4 − 0.5·4 = 2.0, step 2 gives 2.0 − 0.5·2.0 = 1.0, step 3 gives 1.0 − 0.5·1.0 = 0.5. The program prints 0.5.
Everything else on this page changes either g (how it is estimated, batched, averaged or rescaled) or η (how far to move). The habit that pays off in an interview is to name which of the two a proposed change touches.
Learning rate: the one knob that decides whether you land
On a one-dimensional quadratic L(w) = a·w²/2, the gradient is a·w and the update is w ← w(1 − ηa). Convergence needs |1 − ηa| < 1, so 0 < η < 2/a. For small η the iterate creeps monotonically toward zero. Between 1/a and 2/a it still converges, but it flips sign each step while shrinking. Above 2/a the magnitude grows every step.
Worked trace, with a = 4 and w0 = 1 (hypothetical figures):
| η | step factor 1−ηa | w after 3 steps | behaviour |
|---|---|---|---|
| 0.05 | 0.8 | 0.512 | slow monotone crawl |
| 0.25 | 0 | 0 | exact landing in one step |
| 0.4 | −0.6 | −0.216 | sign flips, shrinking |
| 0.6 | −1.4 | −2.744 | blow-up, since 0.6 > 2/4 = 0.5 |
For many dimensions the threshold is η_max = 2 / λ_max, where λ_max is the largest Hessian eigenvalue. That is why the same η can be stable for one layer and divergent for another. Show the arithmetic when you state the bound.
A weak answer says "too high a learning rate makes training unstable." A strong one gives the bound, explains that it depends on curvature, and says what changes when the curvature differs across layers.
Batch, stochastic and mini-batch: cost per step versus noise
Batch gradient descent averages the gradient over all N examples. It is exact, but each step costs O(N·d) for d parameters. Stochastic gradient descent uses one example per step: cheap and noisy, and without a decaying step size it never settles at a fixed point. Mini-batch sits between them, and in practice "SGD" on GPUs almost always means mini-batch.
The batch-size trade-off is cost per step against gradient noise. Doubling the batch halves the variance of the gradient estimate but doubles the work per step. Whether a bigger batch helps depends on whether you are compute-bound or step-bound. Large batches usually need a learning-rate adjustment, commonly linear scaling with batch size plus warmup, and they can generalise worse at the same epoch count.
Convex, non-convex, saddles and plateaus
On a convex loss every local minimum is global, so a small enough step converges to the optimum. Neural network losses are non-convex. The obstacles that matter there are saddle points, where the gradient is zero but the Hessian has both positive and negative eigenvalues, and flat plateaus where gradients are tiny. In high dimensions saddles are much more common than poor local minima, and mini-batch noise helps escape them.
What interviewers probe is whether you know that "stuck" usually means a plateau or a saddle. A weak answer says the optimizer "falls into a local minimum" and stops.
Momentum and Nesterov: accumulating direction
Momentum keeps a velocity: v ← β·v + g, then θ ← θ − η·v, with β typically 0.9. Components that oscillate across a ravine cancel between steps, while consistent components add up. The effective step along a steady direction is about 1/(1−β) times the raw step, so β = 0.9 gives roughly a tenfold boost.
Trace with a constant gradient g = 1 and v0 = 0 (hypothetical figures):
| step | v before | v after 0.9·v + 1 |
|---|---|---|
| 1 | 0 | 1.0 |
| 2 | 1.0 | 1.9 |
| 3 | 1.9 | 2.71 |
| 4 | 2.71 | 3.439 |
The velocity approaches the limit 1/(1−0.9) = 10 and the step per update grows toward η·10.
Nesterov momentum evaluates the gradient at the look-ahead point θ − η·β·v rather than at θ. The velocity gets corrected earlier, which usually reduces overshoot. With the same β, the gain is usually small, not a different regime.
Adaptive methods: AdaGrad, RMSProp, Adam and AdamW
Adaptive methods give each parameter its own effective step by dividing by a running measure of gradient magnitude.
- AdaGrad accumulates the sum of squared gradients. The denominator only grows, so the effective rate decays monotonically. This suits sparse features, but it can stall in long runs.
- RMSProp replaces the sum with an exponential moving average, so the rate no longer decays to zero.
- Adam combines momentum on the gradient (first moment,
β1 = 0.9) with RMSProp-style scaling (second moment,β2 = 0.999), plus bias correction for the first steps. Defaultη = 1e-3. - AdamW decouples weight decay from the adaptive update. In Adam, L2 regularisation folded into the gradient gets divided by the second-moment denominator, which weakens it for parameters with large gradients. AdamW applies
θ ← θ − η·λ·θas a separate term.
On the first step, bias correction makes m̂ = g and v̂ = g², so the update is about η · g / |g| = η, whatever the gradient's size (the 1e-8 epsilon is negligible here). Hypothetical check with η = 1e-3:
| gradient g | m̂ | v̂ | step η·m̂/(√v̂+ε) |
|---|---|---|---|
| 0.01 | 0.01 | 0.0001 | ≈ 0.001 |
| 100 | 100 | 10000 | ≈ 0.001 |
This is why Adam's early steps are roughly the same size for every parameter, and why its learning rate needs less tuning than SGD's.
The claim to defend: Adam stores two extra buffers per parameter, so optimizer state brings total parameter-related memory to roughly 3× the weights alone. In exchange you get fast early progress and less learning-rate sensitivity. AdamW is the right choice when you want weight decay to mean what it says.
When plain SGD with momentum still wins
For many vision models trained from scratch on long schedules, SGD with momentum and a step or cosine schedule often generalises better than Adam at the same accuracy budget. One explanation is that adaptive methods can land in sharper minima, but this is not settled. Plain SGD also uses less memory and has fewer knobs. Name the workload before naming the winner.
Schedules, warmup and decay
A fixed η is rarely best across a whole run. Warmup ramps η up from near zero over the first few thousand steps, which stabilises Adam's second-moment estimates and large-batch training. Decay then lowers η through step decay, cosine annealing or linear decay to zero. Warmup followed by cosine decay is a common default for transformers. A schedule is a function of step count, so state whether your batch size changes the step count.
Conditioning, feature scaling and vanishing or exploding gradients
When curvature differs widely across directions, the condition number κ = λ_max / λ_min sets how many steps plain gradient descent needs. Its per-step contraction is about (κ−1)/(κ+1), so the iteration count grows roughly in proportion to κ. Standardising inputs to zero mean and unit variance reduces κ for linear models, which is why scaling often matters more than the optimizer choice on tabular data.
In deep networks, gradients are products of per-layer factors. Factors below 1 compound toward zero and factors above 1 blow up. Residual connections, normalisation layers, careful initialisation and gradient clipping (commonly by global norm) are the standard responses.
Where the gradient comes from: backpropagation
Backpropagation applies the chain rule from the loss back to each parameter, reusing activations stored during the forward pass. The backward pass costs a small constant multiple of the forward pass, not one pass per parameter. That is how ∇L is available for millions of parameters in a single sweep. Be ready to derive the chain rule on a two-layer network, because interviewers ask for it.
Checking convergence and debugging divergence or stalls
Watch three signals: training loss, gradient norm and parameter update size. A loss that becomes NaN or inf within a few steps usually means η is too large, the inputs are badly scaled, or an operation is numerically unstable. A loss that barely moves while the gradient norm stays healthy usually means η is too small or the schedule decayed too early. The update-to-weight ratio is a useful cross-check, with roughly 1e-3 as a common rule of thumb. First rule out a bug: if the model cannot overfit a single batch, the optimizer is not the problem.
Worked example: one update by hand on linear regression
Fit ŷ = w·x + b to one example (x, y) = (2, 5), starting from w = 1, b = 0, with squared loss L = (ŷ − y)² and η = 0.1. (Hypothetical figures.)
- Prediction:
ŷ = 1·2 + 0 = 2. Errorŷ − y = −3. - Gradients:
∂L/∂w = 2·(ŷ − y)·x = 2·(−3)·2 = −12.∂L/∂b = 2·(ŷ − y) = −6. - Update:
w ← 1 − 0.1·(−12) = 2.2.b ← 0 − 0.1·(−6) = 0.6. - New prediction:
2.2·2 + 0.6 = 5.0. Loss is now 0.
Repeat with η = 0.3. Step 1 and 2 are unchanged. Then w ← 1 + 0.3·12 = 4.6 and b ← 0 + 0.3·6 = 1.8. The new prediction is 4.6·2 + 1.8 = 11.0, which overshoots the target 5 by 6. Doubling the step past the point where it lands flips the error from −3 to +6, so this single example shows the same overshoot the quadratic bound predicts.
What interviewers probe is whether you can derive the gradient, apply the update, and predict what a larger step does. A weak answer quotes the update rule without computing it.