Skip to content
Tech Interview Prep
Technical interview guideNew

Gradient Descent and Optimizers

How gradient descent minimises a loss by stepping against its gradient, and how learning rate, batch size, momentum, adaptive methods and schedules change how that descent behaves in practice. Built for ML engineers who must explain and compare optimizers in interviews.

Read
14 min
Practice MCQs
25
Interview Q&A
25
Edition
v1
Editorial status
Reviewed

Scope: Current to PyTorch 2.x (torch.optim defaults: Adam lr 1e-3, betas (0.9, 0.999), eps 1e-8, weight_decay 0; AdamW weight_decay 0.01) and TensorFlow 2.x / Keras 3 optimizers. Convergence statements assume smooth losses unless stated; learning-rate limits quoted for quadratic losses with curvature lambda..

Interview Q&A

Treat each question like a live interview question: answer out loud first (structure, assumptions, tradeoffs), then open the model answer to spot gaps and rehearse a tighter follow-up.

Curated: · Written: · Reviewed:

25 questions · Flow follows Relevance until this set has a learning order.

Q&A-1

Suppose L(w) = 10(w - 3)^2 and you run w <- w - 0.1 * dL/dw from w = 2.0. Does the run converge, and what is the largest learning rate that still converges for this loss?

Answer

Short answer: the run does not converge. With eta = 0.1 the iterates oscillate between 2 and 4 forever. The largest learning rate that converges is just under 0.1, and the boundary value 0.1 itself fails.

Assumptions: one scalar parameter, a smooth convex loss, full-batch gradients, and a fixed learning rate eta.

The gradient is dL/dw = 20(w - 3), so the update is w <- w - eta * 20(w - 3). Write e = w - 3 for the distance to the minimum. Then e <- e(1 - 20 eta). Each step multiplies the error by the factor 1 - 20 eta, so convergence depends on whether that factor has magnitude below 1. In general, for L = a(w - w*)^2 the factor is 1 - 2a eta, and the stability bound is eta < 1/a.

Trace at eta = 0.1, starting from w = 2.0:

step  w     grad = 20(w-3)   w_next = w - 0.1*grad   L(w)
0     2.0   -20.0             4.0                      10.0
1     4.0    20.0             2.0                      10.0
2     2.0   -20.0             4.0                      10.0

error factor: 1 - 20*0.1 = -1

The factor is exactly -1, so the error flips sign and never shrinks. The loss stays at 10 and never decreases.

What happens for other values:

  • 0 < eta < 0.1: the factor lies in (-1, 1), so the error shrinks geometrically. At eta = 0.05 the factor is 0 and w reaches 3 in one step (2 - 0.05 * -20 = 3). Values near 0.1 converge slowly and oscillate with decaying amplitude.
  • eta = 0.1: factor -1, no convergence, oscillation at constant amplitude.
  • eta > 0.1: the factor has magnitude above 1, so the oscillation amplitude grows and the loss rises each step. At eta = 0.15 the factor is -2 and the error doubles in magnitude per step.

So the bound is 0 < eta < 0.1, and there is no separate threshold at 0.2. Any rate above 0.1 diverges.

The same bound scales with curvature. If a = 1, the factor is 1 - 2 * 0.1 = 0.8 and eta = 0.1 converges. This is why a rate tuned on one model can fail on another with a different loss scale.

Failure detection in practice: a loss that falls and then flattens suggests a workable rate. A loss that oscillates between the same values, as here, indicates a rate at or above the bound. Growing loss or NaN indicates a rate well above it. Adaptive optimizers such as Adam normalize each coordinate by a running gradient scale, so the effective step no longer follows this simple bound, and the base learning rate still needs tuning with the same kind of check.

Curated: · Written: · Reviewed:

Q&A-2

For gradient descent on a quadratic loss with curvature λ, what is the range of learning rates η that converges?

Answer

The bound is 0 < η < 2/λ. A strong answer states it at once, then adds the reason in a sentence.

For a loss L(w) = (λ/2)w², each step multiplies the error by (1 − ηλ). The error shrinks exactly when |1 − ηλ| < 1, which unpacks to 0 < η < 2/λ. Below that range the steps are small enough to close the gap; above it the factor's magnitude exceeds 1 and the iterate grows.

Worked check with λ = 4: the bound is η < 0.5. At η = 0.1 the factor is 0.6 and the error shrinks by 40% per step. At η = 0.6 the factor is −1.4 and the error grows by 40% per step, alternating in sign.

In a problem with several directions, the bound is set by the sharpest one, the largest curvature λ_max, so the condition becomes η < 2/λ_max. Real networks rarely have this value known in advance, which is one reason learning-rate warmup and schedules exist.

Curated: · Written: · Reviewed:

Q&A-3

You train on 10 million examples with a dense network on one accelerator. A 1,024-example batch takes 40 ms and a single example takes 35 ms. You cut the batch to 256 to fit a larger model's activations, and validation loss at a fixed epoch count gets worse. Walk me through the cost per step, per epoch and in utilisation, and what you would change before blaming the batch size.

Answer

Opening. The batch cut changes the update count and the gradient noise, not the number of examples seen. Per epoch every example is still touched once. Updates go from 9,766 to 39,063, and per-step gradient variance (for a mean loss, roughly sigma squared over B) is four times larger. Before blaming the batch size I would rescale the learning rate, because a fixed rate on four times as many noisier steps is a different optimiser.

Per step. The two timings give a linear fit, time(B) ≈ a + bB. From a + b = 35 ms and a + 1024b = 40 ms, b ≈ 0.0049 ms per example and a ≈ 35.0 ms. A 256-example step therefore costs about 36.2 ms. Fixed per-step cost dominates here, which is why small batches are inefficient. These are hypothetical figures from a two-point fit, not a measurement.

BatchUpdates per epochStep timeTime per epochExamples per secondGradient noise (relative)
1,0249,76640 ms9,766 x 40 ms ≈ 6.5 min25,6001
25639,06336.2 ms39,063 x 36.2 ms ≈ 23.6 min7,0704

Throughput at 256 is about 28% of the 1,024 run, and an epoch takes about 3.6 times as long. The cause is per-step overhead, not arithmetic.

Before blaming the batch size. Three candidate causes:

  1. The learning rate was not rescaled. For SGD with momentum and a mean loss, the linear scaling rule (Goyal et al., 2017) suggests lr / 4 as a starting point for a 4x smaller batch. For Adam-family optimisers, square-root scaling (lr / 2) is a common heuristic. Both are starting points only. The right value comes from a sweep at batch 256.
  2. Batch-norm statistics over 256 examples are noisier, and the running estimates used at eval time can lag. Compare validation with the running statistics against statistics recomputed over a pass of training data.
  3. The larger model changed capacity, so model size and batch size are confounded.

Run three controls: (a) batch 256 with a swept learning rate, (b) batch 256 at the original learning rate, (c) the original model at batch 1,024. If (a) closes the gap, the optimiser was mistuned. If the gap persists under (a), the model change is responsible.

Failure modes. A learning rate left at its 1,024 value with 256-example batches can oscillate. Training loss flattens and seed-to-seed variance grows. The learning-rate sweep detects this. Gradient accumulation to restore an effective batch of 1,024 matches the large-batch gradient only if no normalisation layer computes statistics per micro-batch. Batch norm does, so four accumulated 256 micro-batches produce different normalisation statistics from one 1,024 batch. Check the running statistics against a single-pass reference to confirm.

What I'd do. Use gradient accumulation to recover the effective batch if the normalisation layers allow it. Otherwise sweep the learning rate at batch 256. Judge the model change only after that.

Curated: · Written: · Reviewed:

Q&A-4

For a loss that is L-smooth and mu-strongly convex with L/mu = 100, how many steps of gradient descent with step 1/L cut the suboptimality by 10^6, and why does SGD with a constant step stop short of the minimum?

Answer

Start with the assumptions, because the rates depend on them. Smooth means the gradient is L-Lipschitz. Strongly convex with modulus mu > 0 means the loss has a unique minimiser x*, and the curvature is at least mu everywhere. With both, gradient descent using step 1/L contracts the suboptimality on every iteration:

f(x_k) - f* <= (1 - mu/L)^k (f(x_0) - f*)

The per-step factor is 1 - 1/kappa, where kappa = L/mu is the condition number. For kappa = 100 the factor is 0.99. Cutting the gap by 10^6 needs k >= ln(10^6) / (-ln 0.99) = 13.82 / 0.01005, about 1,375 steps. Doubling kappa roughly doubles that count, so the cost grows linearly in kappa and logarithmically in the accuracy you ask for.

If the loss is convex but not strongly convex, mu = 0 and the geometric factor disappears. The standard bound with step 1/L is

f(x_k) - f* <= L ||x_0 - x*||^2 / (2k)

This is O(1/k). With L ||x_0 - x*||^2 = 1 (a hypothetical normalisation), reaching 10^-6 takes 500,000 steps, about 360 times the strongly convex count. The gap now falls as a power law instead of a geometric decay, so each extra digit of accuracy costs ten times more steps.

SGD replaces the gradient with a sample g_t that is unbiased, E[g_t] = grad f(x_t), but has variance sigma^2 at the optimum. The gradient at x* is zero, yet individual sample gradients are not, so every step injects noise that does not shrink as the iterate approaches the minimiser. Take the quadratic f = (mu/2) x^2 with noisy gradient mu x + noise of variance s^2 and constant step eta. For small eta the stationary variance of x is about eta s^2 / (2 mu). The iterate keeps wandering in a ball whose size scales with eta, and running more steps does not shrink that ball. Halving eta halves the floor but doubles the steps needed to forget the starting point, since the contraction per step is about 1 - eta mu.

The fix is a decaying schedule with two conditions. Sum of eta_t must diverge, so the iterate can travel from any start to the minimiser. Sum of eta_t^2 must converge, so the accumulated noise stays bounded. eta_t = c/t satisfies both. For mu-strongly convex losses, eta_t = 1/(mu t) gives an expected suboptimality of O(1/t), which matches the deterministic O(1/k) rate for convex losses, not the geometric rate.

Two failure modes. First, a schedule that decays too fast, so the sum of steps is finite: training stalls at a loss above the minimum while the full-batch gradient norm is still large. Detect it by logging the full-batch gradient norm against the step count. Second, a schedule that decays too slowly: the validation loss keeps oscillating at the noise floor. Detect it by the spread of the loss over a window of steps. These guarantees are for convex losses. Neural network losses are non-convex, so they tell you what to expect from the schedule's shape, not a bound on a deep model's final loss.

Curated: · Written: · Reviewed:

Q&A-5

Why does gradient descent reach the global minimum on a convex loss but not on a neural network loss, and why are saddle points usually the bigger obstacle?

Answer

The 60-second answer. On a convex loss every local minimum is global. For an L-smooth convex loss, gradient descent with step size η ≤ 1/L satisfies f(x_k) − f* ≤ ‖x_0 − x*‖² / (2ηk), so the gap shrinks at a known rate. A neural network loss is non-convex. The same update rule still reaches a point where the gradient is zero, but that point may be a saddle or a poor minimum, and the rate bound no longer applies. In high dimensions the usual obstacle is a saddle, because a critical point is a local minimum only if every Hessian eigenvalue is positive.

Mechanism. At a critical point the gradient vanishes and the Hessian H decides the type of point: all eigenvalues positive is a local minimum, all negative is a local maximum, mixed signs is a saddle. Take f(x,y) = x² − y² at the origin. H = diag(2, −2) and the gradient is zero. Gradient descent started on the x-axis converges to the origin, but the y-direction is unstable: any small offset in y grows, so the iterate eventually leaves. For random symmetric matrices, the chance that all eigenvalues share one sign falls off quickly as the dimension grows, so high-dimensional critical points are mostly saddles. Dauphin et al. (2014) argued this case for neural network losses.

point type      Hessian eigenvalues    plain GD behaviour
local minimum   all > 0                converges to it
saddle          mixed signs            stalls if the gradient is tiny nearby
local maximum   all < 0                unstable, not reached from noise

Plain GD stalls near a saddle only when the gradient is small along the descent directions, which happens on plateaus. Stochastic gradient noise or a small deliberate perturbation pushes the iterate off a strict saddle. Lee et al. (2016) show that plain GD from a random initialisation almost surely avoids strict saddles. That is a probability statement about random starts, not a guarantee about your run.

What is lost.

  1. Global optimality. GD returns a stationary point, and the loss it reaches may be well above the global minimum.
  2. The rate bound. Convex smooth losses get O(1/k) on the objective. Non-convex smooth losses get only that the gradient norm falls below ε within O(1/ε²) steps.
  3. Uniqueness. Runs from different seeds can land in different minima with different test error.
  4. A usable step-size rule. η ≤ 1/L still holds locally, but L is a global constant that is large or unknown for a deep network, so it cannot be used as a recipe.

Trade-offs. Momentum and Adam change the path, not the guarantee. They help cross plateaus and ill-conditioned valleys, but neither proves global convergence. Newton's method is attracted to saddles unless the Hessian is modified, so a second-order method does not remove the problem.

Failure you can detect. A run whose loss flattens at a high value while the gradient norm is near zero has stopped at a stationary point. Estimate the smallest Hessian eigenvalue with Lanczos iteration. If it is negative, the run sits on a saddle. If restarts with different seeds and noise end at widely spread losses, the landscape is the limit, not the optimiser.

Curated: · Written: · Reviewed:

Q&A-6

Why does a gradient that is nearly zero on a flat plateau slow training to a crawl, even when the minimum is far away?

Answer

Start by saying the step is eta times the gradient, so any slowdown has to come from a small gradient. On a plateau the loss surface is nearly flat, so dL/dw is close to zero and w <- w - eta * dL/dw moves w by almost nothing, however far the minimum is. The learning rate does not rescue this: a larger eta multiplies a tiny gradient and still produces a tiny step. Assumptions: plain gradient descent on a smooth loss, with no normalization of the gradient.

A worked check with L(w) = 10(w - 3)^2, so dL/dw = 20(w - 3). At w = 3.5 the gradient is 10, and with eta = 0.01 the step is 0.1. At w = 3.001 the gradient is 0.02, and the step is 0.0002. The slowdown comes from the gradient itself, not from the optimizer's bookkeeping.

The most common source of a flat gradient in deep networks is saturation. A sigmoid or tanh unit with large-magnitude input has a local derivative near zero (sigmoid's derivative is s(1 - s), which is at most 0.25 and falls toward 0 as the output approaches 0 or 1). The gradient reaching earlier layers is multiplied by these factors, so it shrinks before the optimizer acts. Detection: log per-layer gradient norms and check whether the early layers are orders of magnitude below the last layer. Also record the fraction of activations near the saturated ends of the range, and the ratio of update size to weight norm per layer. A ratio that stays very small for many steps means the layer is not learning. The cost is wasted compute: the loss curve looks flat for many epochs, and the run gets blamed on the learning rate when the real problem is in the early layers.

Momentum helps when the gradient keeps the same sign across steps, as in a long shallow valley. It accumulates that consistent signal, so the effective step grows. It does not help when the gradient is tiny and noisy everywhere, or when signs flip from step to step, because the accumulated sum cancels out. Adaptive methods such as Adam and RMSprop divide each parameter's update by a running estimate of its gradient scale, so they can take meaningful steps even when raw gradients are small. The limit is epsilon. Adam's update is roughly m / (sqrt(v) + eps), and with the common default eps = 1e-8, parameters whose gradients are far below that threshold still barely move. So adaptive methods help when the plateau is a per-parameter scale mismatch, and fail when gradients are below epsilon across the board.

For a saturated network, the choice of fix depends on the cause. Use Xavier or Glorot initialization for tanh and sigmoid, and Kaiming initialization for ReLU, so activations start in their responsive range. Replace saturating units with ReLU or GELU where the architecture allows it. Add normalization layers such as batch or layer norm to keep pre-activation scales stable. Choose an optimizer last, and only after checking that the gradient is small for a reason the optimizer can compensate for.

Curated: · Written: · Reviewed:

Q&A-7

With heavy-ball momentum, v <- beta*v + g and w <- w - eta*v, what step does a consistent gradient g produce once v settles, and how should eta change when you raise beta from 0.9 to 0.99?

Answer

Assumptions: PyTorch-style heavy-ball with dampening 0, so v <- betav + g and w <- w - etav. The gradient g stays constant for many steps, and the curvature is roughly quadratic near the minimum.

Once v settles, v = g/(1-beta), so each step moves the weights by eta*g/(1-beta). In directions where the gradient keeps its sign, momentum acts as a learning-rate multiplier of 1/(1-beta). Raising beta from 0.9 to 0.99 multiplies that by 10. To keep the flat-direction step unchanged, eta drops from 0.01 to 0.001.

Trace with g = 1 and beta = 0.9:

k=1: v = 1.0 k=2: v = 0.91.0 + 1 = 1.9 k=3: v = 0.91.9 + 1 = 2.71 k=10: v = 6.513 limit: 1/(1-0.9) = 10

Stability is the part the retune does not address. On a quadratic with curvature lambda, the update becomes w_(k+1) = (1 + beta - etalambda) w_k - betaw_(k-1). Both roots stay inside the unit circle exactly when 0 < eta*lambda < 2(1+beta) and beta < 1. So the ceiling on eta is 2(1+beta)/lambda. At beta = 0.9 that is 3.8/lambda. At beta = 0.99 it is 3.98/lambda. For lambda = 50 the ceilings are 0.076 and 0.0796. For lambda = 500 they are 0.0076 and 0.00796.

The eta ceiling barely moves when beta rises, but the flat-direction step grows tenfold. That asymmetry is the real risk. Retuning eta to 0.001 keeps the effective step at 0.1 per unit gradient and keeps eta*lambda well under the ceiling for curvatures up to about 3980 at beta = 0.99. Keeping eta = 0.01 with beta = 0.99 diverges once lambda passes about 398.

Failure modes and detection:

  1. Divergence after raising beta with eta unchanged. Detect: the loss rises or goes NaN within tens of steps. Estimate the top Hessian eigenvalue with power iteration on Hessian-vector products and compare it with 2(1+beta)/eta.
  2. Oscillation near the ceiling. The roots become complex and the loss rings. Detect: log the loss every step and count sign changes of the gradient along the top Hessian eigenvector. Average loss can still fall while this happens.
  3. Stalls in flat directions after an incorrect retune. Detect: the per-parameter update norm stays near eta*|g|/(1-beta), far below what the loss curve needs.
  4. Resuming from a checkpoint with a different beta. The velocity buffer holds history built under the old beta. Reset it or warm it up over a few hundred steps.

Trade-off: on a convex quadratic, momentum leaves the minimizer and the asymptotic rate alone and changes the transient. Gradient noise is amplified by a similar 1/(1-beta) factor in the variance of the iterates, so high beta is a poor fit for noisy losses. Reduce beta when gradients are noisy or curvature varies sharply across parameters, and use the retuned eta only after checking the top eigenvalue.

Curated: · Written: · Reviewed:

Q&A-8

Classical momentum and Nesterov momentum both keep a velocity term. What exactly does Nesterov change, why does that reduce overshoot in a curved valley, and how much does the difference show up in practice?

Answer

The 60-second answer. Both methods keep a velocity v that decays by mu each step and accumulates the negative gradient. Classical (heavy-ball) momentum evaluates the gradient at the current weight w. Nesterov momentum first moves to the look-ahead point w + mu*v, which is where the accumulated velocity is about to take you, and evaluates the gradient there. The gradient at the look-ahead point already points back toward the valley floor when the velocity has carried you past it, so the update brakes earlier. The cost is one extra gradient reference per step, which is nothing in a framework that handles it for you.

classical:  v <- mu*v - lr*grad(w)         w <- w + v
nesterov:   v <- mu*v - lr*grad(w + mu*v)  w <- w + v

Trace. Hypothetical figures: L(w) = 5w^2 (curvature 10), lr = 0.01, mu = 0.9, w0 = 1, v0 = 0.

step  classical: grad, v, w          nesterov: lookahead, grad, v, w
1     10.0, -0.100, 0.900           1.000, 10.00, -0.100, 0.900
2      9.0, -0.180, 0.720           0.810,  8.10, -0.171, 0.729
3      7.2, -0.234, 0.486           0.575,  5.75, -0.211, 0.518

On a single quadratic the two paths are close for the first steps, and the lookahead gradient is smaller while the velocity points toward the minimum. The difference matters when the velocity points past the floor of a narrow, curved valley. Classical momentum keeps accumulating along the slope and overshoots, then needs several steps of reversed gradient to recover. Nesterov sees the reversal one step early, so the oscillation damps faster. The standard result is a faster rate on smooth convex problems, with the usual caveat that the constant depends on the curvature and the chosen lr and mu.

Practice. PyTorch's torch.optim.SGD(nesterov=True) does not literally evaluate at w + muv. It uses an equivalent reparametrisation: buf <- mubuf + g, then the step uses g + mu*buf. The gradient is still computed at the current parameters, so autograd needs no change, and the iterates match the look-ahead form under that change of variables. The practical differences are small for most deep networks with stochastic gradients, and you should check the claim against the optimizer's own documentation for your version rather than assume a large gain.

Failure modes. With a high mu and a large lr, both methods diverge; Nesterov does not fix an lr that is above the stability limit. Nesterov is also sensitive to the schedule: a lr decay applied at a bad moment can leave a large velocity that overshoots. Detect it as loss spikes right after a schedule step.

Curated: · Written: · Reviewed:

Q&A-9

AdaGrad divides each parameter's step by the root of its accumulated squared gradients. Explain what that does to the effective learning rate over training, where it helps, and where it stalls.

Answer

Start with the update. For each parameter i, AdaGrad keeps G_i, the running sum of squared gradients seen so far, and steps by w_i <- w_i - eta * g_i / (sqrt(G_i) + eps). The global learning rate eta stays fixed; what changes is the effective rate for each coordinate, eta_eff,i = eta / (sqrt(G_i) + eps). Because G_i only ever grows (each term g_i^2 is nonnegative), eta_eff,i can only shrink. That monotone decay is the defining property, and it is also the source of the main failure.

The per-parameter part matters as much as the decay. A coordinate that receives large gradients accumulates a large G_i and gets a small step. A coordinate that is rarely touched keeps a small G_i and its first real gradient moves it a lot. Take two features: a common one that is nonzero in 90% of examples with gradient about 1.0 each step, and a rare one nonzero in 0.1% of examples with gradient about 1.0 when it fires. After 10,000 steps the common feature has G of roughly 9,000, so its effective rate is about eta / 95. The rare feature has G of roughly 10, so its effective rate is about eta / 3.2. The rare feature keeps learning about thirty times faster per update, which is what you want for sparse one-hot or hashed features in linear models and early embedding layers.

Where it stalls: the decay is unconditional. Suppose the loss surface is steep early, so G_i grows to 10^4 in the first few hundred steps, and then the run reaches a region where the useful gradient is about 10^-2. The effective rate is now eta / 100 and still falling, and the parameter barely moves even though it is nowhere near a minimum. AdaGrad cannot forget old gradients, so an early burst of large gradients taxes the whole run. In deep networks this shows up as training that flattens out long before the loss is good, while Adam, which uses an exponential moving average of squared gradients, recovers the effective rate as old gradients fade.

When I would choose it: sparse linear models or problems where each feature's gradient is rare and informative, and a short run where the decay will not bite. When I would not: dense deep networks trained for many epochs, or any setting where the loss landscape changes after early training. The practical fix if I saw the stall is to switch to RMSprop or Adam (decaying average), or to restart the accumulator, rather than tune eta harder.

What I would check first: log the per-parameter effective rate, or the norm of G, over steps. A stalled run shows the accumulator grown by orders of magnitude while the loss is still high, which distinguishes this failure from a bad learning rate or a vanishing gradient.

Curated: · Written: · Reviewed:

Q&A-10

RMSProp keeps a running average of squared gradients instead of AdaGrad's running sum. What problem does that fix, and what is epsilon for?

Answer

The 60-second answer. RMSProp divides each parameter's step by the square root of an exponential moving average of its squared gradients. AdaGrad divides by the square root of the sum of all past squared gradients. That sum only grows, so every parameter's effective learning rate shrinks monotonically, and on long runs it can shrink to almost nothing before the model has converged. RMSProp replaces the sum with a decaying average, so old gradients are forgotten and the denominator can fall as well as rise. The step size stays responsive for the whole run.

The update, with decay rate rho (commonly 0.9) and epsilon:

# PyTorch 2.x, torch.optim.RMSprop defaults: alpha=0.99, eps=1e-8
# g: gradient at step t, v: running average of g**2, lr: learning rate
v = rho * v + (1 - rho) * g * g
w = w - lr * g / (v.sqrt() + eps)

The effect of the decay is easy to check by hand. With rho = 0.9, a squared gradient from 50 steps ago contributes 0.9^50 ≈ 0.005 of its original weight, so the denominator tracks recent gradient scale. AdaGrad's sum keeps that old term at full weight forever.

Epsilon. The denominator is sqrt(v). At the start v is zero, and any parameter whose gradient is exactly zero would divide by zero. Epsilon, usually 1e-8 in PyTorch and 1e-7 in Keras, keeps the denominator positive. It is a numerical guard, not a tuning knob for convergence. Raising it to 1e-3 or more pulls the step toward plain SGD with a smaller learning rate for parameters with small gradients, which changes the optimizer's behaviour and should be treated as a hyperparameter change.

Trade-offs and failure modes. RMSProp does not fix the learning rate itself. If the base lr is too large, the normalised step is still roughly lr per parameter per step, so the run can oscillate. Because the step is normalised, parameters with tiny gradients take steps of similar size to parameters with large ones, which can amplify noise on parameters that barely matter. Detect this by logging per-parameter update norms: a parameter whose update norm stays near lr while its loss contribution is flat is a sign of noise amplification. Adam combines this second-moment average with a momentum term; RMSProp alone has no momentum.

Curated: · Written: · Reviewed:

Q&A-11

With beta1 = 0.9 and beta2 = 0.999, how far can Adam move one parameter on its first step, with and without bias correction, and what bounds a later step after a long quiet stretch?

Answer

Start with the setup. Adam keeps an exponential average of gradients m (beta1, default 0.9) and of squared gradients v (beta2, default 0.999), both initialised to zero. Each step computes m_hat = m / (1 - beta1^t) and v_hat = v / (1 - beta2^t), then moves theta by eta * m_hat / (sqrt(v_hat) + eps), with eps = 1e-8 by default. The claim I would defend: with consistent gradients a parameter moves about eta per step, and a single spike after a quiet stretch moves it about eta * (1 - beta1) / sqrt(1 - beta2), roughly 3.2 eta with the defaults. Bias correction is what makes the early steps match that claim, and beta2 near 1 is why the correction matters so much.

Mechanism. Let g be the gradient of one parameter. After t steps, v_t = (1 - beta2) * sum over i of beta2^(t-i) * g_i^2. If the gradients have mean square s, E[v_t] = (1 - beta2^t) * s, so dividing by (1 - beta2^t) recovers s. At t = 1 that factor is 0.001. With beta2 = 0.999 the average spans about 1000 steps, so v needs roughly that long to warm up, and until it does the uncorrected denominator is far too small.

Trace with constant g = 1 and eta = 0.001: t = 1: m = 0.1, v = 0.001. Uncorrected step = 0.001 * 0.1 / sqrt(0.001) = 3.16 eta. Corrected: m_hat = 0.1 / 0.1 = 1, v_hat = 0.001 / 0.001 = 1, step = eta. t = 2: m = 0.19, v = 0.001999. Uncorrected step = 0.19 / sqrt(0.001999) * eta = 4.25 eta. Corrected: m_hat = 0.19 / 0.19 = 1, v_hat = 0.001999 / 0.001999 = 1, step = eta. So the uncorrected run moves 3 to 4 times too far at the start, and only drifts back toward eta as v fills. The corrected run is exactly eta from the first step.

The spike bound. Suppose v and m have decayed to near zero, then one gradient G arrives. m rises by 0.1 G and v by 0.001 G^2. The corrected step is about eta * 0.1 |G| / (0.0316 |G|), which is 3.16 eta. The paper (Kingma and Ba, 2015) gives |step| <= eta * (1 - beta1) / sqrt(1 - beta2) when 1 - beta1 > sqrt(1 - beta2), and <= eta otherwise. Read it as an analysis bound under stated assumptions, not a guarantee for every run. The practical point is that the bound is a small multiple of eta, not unbounded, and it applies per parameter, so features with rare large gradients see the biggest relative jumps.

Failure modes and detection. First, a spike after a quiet period on sparse parameters such as rare embedding rows: the loss jumps on a single step. Detect it by logging |delta theta| / eta per parameter group and the gradient norm histogram, and look for update ratios near 3 on rows that were almost never touched. Second, beta2 set too low, for example 0.9: the window shrinks to about 10 steps, the bound on a spike approaches (1 - beta1)/sqrt(1 - beta2) = 0.1/0.316 which is about 0.3 eta, and the optimiser loses per-parameter scale information. Detect it by comparing v_hat against a long-window estimate on a few layers.

Cost. Adam stores two extra tensors per parameter. A 1B-parameter model in fp32 holds 4 GB of weights, 8 GB of Adam state and another 4 GB of gradients, so about 16 GB before activations. When that memory matters I would reach for SGD with momentum or a factored second moment such as Adafactor, and accept the tuning cost.

Curated: · Written: · Reviewed:

Q&A-12

Why does AdamW decouple weight decay from the adaptive gradient scaling, and why does adding L2 regularisation to the loss behave differently under Adam than under SGD?

Answer

The 90-second answer. For plain SGD, adding (λ/2)‖w‖² to the loss and taking a gradient step is the same as shrinking each weight by a factor (1 − ηλ) before the step. The two are identical, so the question does not arise. Adam breaks the equivalence. Adam divides the gradient by √v̂, a per-parameter running estimate of squared gradients. If the L2 term λw is folded into the gradient, it is divided by that same estimate, so the effective decay depends on each parameter's gradient history rather than being the λ you configured. AdamW keeps the decay term outside the adaptive scaling, so every weight shrinks by the same ηλ per step.

Worked trace (hypothetical figures, Adam with β1 = 0.9, β2 = 0.999, ε = 1e-8, η = 0.001, λ = 0.1, data gradient set to zero so only the decay term acts). Step 1, w = 1 for both parameters:

L2 folded into the gradient, under SGD: update = η·λw = 0.001 · 0.1 · 1 = 0.0001 for both weights.

L2 folded into the gradient, under Adam: g = 0.1, m̂ = 0.1, v̂ = 0.01, √v̂ = 0.1, so update = 0.001 · 0.1 / 0.1 = 0.001. The step is η regardless of λ, and it is ten times the SGD step.

Now give the two parameters different histories. Parameter A has a √v̂ of about 0.1, as above. Parameter B has seen data gradients of magnitude about 10, so √v̂ is about 10 and its decay term is 0.001 · 0.1 / 10 = 0.00001. Parameter B decays a hundred times more slowly, though both carry the same λ. The regulariser has become adaptive, so the weights that most need shrinking can be the ones it reaches least.

AdamW: w ← w − η·(m̂/(√v̂ + ε) + λw). Both parameters shrink by ηλ = 0.0001 per step, and λ means what it says.

Trade-offs and when not to bother. Decoupled decay is the default for transformers and most modern Adam training. For SGD with or without momentum, the coupled and decoupled forms agree once you scale the decay by the learning rate, so you can use either. The coupling matters when λ is tuned against a schedule, because the effective regularisation strength then moves with the learning rate under coupled L2 and stays put under AdamW.

Failure modes and how to detect them. (1) You tune λ under coupled L2 and the best value changes whenever you change the learning rate or warmup. Detect it by sweeping η and λ on a grid and checking whether the optimum lies on a diagonal. (2) Layers with sparse or small gradients, such as embeddings, collapse toward zero while dense layers barely move. Detect it by logging per-layer weight norms over training. (3) Someone assumes PyTorch's torch.optim.Adam(weight_decay=…) and AdamW share semantics. Detect it by reading the optimizer's source, since Adam applies the decay to the gradient.

Curated: · Written: · Reviewed:

Q&A-13

You train the same image classifier with SGD plus momentum and with AdamW, both at their defaults, and SGD ends with lower test error. Explain the mechanism that could cause this, and name one thing you would check before believing it.

Answer

Say first that this is one comparison on one workload, so it supports a narrow claim at most. Adam and AdamW are not worse optimizers in general.

Mechanism. SGD with momentum keeps one velocity per parameter: v <- betav + g, then w <- w - etav. Every coordinate moves with the same global step size, scaled by its gradient history. AdamW divides each coordinate's step by a running root-mean-square of that coordinate's own gradients:

m <- b1*m + (1-b1)*g
v <- b2*v + (1-b2)*g^2
w <- w - eta*(m_hat/(sqrt(v_hat)+eps) + lambda*w)

Coordinates with small historical gradients therefore take relatively large steps. A common hypothesis is that this normalised, noisier path settles into sharper minima, which tend to be more sensitive to small weight perturbations and so generalise worse. Present that as a hypothesis from some experiments, not as settled geometry.

The check. The gap may come from how each optimizer was run rather than from the update rule. Before accepting it, I would check that both runs had the same tuning budget, since SGD often gets a learning-rate sweep while the Adam run keeps its defaults. I would also compare test error at matched training loss, because an optimizer that fits the training set faster can look better or worse on test for reasons unrelated to generalisation. A third seed per configuration would show whether the gap sits outside seed variance.

What would change my view. If a tuned AdamW with a small learning-rate sweep closes the gap, the result was about tuning. If the gap holds at matched training loss across seeds, it is a real tendency for this workload, and I would still not extend it to a language model without running the same comparison there.

Curated: · Written: · Reviewed:

Q&A-14

What does a learning-rate warmup schedule do, and why do large-batch training runs use it?

Answer

The 60-second answer. Warmup starts the learning rate near zero and raises it linearly to its target over the first few hundred to few thousand steps, then hands over to the main schedule. The point is to keep the earliest updates small. At the start the weights are far from a good region and the optimizer's running statistics have seen almost no gradients, so a full-size step is risky. Large-batch runs need it most because the learning rate is usually scaled up with the batch size, and that scaled rate is too aggressive at step 0.

Why the early steps are the risky ones. Adam divides each step by the square root of a running average of squared gradients. That average is bias-corrected, so at step 1 the estimate equals the first squared gradient and every parameter moves by about the learning rate, whatever its gradient size. With beta2 = 0.999 the average covers about 1,000 steps, so the early estimates rest on a few noisy minibatch gradients. Warmup keeps those first normalized steps small until the estimate has more data behind it.

The large-batch case. The linear scaling rule (Goyal et al., 2017, for SGD with momentum) multiplies the learning rate by k when the batch grows k times. Hypothetical figures: base learning rate 0.1 at batch 256, so batch 8,192 (k = 32) targets 3.2. Applied from step 0, that step can push activations and BatchNorm statistics into a bad regime. Ramping from 0.1 to 3.2 over five epochs, as Goyal et al. did for ResNet-50, keeps early steps within the range the loss tolerates.

What it does not do. Warmup does not make a too-high rate safe later in training, and it does not replace gradient clipping for exploding gradients. If a run diverges after warmup, check the peak rate and the batch-size scaling first.

Failure modes. Warmup that is too short shows as a loss spike in the first epoch. Warmup that is too long wastes steps at a low rate, which shows as a flat early loss and a run that trails a shorter-warmup run at the same step count. Plot the learning rate and the loss over the first 1 to 5 percent of steps to see either.

Curated: · Written: · Reviewed:

Q&A-15

You have one 30-epoch training run and no budget for a second sweep, but you must keep the checkpoint with the best validation loss. Which learning-rate schedule would you commit to (step decay, cosine, linear warmup then linear decay, or one-cycle), and how would you set the peak?

Answer

The 90-second answer. With one run and no second sweep, I commit to a short linear warmup followed by cosine decay to a small floor, and I set the peak with a cheap range test before the run. Cosine keeps the step size in a useful range for most of the budget and ends near zero, so the last checkpoint is a settled one. Step decay is the better choice only when something downstream needs weights frozen at known plateaus. One-cycle is a strong default for short budgets, but its peak is high and its result is most sensitive to the warmup fraction, which a single run cannot tune.

Assumptions: 30 epochs, fixed batch size, validation loss every epoch, and the stopping rule is 'keep the checkpoint with the lowest validation loss'.

Mechanism and arithmetic. For cosine over T epochs, lr(t) = eta_min + 0.5 * (eta_max - eta_min) * (1 + cos(pi * t / T)). Halfway through, lr is 0.5 * eta_max. At epoch 20 of 30 it is 0.25 * eta_max, and at epoch 29 it is about 0.003 * eta_max. Compare a step schedule with the same peak that multiplies by 0.1 at epochs 15 and 25. Its average step size over the 30 epochs is (15 * 1 + 10 * 0.1 + 5 * 0.01) / 30 = 0.535 * eta_max, against 0.5 * eta_max for cosine. Same peak, so step decay moves faster early and is closer to divergence at the peak. That is a reason to lower its peak, not a reason to prefer it.

# Python 3.12
import math
def cosine_lr(t, T, eta_max, eta_min=0.0):
    return eta_min + 0.5 * (eta_max - eta_min) * (1 + math.cos(math.pi * t / T))

Setting the peak. Run a learning-rate range test: ramp the rate from 1e-7 to 1 over roughly 200 steps and record loss per step. Take the rate where loss falls fastest, then set the peak a little below the point where loss starts to climb. This costs a few hundred steps, well under 5% of the budget. It is a heuristic, so confirm it with the first two epochs of the real run.

Failure modes and detection.

  1. Peak too high: loss spikes or goes NaN during warmup. Detect it from per-step loss and gradient-norm logs. Fix it by lowering the peak. Clipping alone hides the problem.
  2. Warmup too short for the batch size: early divergence that looks like a bad seed. Detect it by repeating the first two epochs with double the warmup.
  3. Floor too high, or decay ending before convergence: validation still falls over the last five epochs while training loss is flat. Detect it by plotting the validation slope at the end.
  4. Early stopping fighting the schedule: with cosine, the best validation loss usually comes in the final fifth of the run. A patience rule measured in epochs can fire during the high-rate phase, when validation is noisy. Measure patience against the schedule, or only start it after the decay has passed a set fraction.

Curated: · Written: · Reviewed:

Q&A-16

Your least-squares fit has features in metres and in millimetres, and gradient descent needs far more steps than the feature count suggests. Why, and what would you change before tuning the learning rate?

Answer

The 90-second answer. For mean squared error on a linear model, L(w) = (1/2n)||Xw − y||², the gradient is (1/n)X^T(Xw − y) and the Hessian is H = (1/n)X^T X. H does not depend on w, so gradient descent on this loss is a linear iteration whose speed is set by the eigenvalues of H. With λmax and λmin the largest and smallest eigenvalues and κ = λmax/λmin, the step must satisfy η < 2/λmax, so the stiffest direction caps the learning rate. The flattest direction then closes only a fraction λmin/λmax of its remaining error per step. A feature in millimetres rather than metres multiplies its diagonal entry of H by a million, which raises κ without changing the model.

Mechanism. Along eigenvector i, the error is multiplied by (1 − η·λi) each step. With η = 1/λmax, that factor is 1 − λi/λmax. Cutting the error along the flattest direction by 10^6 takes about ln(10^6)/(λmin/λmax) ≈ 13.8κ steps. The iteration count grows with κ, not with the number of features.

Worked example (hypothetical figures). Feature A is in metres with variance about 1. Feature B is in millimetres with variance about 10^6. Assume they are uncorrelated, so H ≈ diag(1, 10^6), κ = 10^6, and the flat direction needs about 1.4 × 10^7 steps. Standardise each column using training-split mean and standard deviation, and diag(H) becomes 1. With no correlation, H = I, κ = 1, and η = 1 solves the problem in one step. Fit the scaler on the training split only and reapply it unchanged to validation and serving inputs, or validation statistics leak into training.

What standardisation does not fix. It corrects scale on the diagonal, not correlation. If two features are nearly copies, κ stays large after scaling. Check the eigenvalues of the standardised X^T X/n rather than assuming the scaling step solved the problem. Remedies differ in what they change. Dropping or merging redundant features shrinks the small eigenvalue directly. Ridge adds α to every eigenvalue, so κ becomes (λmax + α)/(λmin + α), at the cost of a biased fit. A quasi-Newton method such as L-BFGS approximates H^-1 and handles correlation without changing the model, at the cost of more memory and a less predictable path. Adam's per-coordinate scaling corrects the diagonal mismatch but not correlations, so it does not remove the need to standardise.

Detection. The symptom is a loss that falls quickly, then flattens while weights on the small-scale features keep drifting. Compute the eigenvalues of X^T X/n on the training matrix, or cond(X), before tuning the learning rate, and compare the spectrum before and after standardising.

Curated: · Written: · Reviewed:

Q&A-17

A 12-layer plain MLP with sigmoid activations trains its last layers but barely moves the first ones. Explain why, with numbers, and what you would change to fix it.

Answer

Start with the product. Backpropagation multiplies the gradient arriving from above by each layer's local Jacobian. For an elementwise sigmoid the derivative is sigma(x)(1 - sigma(x)). It peaks at 0.25 when x = 0 and falls toward 0 as |x| grows. So the activation factors alone scale the gradient reaching layer k from a loss L layers above by at most 0.25^(L-k). Ten layers gives 0.25^10, about 9.5e-7. The weight matrices multiply in their own gain on top of that, so the true factor is usually smaller still once units saturate. The first layers receive almost no signal and the last layers train, which is the symptom in the question. Confirm it before fixing it: after one backward pass, log the gradient norm per layer. A first-to-last ratio near 1e-6 is the signature.

ReLU changes the activation factor. Its derivative is 1 for a positive pre-activation and 0 otherwise. Along any active path the factor is exactly 1, so the product stops shrinking with depth. The cost is that a unit whose pre-activation is negative for every input passes no gradient and never recovers. That is the dying-ReLU failure. Detect it by counting, per layer, the fraction of activations that are zero over a batch; a layer near 100% zero is dead. Leaky ReLU or GELU keep a small nonzero slope for that trade.

Initialisation sets the weight gain. He initialisation draws weights with variance 2/fan_in, which keeps the forward signal and the backward gradient at roughly constant variance through ReLU layers. Glorot (Xavier) uses 2/(fan_in + fan_out), which suits sigmoid and tanh. Wrong initialisation can recreate the problem even with ReLU, and the same arithmetic explains why it can also explode.

Residual connections change the structure. With y = x + F(x), the Jacobian is I + dF/dx. The identity term carries the gradient to earlier layers with factor 1 whatever dF/dx is. This guarantees a route for the signal, not a magnitude, so the blocks still need sensible scaling.

Normalisation keeps units out of the saturated tails. BatchNorm or LayerNorm rescales pre-activations to zero mean and unit variance, which puts sigmoid inputs near 0 where the derivative is largest. It cannot beat the 0.25 ceiling, so for a sigmoid network it reduces saturation rather than removing the bound. BatchNorm depends on batch statistics and behaves badly at very small batch sizes, which is why LayerNorm is common in transformers.

What I would change in this network: move to ReLU or GELU with He initialisation, add residual connections if the architecture allows, and keep the normalisation layer. What none of these fixes is exploding gradients from large weights, which needs gradient clipping and is a separate failure with the same product arithmetic.

Curated: · Written: · Reviewed:

Q&A-18

In global-norm gradient clipping, what happens to the gradient when its norm exceeds the threshold?

Answer

The 60-second answer. Global-norm clipping computes one norm over every parameter's gradient, treated as a single vector. If that norm is above the threshold c, every gradient is multiplied by the same factor, c divided by the norm. Gradients below the threshold are left alone. The update therefore keeps its direction and only its length shrinks to c.

The rule is the point. Because one positive scalar multiplies the whole vector, each component shrinks by the same ratio. Per-value clipping instead clamps each component to the range minus c to c, so a large component is cut while the others stay put. That can change the direction of the update, which global norm never does.

# PyTorch 2.x
loss.backward()
total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

clip_grad_norm_ returns the norm before clipping. Log that value.

Failure modes.

  1. Clipping hides a real bug. If the pre-clip norm sits above the threshold on most steps, clipping is no longer a safety net. It is silently setting the effective learning rate. A loss scale mistake or corrupted batch can look like a healthy run with a low learning rate. Detect it by plotting the pre-clip norm: a flat line far above c means the threshold is masking the cause.

  2. Clipping does not fix non-finite gradients. If any gradient is NaN or inf, the norm is non-finite and the scaled gradient stays non-finite. Check for this before clipping.

When not to clip. If the norm is always small, clipping does nothing and costs a reduction over all parameters every step.

Curated: · Written: · Reviewed:

Q&A-19

Your model has 40 million parameters and a scalar loss. Walk me through how backpropagation produces every parameter gradient, why one backward pass costs about the same as a forward pass, and what it costs in memory.

Answer

Start with what backprop is for. It does not choose the update rule. It computes dL/dw for every parameter so the optimizer can use it. The mechanism is the chain rule applied to the graph the forward pass recorded, walked from the loss back to the parameters.

Worked trace. Let a = w*x, b = a + c, L = b^2, with x = 2, w = 3, c = 1. Forward: a = 6, b = 7, L = 49. Reverse, each node multiplies the upstream gradient by its local derivative:

dL/db = 2b = 14 dL/da = dL/db * db/da = 14 * 1 = 14 dL/dw = dL/da * x = 14 * 2 = 28 dL/dc = dL/db * 1 = 14

One sweep produced both parameter gradients. Two rules make that work. Each node needs only its local derivative, or the saved inputs to compute it, so the backward step is a function of the upstream gradient. And when a value feeds several consumers, its gradient is the sum of the contributions along each path. If b were used twice, dL/db would be the sum of both.

Why the cost is about one forward pass. Forward-mode differentiation carries one input direction through the graph per pass, so n parameters need n passes. Reverse mode carries one output's sensitivity back through the graph, so a scalar loss needs one pass however many parameters there are. Each node's backward cost is a constant multiple of its forward cost. For y = Wx with W of shape m by n, the forward is one matrix-vector or matrix-matrix product. The backward computes dx = W^T dy and dW = dy x^T, two products of comparable size. A backward pass is therefore usually about two times a forward pass in FLOPs, and a full training step about three times forward. The claim that holds is that the ratio does not grow with parameter count. The claim that fails is that backward costs exactly one forward.

The cost that bites is memory. Backward needs the forward activations, so training memory scales with batch size times depth and exceeds inference memory. Gradient checkpointing trades that memory for recomputation: you keep activations only at segment boundaries and rerun the forward inside each segment during backward, typically paying roughly one extra forward.

Failure modes and how you catch them.

  1. An in-place write to a tensor the graph saved. PyTorch 2.x raises a runtime error at backward, naming the modified variable. It fails loudly and early, so you fix the op rather than the symptom.
  2. A silently detached branch, such as a .detach() or a conversion to NumPy inside the model. The loss is finite, the optimizer runs, and the parameters behind the break keep .grad as None. Detect it by asserting that every trainable parameter has a non-None gradient after the first backward.
  3. Out-of-memory at the first backward, not the first forward. Detect it by reading peak allocated memory before and after backward on one batch.

Curated: · Written: · Reviewed:

Q&A-20

Your training loss has been flat at 0.41 for 4,000 steps with a gradient norm of 2e-4. Is the run converged, and what is the single quantity you would check first to tell a plateau from a minimum?

Answer

The 60-second answer. Not yet. A flat loss with a small gradient is consistent with a minimum, but also with a plateau, a saddle, or a learning rate so small that the parameters barely move. The quantity I would check first is the update size, learning rate times gradient norm, because it tells me whether the parameters are still moving at all. If the update is negligible, the run is frozen rather than converged, and I need to know which before I stop.

Applying it to the run. Learning rate 0.01 and gradient norm 2e-4 give a per-step update of about 2e-6 in norm. Over 4,000 steps the parameters can move at most about 8e-3 in total if the gradient stayed that size, which is small relative to a unit-scale weight vector. So the run is effectively not moving. That is the frozen case, and it is different from a minimum. A minimum is a place where the gradient is zero and stays zero; a frozen run has a non-zero gradient that is simply too small to matter at this learning rate.

Distinguishing the cases. Each cause predicts a different signal:

CauseGradient normUpdate sizeValidation loss
Minimumnear zeronear zeroflat or best
Tiny learning ratesmall, non-zeronear zeroslowly improving
Saturated layersmall globally, one layer near zeronear zero thereflat

If validation is still improving, a tiny learning rate is the likely cause and the fix is to raise it. If a single layer's gradient is near zero while others are not, the problem is saturation in that layer, and a global norm hides it. Only if the global and per-layer norms are both small and validation has stopped improving do I accept a minimum.

Stopping rule. Before the run starts I would write the rule against three things at once: the relative change in validation loss over a patience window, the update size relative to the initial one, and the per-layer gradient norms. I would stop only when all three are small and the checkpoint with the best validation loss is kept. A fixed step cap guards against a run that never settles.

Failure modes. Minibatch noise keeps the gradient norm above zero at a true minimum, so the threshold has to come from a measured noise level, not from zero. Validation on a small set can move by half a percent from sampling alone, so the patience window must be longer than that noise. Thresholds are heuristics; the guarantee is only that a zero gradient is necessary for an interior minimum.

Curated: · Written: · Reviewed:

Q&A-21

Training loss fell for 200 steps, then jumped about tenfold in one step without turning into NaN. Which single measurement would you take first, and what would each outcome tell you?

Answer

Opening answer. I would take one measurement first: the loss on the batch that caused the spike, computed once before the update and once after it, with the weights frozen between the two evaluations. The sign and size of that change decide which cause to pursue.

Assumptions. The loss is a mean over the batch, the optimizer is plain SGD or Adam with a fixed learning rate at this step, and the spike is visible in the logged training loss at the step where the update happened. I would also confirm that the batch is replayable, meaning its exact samples and augmentations can be fed back in, so the two evaluations see identical inputs.

Mechanism. A gradient step raises the loss on the batch it was computed from only if it overshoots the local curvature. On a quadratic with curvature λ along some direction, one step multiplies the error along that direction by (1 − ηλ), so the step is unstable when ηλ > 2. Suppose λ = 40 near the current weights (a hypothetical figure). With η = 0.06, ηλ = 2.4 and the error grows by a factor of 1.4 per step. With η = 0.04, ηλ = 1.6 and the error shrinks by 0.6 per step. A rate that was stable for 200 steps can cross this line when curvature rises, for example after the weights move into a sharper region. Real networks are not quadratics, so treat this as the mechanism for the overshoot case, not as a fitted model.

Outcomes.

  • Loss on the same batch rises after the update. The step overshot. Either the learning rate is too high for the curvature at this point, or the gradient was unusually large. Next I would log the global gradient norm for the last 20 steps. A norm that climbs steadily before the jump points to a rate that is too high. A norm that jumps by orders of magnitude on one step points to a bad batch or an input problem.
  • Loss on that batch is unchanged or falls. The spike came from a different batch than the one I measured, or the measurement missed it. Check that the logged loss and the batch were aligned to the same step. If they were, the batch is the outlier: a label error, an extreme feature value, or a corrupted sample. Inspect its inputs and labels before changing the schedule.

Why not a sweep first. A learning-rate sweep costs a full set of runs and cannot separate an outlier from an overshoot. The paired evaluation answers the question on one batch.

Trade-off. Global-norm clipping would stop the explosion but hides which cause is present. I would add it after the measurement and keep it as a guard, not as the diagnosis. If the spike is an outlier, clipping only limits the damage; the bad sample still needs removing.

Curated: · Written: · Reviewed:

Q&A-22

A loss has stopped falling. Its gradient norm is exactly zero, and it stays at zero for every step. What does that tell you about the weights, and why would more steps at the same learning rate not fix it?

Answer

The gradient norm is the L2 norm over every parameter's gradient, so an exact zero means every component is zero. With plain SGD, the update is w ← w − η·g, so with g = 0 the weights do not move on that step, and the next loss equals the last one. Two conditions could produce this: the weights sit at a point where the loss has zero slope in every parameter direction, or the backward pass is not carrying signal to them at all. A per-parameter printout tells you which.

More steps at the same learning rate repeat the same zero. A larger learning rate multiplies the same zero. Both levers are wrong, because the gradient itself has to change, which means changing the model, the data path, or the graph.

Separate the two causes:

  1. Critical point. The point is a local minimum or saddle of the loss. Exact zeros are rare in float32 on real data. Check the loss value: if it is near the target, training is done. If it is high, you are at a saddle, and more steps at the same rate will not leave it.
  2. Dead or disconnected signal. The graph cuts the path to these weights. Typical causes: a ReLU whose pre-activations are all negative, a detach or no_grad on the path, a mask that zeroes every contribution, indexing or argmax with no gradient, or parameters that never enter the forward pass.

A small case that produces an exact zero:

import torch x = torch.tensor([-1.0, -2.0]) w = torch.tensor([1.0, 1.0], requires_grad=True) y = torch.relu(x * w).sum() y.backward() print(w.grad)

Trace: x * w = [-1, -2]. relu gives [0, 0], so the sum is 0. The derivative of relu is 0 wherever its input is negative, so the chain rule gives w.grad = x * 0 = [0, 0]. It prints tensor([0., 0.]). Both entries are exact zeros, and no learning rate changes that.

Check the optimizer before trusting the zero-update argument. With momentum or Adam, buffered state from earlier steps can still move the weights after the current gradient is zero, and the movement decays over steps rather than stopping at once. Weight decay (for example AdamW, or SGD with weight_decay) shrinks weights even when the gradient is zero. So the claim that nothing moves holds for plain SGD on that step. Read the gradients directly rather than inferring from the loss curve.

Diagnostic: after loss.backward() and before optimizer.step(), print each parameter's grad norm. Treat None as not in the graph and 0.0 as in the graph with no signal. Count active ReLU units per layer on a batch. If the zeros come from an upstream layer, inspect the activations there, since saturated or negative pre-activations often trace back to initialization or input scale.

Fix the path: remove the stray detach, correct the mask, check the initialization and input normalization that pushed units negative, and confirm the parameters are passed to the optimizer. Lower the learning rate only if the diagnosis shows instability rather than a missing signal.

Curated: · Written: · Reviewed:

Q&A-23

A mini-batch holds the points (2, 3) and (3, 6), and the model is ŷ = wx + b under mean squared loss over that batch. Starting from w = 0 and b = 0 with η = 0.01, what are the gradients and the parameters after one step, and what is the largest stable learning rate for this batch?

Answer

Assumptions first: the loss is the mean over the batch of size N = 2, L = (1/N) Σ (ŷ_i − y_i)², and the update is w ← w − η ∂L/∂w, b ← b − η ∂L/∂b. At w = 0, b = 0 the residuals are e = (−3, −6). The gradients are ∂L/∂w = (2/N) Σ e_i x_i = (2/2)(−3·2 + −6·3) = −24 and ∂L/∂b = (2/N) Σ e_i = (2/2)(−9) = −9. With η = 0.01 the step gives w₁ = 0.24 and b₁ = 0.09. Check: the batch loss falls from (9 + 36)/2 = 22.5 to about 14.9, so the step lowers the loss on this batch.

Stability depends on the batch Hessian, not on the full dataset. For this batch H = (2/N)[[Σx², Σx], [Σx, N]] = [[13, 5], [5, 2]]. Its eigenvalues solve λ² − 15λ + 1 = 0, giving about 14.93 and 0.067. Plain gradient descent converges for 0 < η < 2/14.93 ≈ 0.134, so η = 0.01 is stable here. The full three-point dataset has a largest eigenvalue of about 11.09 and a bound of about 0.18, so a batch-tuned rate is not automatically safe for the full run, and a rate tuned on the full run can be unstable on a batch that is more concentrated on high-leverage points.

Now the convention. If the loss is a sum instead of a mean, the gradients double to −48 and −18, the step at η = 0.01 becomes 0.48 and 0.18, and the stable bound halves to about 0.067. The rate still converges here, but the first step is twice as large and the margin to instability is halved. Batch size enters the same way: with a mean loss the gradient scale is roughly independent of N, while with a sum it grows with N. A learning rate chosen on a 2-example batch and reused on a 256-example batch is therefore wrong by a factor that depends on the loss convention, not just the batch size.

The subtle failure: the (3, 6) point contributes −18 of the −24 in the w gradient. A single high-leverage example dominates the first step, and a noisy batch like this one can point the update away from the full-data direction even while it lowers the batch loss.

Curated: · Written: · Reviewed:

Q&A-24

You implemented a backward pass by hand for a small network. How would you check it with finite differences, and what error would you accept?

Answer

Compare the analytic gradient with a numerical one for each coordinate. The central difference is g_num = (L(w + h e_i) - L(w - h e_i)) / (2h), and the comparison is the relative error |g_a - g_num| / (|g_a| + |g_num|). Run it in float64 on a tiny model with a fixed seed, so that arithmetic noise does not hide a real mismatch.

# NumPy float64, tiny model, fixed seed
def num_grad(f, w, i, h=1e-6):
    wp = w.copy(); wp[i] += h
    wm = w.copy(); wm[i] -= h
    return (f(wp) - f(wm)) / (2 * h)

Why the central form. Take L(w) = w^3 at w = 2, where the true gradient is 12. The central difference gives (2+h)^3 - (2-h)^3 over 2h, which equals 12 + h^2. Its error shrinks with h squared. A forward difference has error proportional to h, so at the same two evaluations the central form is far more accurate.

Step size. Two errors compete. Truncation error falls as h shrinks. Rounding error grows as h shrinks, because the two loss values are subtracted and divided by 2h. In float64 the rounding term is near eps*|L|/(2h) with eps about 2.2e-16. For losses of order 1, h around 1e-5 to 1e-6 keeps both terms small. Going much smaller, to 1e-12 say, lets rounding dominate and the reported error gets worse.

Reading the number. On smooth code, a correct gradient with h near 1e-6 usually lands around 1e-9 to 1e-7. An error near 1e-2 or larger points to a real bug in the backward pass. Between those, check for a kink, a stochastic layer, or a very small gradient. These thresholds are conventions I use, so I would state them as my cut-offs and not as laws.

Why float64 and a tiny model. Float32 has eps near 1.2e-7, so the same accuracy cannot be reached and a correct gradient can land near the failure threshold. The model must be small, since each coordinate needs two full forward passes. I would check a few hundred coordinates on a network with a few dozen parameters.

Failure modes that look like bugs. A ReLU evaluated exactly at zero has no derivative, so the two sides can disagree. Dropout or random augmentation changes L between calls, so fix the seed or switch them off for the check. Batch norm in training mode depends on the batch, so evaluate both sides on the same batch.

Curated: · Written: · Reviewed:

Q&A-25

You have a fixed budget of 8 GPU-hours to fine-tune a 1.3B-parameter decoder transformer on 500M tokens. Choose the optimizer, peak learning rate, batch size and schedule, and name what you would monitor to tell that you chose wrong.

Answer

The 90-second answer. I would fix the budget in tokens first, because that is what the data and the GPU-hours actually buy. Then I would use AdamW with betas (0.9, 0.999) and weight decay 0.01, a peak learning rate of 2e-5, a batch of about 256 sequences of 1024 tokens, a linear warmup over the first 5% of steps, then linear decay to zero. I would clip the global gradient norm at 1.0. I would not sweep this. I would run a 200-step probe at the planned settings, check that the loss falls without a spike, and only then commit the 8 GPU-hours. The choice is defensible because a transformer's parameters have very different curvature (embeddings, layer norms, attention projections), which adaptive per-parameter scaling handles and a single global step size does not.

Arithmetic. 500M tokens at 262,144 tokens per step (256 × 1024) is about 1,907 optimizer steps for one epoch. If the run takes 8 hours at a measured 14 s per step, that is 2,057 steps, so one epoch fits with a small margin. Doubling the batch to 512 halves the steps to about 954 and roughly doubles the time per step, so the token budget is unchanged but optimizer steps fall. That is the trade: larger batches give steadier gradients and better hardware use, but fewer updates per token. If memory forces a smaller micro-batch, I use gradient accumulation to keep the effective batch at 256 rather than changing the learning rate.

Why not the alternatives. Plain SGD with momentum usually needs a learning rate one to two orders of magnitude larger and tends to fit transformers worse, because one step size cannot suit both the embedding rows and the attention weights. Adafactor or 8-bit Adam save optimizer memory, which matters when the model does not fit, but they change the update rule and need their own check. A cosine schedule is a fine choice; I prefer linear decay here only because it is a common fine-tuning default and needs no extra parameter.

Failure modes, and how you detect each. A peak learning rate that is too high shows as a loss spike within the first few hundred steps, a gradient norm that climbs past its early median, and validation loss that rises at the first checkpoint while training loss looks normal. Forgetting shows as a held-out general benchmark dropping while the target task improves. A warmup that is too short produces spikes when Adam's second-moment estimates are still noisy, so the spike clusters in the first few percent of steps. A learning rate that is too low gives a validation curve that is flat from the start and a per-step update ratio, the norm of the update over the norm of the weights, well under 1e-4. A healthy run usually sits near 1e-3. I log these at fixed token intervals, not fixed steps, so a batch change does not hide a regression.

Before blaming the batch size. I check the learning rate scale against the new batch, whether warmup still covers the same token count, and whether gradient clipping is firing on most steps. Clipping on every step means the learning rate is effectively lower than configured.

Curated: · Written: · Reviewed: