Q&A-1Suppose L(w) = 10(w - 3)^2 and you run w <- w - 0.1 * dL/dw from w = 2.0. Does the run converge, and what is the largest learning rate that still converges for this loss?
Answer
Short answer: the run does not converge. With eta = 0.1 the iterates oscillate between 2 and 4 forever. The largest learning rate that converges is just under 0.1, and the boundary value 0.1 itself fails.
Assumptions: one scalar parameter, a smooth convex loss, full-batch gradients, and a fixed learning rate eta.
The gradient is dL/dw = 20(w - 3), so the update is w <- w - eta * 20(w - 3). Write e = w - 3 for the distance to the minimum. Then e <- e(1 - 20 eta). Each step multiplies the error by the factor 1 - 20 eta, so convergence depends on whether that factor has magnitude below 1. In general, for L = a(w - w*)^2 the factor is 1 - 2a eta, and the stability bound is eta < 1/a.
Trace at eta = 0.1, starting from w = 2.0:
step w grad = 20(w-3) w_next = w - 0.1*grad L(w)
0 2.0 -20.0 4.0 10.0
1 4.0 20.0 2.0 10.0
2 2.0 -20.0 4.0 10.0
error factor: 1 - 20*0.1 = -1
The factor is exactly -1, so the error flips sign and never shrinks. The loss stays at 10 and never decreases.
What happens for other values:
- 0 < eta < 0.1: the factor lies in (-1, 1), so the error shrinks geometrically. At eta = 0.05 the factor is 0 and w reaches 3 in one step (2 - 0.05 * -20 = 3). Values near 0.1 converge slowly and oscillate with decaying amplitude.
- eta = 0.1: factor -1, no convergence, oscillation at constant amplitude.
- eta > 0.1: the factor has magnitude above 1, so the oscillation amplitude grows and the loss rises each step. At eta = 0.15 the factor is -2 and the error doubles in magnitude per step.
So the bound is 0 < eta < 0.1, and there is no separate threshold at 0.2. Any rate above 0.1 diverges.
The same bound scales with curvature. If a = 1, the factor is 1 - 2 * 0.1 = 0.8 and eta = 0.1 converges. This is why a rate tuned on one model can fail on another with a different loss scale.
Failure detection in practice: a loss that falls and then flattens suggests a workable rate. A loss that oscillates between the same values, as here, indicates a rate at or above the bound. Growing loss or NaN indicates a rate well above it. Adaptive optimizers such as Adam normalize each coordinate by a running gradient scale, so the effective step no longer follows this simple bound, and the base learning rate still needs tuning with the same kind of check.
Curated: · Written: · Reviewed: