Adaptive Optimizers

Vanilla gradient descent has a fundamental problem: it treats every gradient update as if it exists in isolation. Each step looks only at the current gradient, with no memory of previous steps. This myopic behavior leads to two devastating inefficiencies that cripple training in practice.

Problem 1: Oscillation in Ravines

Consider a loss surface shaped like a long, narrow valley—a ravine. The gradient points steeply toward the walls but only gently toward the valley floor where the minimum lies. Vanilla gradient descent bounces back and forth between the walls, making tiny progress along the valley floor. Each step undoes much of the previous step's work.

Problem 2: Slow Progress on Plateaus

On flat regions of the loss surface, gradients become vanishingly small. Without any accumulated velocity, vanilla gradient descent slows to a crawl, taking thousands of iterations to cross what could be traversed quickly with momentum.

The breakthrough insight: give the optimizer memory. Instead of using only the current gradient, accumulate a velocity vector that builds up speed in consistent directions and dampens oscillations in inconsistent directions. This simple idea—borrowed directly from physics—transforms optimization.

The name "momentum" comes directly from classical mechanics, and the analogy is remarkably precise. Imagine a heavy ball rolling down a hilly landscape where the height represents the loss function value.

Without momentum (vanilla gradient descent):

The ball is infinitely sticky—it has no inertia. At each moment, it moves in the direction of steepest descent at that exact location, then stops completely. The next moment, it evaluates the slope again from scratch. In a ravine, this ball bounces back and forth endlessly.

With momentum:

The ball has mass and velocity. It accelerates in the direction of the slope, but its velocity persists. When the gradient changes direction (walls of the ravine), the ball's existing velocity partially cancels the oscillation. When the gradient points consistently in one direction (down the valley), velocities accumulate, building speed.

The key physical concepts that transfer directly to optimization:

Why this analogy matters:

This isn't just a pedagogical metaphor—it's the actual derivation. Momentum SGD solves a discrete approximation to the physics equation:

$$m\frac{d^2 x}{dt^2} = - abla L(x) - \gamma \frac{dx}{dt}$$

Where m is mass, γ is friction, and ∇L(x) is the loss gradient. The momentum coefficient β controls the ratio of inertia to friction, determining how much the ball "remembers" its previous velocity versus responding to current gradients.

Let's formalize momentum optimization with precise mathematical notation. Given parameters θ, loss function L(θ), learning rate α, and momentum coefficient β ∈ [0, 1):

Classical Momentum (Polyak's Heavy Ball Method):

$$v_t = \beta v_{t-1} + abla L(\theta_{t-1})$$ $$\theta_t = \theta_{t-1} - \alpha v_t$$

Here, v_t is the velocity vector, initialized to zero. The first equation updates velocity: multiply previous velocity by β (preserving momentum), then add the current gradient. The second equation updates parameters: move in the direction of velocity, scaled by the learning rate.

Alternative Formulation (with gradient scaling):

Some frameworks use a slightly different formulation:

$$v_t = \beta v_{t-1} + \alpha abla L(\theta_{t-1})$$ $$\theta_t = \theta_{t-1} - v_t$$

This is mathematically equivalent after absorbing α into the velocity definition. The choice affects hyperparameter interpretation but not the algorithm's behavior.

Expanding the recursion reveals the exponential averaging:

Interpretation as Exponential Moving Average:

The velocity v_t is an exponentially weighted moving average (EMA) of past gradients. This has profound implications:

1. Noise reduction: Random noise in stochastic gradients has inconsistent signs. The EMA cancels this noise while preserving the consistent signal.

2. Direction stabilization: If gradients consistently point in one direction, the EMA accumulates, increasing the effective step size. If gradients oscillate, the EMA dampens.

3. Implicit learning rate adaptation: The effective step in any direction is proportional to how consistently the gradients point that way.

The critical relationship between β and effective step count:

The "half-life" of the momentum—how many steps until a gradient's influence is halved—is:

$$\text{half-life} = \frac{\ln(0.5)}{\ln(\beta)} = \frac{-0.693}{\ln(\beta)}$$

For β = 0.9, half-life ≈ 6.6 steps. For β = 0.99, half-life ≈ 69 steps.

Understanding why momentum accelerates convergence requires analyzing the algorithm's behavior on well-understood loss surfaces. The canonical test case is a quadratic loss function, where we can derive exact convergence rates.

Quadratic Analysis:

Consider minimizing L(θ) = ½θᵀHθ where H is a positive definite Hessian. The eigenvalues λ₁ ≤ λ₂ ≤ ... ≤ λₙ of H determine the curvature in each eigendirection.

Condition number: κ = λₙ/λ₁

This ratio measures how "stretched" the loss surface is. A high condition number means a ravine—steep in some directions, shallow in others.

Vanilla gradient descent convergence:

The optimal learning rate is α* = 2/(λ₁ + λₙ), giving convergence rate:

$$\rho_{\text{GD}} = \frac{\kappa - 1}{\kappa + 1}$$

For κ = 100, ρ_GD ≈ 0.98, meaning each step reduces error by only 2%.

Momentum convergence:

With optimal hyperparameters α* and β*, momentum achieves:

$$\rho_{\text{momentum}} = \frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}$$

For κ = 100, ρ_momentum ≈ 0.818, reducing error by ~18% per step—a 9× speedup!

The square root phenomenon:

The key insight is that momentum reduces the square root of the condition number's effect. This is profound: problems that would take 10,000 iterations with GD take ~100 with momentum.

Optimal hyperparameters for quadratics:

$$\alpha^* = \left(\frac{2}{\sqrt{\lambda_1} + \sqrt{\lambda_n}}\right)^2$$

$$\beta^* = \left(\frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}\right)^2$$

Note that optimal β approaches 1 as the condition number increases. This explains the common practice of using high momentum values (0.9-0.99) in deep learning, where loss surfaces often have high condition numbers.

Beyond quadratics:

Real neural network loss surfaces aren't quadratic, but the intuition transfers:

• High curvature directions → momentum prevents overshooting
• Low curvature directions → momentum accumulates velocity
• The effective condition number of the local Hessian determines speedup

To truly understand momentum, we need to visualize what happens geometrically in different loss landscape scenarios.

Scenario 1: The Ravine (High Condition Number)

Imagine a loss surface shaped like a long, narrow valley—a paraboloid stretched in one direction. The gradient at any point has two components:

1. A large component pointing toward the nearest wall (high curvature direction)
2. A small component pointing along the valley floor (low curvature direction)

Without momentum: Each step bounces toward a wall, then back. The valley-floor component makes tiny progress each iteration. Convergence is painfully slow.

With momentum: The wall-pointing components oscillate in sign and cancel in the velocity. The valley-floor components have consistent sign and accumulate. The ball accelerates along the valley while its wall-to-wall oscillations are damped.

Scenario 2: The Plateau (Low Gradient Region)

On flat regions of the loss surface, gradients are small. Without momentum, progress stalls. With momentum, velocity persists from previous steps, carrying the optimizer across the plateau.

Scenario 3: The Saddle Point

Saddle points have zero gradient but aren't minima—the surface curves up in some directions and down in others. Vanilla GD stops at saddle points (zero gradient = zero update). Momentum, with its accumulated velocity, can escape saddle points by continuing in the direction it was already moving.

Scenario 4: Sharp Minima vs Flat Minima

There's evidence that momentum prefers flat minima over sharp minima. Sharp minima have high curvature; momentum's oscillation-damping effect prevents settling into these. Flat minima have low curvature; momentum doesn't oscillate and settles smoothly. This may contribute to better generalization.

The velocity-gradient angle:

A useful diagnostic is the angle between velocity and current gradient. In smooth optimization, this angle should be small (velocity aligns with gradient). In oscillatory regimes, this angle becomes large (velocity fights gradient). Tracking this angle can reveal optimization issues.

Implementing momentum correctly requires attention to several practical details that aren't obvious from the mathematical formulation.

Memory Requirements:

Momentum requires storing one velocity vector v for each parameter. For a model with N parameters:

• Vanilla SGD: O(N) memory (just parameters)
• Momentum SGD: O(2N) memory (parameters + velocities)

For large models, this doubles the optimizer state. Most frameworks store this efficiently alongside parameters.

Initialization:

The standard approach initializes velocity to zero: v₀ = 0. This means the first step is identical to vanilla gradient descent—no momentum contribution. The momentum effect builds up over the first 1/(1-β) steps.

Some practitioners use "warmup" strategies, starting with low momentum and increasing it, to avoid initial instability.

Numerical Considerations:

With high β values (0.99+), velocity can grow large. Combined with large gradients, this can cause numerical overflow. Framework implementations typically use float32, which handles this well, but monitoring velocity norms is wise for debugging.

Momentum SGD has two primary hyperparameters—learning rate α and momentum coefficient β—with a complex interaction that requires careful tuning.

The Learning Rate (α):

With momentum, the effective step size is larger than with vanilla GD. The velocity accumulates gradients, amplifying the step. As a rule of thumb:

• If vanilla GD uses learning rate α₀, momentum SGD should use α ≈ α₀ × (1 - β)
• For β = 0.9, this means ~10× smaller learning rate than vanilla GD

The Momentum Coefficient (β):

Higher β means stronger momentum effect:

• β = 0: No momentum (vanilla SGD)
• β = 0.5: Weak momentum, mild acceleration
• β = 0.9: Standard choice, ~10× speedup on ill-conditioned problems
• β = 0.99: Strong momentum, for very ill-conditioned problems
• β → 1: Infinite momentum, unstable (never converges)

The α-β Tradeoff:

There's an inherent tradeoff: higher momentum allows faster convergence on smooth paths, but increases the risk of overshooting and oscillation at minima. A common strategy:

1. Start with β = 0.9 (standard default)
2. If training is unstable, reduce α (not β)
3. If training is too slow, consider β = 0.99 with reduced α
4. For fine-tuning, reduce both α and β for precise convergence

Momentum Warmup:

Some practitioners start with low momentum (β = 0.5) and increase to the target value (β = 0.9) over the first few epochs. This prevents the initial "cold start" where zero velocity makes the first updates behave like vanilla GD.

Interaction with Batch Size:

Larger batches produce less noisy gradients. With low noise, momentum has fewer oscillations to dampen, so its benefit decreases. Very large batch training often uses lower momentum or other techniques (LARS, LAMB) that we'll cover later.

Momentum is not universally beneficial. Understanding when it helps—and when it hurts—is essential for effective optimization.

The Sparsity Problem:

In models with sparse gradients (embeddings, attention patterns), many parameters receive zero gradients on most steps. Momentum accumulates all past gradients, including the zeros. When a sparse parameter finally gets a gradient, its velocity might be dominated by decay from old updates, not the current signal.

This is one motivation for per-parameter adaptive methods (AdaGrad, Adam) covered later.

Online Learning Caution:

In online learning, the data distribution may shift over time. Momentum averages old gradients, which reflect the old distribution. If the distribution changes faster than 1/(1-β) steps, momentum actively hurts by pulling toward outdated optima. Use lower momentum (β ≤ 0.5) or none for online settings.

Near-Convergence Behavior:

As training approaches the minimum, the gradient becomes small. But momentum carries residual velocity from previous steps, causing the optimizer to overshoot and oscillate around the minimum. This is why learning rate schedules are important—reducing α near the end mitigates this oscillation.

Momentum represents a fundamental insight in optimization: past gradients contain valuable information. By accumulating an exponentially weighted average of gradients into a velocity vector, momentum achieves:

The momentum coefficient β = 0.9 has become a standard default in deep learning, providing substantial acceleration on typical neural network loss surfaces while remaining stable with appropriate learning rates.

What's Next:

Classical momentum has a subtle inefficiency: it computes the gradient at the current position, then updates velocity. But what if we could compute the gradient at the position we're about to move to? This "lookahead" idea leads to Nesterov momentum, which further accelerates convergence with a simple modification. The next page explores this elegant improvement.