Momentum

Momentum is an optimization technique that adds a running average of past gradients to each update, so the parameters keep moving in a consistent direction instead of reacting only to the current slope. The standard form keeps a velocity v and updates it as v ← β × v − η × g, then moves the weights by w ← w + v, where g is the current gradient, η the learning rate and β, commonly 0.9, the fraction of the previous velocity retained. The physical picture is a heavy ball rolling down a surface: it gathers speed along persistent slopes and is not knocked off course by every small bump.

The method comes from Boris Polyak's 1964 paper on speeding up the convergence of iteration methods, where it is known as the heavy ball method. Its motivation is the narrow ravine: on a loss surface that is steep across one direction and shallow along another, plain gradient descent bounces between the walls while crawling along the floor. Momentum cancels the oscillating components, since successive gradients across the ravine point in opposite directions and average out, while the components along the floor keep the same sign and accumulate. With a steady gradient, the effective step becomes roughly η/(1 − β), so β = 0.9 behaves like a learning rate about ten times larger along consistent directions.

Nesterov's refinement evaluates the gradient at a look-ahead point and carries a provable speedup in the right setting. Yurii Nesterov's 1983 method computes the gradient after provisionally applying the momentum step, which lets it correct course earlier when the ball is about to overshoot. For smooth convex problems it reaches an error that shrinks like 1/k² after k iterations, against 1/k for plain gradient descent, and this rate is optimal among first-order methods. In deep learning, the version most often used is a reformulation by Sutskever, Martens, Dahl and Hinton (2013), who showed in "On the importance of initialization and momentum in deep learning" that carefully tuned momentum let plain SGD train deep networks that had previously seemed to need more elaborate second-order methods.

A frequent misconception is that the acceleration guarantee carries over unchanged to neural networks. The 1/k² result assumes a deterministic, smooth, convex objective. Real training is stochastic and non-convex, and there the benefit is mostly empirical: smoothing noisy gradients, speeding movement through flat regions, and carrying the iterate across small barriers. A related pitfall is mixing up conventions. Frameworks differ in whether the learning rate multiplies the velocity before or after accumulation, so the same numeric hyperparameters can behave differently between libraries, and changing the learning rate mid-training interacts with stored velocity.

Momentum is also the conceptual ancestor of adaptive optimizers. Adam keeps an exponential moving average of the gradient, which is exactly momentum, and combines it with per-parameter scaling by a running average of squared gradients. High momentum can overshoot and oscillate if the learning rate is too large, and too little leaves the benefit unused. Typical settings are β between 0.9 and 0.99, and large-batch training often relies on warmup so that early, unreliable velocity does not destabilize the run.

Read momentum as inertia for optimization: by remembering where recent gradients pointed, the update damps the sideways jitter that wastes motion and builds speed along the directions that keep proving to be downhill, from Polyak's heavy ball to Nesterov's look-ahead correction.