Stochastic Gradient Descent

Stochastic gradient descent is an optimization method that minimizes a loss by repeatedly stepping the parameters against a gradient estimated from a small random sample of the training data rather than from the full dataset. Each update has the form w ← w − η × g, where η is the learning rate and g is the gradient computed on the sampled examples. Because the sample is random, g is a noisy but, on average, correct estimate of the true gradient, and that trade of exactness for speed is what makes training on millions or billions of examples feasible.

The idea predates deep learning by decades and comes from statistics. Herbert Robbins and Sutton Monro, in "A Stochastic Approximation Method" (Annals of Mathematical Statistics, 1951), showed that noisy measurements can still drive an iterative procedure to the right answer if the step sizes shrink appropriately: their sum must diverge so the process can travel anywhere, while the sum of their squares must stay finite so the noise eventually averages out. Those Robbins-Monro conditions explain why a constant learning rate only hovers around a minimum while a decaying schedule can converge, and why modern training almost always warms up and then decays the learning rate.

Minibatches sit between two extremes and set the balance between noise and hardware efficiency. Pure stochastic gradient descent uses one example per step, giving the noisiest estimate, while batch gradient descent uses the whole dataset, giving a precise but expensive step. A minibatch of B examples reduces the variance of the gradient estimate roughly in proportion to 1/B, while a GPU processes a batch in parallel almost as cheaply as a single example up to a point. In practice the term SGD in deep learning nearly always means minibatch SGD. Goyal and colleagues showed in 2017 that scaling the learning rate linearly with batch size, with warmup, allowed ImageNet training with batches of 8,192 images to match small-batch accuracy.

The noise is not merely tolerated; it is often useful. In the non-convex landscapes of neural networks, the randomness of each step helps the iterate escape saddle points and shallow traps where an exact gradient would stall. Evidence also suggests it acts as an implicit regularizer: Keskar and coauthors reported in 2017 that very large batches tended to settle in sharp minima that generalized worse, while smaller batches favored flatter ones, although the effect can be reduced by retuning the learning rate and schedule, so it is not an iron law.

Momentum and adaptive methods build on the same loop rather than replacing it. Momentum smooths successive noisy gradients, and Adam, introduced by Kingma and Ba in 2014, rescales each parameter's step using running averages of the gradient and its square. Adam, and its decoupled-weight-decay variant AdamW, is the default for transformers, while plain SGD with momentum remains common in image models. All of them are still stochastic gradient methods in the Robbins-Monro lineage, differing in how the noisy gradient is turned into a step.

Several pitfalls recur. An epoch is one pass over the data, not one update, so counting steps and counting epochs are different things. A learning rate that is too large makes the loss diverge or oscillate, while one that is too small wastes compute; it is usually the single most important hyperparameter. Forgetting to shuffle can introduce correlated batches and biased updates. And the guarantees from classical theory apply cleanly to convex problems, so the success of SGD on deep networks is partly an empirical fact that theory is still catching up with.

Read stochastic gradient descent as steering by a rough compass checked constantly rather than a perfect map consulted rarely: each minibatch gives a noisy but unbiased hint about downhill, and taking an enormous number of such cheap steps, with the noise sometimes helping, is how large models are actually trained.