Gradient Descent

Gradient descent is the algorithm that actually does the "learning" in machine learning: it repeatedly nudges a model's parameters in the direction that reduces the loss function the fastest, using the gradient — the multi-dimensional slope of the loss with respect to every parameter — as the compass. Nearly every trained neural network, from a tiny logistic regression to a trillion-parameter language model, is the product of this same basic loop run billions of times, which is why gradient descent is arguably the single most economically important algorithm in modern computing, not an abstract math footnote. At a high level, one step of the loop looks like this:

The learning rate is the single hyperparameter most likely to make or break a training run. It controls how big a step gradient descent takes at each update — too large, and the optimizer overshoots the loss surface's minimum and can diverge entirely (loss exploding to NaN is the classic symptom); too small, and training crawls so slowly it never converges in a practical amount of GPU time. Modern training almost never uses a fixed learning rate: a warmup phase ramps it up gradually at the start (large random initial gradients can be unstable), followed by a decay schedule (cosine or linear) that shrinks it as training progresses, letting the model take large confident steps early and small precise steps as it nears a good solution.

Batch, mini-batch, and stochastic gradient descent trade gradient accuracy for GPU throughput, and the industry settled firmly on mini-batch. Computing the gradient over the entire training set (batch) gives the mathematically exact direction but is far too slow to be practical past a tiny dataset; computing it from a single example (stochastic) is cheap but noisy. Mini-batch — typically 32 to a few thousand examples per step on modern hardware — strikes the balance that actually matches how GPUs work: a GPU's thousands of parallel cores are most efficient processing a batch of examples simultaneously, so batch size isn't just a statistics choice, it's a hardware-utilization choice tuned against a specific GPU's core count and memory bandwidth.

Adam and its variants replaced plain gradient descent as the default optimizer because they adapt the step size per parameter automatically. Plain SGD uses the same learning rate for every single parameter; Adam tracks a running average of each parameter's recent gradients (momentum) and recent squared gradients (adaptive scaling), giving parameters that have been consistently updated in one direction a bigger effective step and noisy, inconsistent parameters a smaller one — which is why Adam and AdamW are the default optimizer for virtually every modern transformer training run, despite costing extra GPU memory to store that per-parameter running state.

Read gradient descent as the actual physical process that turns GPU-hours into a trained model: every training run, regardless of architecture, is this one simple update rule executed at enormous scale, and the real engineering skill in modern AI is tuning its few knobs — learning rate schedule, batch size, optimizer choice — well enough that millions of GPU-hours converge to a useful model instead of diverging, stalling, or wasting the compute budget entirely.