The learning rate is the single most consequential number in a training run: it sets how far each optimizer step moves the weights. Set it too high and the loss diverges; set it too low and training crawls or settles into a poor minimum. A learning-rate schedule is the recognition that no single value is right for the whole run — the ideal step size early in training, when the weights are random and gradients are large, is not the ideal step size late in training, when the model is fine-tuning its way into a minimum. The canonical modern recipe, warmup followed by cosine decay, encodes exactly this intuition.\n\nWarmup starts the learning rate near zero and ramps it up over the first few percent of training. This looks wasteful but is essential for large models, and for two reasons. At initialization the weights are random, so gradients are large and pointing in inconsistent directions; a full-size step here can knock the model into a bad region it never recovers from. And adaptive optimizers like Adam estimate a running variance of the gradients that is unreliable for the first few hundred steps, so their effective step size is erratic until those statistics settle. A linear warmup holds the step size small while both problems resolve, then hands off to the peak learning rate once training is on stable footing. Large-batch training makes warmup even more important.\n\nDecay then walks the learning rate back down toward zero over the rest of training. The logic is explore-then-settle: a high learning rate covers ground quickly and escapes shallow traps, but you cannot converge to a sharp minimum while taking large steps, so you gradually shrink the step size to let the model settle. Cosine decay is the dominant choice — it follows a smooth half-cosine from the peak down to near zero, spending a lot of the run at a moderately high rate and only slowing sharply at the very end. Its smoothness avoids the abrupt loss jumps that hard step-decay schedules can cause.\n\nWarmup plus cosine decay is the default for essentially all large-model training. You pick a peak learning rate, a warmup length (often 1-4% of total steps), and a total step budget the cosine decays across; that budget coupling is why you generally must know your total training length up front. Other schedules still have their places: the original Transformer used an inverse-square-root decay tied to warmup; step decay (cut the rate by a factor at fixed milestones) remains common in vision; and a constant rate with a short decay at the end is used when the total length is not known in advance. The through-line is always the same shape of idea — ramp up carefully, run hot, then cool down to converge.\n\n| Schedule | Shape | Needs total steps? | Typical home |\n|---|---|---|---|\n| Constant | Flat | No | Debugging, small jobs |\n| Step decay | Cut at milestones | No | Classic vision (ResNets) |\n| Inverse sqrt | 1/sqrt(step) after warmup | No | Original Transformer |\n| Warmup + linear | Ramp up, linear down | Yes | Fine-tuning (BERT-style) |\n| Warmup + cosine | Ramp up, cosine down | Yes | LLM pretraining (default) |\n\n``svg\n\n``\n\nIt is tempting to treat the learning rate as one number you sweep for and forget. The schedule reframes it as a story the training run tells over time: begin timidly because the model is fragile and the optimizer's own statistics are still forming, open up to a high rate once things are stable to make fast progress, then quiet down to converge cleanly. Read a schedule through an explore-then-settle lens rather than a set-and-forget lens, and warmup, cosine decay, and the coupling to your total step budget stop being ritual and become a direct expression of what the model needs at each phase of its training.