Gradient descent on a quadratic with condition number 1,000 is fastest at a learning rate that is 99.9 percent of the one that blows the run up. Not 50 percent, not the middle of a plateau — 99.9 percent, and the figure is exact rather than empirical. That single fact sets the character of every learning-rate decision that follows: the quantity being tuned is not a dial with a comfortable middle, it is a distance from a wall, and where the wall stands is decided by the curvature of the loss, the width of the network, the size of the batch and the units the gradient happens to carry. Change any of those without changing the model at all and the right learning rate moves by orders of magnitude. The number is a statement about the whole training configuration, and treating it as a property of the architecture is the source of most of the folklore around it.

<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 470" width="100%" style="max-width:760px" font-family="Helvetica,Arial,sans-serif">
<rect width="760" height="470" fill="#0d1117"/>
<text x="28" y="34" fill="#e6edf3" font-size="19" font-weight="700">A learning rate is a statement about curvature, width and batch.</text>
<text x="28" y="54" fill="#8b98a5" font-size="13">Measured, not asserted &#8212; every figure below comes out of the same sweeps that generated the table.</text>
<text x="28" y="92" fill="#e6edf3" font-size="13" font-weight="700">the optimum sits against the wall</text>
<text x="28" y="108" fill="#8b98a5" font-size="11">steps to cut the loss a millionfold, kappa = 100</text>
<line x1="62" y1="330.0" x2="356" y2="330.0" stroke="#21262d" stroke-width="1"/>
<text x="56" y="334.0" fill="#6e7681" font-size="10" text-anchor="end">100</text>
<line x1="62" y1="197.7" x2="356" y2="197.7" stroke="#21262d" stroke-width="1"/>
<text x="56" y="201.7" fill="#6e7681" font-size="10" text-anchor="end">1,000</text>
<line x1="62" y1="118.0" x2="356" y2="118.0" stroke="#21262d" stroke-width="1"/>
<text x="56" y="122.0" fill="#6e7681" font-size="10" text-anchor="end">4,000</text>
<rect x="329.3" y="112" width="26.7" height="218.0" fill="#f85149" opacity="0.13"/>
<line x1="329.3" y1="112" x2="329.3" y2="330" stroke="#f85149" stroke-width="2"/>
<text x="321.3" y="127" fill="#f85149" font-size="11" font-weight="700" text-anchor="end">2 / lambda_max</text>
<text x="321.3" y="141" fill="#f85149" font-size="11" text-anchor="end">101% diverges</text>
<polyline points="75.4,132.6 128.8,225.2 195.6,265.2 302.5,299.2 326.6,273.3" fill="none" stroke="#58a6ff" stroke-width="2"/>
<circle cx="75.4" cy="132.6" r="3" fill="#58a6ff"/>
<text x="83.4" y="136.6" fill="#58a6ff" font-size="10" text-anchor="start">3,101</text>
<circle cx="128.8" cy="225.2" r="3" fill="#58a6ff"/>
<text x="134.8" y="218.2" fill="#58a6ff" font-size="10" text-anchor="start">619</text>
<circle cx="195.6" cy="265.2" r="3" fill="#58a6ff"/>
<text x="201.6" y="258.2" fill="#58a6ff" font-size="10" text-anchor="start">309</text>
<circle cx="302.5" cy="299.2" r="4" fill="#3fb950"/>
<text x="302.5" y="316.2" fill="#3fb950" font-size="10" text-anchor="middle">171</text>
<circle cx="326.6" cy="273.3" r="3" fill="#58a6ff"/>
<text x="318.6" y="277.3" fill="#58a6ff" font-size="10" text-anchor="end">268</text>
<line x1="62" y1="330" x2="356" y2="330" stroke="#30363d" stroke-width="1"/>
<text x="75.4" y="345" fill="#6e7681" font-size="10" text-anchor="middle">5%</text>
<text x="128.8" y="345" fill="#6e7681" font-size="10" text-anchor="middle">25%</text>
<text x="195.6" y="345" fill="#6e7681" font-size="10" text-anchor="middle">50%</text>
<text x="302.5" y="345" fill="#6e7681" font-size="10" text-anchor="middle">90%</text>
<text x="329.3" y="345" fill="#6e7681" font-size="10" text-anchor="middle">100%</text>
<text x="209" y="361" fill="#8b98a5" font-size="11" text-anchor="middle">learning rate, as a percentage of the divergence threshold</text>
<text x="28" y="386" fill="#3fb950" font-size="11" font-weight="700">fastest at 90% of the threshold.</text>
<text x="28" y="401" fill="#8b98a5" font-size="11">5% of it costs 18x the steps. 101% costs the run.</text>
<text x="28" y="416" fill="#8b98a5" font-size="11">The optimum is kappa/(kappa+1) of it: 99.9% at kappa=1,000.</text>
<line x1="392" y1="76" x2="392" y2="424" stroke="#21262d" stroke-width="1"/>
<text x="424" y="92" fill="#e6edf3" font-size="13" font-weight="700">how far the optimum moves when you change</text>
<text x="424" y="108" fill="#e6edf3" font-size="13" font-weight="700">something that is not the model</text>
<text x="424" y="124" fill="#8b98a5" font-size="11">nothing on this axis is a fact about the network</text>
<line x1="424.0" y1="146" x2="424.0" y2="395" stroke="#21262d" stroke-width="1"/>
<text x="424.0" y="407" fill="#6e7681" font-size="10" text-anchor="start">1x</text>
<line x1="516.0" y1="146" x2="516.0" y2="395" stroke="#21262d" stroke-width="1"/>
<text x="516.0" y="407" fill="#6e7681" font-size="10" text-anchor="middle">100x</text>
<line x1="608.0" y1="146" x2="608.0" y2="395" stroke="#21262d" stroke-width="1"/>
<text x="608.0" y="407" fill="#6e7681" font-size="10" text-anchor="middle">10,000x</text>
<line x1="700.0" y1="146" x2="700.0" y2="395" stroke="#21262d" stroke-width="1"/>
<text x="700.0" y="407" fill="#6e7681" font-size="10" text-anchor="middle">1,000,000x</text>
<text x="424" y="147" fill="#c9d1d9" font-size="11">rescale the loss by 1,000,000, under SGD</text>
<rect x="424" y="152" width="276.0" height="17" fill="#f85149" rx="2"/>
<text x="693.0" y="165" fill="#0d1117" font-size="11" font-weight="700" text-anchor="end">1,000,000x</text>
<text x="424" y="182" fill="#6e7681" font-size="10">the rate carries the units of the gradient</text>
<text x="424" y="199" fill="#c9d1d9" font-size="11">rescale the loss by 1,000,000, under Adam</text>
<rect x="424" y="204" width="9.2" height="17" fill="#3fb950" rx="2"/>
<text x="439.2" y="217" fill="#3fb950" font-size="11" font-weight="700">1.6x</text>
<text x="424" y="234" fill="#6e7681" font-size="10">one cell of the search grid; the trajectories are identical</text>
<text x="424" y="251" fill="#c9d1d9" font-size="11">widen the network 16x, standard parameterization</text>
<rect x="424" y="256" width="36.8" height="17" fill="#f85149" rx="2"/>
<text x="466.8" y="269" fill="#f85149" font-size="11" font-weight="700">6.3x</text>
<text x="424" y="286" fill="#6e7681" font-size="10">and the width-64 rate transferred to width 1,024 is 910x worse</text>
<text x="424" y="303" fill="#c9d1d9" font-size="11">widen the network 16x, maximal update</text>
<rect x="424" y="308" width="9.2" height="17" fill="#3fb950" rx="2"/>
<text x="439.2" y="321" fill="#3fb950" font-size="11" font-weight="700">1.6x</text>
<text x="424" y="338" fill="#6e7681" font-size="10">one grid cell; the same transfer costs 7.4x</text>
<text x="424" y="355" fill="#c9d1d9" font-size="11">grow the batch from 1 to 512</text>
<rect x="424" y="360" width="86.8" height="17" fill="#d29922" rx="2"/>
<text x="503.8" y="373" fill="#0d1117" font-size="11" font-weight="700" text-anchor="end">77x</text>
<text x="424" y="390" fill="#6e7681" font-size="10">and then stop &#8212; the critical batch here is 165</text>
<line x1="28" y1="424" x2="732" y2="424" stroke="#21262d" stroke-width="1"/>
<text x="28" y="444" fill="#8b98a5" font-size="12">And the curvature does not constrain the rate &#8212; it moves to meet it. Full-batch GD at lr=0.25 ends with sharpness</text>
<text x="28" y="460" fill="#8b98a5" font-size="12">8.00 against 2/lr = 8.00; at lr=0.50, 4.03 against 4.00. Same network, same data, different sharpness.</text>
</svg>

The stability bound is a spectral radius, and the optimum sits directly underneath it. For $f(x) = \tfrac{1}{2}x^{\top}Hx$, gradient descent multiplies the error in eigendirection $i$ by $(1 - \eta\lambda_i)$ every step, so the iteration converges if and only if that factor has magnitude below one in every direction. The binding direction is the sharpest, which gives the familiar $\eta < 2/\lambda_{\max}$, and the rate that minimises the worst-case contraction balances the sharpest direction against the flattest:

At $\kappa = 10$ the optimum is 90.9 percent of the divergence threshold, at $\kappa = 100$ it is 99.01 percent, at $\kappa = 1{,}000$ it is 99.90 percent and at $\kappa = 10{,}000$ it is 99.99 percent. Better conditioning buys headroom and no real problem has any. Measuring the cost of standing back from the wall — steps to cut the loss a millionfold at $\kappa = 100$ — gives 3,101 steps at 5 percent of the threshold, 619 at 25 percent, 309 at 50 percent, 171 at 90 percent and 268 at 99 percent, where the curvature of the second-slowest direction starts to bite. Half the optimal rate costs 1.2 times the steps and a twentieth of it costs 18 times. One percent over the threshold costs the run. The asymmetry is the whole reason learning-rate tuning feels like a search rather than an optimisation: the penalty for being low is polynomial and the penalty for being high is total.

The curvature does not constrain the learning rate so much as move to meet it. Cohen and colleagues observed that a network trained by full-batch gradient descent does not keep the sharpness that initialisation gave it. The sharpness rises until it reaches $2/\eta$, then hovers there, oscillating across the stability boundary while the loss continues to fall — the regime now called the edge of stability. Reproducing it on a small tanh network, identical data and identical initialisation at three learning rates: at $\eta = 0.25$ the sharpness starts at 5.31 and ends at 7.998 against a threshold of 8.000, a ratio of 1.000; at $\eta = 0.50$ it ends at 4.03 against 4.00, a ratio of 1.008. The same network on the same data ends up with twice the curvature when trained at half the learning rate. At $\eta = 0.10$, where the threshold of 20.0 is far above where the network naturally sits, sharpness climbs from 5.31 to 10.84 over 24,000 steps and is still climbing — it has not reached the boundary yet, and the ratio is 0.54. This inverts the usual mental model. Sharpness is not a fixed obstacle that the learning rate must respect; it is a quantity the learning rate selects, which is why measuring the Hessian of a trained network tells you about the schedule that produced it as much as about the loss landscape it lives on.

Under SGD the learning rate carries the units of the gradient; under Adam it carries none. Multiply the loss by a constant $c$ and every gradient scales by $c$, so SGD needs $\eta/c$ to take the same step. Adam forms $\hat m/\sqrt{\hat v}$, and $\hat m$ scales by $c$ while $\sqrt{\hat v}$ also scales by $c$, so the update is unchanged and the optimal rate does not move at all. Sweeping this rather than asserting it: across a millionfold rescaling of the loss the best SGD rate moved by exactly 1,000,000 times, and the best Adam rate moved by 1.58 times, which is one cell of a search grid whose spacing is 1.58. The invariance is exact and not statistical — running Adam from the same seed at the same rate with the loss multiplied by 1,000 gives weight trajectories that differ by 4.6 parts in a hundred thousand after 200 steps, which is floating-point noise in the $\epsilon$ term. This is why 3e-4 functions as a usable default across models three orders of magnitude apart in size while no single SGD rate does, and it is also why a loss-scaling change, a switch from sum reduction to mean reduction, or a change in gradient-accumulation arithmetic silently retunes an SGD run and leaves an Adam run alone.

Warmup exists because the first Adam step is the one step that ignores the loss surface entirely. With bias correction, at $t = 1$ the first moment is $(1-\beta_1)g/(1-\beta_1) = g$ and the second is $(1-\beta_2)g^2/(1-\beta_2) = g^2$, so $\hat m/\sqrt{\hat v} = \operatorname{sign}(g)$ exactly and every parameter moves by exactly the learning rate regardless of its gradient. Checked across gradients from 1e-4 to 1e4, the ratio of update magnitude to learning rate is 1.000000 in every case. A weight initialised at $1/\sqrt{\text{fan\_in}}$ therefore moves a fixed fraction of its own size on step one, set by the learning rate and the fan-in and by nothing else: at $\eta$ = 3e-4 that is 0.96 percent of a weight at fan-in 1,024, 1.92 percent at 4,096 and 3.84 percent at 16,384, so a wide model displaces its weights four times as far on its first step as a narrow one at the same rate. Measuring how erratic that step is, using 4,000 independent coordinates drawn from a fixed gradient distribution, the first update is 3.0 times the magnitude of the same optimizer's settled update and 4.4 times as variable, and the magnitude comes within 5 percent of its settled value only around step 25. Warmup is the ramp that covers that interval, and its natural length is set by the optimizer's moment windows — $1/(1-\beta_1) = 10$ steps and $1/(1-\beta_2) = 1{,}000$ steps — rather than by anything about the model.

Widening a network moves its optimal learning rate, unless the parameterization is built so that it does not. Sweeping sixteen learning rates at five widths from 64 to 1,024 under the textbook $1/\sqrt{\text{fan\_in}}$ initialisation with one global Adam rate — which is what every framework does by default — the optimum drifts from 6.31e-2 down to 1.00e-2, a factor of 6.3 across a 16-fold change in width. Under maximal update parameterization, which scales the output multiplier and the per-layer Adam step by $\text{base\_width}/\text{width}$, the same sweep moves the optimum by a factor of 0.6, which is one grid cell in the other direction, so it has not meaningfully moved at all. The consequence is what makes this worth the trouble: tune at width 64 and transfer that number to width 1,024, and the standard parameterization lands at a loss of 3.56e-3 where 3.91e-6 was attainable, 910 times worse, while maximal update lands 7.35 times off. Yang and Hu's argument is that this is not a tuning trick but a statement about which infinite-width limit a parameterization converges to — the standard one drives feature learning to zero as width grows, so the best rate must shrink to compensate, and $\mu$P is the scaling under which features keep moving. A learning-rate sweep at a small width is only informative about a large model if the parameterization was chosen to make it so.

Batch size buys steps until it does not, and the learning rate stops rising at exactly the same place. On a noisy quadratic where the gradient variance falls as $1/B$, searching for the rate that reaches a fixed target in the fewest steps, the optimal rate climbs from 0.0196 at $B = 1$ to 1.5115 at $B = 512$ — a factor of 77 — and then stops, returning the identical value at 1,024 and at 4,096. The step count falls from 765 to 8 over the same range, a factor of 109, and then floors at 7. Fitting the standard form $\text{steps} = S_{\min}(1 + B_{\text{crit}}/B)$ across the sweep gives a median implied critical batch of 165, and both quantities saturate together because they saturate for the same reason: below $B_{\text{crit}}$ the run is noise-limited and a bigger batch is a better gradient that supports a bigger step, and above it the run is curvature-limited and the ceiling is $2/\lambda_{\max}$, which no amount of averaging raises. This is the honest version of the linear-scaling rule that Goyal and colleagues used to train ImageNet in an hour. Scaling $\eta$ with $B$ works while the noise dominates and silently stops working at a threshold that is a property of the task and the point in training, not of the hardware, which is why the rule needs warmup to survive the early steps and why McCandlish and colleagues framed the whole question as a measurement to make rather than a formula to apply.

A cosine schedule is a commitment to a step count made before step one, and the step count is an untuned hyperparameter. The declared horizon determines the entire shape of the decay, it is never trained on, never validated and rarely reported, and it changes the outcome substantially. Running the same network at the same peak rate for the same 4,000 steps under cosines declared for six different horizons, plus warmup-stable-decay, which declares nothing and decays only over the final fifth of whatever run it is given:

Across those six settings the loss at the stopping point spans a factor of 60, and the conventional choice — declare the horizon you intend to run — is the worst of the six, sixty times behind the best, which declared three times the run it actually got. The schedule that declares nothing beats every cosine at every checkpoint, and beats the matched cosine by a factor of 103 at the end. The mechanism is not mysterious: a matched cosine spends most of its budget already decayed, so it trades late-stage progress for an early-stopping curve nobody reads, while WSD holds the peak rate through the bulk of the run and collapses it only at the end. This is the finding that made warmup-stable-decay standard in the MiniCPM and DeepSeek recipes, and its practical value is not primarily the loss — it is that a run whose schedule does not depend on a declared endpoint can be extended, checkpointed and branched without invalidating everything before the branch point.

{ "rows": [
  { "type": "nodes", "items": [
    { "title": "\"what learning rate should I use?\"", "sub": "the answer is not a property of the model", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "First: fix the things that move the optimum for non-model reasons", "note": "each of these shifts the answer without changing a single weight", "items": [
    { "title": "Optimizer units", "sub": "SGD moves 1,000,000x under a loss rescale; Adam moves 1.6x, one grid cell", "tone": "green" },
    { "title": "Parameterization", "sub": "standard drifts 6.3x over 16x width; muP drifts 0.6x and transfers", "tone": "green" },
    { "title": "Batch size", "sub": "the usable rate rises 77x to B_crit and not one step past it", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "Then find the ceiling, because the optimum is right underneath it", "note": "kappa/(kappa+1) - at kappa=1,000 the best rate is 99.9% of the divergent one", "items": [
    { "title": "Sweep up until it diverges", "sub": "back off one grid cell, not one order of magnitude", "tone": "green" },
    { "title": "Being 20x low costs 18x the steps", "sub": "being 1% high costs the entire run", "tone": "orange" },
    { "title": "Expect sharpness to follow", "sub": "full-batch GD drives it to 2/lr and pins it there", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "Warmup covers the interval where the optimizer has no statistics", "note": "length set by 1/(1-b1)=10 and 1/(1-b2)=1000 steps, not by the model", "items": [
    { "title": "Step 1 moves every weight by exactly lr", "sub": "bias correction makes m/sqrt(v) = sign(g), whatever the gradient", "tone": "red" },
    { "title": "3.0x the settled step, 4.4x the spread", "sub": "within 5% of settled magnitude only by step 25", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "Last, and treat the horizon as a hyperparameter", "note": "60x spread in final loss across six declared cosine lengths", "items": [
    { "title": "Cosine matched to the run", "sub": "the conventional choice and the worst of the six tested", "tone": "red" },
    { "title": "Warmup-stable-decay", "sub": "declares nothing, won every checkpoint, and survives run extension", "tone": "green" }
  ] }
] }

Read the learning rate through a stability lens rather than a step-size lens. As a step size it is a knob with a good value somewhere in the middle, and that picture predicts none of the behaviour above. As a distance from the divergence threshold it predicts all of it: the optimum sits at $\kappa/(\kappa+1)$ of the threshold because the threshold is what binds; the sharpness rises to $2/\eta$ because the threshold is what the dynamics equilibrate against; the batch-size ceiling appears at $B_{\text{crit}}$ because past that point the threshold rather than the noise is what limits the step; and $\mu$P works because it holds the threshold fixed as width grows. Every reliable rule about learning rates is a rule about where that boundary sits and what moves it, and every quantity that moves it — the optimizer's units, the parameterization, the batch, the loss scale, the schedule's declared endpoint — is a decision made outside the model. Tune the rate last, after those are settled, and it becomes a one-dimensional search for a cliff edge. Tune it first and it absorbs the error in all of them.