batch size

Batch size is how many training examples a model averages together before it takes a single step of gradient descent, and it is one of the most consequential knobs in deep learning precisely because it looks like a mere efficiency setting. Turn it up and each step uses a smoother, more accurate estimate of the gradient and the hardware runs more efficiently; turn it down and each step is cheap but noisy. That noise is not simply a nuisance to be minimized — it is a real force shaping where training ends up — and the *gradient noise scale* is the tool that tells you how much batch size you can actually use before the noise is gone and bigger stops buying you anything.\n\n**Every mini-batch gradient is a noisy estimate of the true gradient, and batch size sets the noise level.** The gradient you really want is the average over the entire dataset, but computing it every step is far too expensive, so you estimate it from a random mini-batch. That estimate has variance that falls as 1/batch_size: a batch of 1 (pure stochastic gradient descent) gives a very noisy direction, a batch of thousands gives a smooth one, and full-batch gradient descent gives the exact direction at ruinous cost. The noise is a double-edged sword — it slows convergence and destabilizes training, but it also helps the optimizer escape sharp minima and is widely believed to bias training toward *flatter* minima that generalize better, which is why the smoothest possible large-batch gradient is not automatically the best.\n\n**The gradient noise scale predicts the "critical batch size" where doubling the batch stops halving the step count.** OpenAI's gradient noise scale measures the ratio of the gradient's variance to its squared magnitude — intuitively, how much the per-example gradients disagree with each other. Below the critical batch size this number implies, training is *noise-dominated*: doubling the batch roughly halves the number of steps needed, so you get near-perfect speedup and should scale up. Above it, training is *curvature-dominated*: the gradient estimate is already accurate, extra examples barely improve the direction, and you burn compute for almost no reduction in steps. The critical batch size is not a constant — it grows as the loss falls and as the task gets harder, which is why big models late in training can profitably use enormous batches that would be wasteful early on.\n\n**Batch size and learning rate are coupled, so you can never change one alone.** A larger batch gives a lower-variance gradient, which lets you safely take a larger step, so the two scale together. The *linear scaling rule* (multiply the learning rate by the same factor as the batch size, with a warmup) works remarkably well up to the critical batch size; beyond that, returns diminish and a gentler *square-root scaling* often fits better. Get this coupling wrong and a "just make the batch bigger" change silently degrades the final model, because the effective step size collapsed relative to the noise the optimizer was tuned for.\n\n| Batch regime | Gradient character | Practical consequence |\n|---|---|---|\n| Batch = 1 (SGD) | Very high noise | Cheap steps, erratic, strong implicit regularization |\n| Small mini-batch | Moderate noise | Good generalization, common default |\n| Near critical batch size | Noise ≈ curvature | Best speed-per-compute sweet spot |\n| Far above critical | Noise gone | Diminishing returns, wasted compute |\n| Full batch | Exact gradient | Smoothest, expensive, can overfit sharp minima |\n\n```svg\n\n```\n\nThe unhelpful way to think about batch size is as a memory-and-throughput setting you crank as high as the GPU allows. The useful way is to see it as the dial that sets how noisy each gradient estimate is: small batches give cheap, jittery steps that regularize and seek flat minima, large batches give smooth, expensive steps, and the gradient noise scale tells you the *critical batch size* where the noise is essentially gone and further doubling stops halving your step count. Because that noise also determines how large a learning rate you can survive, batch size and learning rate must move together — linearly up to the critical size, then more gently. Read batch size through a how-noisy-is-each-gradient-estimate lens rather than a just-fit-it-in-memory lens, and the scaling rules, the warmups, and the point where more compute stops buying speed all fall out of one quantity instead of feeling like separate rules of thumb.