warmup for large batch

**Warmup for large batch** is the **learning-rate scheduling technique that gradually ramps optimization step size during early training** - it prevents divergence when large-batch configurations require high target learning rates from the linear scaling regime. **What Is Warmup for large batch?** - **Definition**: Controlled increase of learning rate from low initial value to target value over a warmup window. - **Instability Context**: Early gradients can be volatile, and immediate high LR often causes overshoot or loss spikes. - **Schedule Types**: Linear warmup, cosine warmup, or staged ramps integrated with main LR policy. - **Tuning Variables**: Warmup duration, initial LR floor, and target LR transition shape. **Why Warmup for large batch Matters** - **Stability**: Reduces early-training divergence risk in high-batch, high-LR configurations. - **Convergence Quality**: Improves chance of reaching strong final accuracy in aggressive scaling setups. - **Operational Reliability**: Lower failure rates mean fewer expensive aborted large-cluster runs. - **Scaling Enablement**: Warmup is often required to realize benefits of linear LR scaling at larger batches. - **Tuning Consistency**: Provides repeatable startup behavior across different hardware scales. **How It Is Used in Practice** - **Ramp Design**: Set warmup length as fraction of total steps based on model and optimizer sensitivity. - **Monitoring**: Track loss curvature and gradient norms during warmup to detect instability early. - **Policy Coupling**: Transition smoothly from warmup into decay schedule without abrupt LR discontinuities. Warmup for large batch is **a critical stabilization mechanism for scaled training** - controlled learning-rate ramping protects convergence while enabling high-throughput optimization regimes.