linear scaling rule
**Linear scaling rule** is the **heuristic that increases learning rate proportionally with batch-size growth** - it is a common starting point for large-batch training but must be validated with stability controls.
**What Is Linear scaling rule?**
- **Definition**: If global batch is multiplied by k, initial learning rate is multiplied by k as first-order adjustment.
- **Intuition**: Larger batches reduce gradient noise, allowing larger optimization step sizes in many regimes.
- **Applicability**: Works best within bounded scaling ranges and with suitable optimizer settings.
- **Failure Cases**: At extreme batch sizes, linear scaling can destabilize training or hurt final quality.
**Why Linear scaling rule Matters**
- **Practical Baseline**: Provides simple, widely used initialization rule for distributed scaling experiments.
- **Tuning Efficiency**: Reduces search space when moving from single-node to multi-node batch sizes.
- **Speed Potential**: Correctly scaled LR can preserve convergence speed at higher throughput.
- **Knowledge Transfer**: Rule offers shared language across teams for scaling discussions.
- **Optimization Discipline**: Encourages structured rather than arbitrary hyperparameter adjustment.
**How It Is Used in Practice**
- **Baseline Establishment**: Start from well-performing small-batch configuration and apply proportional LR scaling.
- **Warmup Integration**: Use gradual LR ramp-up to avoid early divergence at high effective step sizes.
- **Validation Sweep**: Run narrow LR sweeps around linear target and choose by time-to-quality outcome.
Linear scaling rule is **a useful first-order guide for large-batch optimization** - it accelerates tuning, but robust results still require empirical validation and stability safeguards.