lars

**LARS** (Layer-wise Adaptive Rate Scaling) is an **optimizer designed for large-batch distributed training** — scaling the learning rate for each layer by the ratio of the layer's weight norm to its gradient norm, enabling stable training with batch sizes up to 32K or more. **How Does LARS Work?** - **Trust Ratio**: For each layer $l$: $lambda_l = eta cdot ||w_l|| / ||g_l||$ where $eta$ is a trust coefficient. - **Intuition**: Layers with large weights and small gradients get larger learning rates. Layers with small weights and large gradients get smaller rates. - **Base**: Applied on top of SGD with momentum (LARS) or Adam (LAMB). - **Paper**: You et al., "Large Batch Training of Convolutional Networks" (2017). **Why It Matters** - **Large Batch Training**: Enables near-linear scaling of SGD to thousands of GPUs without accuracy loss. - **ResNet in Minutes**: LARS enabled training ResNet-50 on ImageNet in under 1 hour with 64 GPUs. - **Foundation**: LAMB (Layer-wise Adam) extends the same principle to Adam for BERT pre-training. **LARS** is **the layer balancer for massive batches** — preventing any single layer from destabilizing training by adaptively scaling learning rates per layer.