LARS (Layer-wise Adaptive Rate Scaling) is an optimizer designed for large-batch distributed training — scaling the learning rate for each layer by the ratio of the layer's weight norm to its gradient norm, enabling stable training with batch sizes up to 32K or more.

How Does LARS Work?

• Trust Ratio: For each layer $l$: $lambda_l = eta cdot ||w_l|| / ||g_l||$ where $eta$ is a trust coefficient.
• Intuition: Layers with large weights and small gradients get larger learning rates. Layers with small weights and large gradients get smaller rates.
• Base: Applied on top of SGD with momentum (LARS) or Adam (LAMB).
• Paper: You et al., "Large Batch Training of Convolutional Networks" (2017).

Why It Matters

• Large Batch Training: Enables near-linear scaling of SGD to thousands of GPUs without accuracy loss.
• ResNet in Minutes: LARS enabled training ResNet-50 on ImageNet in under 1 hour with 64 GPUs.
• Foundation: LAMB (Layer-wise Adam) extends the same principle to Adam for BERT pre-training.

LARS is the layer balancer for massive batches — preventing any single layer from destabilizing training by adaptively scaling learning rates per layer.