lamb

**LAMB** (Layer-wise Adaptive Moments optimizer for Batch training) is an **optimizer specifically designed for large-batch distributed training** — extending Adam with layer-wise trust ratios that normalize the update magnitude per layer, enabling stable training with batch sizes up to 65K or more. **How Does LAMB Work?** - **Base**: Standard Adam momentum and adaptive learning rate computation. - **Trust Ratio**: Scale each layer's update by $phi(||w||) / ||Adam\_update||$ (ratio of weight norm to update norm). - **Effect**: Prevents any single layer from receiving disproportionately large or small updates. - **Paper**: You et al. (2020). **Why It Matters** - **Large Batch Training**: Enables training BERT in 76 minutes (was 3 days with smaller batches). - **Scaling Efficiency**: Near-linear scaling up to thousands of GPUs. - **Distributed Training**: The go-to optimizer for large-scale distributed pre-training runs. **LAMB** is **the team coordinator for distributed training** — ensuring that large-batch updates are balanced across layers for maximum training throughput.