zero

**Zero Redundancy Optimizer ZeRO DeepSpeed** is **a memory-efficient distributed training framework eliminating parameter, gradient, and optimizer state redundancy enabling training of trillion-parameter models** — DeepSpeed ZeRO addresses memory limitations preventing training of large models, reducing per-GPU memory consumption 10-100x through intelligent state partitioning. **Stage 1: Optimizer State Partitioning** distributes optimizer states (Adam momentum and variance) across GPUs, each GPU maintaining only states for its parameters, reducing memory 4x. **Stage 2: Gradient Partitioning** partitions gradients across GPUs, each GPU maintaining gradients only for assigned parameters, further reducing memory 2x. **Stage 3: Parameter Partitioning** partitions model parameters across GPUs, enabling training of models larger than cluster total memory through on-demand fetching. **Gradient Accumulation** accumulates gradients from multiple mini-batches reducing communication frequency and memory consumption. **Communication Hiding** overlaps parameter fetching with computation, hides gradient reduction behind forward/backward passes. **Offloading** spills optimizer states and parameters to CPU memory enabling GPU memory for computation, trades memory for CPU-GPU communication. **Mixed Precision Training** leverages FP16 computation with FP32 master weights maintaining convergence, reducing memory and improving speed. **Zero Redundancy Optimizer ZeRO DeepSpeed** enables training models previously requiring entire datacenters.