zero

ZeRO (Zero Redundancy Optimizer) from DeepSpeed partitions optimizer states, gradients, and model parameters across GPUs to eliminate memory redundancy, enabling training of models with tens or hundreds of billions of parameters. Memory problem: optimizer states (Adam momentum, variance) use 2× model size; gradients add another 1×; with FP32 master weights, one copy of model needs 16 bytes per parameter. ZeRO stages: Stage 1 shards optimizer states (4× memory reduction), Stage 2 also shards gradients (8× reduction), and Stage 3 additionally shards parameters (linear scaling). How it works: each GPU owns 1/N of each tensor (N = GPU count); gather full tensor when needed, compute, then scatter results back to shards. Communication: Stage 3 requires all-gather before forward/backward, reduce-scatter after; communication overhead traded for memory savings. ZeRO-Offload: extend to CPU and NVMe storage; train even larger models on limited GPU memory. Integration: DeepSpeed wraps optimizer; minimal code changes required. Comparison to model parallelism: ZeRO is data parallelism with sharding; doesn't require model restructuring. ZeRO-Infinity: extends offload to NVMe for maximum model size. Hybrid: combine ZeRO with pipeline/tensor parallelism for largest models. Performance: overlap communication with computation where possible; efficiency depends on hardware and model. ZeRO revolutionized large model training accessibility.