Zero Redundancy Optimizer ZeRO DeepSpeed is a memory-efficient distributed training framework eliminating parameter, gradient, and optimizer state redundancy enabling training of trillion-parameter models — DeepSpeed ZeRO addresses memory limitations preventing training of large models, reducing per-GPU memory consumption 10-100x through intelligent state partitioning. Stage 1: Optimizer State Partitioning distributes optimizer states (Adam momentum and variance) across GPUs, each GPU maintaining only states for its parameters, reducing memory 4x. Stage 2: Gradient Partitioning partitions gradients across GPUs, each GPU maintaining gradients only for assigned parameters, further reducing memory 2x. Stage 3: Parameter Partitioning partitions model parameters across GPUs, enabling training of models larger than cluster total memory through on-demand fetching. Gradient Accumulation accumulates gradients from multiple mini-batches reducing communication frequency and memory consumption. Communication Hiding overlaps parameter fetching with computation, hides gradient reduction behind forward/backward passes. Offloading spills optimizer states and parameters to CPU memory enabling GPU memory for computation, trades memory for CPU-GPU communication. Mixed Precision Training leverages FP16 computation with FP32 master weights maintaining convergence, reducing memory and improving speed. Zero Redundancy Optimizer ZeRO DeepSpeed enables training models previously requiring entire datacenters.