distributed training framework
**Distributed Training Frameworks** are the **software systems that coordinate the training of large machine learning models across multiple GPUs and multiple machines** — handling data distribution, gradient synchronization, communication optimization, and fault tolerance to enable training of models that exceed single-GPU memory capacity and to reduce training time from months to days through horizontal scaling.
**Major Distributed Training Frameworks**
| Framework | Developer | Key Feature | Typical Use |
|-----------|----------|------------|------------|
| PyTorch DDP | Meta | Native PyTorch distributed | Standard multi-GPU training |
| DeepSpeed | Microsoft | ZeRO optimizer, pipeline parallelism | Large language models |
| Horovod | Uber → LF AI | Ring-allreduce, easy adoption | Multi-framework support |
| Megatron-LM | NVIDIA | Tensor + pipeline + data parallelism | GPT-scale training |
| JAX/pjit | Google | XLA compiler, automatic sharding | TPU and GPU training |
| ColossalAI | HPC-AI Tech | Heterogeneous, auto-parallelism | Research and production |
**PyTorch DDP (DistributedDataParallel)**
- Each GPU holds full model replica.
- Each GPU processes different data batch (data parallelism).
- Gradient synchronization: All-reduce across GPUs after backward pass.
- **Bucket gradient all-reduce**: Overlaps communication with computation.
- Scales to hundreds of GPUs efficiently for models that fit in single GPU memory.
**DeepSpeed ZeRO Stages**
| Stage | What's Partitioned | Memory Saving |
|-------|-------------------|---------------|
| ZeRO-1 | Optimizer states (Adam momentum, variance) | ~4x |
| ZeRO-2 | + Gradients | ~8x |
| ZeRO-3 | + Model parameters | ~Nx (N = GPU count) |
| ZeRO-Infinity | Offload to CPU/NVMe | Nearly unlimited |
- ZeRO-3 enables training models larger than single GPU memory.
- Communication cost: All-gather parameters before forward/backward, reduce-scatter gradients after.
**Megatron-LM 3D Parallelism**
- **Data Parallelism**: Replicate model, split data.
- **Tensor Parallelism**: Split individual layers across GPUs (within a node, needs fast NVLink).
- **Pipeline Parallelism**: Split model layers sequentially across GPUs.
- Combined: GPT-3 (175B parameters) trained on 1024 A100 GPUs using 3D parallelism.
**Communication Patterns**
| Pattern | Operation | Used By |
|---------|----------|--------|
| All-Reduce | Sum gradients across all GPUs | DDP, Horovod |
| All-Gather | Collect full parameter from shards | ZeRO-3, FSDP |
| Reduce-Scatter | Reduce + distribute shards | ZeRO-2/3 |
| Point-to-Point | Send activation between pipeline stages | Pipeline parallelism |
**Fault Tolerance**
- Checkpointing: Save model/optimizer state periodically.
- Elastic training: Add/remove workers without restart (PyTorch Elastic, Horovod Elastic).
- Communication timeout: Detect and handle straggler or failed nodes.
Distributed training frameworks are **the essential infrastructure for training modern AI** — without them, training a GPT-4-class model (estimated > 1 trillion parameters on tens of thousands of GPUs) would be impossible, making these frameworks as critical to AI progress as the hardware itself.