data parallel training

**Data Parallelism in Distributed Training** is the **most widely used distributed deep learning strategy where the model is replicated across N GPUs, each processing 1/N of the training batch independently, then all GPUs synchronize their gradients through an all-reduce operation before updating the identical model copies — achieving near-linear throughput scaling with GPU count while requiring no model partitioning, making it the default approach for training models that fit in a single GPU's memory**. **How Data Parallelism Works** 1. **Replication**: The same model (weights, optimizer states) is copied to each of N GPUs. 2. **Data Sharding**: Each mini-batch is divided into N micro-batches. GPU i processes micro-batch i. 3. **Forward + Backward**: Each GPU independently computes forward pass and gradients on its micro-batch. 4. **Gradient All-Reduce**: All GPUs sum their gradients using an all-reduce collective operation (ring, tree, or NCCL-optimized algorithm). After all-reduce, every GPU has the identical averaged gradient. 5. **Weight Update**: Each GPU applies the averaged gradient to update its local model copy. Since all GPUs start with the same weights and apply the same gradient, models remain synchronized. **Scaling Efficiency** - **Ideal**: N GPUs → N× throughput (samples/second). - **Actual**: Communication overhead reduces efficiency. At 8 GPUs on NVLink (900 GB/s), efficiency is typically 95-99%. At 1000 GPUs across network (200 Gbps InfiniBand per GPU), efficiency drops to 70-90% depending on model size and batch size. - **Communication Cost**: All-reduce transfers 2×(N-1)/N × model_size bytes. For a 7B parameter model in FP16 (14 GB), each all-reduce moves ~28 GB. At 200 Gbps per GPU, this takes ~1.1 seconds — acceptable only if the compute time per micro-batch is significantly longer. **Large Batch Training Challenges** Scaling from N=1 to N=1024 multiplies the effective batch size by 1024. Large batches can degrade model quality: - **Learning Rate Scaling**: Linear scaling rule — multiply LR by N when multiplying batch size by N (up to a threshold). Gradual warmup (start with small LR, ramp up over 5-10 epochs) stabilizes early training. - **LARS/LAMB Optimizers**: Layer-wise Adaptive Rate Scaling adjusts LR per parameter layer based on the ratio of weight norm to gradient norm. Enables stable training at batch sizes of 32K-64K. **PyTorch DistributedDataParallel (DDP)** The standard implementation: - **Gradient Bucketing**: Gradients are grouped into buckets (~25 MB) for all-reduce. Bucketing amortizes all-reduce overhead and enables overlap — all-reduce of bucket 1 starts while backward pass computes gradients for bucket 2. - **Gradient Compression**: Optional gradient quantization (1-bit, top-k sparsification) reduces communication volume at the cost of convergence speed. Data Parallelism is **the workhorse of distributed training** — simple to implement, requiring no model architecture changes, and scaling efficiently to hundreds of GPUs for models that fit in single-GPU memory, processing training datasets at throughputs that make large-scale AI development practical.