distributed training data parallelism

**Data Parallel Distributed Training** is **the most widely used strategy for scaling deep learning training across multiple GPUs or nodes by replicating the entire model on each worker, partitioning training data across workers, and synchronizing gradients after each mini-batch to maintain model consistency**. **DDP Architecture (PyTorch):** - **Process Group**: each GPU runs in its own process with a full model replica — NCCL backend provides optimized GPU-to-GPU collective communication (ring AllReduce, tree AllReduce) - **Gradient Bucketing**: instead of reducing each parameter individually, gradients are grouped into buckets (25 MB default) and AllReduced bucket-by-bucket — bucketing amortizes communication launch overhead and enables overlap with backward pass - **Backward-Communication Overlap**: AllReduce for a gradient bucket begins as soon as all gradients in that bucket are computed — while later layers are still computing backward pass, earlier layer gradients are already being communicated - **Gradient Compression**: optional gradient compression (quantization to FP16/INT8, sparsification keeping only top-K%) reduces communication volume at the cost of slight accuracy degradation — most effective when communication is the bottleneck **Scaling Considerations:** - **Batch Size Scaling**: total effective batch size = per-GPU batch size × number of GPUs — learning rate typically scaled linearly with batch size (linear scaling rule) with warmup period for first few epochs - **Communication Overhead**: AllReduce time scales as 2(N-1)/N × model_size / bandwidth — for a 10B parameter model on a 400 Gbps network, AllReduce takes ~40 ms per step - **Computation-Communication Ratio**: scaling efficiency = time_single_GPU / (time_N_GPUs × N) — efficiency >90% achievable when computation time >> communication time (large models, large batch sizes) - **Gradient Staleness**: synchronous DDP guarantees zero staleness but synchronization barriers limit scalability — asynchronous alternatives (Hogwild, local SGD) reduce barriers but may affect convergence **Advanced Techniques:** - **FSDP (Fully Sharded Data Parallel)**: each GPU holds only a shard of each parameter tensor; parameters gathered just before forward/backward computation and discarded after — reduces per-GPU memory from O(model_size) to O(model_size/N), enabling training of models too large for single-GPU memory - **ZeRO Optimization**: DeepSpeed ZeRO partitions optimizer states (Stage 1), gradients (Stage 2), and parameters (Stage 3) across GPUs — Stage 1 alone reduces per-GPU memory by 4× for Adam optimizer - **Gradient Accumulation**: perform multiple forward/backward passes before reducing gradients — simulates larger batch sizes without additional GPUs, useful when GPU memory limits per-step batch size **Data parallel training is the foundational distributed technique that has enabled training billion-parameter models — understanding DDP, FSDP, and communication optimization is essential for any engineer working on large-scale AI training infrastructure.**