data parallelism

Data parallelism replicates the model on each device and processes different data batches in parallel. **How it works**: Copy complete model to each GPU, each processes different mini-batch, average gradients across devices, update weights synchronously. **Gradient synchronization**: All-reduce operation aggregates gradients across devices. Communication overhead scales with parameter count. **Scaling**: Effective batch size = per-device batch size x number of devices. More devices = larger effective batch. **Advantages**: Simple to implement, near-linear speedup for compute-bound training, well-supported in frameworks. **Limitations**: Each device must fit entire model in memory. Doesnt help if model too large for single GPU. **Communication bottleneck**: Gradient sync can become bottleneck at scale. Gradient compression, async methods help. **Implementation**: PyTorch DDP (DistributedDataParallel), Horovod, DeepSpeed ZeRO (hybrid). **Best practices**: Tune batch size with learning rate (linear scaling rule), use gradient accumulation for larger effective batch. **Combination**: Often combined with other parallelism strategies for large models (e.g., ZeRO, pipeline parallelism).