distributed training

Distributed training splits the computational workload of training neural networks across multiple GPUs, TPUs, or machines to handle models and datasets too large for a single device, reducing training time from months to days or hours through parallel computation. As model sizes have grown from millions to trillions of parameters, distributed training has evolved from a convenience to an absolute necessity — no single device can hold or process modern large language models. Distributed training paradigms include: data parallelism (the most common approach — each device holds a complete model copy and processes a different mini-batch of data, gradients are averaged across devices via all-reduce operations, effectively increasing batch size proportional to device count), model parallelism (splitting the model itself across devices when it exceeds single-device memory — tensor parallelism splits individual layers across devices, pipeline parallelism assigns different layers to different devices), expert parallelism (for MoE models — placing different experts on different devices), fully sharded data parallelism (FSDP/ZeRO — combining aspects of data and model parallelism by sharding model parameters, gradients, and optimizer states across devices while computing with the full model through all-gather operations), and hybrid parallelism (combining multiple strategies — e.g., tensor parallelism within a node and data parallelism across nodes). Communication frameworks include: NCCL (NVIDIA Collective Communications Library — optimized GPU-to-GPU communication), Gloo (CPU-based collective operations), and MPI (traditional message passing). Key challenges include: communication overhead (gradient synchronization becomes a bottleneck — mitigated through gradient compression, asynchronous updates, or communication-computation overlap), memory management (each parallelism strategy has different memory profiles), fault tolerance (handling device failures during multi-day training runs — checkpoint/restart), and scaling efficiency (maintaining near-linear speedup as device count increases). Training frameworks like PyTorch FSDP, DeepSpeed, Megatron-LM, and JAX/XLA with pjit provide implementations of these strategies.