model parallelism strategies

**Model Parallelism Strategies** are **the techniques for distributing a single neural network across multiple GPUs or nodes when the model is too large to fit on a single device — including tensor parallelism (splitting individual layers), pipeline parallelism (distributing layers across devices), and sequence parallelism (partitioning sequence dimension), enabling training and inference of models with hundreds of billions of parameters**. **Tensor Parallelism:** - **Layer Splitting**: splits weight matrices of individual layers across GPUs; for linear layer Y = XW with W of size [d_in, d_out], split W column-wise across N GPUs; each GPU computes partial output, then all-gather to combine results - **Megatron-LM Approach**: splits attention and MLP layers in Transformers; attention: split Q, K, V projections column-wise, output projection row-wise; MLP: split first linear column-wise, second linear row-wise; minimizes communication (2 all-reduce per layer) - **Communication Overhead**: requires all-reduce or all-gather after each split layer; communication volume = batch_size × sequence_length × hidden_dim; high-bandwidth interconnect (NVLink, InfiniBand) essential for efficiency - **Scaling Efficiency**: near-linear scaling up to 8 GPUs per node (NVLink); efficiency drops with inter-node communication; typically combined with data parallelism for larger scale **Pipeline Parallelism:** - **Layer Distribution**: assigns consecutive layers to different GPUs; GPU 0: layers 0-7, GPU 1: layers 8-15, etc.; forward pass flows through pipeline, backward pass flows in reverse - **Naive Pipeline Problem**: GPU 0 processes batch, sends to GPU 1, then idles while GPU 1 processes; severe underutilization (1/N efficiency for N GPUs) - **Micro-Batching (GPipe)**: splits batch into micro-batches; GPU 0 processes micro-batch 1, sends to GPU 1, then processes micro-batch 2; overlaps computation across GPUs; achieves ~80-90% efficiency - **Pipeline Bubble**: idle time at pipeline start (filling) and end (draining); bubble size = (num_stages - 1) × micro_batch_time; smaller micro-batches reduce bubble but increase communication overhead **Advanced Pipeline Techniques:** - **1F1B (One-Forward-One-Backward)**: alternates forward and backward micro-batches; reduces memory usage compared to GPipe (stores fewer activations); PipeDream and Megatron use this schedule - **Interleaved Pipeline**: each GPU handles multiple non-consecutive stages; GPU 0: layers [0-3, 12-15], GPU 1: layers [4-7, 16-19]; reduces bubble size by enabling more overlapping - **Virtual Pipeline Stages**: splits each GPU's layers into multiple virtual stages; increases scheduling flexibility; further reduces bubble at cost of more communication - **Asynchronous Pipeline**: doesn't wait for all micro-batches to complete; uses stale gradients for some updates; trades consistency for throughput; requires careful learning rate tuning **Sequence Parallelism:** - **Sequence Dimension Splitting**: partitions sequence length across GPUs; each GPU processes subset of tokens; used in addition to tensor/pipeline parallelism for very long sequences - **Communication Pattern**: requires all-gather for attention (each token attends to all tokens); all-reduce for gradients; communication volume proportional to sequence length - **Megatron Sequence Parallelism**: splits sequence dimension for LayerNorm and Dropout (operations outside attention/MLP); reduces activation memory without additional communication - **Ring Attention**: processes attention in chunks using ring all-reduce; enables extremely long sequences (millions of tokens); communication overlapped with computation **3D Parallelism:** - **Combining Strategies**: data parallelism (DP) × tensor parallelism (TP) × pipeline parallelism (PP); example: 1024 GPUs = 8 DP × 8 TP × 16 PP - **Dimension Selection**: TP within nodes (high bandwidth), PP across nodes (lower bandwidth), DP for remaining GPUs; matches parallelism strategy to hardware topology - **Megatron-DeepSpeed**: combines Megatron's tensor/pipeline parallelism with DeepSpeed's ZeRO optimizer; enables training trillion-parameter models - **Optimal Configuration Search**: profile different DP/TP/PP combinations; consider model size, batch size, hardware topology; automated tools (Alpa) search configuration space **Memory Optimization:** - **Activation Checkpointing**: recomputes activations during backward pass instead of storing; trades computation for memory; enables 2-4× larger models; selective checkpointing (checkpoint every N layers) balances trade-off - **ZeRO (Zero Redundancy Optimizer)**: partitions optimizer states, gradients, and parameters across data parallel ranks; ZeRO-1 (optimizer states), ZeRO-2 (+gradients), ZeRO-3 (+parameters); reduces memory by DP factor - **Offloading**: stores optimizer states or parameters in CPU memory; loads on-demand during computation; ZeRO-Offload, ZeRO-Infinity enable training models larger than total GPU memory - **Mixed Precision**: uses FP16/BF16 for activations and gradients, FP32 for optimizer states; reduces memory by 50% for activations; requires loss scaling (FP16) or is numerically stable (BF16) **Communication Optimization:** - **Gradient Accumulation**: accumulates gradients over multiple micro-batches before communication; reduces communication frequency; effective batch size = micro_batch_size × accumulation_steps × DP_size - **Communication Overlap**: overlaps gradient all-reduce with backward computation; starts communication as soon as layer gradients are ready; requires careful scheduling - **Compression**: compresses gradients before communication; FP16 instead of FP32 (2× reduction), or quantization to INT8 (4× reduction); trades accuracy for bandwidth - **Hierarchical Communication**: all-reduce within nodes (fast NVLink), then across nodes (slower InfiniBand); reduces cross-node traffic; NCCL automatically optimizes communication topology **Framework Support:** - **Megatron-LM (NVIDIA)**: tensor and pipeline parallelism for Transformers; highly optimized for NVIDIA GPUs; used for training GPT, BERT, T5 at scale - **DeepSpeed (Microsoft)**: ZeRO optimizer, pipeline parallelism, and 3D parallelism; supports PyTorch; extensive optimization for large-scale training - **Alpa (UC Berkeley)**: automatic parallelization; searches for optimal DP/TP/PP configuration; compiler-based approach; supports JAX - **Fairscale (Meta)**: modular parallelism components for PyTorch; FSDP (Fully Sharded Data Parallel) similar to ZeRO-3; easier integration than DeepSpeed **Practical Considerations:** - **Batch Size Scaling**: larger parallelism requires larger batch sizes for efficiency; global_batch_size = micro_batch_size × gradient_accumulation × DP_size; very large batches may hurt convergence - **Learning Rate Tuning**: linear scaling rule (LR ∝ batch_size) often works; warmup critical for large batches; may need to tune for specific model/dataset - **Debugging Complexity**: distributed training failures are hard to debug; use smaller scale for initial debugging; comprehensive logging and monitoring essential - **Cost-Performance Trade-off**: more GPUs = faster training but higher cost; find sweet spot where training time is acceptable and cost is reasonable; consider spot instances for cost savings Model parallelism strategies are **the enabling technology for frontier AI models — without tensor, pipeline, and sequence parallelism, training GPT-4, Llama 3, and other hundred-billion-parameter models would be impossible, making these techniques essential for pushing the boundaries of AI capability**.