pipeline parallelism llm training

**Pipeline Parallelism for LLM Training** is **a model parallelism strategy that partitions a large neural network into sequential stages assigned to different devices, processing multiple micro-batches simultaneously through the pipeline to maximize hardware utilization** — this approach is essential for training models too large to fit on a single GPU while maintaining high throughput. **Pipeline Parallelism Fundamentals:** - **Stage Partitioning**: the model is divided into K contiguous groups of layers (stages), each assigned to a separate GPU — for a 96-layer transformer, 8 GPUs would each handle 12 layers - **Micro-Batching**: the global mini-batch is split into M micro-batches that flow through the pipeline sequentially — while stage K processes micro-batch m, stage K-1 can process micro-batch m+1, enabling concurrent execution - **Pipeline Bubble**: at the start and end of each mini-batch, some stages are idle waiting for data to flow through — the bubble fraction is approximately (K-1)/(M+K-1), so more micro-batches reduce overhead - **Memory vs. Throughput Tradeoff**: more stages reduce per-GPU memory requirements but increase pipeline bubble overhead and inter-stage communication **GPipe Schedule:** - **Forward Pass First**: all M micro-batches execute their forward passes sequentially through all K stages before any backward pass begins — requires storing O(M×K) activations in memory - **Backward Pass**: after all forwards complete, backward passes execute in reverse order through the pipeline — gradient accumulation across micro-batches before optimizer step - **Bubble Fraction**: with M micro-batches and K stages, the bubble is (K-1)/M of total compute time — GPipe recommends M ≥ 4K to keep bubble under 25% - **Memory Impact**: storing all intermediate activations for M micro-batches is costly — activation checkpointing reduces memory from O(M×K×L) to O(M×K) by recomputing activations during backward pass **1F1B (One Forward One Backward) Schedule:** - **Interleaved Execution**: after the pipeline fills (K-1 forward passes), each stage alternates between one forward and one backward pass — steady-state pattern is F-B-F-B-F-B - **Memory Advantage**: only K micro-batches' activations are stored simultaneously (rather than M in GPipe) — reduces peak memory by M/K factor - **Same Bubble**: the 1F1B schedule has the same bubble fraction as GPipe — (K-1)/(M+K-1) — but dramatically lower memory requirements - **PipeDream Flush**: variant that accumulates gradients across micro-batches and performs a single optimizer step per mini-batch — avoids weight staleness issues of the original PipeDream **Interleaved Pipeline Parallelism (Megatron-LM):** - **Virtual Stages**: each GPU holds multiple non-contiguous stages (e.g., GPU 0 handles stages 0, 4, 8 in a 12-stage pipeline across 4 GPUs) — creates a virtual pipeline of V×K stages - **Reduced Bubble**: bubble fraction decreases to (K-1)/(V×M+K-1) where V is the number of virtual stages per GPU — with V=4, bubble overhead drops by ~4× compared to standard pipeline - **Increased Communication**: non-contiguous stage assignment requires more inter-GPU communication since activations must travel between GPUs more frequently - **Optimal Balance**: typically V=2-4 provides the best tradeoff between reduced bubble and increased communication overhead **Integration with Other Parallelism Dimensions:** - **3D Parallelism**: combines pipeline parallelism (inter-layer), tensor parallelism (intra-layer), and data parallelism — standard approach for training 100B+ parameter models - **Megatron-LM Configuration**: for a 175B parameter model across 1024 GPUs — 8-way tensor parallelism × 16-way pipeline parallelism × 8-way data parallelism - **Stage Balancing**: unequal computation per stage (embedding layers vs. transformer blocks) creates load imbalance — careful partitioning ensures <5% imbalance across stages - **Cross-Stage Communication**: activation tensors transferred between pipeline stages via point-to-point GPU communication (NCCL send/recv) — bandwidth requirement scales with hidden dimension and micro-batch size **Challenges and Solutions:** - **Weight Staleness**: in async pipeline approaches, different micro-batches see different weight versions — PipeDream-2BW maintains two weight versions to bound staleness - **Batch Normalization**: running statistics computed on micro-batches within a single stage don't reflect global batch statistics — Layer Normalization (used in transformers) avoids this issue entirely - **Fault Tolerance**: if one stage's GPU fails, the entire pipeline stalls — elastic pipeline rescheduling can reassign stages to remaining GPUs with temporary throughput reduction **Pipeline parallelism enables training models with trillions of parameters by distributing memory requirements across many devices, but achieving >80% hardware utilization requires careful balancing of micro-batch count, stage partitioning, and integration with tensor and data parallelism.**