pipeline parallelism deep learning

**Pipeline Parallelism for Deep Learning** is **the model parallelism strategy that partitions neural network layers across multiple GPUs in a sequential pipeline, processing different micro-batches simultaneously at different stages — enabling training of models that exceed single-GPU memory while maintaining high utilization through careful scheduling**. **Pipeline Partitioning:** - **Layer Assignment**: neural network layers divided into K stages, each assigned to one GPU — stage k processes layers assigned to it and passes activations to stage k+1 - **Memory Balancing**: each stage should consume roughly equal memory — earlier stages often have larger activation tensors while later stages have larger parameter tensors; careful partitioning achieves ±10% memory imbalance - **Communication**: only activation tensors (forward) and gradient tensors (backward) at stage boundaries need cross-GPU transfer — intra-stage communication uses local GPU memory, minimizing communication overhead - **Stage Count**: typically 4-16 stages — more stages reduce per-GPU memory but increase pipeline bubble overhead and inter-stage communication **Pipeline Schedules:** - **GPipe (Synchronous)**: inject all M micro-batches sequentially through the pipeline before performing backward passes — simple to implement but creates large pipeline bubble at startup and drainage (bubble fraction = (K-1)/(M+K-1)) - **1F1B (One Forward One Backward)**: interleaves forward and backward passes — each stage alternates between processing forward micro-batches and backward micro-batches once the pipeline is full, reducing bubble to (K-1)/(M) of steady-state time - **Interleaved 1F1B**: each GPU holds multiple non-consecutive stages (e.g., GPU 0 has stages 0 and 4) — reduces bubble fraction by factor of V (number of chunks per GPU) at cost of additional communication for non-adjacent stages - **Zero-Bubble Pipeline**: recent research schedules backward passes for weight gradients (B) and input gradients (W) independently — achieves near-zero bubble overhead by filling idle time with weight gradient computation **Memory Optimization:** - **Activation Checkpointing**: recompute activations during backward pass instead of storing them — reduces memory from O(layers × batch) to O(sqrt(layers) × batch) at cost of ~33% additional computation - **Micro-Batch Size**: smaller micro-batches reduce per-stage memory but increase pipeline startup/drainage overhead — optimal micro-batch count M is typically 4-8× the pipeline depth K - **Tensor Offloading**: temporarily offload inactive stage's optimizer states to CPU memory — swap back just before needed; effective when CPU-GPU bandwidth is sufficient **Pipeline parallelism is essential for training the largest neural networks (100B+ parameters) — combined with data parallelism and tensor parallelism in 3D parallelism configurations, it enables models like GPT-4 and PaLM to be trained across thousands of GPUs.**