Pipeline Parallelism for Deep Learning is the model parallelism strategy that partitions neural network layers across multiple GPUs in a sequential pipeline, processing different micro-batches simultaneously at different stages — enabling training of models that exceed single-GPU memory while maintaining high utilization through careful scheduling.

Pipeline Partitioning:

• Layer Assignment: neural network layers divided into K stages, each assigned to one GPU — stage k processes layers assigned to it and passes activations to stage k+1
• Memory Balancing: each stage should consume roughly equal memory — earlier stages often have larger activation tensors while later stages have larger parameter tensors; careful partitioning achieves ±10% memory imbalance
• Communication: only activation tensors (forward) and gradient tensors (backward) at stage boundaries need cross-GPU transfer — intra-stage communication uses local GPU memory, minimizing communication overhead
• Stage Count: typically 4-16 stages — more stages reduce per-GPU memory but increase pipeline bubble overhead and inter-stage communication

Pipeline Schedules:

• GPipe (Synchronous): inject all M micro-batches sequentially through the pipeline before performing backward passes — simple to implement but creates large pipeline bubble at startup and drainage (bubble fraction = (K-1)/(M+K-1))
• 1F1B (One Forward One Backward): interleaves forward and backward passes — each stage alternates between processing forward micro-batches and backward micro-batches once the pipeline is full, reducing bubble to (K-1)/(M) of steady-state time
• Interleaved 1F1B: each GPU holds multiple non-consecutive stages (e.g., GPU 0 has stages 0 and 4) — reduces bubble fraction by factor of V (number of chunks per GPU) at cost of additional communication for non-adjacent stages
• Zero-Bubble Pipeline: recent research schedules backward passes for weight gradients (B) and input gradients (W) independently — achieves near-zero bubble overhead by filling idle time with weight gradient computation

Memory Optimization:

• Activation Checkpointing: recompute activations during backward pass instead of storing them — reduces memory from O(layers × batch) to O(sqrt(layers) × batch) at cost of ~33% additional computation
• Micro-Batch Size: smaller micro-batches reduce per-stage memory but increase pipeline startup/drainage overhead — optimal micro-batch count M is typically 4-8× the pipeline depth K
• Tensor Offloading: temporarily offload inactive stage's optimizer states to CPU memory — swap back just before needed; effective when CPU-GPU bandwidth is sufficient

Pipeline parallelism is essential for training the largest neural networks (100B+ parameters) — combined with data parallelism and tensor parallelism in 3D parallelism configurations, it enables models like GPT-4 and PaLM to be trained across thousands of GPUs.