pipeline parallelism model parallel

**Pipeline Parallelism** is a **model parallelism technique that divides neural network layers across multiple devices, enabling concurrent forward and backward passes on different micro-batches to hide latency and maintain high GPU utilization.** **GPipe and Synchronous Pipelining** - **GPipe Architecture (Google)**: First practical pipeline parallelism at scale. Splits model layers across sequential GPU stages (Stage_0 → Stage_1 → ... → Stage_N). - **Micro-Batching Strategy**: Input batch (size B) divided into M micro-batches (size B/M). Each micro-batch propagates sequentially through pipeline stages. - **Forward Pass Pipelining**: Stage 0 computes micro-batch 1 while Stage 1 computes micro-batch 0. Overlaps computation across stages, reducing idle time. - **Gradient Accumulation**: Gradients from M micro-batches accumulated and applied once (equivalent to large-batch training). Effective batch size increases without memory pressure. **1F1B (One-Forward-One-Backward) Pipeline Schedule** - **Synchronous Schedule**: GPipe maintains fixed schedule (all F passes before all B passes). Requires buffering all activations until backward phase. - **1F1B Asynchronous Schedule**: Interleaves forward and backward passes. When backward computation available, immediately execute instead of waiting for forward to complete. - **Activation Memory Reduction**: 1F1B reduces peak activation memory from O(N_stage × batch_size × model_depth) to O(batch_size × model_depth) by reusing buffers. - **PipeDream Implementation**: 1F1B extended to handle weight update timing, gradient averaging. Critical for large-scale distributed training. **Pipeline Bubble Overhead** - **Bubble Fraction**: Percentage of GPU cycles spent idle (no useful computation). Bubble = (N_stage - 1) / (N_stage + M - 1), where N_stage = stages, M = micro-batches. - **Minimizing Bubbles**: Increase micro-batches M. With M >> N_stage, bubble fraction approaches (N_stage / M) → 0. Requires sufficient memory bandwidth per GPU. - **Optimal Micro-Batch Count**: Typically M = 3-5 × N_stage balances memory and bubble overhead. For 8 stages, use 24-40 micro-batches. - **Load Imbalance**: Heterogeneous stage sizes (early stages deeper than later) create variable compute time. Faster stages idle, slower stages bottleneck. Requires careful layer partitioning. **Inter-Stage Activation Storage** - **Activation Tensors**: During forward pass, intermediate activations stored at each stage boundary (input to stage, output from stage). Required for backward pass gradient computation. - **Memory Footprint**: Activation memory = (number of micro-batches in-flight) × (activation tensor size per stage) × (number of layers per stage). - **Checkpoint-Recomputation Hybrid**: Store checkpoints at stage boundaries, recompute intermediate activations during backward pass. Reduces memory from O(layers) to O(1) per stage. - **Communication Overhead**: Activations streamed between stages over network (inter-chip or intra-cluster). Bandwidth requirement: ~10-100 GB/s typical for large models. **Communication Overlapping with Computation** - **Pipelining at Machine Level**: While Stage 1 computes backward pass, Stage 0 computes forward pass on next micro-batch. Network communication of activations hidden behind computation. - **Gradient Streaming**: Gradients propagate backward stages asynchronously. All-reduce across replicas (data parallelism + pipeline parallelism) overlapped with forward pass. - **Synchronization Points**: Wait-free pipelines minimize hard synchronization. Soft synchronization (loose coupling) permits stages to operate at slightly different rates. **Real-World Implementation Details** - **Zero Redundancy Optimizer (ZeRO) Integration**: ZeRO stages 1/2/3 combined with pipeline parallelism. Stage 3 (parameter sharding) demands careful activation checkpoint management. - **Gradient Accumulation Steps**: Typically 4-16 gradient accumulation steps combined with 4 micro-batches through 8 pipeline stages. Total effective batch size = 32-128. - **Convergence Properties**: Pipeline parallelism with 1F1B achieves near-identical convergence to sequential training. Hyperparameters transferred between configurations.