Pipeline Parallelism is the distributed deep learning parallelism strategy that partitions a neural network into sequential stages across multiple GPUs, where each GPU computes one stage and passes activations to the next — enabling training of models too large for a single GPU's memory by distributing layers across devices, with micro-batching to fill the pipeline and minimize the idle "bubble" overhead.

Why Pipeline Parallelism

For models with billions of parameters (GPT-3: 175B, PaLM: 540B), neither data parallelism (replicates the entire model) nor tensor parallelism (splits individual layers) alone is sufficient. Pipeline parallelism splits the model vertically by layer groups — GPU 0 holds layers 1-20, GPU 1 holds layers 21-40, etc. Each GPU only stores its stage's parameters and activations, linearly reducing per-GPU memory.

The Pipeline Bubble Problem

Naive pipeline execution has massive idle time: GPU 0 processes one micro-batch and sends activations to GPU 1, then waits idle while subsequent GPUs process. In backward pass, the last GPU computes gradients first while earlier GPUs wait. The idle fraction (pipeline bubble) is approximately (P-1)/M, where P is the number of pipeline stages and M is the number of micro-batches.

Micro-Batching (GPipe)

GPipe splits each mini-batch into M micro-batches, feeding them into the pipeline in sequence. While GPU 1 processes micro-batch 1, GPU 0 starts micro-batch 2. With enough micro-batches (M >> P), the pipeline stays mostly full. Gradients are accumulated across micro-batches and synchronized at the end of the mini-batch.

Advanced Scheduling

• 1F1B (Interleaved Schedule): Instead of processing all forward passes then all backward passes, PipeDream's 1F1B schedule interleaves one forward and one backward micro-batch per step. This reduces peak activation memory because each stage discards activations after backward, rather than buffering all M micro-batches' activations simultaneously.
• Virtual Pipeline Stages: Megatron-LM assigns multiple non-contiguous layer groups to each GPU (e.g., GPU 0 holds layers 1-5 and layers 21-25). This increases the number of virtual stages without adding GPUs, reducing bubble size at the cost of additional inter-GPU communication.
• Zero Bubble Pipeline: Recent research (Qi et al., 2023) achieves near-zero bubble overhead by overlapping forward, backward, and weight-update computations from different micro-batches, filling every idle slot.

Memory vs. Communication Tradeoff

Pipeline parallelism sends only the activation tensor between stages (not the full gradient or parameter set), making inter-stage communication relatively lightweight compared to data parallelism's allreduce. For models with large hidden dimensions, the activation tensor at the pipeline boundary is small relative to the total computation — making pipeline parallelism bandwidth-efficient.

Pipeline Parallelism is the assembly-line strategy for training massive neural networks — dividing the model into stations, feeding data through in overlapping waves, and engineering the schedule to minimize the idle time when any GPU is waiting for work.