pipeline parallelism

Pipeline parallelism is a model-parallel strategy that partitions a neural network by depth into consecutive stages, placing each stage's layers on a different GPU. A batch flows through the devices like items on an assembly line: GPU 0 runs the first block of layers, passes its activations to GPU 1 for the next block, and so on. It lets a model too deep to fit in one accelerator's memory span several devices, with each holding only its slice of the layer stack.\n\n**It splits between layers, not within them.** Unlike tensor parallelism, which shards a single matrix across GPUs, pipeline parallelism assigns whole contiguous layers to each device — inter-layer rather than intra-layer. Communication is therefore light and point-to-point: only the activation tensor at each stage boundary is sent forward (and the gradient sent back), once per stage crossing, instead of a collective on every layer. That modest, localized traffic is why pipeline parallelism tolerates the slower links between servers, where tensor parallelism would choke.\n\n**The catch is the pipeline bubble.** With a single batch, only one stage is busy at a time while the others wait — three of four GPUs idle. The fix is to chop the batch into micro-batches and stream them: as soon as stage 1 finishes micro-batch 1 it starts micro-batch 2, while stage 2 processes micro-batch 1. Once the pipe is full, all stages work in parallel. But filling and draining the pipe still leaves idle time at the edges — the bubble — whose relative cost falls as the number of micro-batches per step rises above the number of stages.\n\n| | Pipeline parallelism | Tensor parallelism |\n|---|---|---|\n| Splits | whole layers into stages | one matrix across GPUs |\n| Communication | activations at stage edges | all-reduce every layer |\n| Traffic pattern | point-to-point | collective |\n| Tolerates slow links | yes (across nodes) | no (needs NVLink) |\n| Main inefficiency | pipeline bubble | per-layer collective |\n\n```svg ```\n\n**Scheduling is the whole game.** Because the bubble and memory footprint depend on how micro-batches are ordered, real systems use schedules — GPipe's fill-then-drain, or interleaved 1F1B (one-forward-one-backward) as in PipeDream/Megatron — to keep more stages busy and cap how many activations must be stored for the backward pass. More micro-batches shrink the bubble but raise activation memory; interleaving stages across GPUs shrinks it further at the cost of extra communication. The art is balancing bubble, memory, and traffic for a given depth and device count.\n\nRead pipeline parallelism through a quant lens rather than a 'chain of GPUs' lens: utilization is 1 − bubble, and the bubble scales roughly as (stages − 1) / (stages − 1 + micro-batches), so throughput is set by how many micro-batches you push per step versus how many stages you span. The design question is that ratio, traded against the activation memory each in-flight micro-batch costs — which is why deep models pick a stage count and micro-batch count together, not a maximum of either.