Pipeline parallelism is a model-parallel strategy that partitions a neural network by depth into consecutive stages, placing each stage's layers on a different GPU. A batch flows through the devices like items on an assembly line: GPU 0 runs the first block of layers, passes its activations to GPU 1 for the next block, and so on. It lets a model too deep to fit in one accelerator's memory span several devices, with each holding only its slice of the layer stack.\n\nIt splits between layers, not within them. Unlike tensor parallelism, which shards a single matrix across GPUs, pipeline parallelism assigns whole contiguous layers to each device — inter-layer rather than intra-layer. Communication is therefore light and point-to-point: only the activation tensor at each stage boundary is sent forward (and the gradient sent back), once per stage crossing, instead of a collective on every layer. That modest, localized traffic is why pipeline parallelism tolerates the slower links between servers, where tensor parallelism would choke.\n\nThe catch is the pipeline bubble. With a single batch, only one stage is busy at a time while the others wait — three of four GPUs idle. The fix is to chop the batch into micro-batches and stream them: as soon as stage 1 finishes micro-batch 1 it starts micro-batch 2, while stage 2 processes micro-batch 1. Once the pipe is full, all stages work in parallel. But filling and draining the pipe still leaves idle time at the edges — the bubble — whose relative cost falls as the number of micro-batches per step rises above the number of stages.\n\n| | Pipeline parallelism | Tensor parallelism |\n|---|---|---|\n| Splits | whole layers into stages | one matrix across GPUs |\n| Communication | activations at stage edges | all-reduce every layer |\n| Traffic pattern | point-to-point | collective |\n| Tolerates slow links | yes (across nodes) | no (needs NVLink) |\n| Main inefficiency | pipeline bubble | per-layer collective |\n\n``svg\n\n``\n\nScheduling is the whole game. Because the bubble and memory footprint depend on how micro-batches are ordered, real systems use schedules — GPipe's fill-then-drain, or interleaved 1F1B (one-forward-one-backward) as in PipeDream/Megatron — to keep more stages busy and cap how many activations must be stored for the backward pass. More micro-batches shrink the bubble but raise activation memory; interleaving stages across GPUs shrinks it further at the cost of extra communication. The art is balancing bubble, memory, and traffic for a given depth and device count.\n\nRead pipeline parallelism through a quant lens rather than a 'chain of GPUs' lens: utilization is 1 − bubble, and the bubble scales roughly as (stages − 1) / (stages − 1 + micro-batches), so throughput is set by how many micro-batches you push per step versus how many stages you span. The design question is that ratio, traded against the activation memory each in-flight micro-batch costs — which is why deep models pick a stage count and micro-batch count together, not a maximum of either.