Pipeline parallelism splits model into sequential stages, each on different device, processing micro-batches in pipeline fashion. How it works: Divide model into N stages (e.g., layers 1-10, 11-20, 21-30, 31-40 for 4 stages). Each device handles one stage. Pipeline execution: Split batch into micro-batches. While device 2 processes micro-batch 1, device 1 processes micro-batch 2. Overlapping computation. Bubble overhead: Pipeline startup and drain time where some devices idle. Larger number of micro-batches reduces bubble fraction. Schedules: GPipe: Simple schedule, all forward then all backward. Large memory (activations stored). PipeDream: 1F1B schedule interleaves forward/backward. Lower memory. Memory trade-off: Must store activations at stage boundaries for backward pass. Activation checkpointing reduces memory at compute cost. Communication: Only stage boundaries communicate (activation tensors). Less frequent than tensor parallelism. Scaling: Useful for very deep models. Combines with tensor and data parallelism for large-scale training. Frameworks: DeepSpeed, Megatron-LM, PyTorch pipelines. Challenges: Load balancing across stages, batch size constraints, complexity of scheduling.