pipeline parallelism

Pipeline parallelism splits model into sequential stages, each on different device, processing micro-batches in pipeline fashion. **How it works**: Divide model into N stages (e.g., layers 1-10, 11-20, 21-30, 31-40 for 4 stages). Each device handles one stage. **Pipeline execution**: Split batch into micro-batches. While device 2 processes micro-batch 1, device 1 processes micro-batch 2. Overlapping computation. **Bubble overhead**: Pipeline startup and drain time where some devices idle. Larger number of micro-batches reduces bubble fraction. **Schedules**: **GPipe**: Simple schedule, all forward then all backward. Large memory (activations stored). **PipeDream**: 1F1B schedule interleaves forward/backward. Lower memory. **Memory trade-off**: Must store activations at stage boundaries for backward pass. Activation checkpointing reduces memory at compute cost. **Communication**: Only stage boundaries communicate (activation tensors). Less frequent than tensor parallelism. **Scaling**: Useful for very deep models. Combines with tensor and data parallelism for large-scale training. **Frameworks**: DeepSpeed, Megatron-LM, PyTorch pipelines. **Challenges**: Load balancing across stages, batch size constraints, complexity of scheduling.