model parallelism

**Model Parallelism** — splitting a model across multiple GPUs when it's too large to fit in a single GPU's memory, using tensor parallelism (split layers) and/or pipeline parallelism (split stages). **Tensor Parallelism (TP)** - Split individual layers across GPUs - Example: A 4096×4096 weight matrix split across 4 GPUs → each holds 4096×1024 - Each GPU computes partial result → AllReduce to combine - Requires high-bandwidth GPU interconnect (NVLink) — very communication-heavy - Typically within a single node (8 GPUs) **Pipeline Parallelism (PP)** - Assign different model layers to different GPUs - GPU0: Layers 1-10, GPU1: Layers 11-20, GPU2: Layers 21-30, ... - Data flows through GPUs sequentially - **Problem**: Naive approach → only one GPU active at a time (pipeline bubble) - **Solution**: Micro-batching (GPipe) — split batch into micro-batches, pipeline them **3D Parallelism (TP + PP + DP)** - Used for the largest models (GPT-4, PaLM, LLaMA 405B) - TP within nodes (8 GPUs), PP across node groups, DP across node groups - Example: 1024 GPUs = 8-way TP × 16-way PP × 8-way DP **When to Use What** - Model fits on 1 GPU → Data Parallelism only - Model fits on 1 node → TP within node + DP across nodes - Model doesn't fit on 1 node → Full 3D parallelism **Model parallelism** is essential for training the frontier models that define modern AI — without it, models beyond ~10B parameters would be impossible to train.