pipeline parallel
**Pipeline and Tensor Parallelism**
**Tensor Parallelism (TP)**
Split individual layers across GPUs, processing the same batch together.
**How It Works**
For a linear layer $Y = XW$:
- Split W column-wise across GPUs
- Each GPU computes partial result
- AllGather to combine
```
GPU 0: X × W[:, :d/2] → Y0
GPU 1: X × W[:, d/2:] → Y1
AllGather: [Y0 | Y1] → Y
```
**Benefits**
- Low memory per GPU
- Same batch processed on all GPUs
- Low latency (within-layer parallelism)
**Challenges**
- Frequent communication (every layer)
- Best for fast interconnects (NVLink)
**Pipeline Parallelism (PP)**
Split layers sequentially across GPUs.
**How It Works**
```
GPU 0: Layers 0-7 → activations →
GPU 1: Layers 8-15 → activations →
GPU 2: Layers 16-23 → activations →
GPU 3: Layers 24-31 → output
```
**Micro-batching**
To avoid GPU idle time (bubble), split batch into micro-batches:
```
Time →
GPU 0: [μ1][μ2][μ3][μ4]
GPU 1: [μ1][μ2][μ3][μ4]
GPU 2: [μ1][μ2][μ3][μ4]
```
**Schedule Types**
| Schedule | Bubble Overhead | Memory |
|----------|-----------------|--------|
| GPipe | High | Low |
| 1F1B | Lower | Higher |
| Interleaved 1F1B | Lowest | Higher |
**Combining Strategies**
**3D Parallelism**
```
[Data Parallel]
↓
[Tensor Parallel across GPUs in same node]
↓
[Pipeline Parallel across nodes]
```
**Example: 32 GPUs, 4 nodes**
- TP=4 (within node, NVLink)
- PP=4 (across nodes)
- DP=2 (replication)
- Total: 4 × 4 × 2 = 32 GPUs
**When to Use What**
| Constraint | Strategy |
|------------|----------|
| Model fits in GPU | DDP only |
| Model larger than GPU | Add FSDP/ZeRO or TP |
| Very large model | Combine TP + PP + DP |
| Slow interconnect | More PP, less TP |
| Fast NVLink | More TP |