Pipeline and Tensor Parallelism

Tensor Parallelism (TP) Split individual layers across GPUs, processing the same batch together.

How It Works For a linear layer $Y = XW$:

• Split W column-wise across GPUs
• Each GPU computes partial result
• AllGather to combine

GPU 0: X × W[:, :d/2] → Y0
GPU 1: X × W[:, d/2:] → Y1
AllGather: [Y0 | Y1] → Y

Benefits

• Low memory per GPU
• Same batch processed on all GPUs
• Low latency (within-layer parallelism)

Challenges

• Frequent communication (every layer)
• Best for fast interconnects (NVLink)

Pipeline Parallelism (PP) Split layers sequentially across GPUs.

How It Works

GPU 0: Layers 0-7    → activations →
GPU 1: Layers 8-15   → activations →
GPU 2: Layers 16-23  → activations →
GPU 3: Layers 24-31  → output

Micro-batching To avoid GPU idle time (bubble), split batch into micro-batches:

Time →
GPU 0: [μ1][μ2][μ3][μ4]
GPU 1:     [μ1][μ2][μ3][μ4]
GPU 2:         [μ1][μ2][μ3][μ4]

Schedule Types

Combining Strategies

3D Parallelism

[Data Parallel]
    ↓
[Tensor Parallel across GPUs in same node]
    ↓
[Pipeline Parallel across nodes]

Example: 32 GPUs, 4 nodes

• TP=4 (within node, NVLink)
• PP=4 (across nodes)
• DP=2 (replication)
• Total: 4 × 4 × 2 = 32 GPUs

When to Use What