fsdp fully sharded
**FSDP (Fully Sharded Data Parallel)** is the **PyTorch-native strategy for training large models across multiple GPUs by sharding model parameters, gradients, and optimizer states across all workers** — reducing per-GPU memory by up to Nx (where N is GPU count) compared to standard data parallelism, enabling training of models that would not fit in a single GPU's memory.
**Why Not Standard Data Parallel?**
- **DDP (DistributedDataParallel)**: Full model replica on every GPU.
- 7B parameter model in fp32: 28GB parameters + 28GB gradients + 56GB optimizer (Adam) = 112GB per GPU.
- Even 80GB A100 cannot hold this.
- **FSDP**: Shards all three across GPUs.
- With 8 GPUs: ~14GB per GPU — fits easily.
**FSDP Memory Savings**
| Strategy | Parameters | Gradients | Optimizer States | Total (per GPU) |
|----------|-----------|-----------|-----------------|----------------|
| DDP | Full copy | Full copy | Full copy | ~16× model size |
| ZeRO Stage 1 | Full | Full | Sharded | ~12× |
| ZeRO Stage 2 | Full | Sharded | Sharded | ~8× |
| FSDP / ZeRO Stage 3 | Sharded | Sharded | Sharded | ~16×/N |
**How FSDP Works**
1. **Initialization**: Model parameters are sharded — each GPU holds only 1/N of parameters.
2. **Forward Pass**: Before computing a layer, FSDP **all-gathers** that layer's parameters from all GPUs.
3. **Compute**: Forward computation using full parameters.
4. **Free**: After forward, full parameters freed — only shard retained.
5. **Backward Pass**: Same all-gather for each layer, compute gradients, then **reduce-scatter** gradients.
6. **Optimizer Step**: Each GPU updates only its shard of parameters.
**PyTorch FSDP API**
```python
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = FSDP(
model,
sharding_strategy=ShardingStrategy.FULL_SHARD,
mixed_precision=MixedPrecision(param_dtype=torch.bfloat16),
auto_wrap_policy=size_based_auto_wrap_policy,
)
```
**Key Configuration**
- **Sharding Strategy**: FULL_SHARD (ZeRO-3), SHARD_GRAD_OP (ZeRO-2), NO_SHARD (DDP).
- **Auto Wrap Policy**: Controls which modules are FSDP-wrapped — affects communication granularity.
- **Mixed Precision**: bfloat16 params + float32 reduce → further memory savings.
- **Activation Checkpointing**: Combined with FSDP for maximum memory efficiency.
**FSDP vs. DeepSpeed ZeRO**
- PyTorch FSDP is the native implementation inspired by DeepSpeed ZeRO.
- DeepSpeed: Third-party library with ZeRO-1/2/3, offloading to CPU/NVMe.
- FSDP: First-class PyTorch citizen — tighter integration with PyTorch ecosystem.
- Both achieve similar memory savings; choice depends on ecosystem preference.
FSDP is **the standard approach for training large language models on GPU clusters** — it democratizes large model training by making billion-parameter models trainable on commodity multi-GPU setups that would otherwise require expensive model parallelism engineering.