accelerate
**Hugging Face Accelerate** is a **library that abstracts away the complexity of running PyTorch training across different hardware configurations** — enabling the same training script to run on a single CPU, single GPU, multi-GPU, multi-node cluster, or TPU without rewriting distributed training boilerplate, by wrapping model, optimizer, and dataloader with a single `accelerator.prepare()` call that handles device placement, gradient synchronization, and mixed precision automatically.
**What Is Accelerate?**
- **Definition**: A Python library by Hugging Face that provides a minimal abstraction layer over PyTorch's distributed training capabilities — handling `torch.distributed`, `DataParallel`, `FullyShardedDataParallel` (FSDP), DeepSpeed, and TPU XLA behind a unified interface.
- **The Problem**: Writing PyTorch code that works on both a laptop and a multi-GPU cluster is hard — you need `torch.distributed.launch`, `local_rank` management, gradient accumulation, mixed precision scaling, and device-specific code paths. Accelerate handles all of this.
- **Minimal Code Changes**: Add 4 lines to any PyTorch training script — `accelerator = Accelerator()`, `model, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)`, replace `loss.backward()` with `accelerator.backward(loss)`. Done.
- **Configuration-Driven**: Run `accelerate config` once to set up your environment (number of GPUs, mixed precision, DeepSpeed stage) — then `accelerate launch train.py` runs your script with the configured distributed strategy.
**Key Features**
- **Hardware Agnostic**: The same script runs on CPU, single GPU, multi-GPU (DDP), multi-node, TPU, and Apple Silicon — Accelerate detects the hardware and applies the correct distributed strategy.
- **Mixed Precision**: Automatic FP16/BF16 mixed precision training — `Accelerator(mixed_precision="bf16")` enables mixed precision with no other code changes.
- **DeepSpeed Integration**: Full DeepSpeed ZeRO Stage 1/2/3 support — configure via `accelerate config` or a DeepSpeed config JSON, no DeepSpeed-specific code in your training script.
- **FSDP Support**: PyTorch FullyShardedDataParallel for training models that don't fit on a single GPU — shard model parameters, gradients, and optimizer states across GPUs.
- **Gradient Accumulation**: `accelerator.accumulate(model)` handles gradient accumulation across steps — essential for simulating large batch sizes on limited GPU memory.
- **Big Model Inference**: `accelerate` can load models larger than GPU memory using device_map="auto" — automatically splitting model layers across multiple GPUs or offloading to CPU/disk.
**Accelerate vs Alternatives**
| Feature | Accelerate | PyTorch DDP (manual) | DeepSpeed (direct) | Lightning |
|---------|-----------|---------------------|-------------------|-----------|
| Code changes | 4 lines | 50+ lines | 30+ lines | Rewrite to LightningModule |
| DeepSpeed support | Yes (config) | No | Native | Yes |
| FSDP support | Yes | Manual | No | Yes |
| TPU support | Yes | No | No | Yes |
| Learning curve | Minimal | High | High | Medium |
| HF ecosystem | Native | Independent | Independent | Independent |
**Hugging Face Accelerate is the "write once, run anywhere" solution for PyTorch distributed training** — adding just 4 lines of code to make any training script hardware-agnostic, with seamless DeepSpeed, FSDP, and mixed precision support that eliminates the distributed training boilerplate that traditionally consumes days of engineering effort.