nccl

**NCCL (NVIDIA Collective Communications Library)** — a high-performance library for multi-GPU and multi-node collective communication operations, essential for distributed deep learning training. **What NCCL Does** - Optimized implementations of collective operations across GPUs: - **AllReduce**: Sum/average gradients across all GPUs (most used in training) - **AllGather**: Each GPU sends its data to all others - **ReduceScatter**: Reduce + scatter result across GPUs - **Broadcast**: One GPU sends to all - **AllToAll**: Full exchange between all GPUs **Why NCCL Matters for Training** - Distributed training: Each GPU computes gradients on its data batch - Before weight update: AllReduce to average gradients across all GPUs - NCCL makes this AllReduce as fast as possible **Communication Backends** - **NVLink**: GPU-to-GPU direct (900 GB/s on H100) - **PCIe**: Older/cheaper (25-64 GB/s) - **InfiniBand**: Multi-node (400 Gbps HDR → 50 GB/s per link) - NCCL automatically selects the best topology and algorithm **Ring vs Tree AllReduce** - **Ring**: Each GPU sends/receives to/from neighbors in a ring. Bandwidth-optimal for large messages - **Tree**: Hierarchical reduce. Latency-optimal for small messages - NCCL auto-selects based on message size and topology **Usage in Frameworks** - PyTorch DDP uses NCCL by default: `torch.distributed.init_process_group(backend='nccl')` - Transparent to user code in most cases **NCCL** is the hidden backbone of all large-scale GPU training — without it, multi-GPU training would be orders of magnitude slower.