collective

NCCL (NVIDIA Collective Communications Library) provides highly optimized GPU-to-GPU communication primitives for multi-GPU and multi-node deep learning training, with AllReduce being the cornerstone operation for gradient synchronization in data-parallel training. AllReduce operation: combines values from all GPUs (sum, mean) and distributes result to all; used to aggregate gradients across workers before parameter update. NCCL optimizations: ring-based algorithms (bandwidth-optimal for uniform topology), tree algorithms (latency-optimal for large messages), and topology-aware routing (exploiting NVLink, PCIe hierarchy). Key collectives: AllReduce (gradient aggregation), Broadcast (distribute model from rank 0), AllGather (collect values from all ranks), and ReduceScatter (reduce then scatter). Multi-node: NCCL extends across nodes using RDMA (InfiniBand, RoCE) or TCP/IP; automatically selects best transport. Integration: PyTorch, TensorFlow, and other frameworks use NCCL backend for distributed training. Performance tuning: NCCL_DEBUG for diagnostics, NCCL_ALGO for algorithm selection, and topology configuration for complex networks. Alternatives: Gloo (CPU-focused), MPI (general-purpose), and custom implementations. NCCL has become the de facto standard for multi-GPU deep learning, making efficient collective communication accessible without manual optimization.