nccl

NCCL (the NVIDIA Collective Communications Library, pronounced 'nickel') is the software layer that implements the collective operations distributed training depends on — all-reduce, all-gather, broadcast, reduce-scatter, and more — and maps each one onto the actual interconnects in the machine. A framework calls a single collective; NCCL detects the topology and executes it as the right ring or tree across NVLink, PCIe, and InfiniBand, so application code never has to know how the GPUs are wired.\n\n**It separates what you want from how it runs.** Training code (PyTorch DDP/FSDP, Megatron, DeepSpeed) issues a logical operation like ncclAllReduce over a group of GPUs. NCCL turns that into a concrete schedule of point-to-point transfers tuned to the hardware — choosing a ring for bandwidth-bound large messages or a tree to cut latency for small ones, and sizing chunks to keep every link busy. The same call runs correctly on one node of NVLink-connected GPUs or a thousand-GPU InfiniBand cluster; only NCCL's plan changes.\n\n**It is topology-aware and hierarchical.** NCCL discovers which GPUs share fast NVLink/NVSwitch links and which are reachable only over PCIe or the InfiniBand/RoCE network, then builds a multi-level collective: reduce within each node over NVLink first, exchange the partial results once across the slow inter-node link, then broadcast back down. That hierarchy minimizes traffic on the scarcest resource — the network between nodes — which is exactly the scale-up-then-scale-out structure defined by NVLink and InfiniBand.\n\n| Concern | Handled by | NCCL's job |\n|---|---|---|\n| What operation | framework (DDP/FSDP) | expose collective API |\n| Which algorithm | NCCL | pick ring vs tree |\n| Which links | NCCL | NVLink / PCIe / IB routing |\n| Overlap with compute | NCCL + CUDA streams | async, non-blocking |\n| Portability | NCCL | same call, any topology |\n\n```svg\n\n```\n\n**It overlaps communication with compute, which is where the speed comes from.** NCCL runs collectives asynchronously on CUDA streams so the network work hides behind ongoing GPU math instead of stalling it — the difference between near-linear scaling and a cluster that spends its time waiting. Because it is the near-universal backend for multi-GPU training, NCCL's algorithm choices and topology detection largely determine how efficiently a given model layout uses the fabric; tuning distributed training is often as much about NCCL's behavior as about the model.\n\nRead NCCL through a quant lens rather than a 'communication library' lens: it is an optimizer that, given a message size and a measured topology, chooses the algorithm (ring for bandwidth, tree for latency) and the link hierarchy that minimize step time — the software realization of the ring-all-reduce and NVLink/InfiniBand tradeoffs. The design question NCCL answers automatically is the one every collective faces: for this many bytes across this many GPUs on these links, which schedule keeps the slowest link busiest while overlapping with compute.