ring all-reduce

Ring all-reduce is a bandwidth-optimal collective communication algorithm that sums a value — typically the gradients in distributed training — across many GPUs so that every GPU ends up with the total. It arranges the workers in a logical ring where each only ever sends to its immediate neighbor, and it moves a fixed amount of data per GPU regardless of how many GPUs participate. That constant per-GPU cost is what lets data-parallel training scale to large clusters without the network collapsing.\n\n**It is the collective under every parallelism scheme.** Data-parallel training must average gradients across replicas each step; tensor parallelism must all-reduce partial matmul outputs every layer. All of these reduce to the same primitive: take one vector per rank, sum them element-wise, and give everyone the result. Ring all-reduce is the standard way to execute that primitive efficiently, which is why it appears inside data parallelism, tensor parallelism, and the collective libraries that drive NVLink and InfiniBand fabrics.\n\n**Two phases move each chunk exactly where it needs to go.** Split each GPU's vector into N chunks. In the first phase, scatter-reduce, the ring runs N−1 steps: at each step every GPU forwards one chunk to its right neighbor and adds the incoming chunk, so after N−1 steps each GPU holds the complete sum of exactly one chunk. In the second phase, all-gather, those completed sums circulate another N−1 steps until every GPU has all N summed chunks — the full result. No central node, no all-to-one bottleneck.\n\n| | Naive (reduce-to-one) | Ring all-reduce |\n|---|---|---|\n| Pattern | all send to a root | neighbor-to-neighbor ring |\n| Steps | ~N (root serializes) | 2(N−1) |\n| Per-GPU bytes | grows with N | ~2× data, flat in N |\n| Bottleneck | root link saturates | none; links balanced |\n| Scales to large N | poorly | yes |\n\n```svg\n\n```\n\n**Bandwidth-optimal, but latency scales with the ring length.** Because every GPU sends and receives simultaneously on each step, all links are kept busy and the total data each GPU moves is about 2·(N−1)/N times the vector size — essentially 2× no matter how big the cluster gets, which is provably minimal for an all-reduce. The cost is latency: the ring takes 2(N−1) hops, so for very large N or tiny messages the per-step latency adds up, and hierarchical or tree-based collectives (reduce within a fast NVLink island, then across nodes) are layered on top to shorten the critical path.\n\nRead ring all-reduce through a quant lens rather than a 'pass it around' lens: the per-GPU traffic is fixed at ~2× the gradient size independent of N, so bandwidth cost does not grow with cluster size — but latency scales as 2(N−1) hops, so the design trade is bandwidth-optimality versus hop count. The engineering question is where the message sits on that curve: large gradients on a fast fabric are bandwidth-bound and love the ring; many small messages are latency-bound and want a tree or a hierarchical collective that reduces the number of sequential hops.