Ring all-reduce is a bandwidth-optimal collective communication algorithm that sums a value — typically the gradients in distributed training — across many GPUs so that every GPU ends up with the total. It arranges the workers in a logical ring where each only ever sends to its immediate neighbor, and it moves a fixed amount of data per GPU regardless of how many GPUs participate. That constant per-GPU cost is what lets data-parallel training scale to large clusters without the network collapsing.\n\nIt is the collective under every parallelism scheme. Data-parallel training must average gradients across replicas each step; tensor parallelism must all-reduce partial matmul outputs every layer. All of these reduce to the same primitive: take one vector per rank, sum them element-wise, and give everyone the result. Ring all-reduce is the standard way to execute that primitive efficiently, which is why it appears inside data parallelism, tensor parallelism, and the collective libraries that drive NVLink and InfiniBand fabrics.\n\nTwo phases move each chunk exactly where it needs to go. Split each GPU's vector into N chunks. In the first phase, scatter-reduce, the ring runs N−1 steps: at each step every GPU forwards one chunk to its right neighbor and adds the incoming chunk, so after N−1 steps each GPU holds the complete sum of exactly one chunk. In the second phase, all-gather, those completed sums circulate another N−1 steps until every GPU has all N summed chunks — the full result. No central node, no all-to-one bottleneck.\n\n| | Naive (reduce-to-one) | Ring all-reduce |\n|---|---|---|\n| Pattern | all send to a root | neighbor-to-neighbor ring |\n| Steps | ~N (root serializes) | 2(N−1) |\n| Per-GPU bytes | grows with N | ~2× data, flat in N |\n| Bottleneck | root link saturates | none; links balanced |\n| Scales to large N | poorly | yes |\n\n``svg\n\n``\n\nBandwidth-optimal, but latency scales with the ring length. Because every GPU sends and receives simultaneously on each step, all links are kept busy and the total data each GPU moves is about 2·(N−1)/N times the vector size — essentially 2× no matter how big the cluster gets, which is provably minimal for an all-reduce. The cost is latency: the ring takes 2(N−1) hops, so for very large N or tiny messages the per-step latency adds up, and hierarchical or tree-based collectives (reduce within a fast NVLink island, then across nodes) are layered on top to shorten the critical path.\n\nRead ring all-reduce through a quant lens rather than a 'pass it around' lens: the per-GPU traffic is fixed at ~2× the gradient size independent of N, so bandwidth cost does not grow with cluster size — but latency scales as 2(N−1) hops, so the design trade is bandwidth-optimality versus hop count. The engineering question is where the message sits on that curve: large gradients on a fast fabric are bandwidth-bound and love the ring; many small messages are latency-bound and want a tree or a hierarchical collective that reduces the number of sequential hops.