network topology

Datacenter network topology is the arrangement of switches and links that decides whether an AI cluster's thousands of GPUs can actually talk fast enough to stay busy. Training a large model is dominated by collective communication, where every GPU must exchange gradients and activations with many others at once, so the fabric that connects them is not a background utility but a first-class part of the machine. The whole field of AI datacenter networking converges on one goal: build a fabric that can carry all-to-all traffic at full bandwidth without becoming the thing that starves the GPUs.\n\n**The workload is all-to-all, so the network has to be effectively non-blocking.** Collective operations such as all-reduce and all-to-all generate simultaneous, full-bandwidth traffic between many pairs of GPUs at the same instant, which is the opposite of the bursty, mostly-idle pattern that classic enterprise networks were designed and oversubscribed for. If the fabric is oversubscribed, those collectives stall and expensive GPUs sit waiting, so AI clusters are built for full bisection bandwidth, meaning any half of the machine can talk to the other half at line rate.\n\n**The spine-leaf Clos fabric, also called a fat-tree, is the workhorse that delivers this.** Each rack's servers connect to a top-of-rack leaf switch, and every leaf connects to every spine switch above it, so any two servers are reachable through the same short, uniform path. Because bandwidth going up to the spine equals bandwidth coming down to the servers, the tree is non-blocking, and it scales to thousands of nodes by adding tiers. Equal-cost multipath routing spreads the many flows of a collective evenly across all the parallel links so no single path becomes a hotspot.\n\n**Rail-optimized topology tailors that fat-tree to the way GPUs actually communicate.** In a rail-optimized design, the same-numbered GPU network port on every server connects to its own dedicated rail switch, so a given GPU can reach the corresponding GPU on any other node in a single hop. This matches the ring and tree patterns that NCCL uses for collectives, letting the high-speed NVLink fabric handle communication inside a node while the rails carry the inter-node legs of the same collective with minimal switch hops and congestion.\n\n**Other topologies trade bandwidth for cost or diameter, and the fabric itself can be InfiniBand or Ethernet.** A torus or mesh wires neighbors directly and is cheap to cable but forces traffic through many hops, while a dragonfly groups nodes and links the groups with a few long global links to keep the network diameter low at extreme scale; both appear in HPC, but the uniform bandwidth of the fat-tree keeps it dominant for AI. Underneath, the links run InfiniBand, with adaptive routing and in-network reduction, or Ethernet with RoCE, now racing to catch up. This whole fabric is the scale-out tier that stitches together the NVLink scale-up domains inside each rack.\n\n| Topology | Bisection bandwidth | Hops / diameter | Cost | Where used |\n|---|---|---|---|---|\n| Spine-leaf / fat-tree | Full, non-blocking | Low, uniform | Higher | Mainstream AI clusters |\n| Rail-optimized fat-tree | Full, GPU-aligned | One hop same-rail | Higher | Large GPU training pods |\n| Torus / mesh | Lower | Many hops | Low | Some HPC systems |\n| Dragonfly | High at scale | Very low diameter | Medium | Large HPC supercomputers |\n\n```svg\n\n```\n\nRead datacenter network topology through a non-blocking-bisection-for-collectives lens rather than a generic-plumbing lens. Once you accept that an AI job is constant all-to-all traffic rather than occasional bursts, the design collapses to a single imperative: give every GPU a fast, uniform path to every other, which is exactly what a spine-leaf fat-tree provides and what rail-optimized wiring sharpens for GPUs, leaving torus and dragonfly as cost-versus-diameter compromises for the HPC world rather than the mainstream AI cluster.