nvl72

Rack-scale computing is the recognition that a single AI training or inference job has outgrown any one server, so the unit of design has moved from the chip to the entire rack. In a rack-scale system like NVIDIA's GB200 NVL72, seventy-two GPUs across a rack are wired together by a switched NVLink fabric into one memory-coherent domain that software can treat as a single, enormous accelerator. The whole point is to make a rack behave like one giant GPU, so that a model far too large for any single package still sees fast, uniform, high-bandwidth access to all the memory and compute in the rack.\n\n**The unit of computing grew from the chip to the rack because models outgrew everything smaller.** A frontier model's weights, activations, and key-value cache no longer fit in one GPU's memory, or even one server's, so the work must be split across many GPUs that constantly exchange data. The speed of that exchange, not raw compute, becomes the bottleneck. Rack-scale design answers this by pulling as many GPUs as possible into a single tightly coupled domain where they talk over a dedicated fabric rather than over the slower, more distant datacenter network.\n\n**A switched NVLink fabric is what makes seventy-two GPUs act like one.** Inside the rack, every GPU connects not point-to-point but through NVSwitch chips that form an all-to-all fabric, so any GPU can reach any other at the full per-link rate with a single hop. Each Blackwell GPU carries on the order of 1.8 TB/s of NVLink bandwidth, and the rack's fabric aggregates to roughly 130 TB/s, orders of magnitude beyond what PCIe or Ethernet between servers can offer. Because the fabric is flat and non-blocking and the GPUs share a coherent address space, a collective like an all-reduce runs over NVLink at memory-like speed instead of crawling over the network.\n\n**The rack is assembled from compute trays and switch trays, and it must be liquid-cooled.** A GB200 compute tray pairs Grace CPUs with Blackwell GPUs on superchip modules; a full NVL72 stacks eighteen such compute trays with nine NVSwitch trays, joined by a copper NVLink spine running along the back of the rack. Copper is used deliberately: over the short reach inside a rack it moves the bandwidth with far less power and latency than optics would. The density that results, well over a hundred kilowatts in one rack, is far past what air can carry, which is why rack-scale and direct-to-chip liquid cooling arrive together as a package.\n\n**The payoff is a larger coherent domain, which means bigger models, faster collectives, and higher utilization.** Keeping the all-reduce and all-to-all traffic of tensor and expert parallelism inside the NVLink domain, rather than spilling it onto Ethernet, sharply cuts the time GPUs spend waiting on each other, so a training step finishes faster and the expensive silicon stays busy. For inference, the shared fabric lets a huge model and its KV cache be sharded across the whole rack while still serving tokens with low latency. Racks are then scaled out to thousands of GPUs over InfiniBand or Ethernet, giving a two-tier picture: scale up inside the rack over NVLink, scale out between racks over the network.\n\n| Attribute | Traditional GPU server | Rack-scale (NVL72-style) |\n|---|---|---|\n| Coherent domain | 8 GPUs in one box | 72 GPUs across the rack |\n| GPU-to-GPU link | NVLink in-box, PCIe/Ethernet out | Switched NVLink fabric, all-to-all |\n| Aggregate fabric BW | Tens of TB/s in-box | ~130 TB/s across the rack |\n| Cooling | Air or hybrid | Direct-to-chip liquid, >100 kW/rack |\n| Scale-out | Ethernet / InfiniBand | Ethernet / InfiniBand between racks |\n\n```svg\n\n```\n\nRead rack-scale computing through a scale-up-domain lens rather than a bigger-cluster lens. Once you see that the goal is to make an entire rack behave as one coherent GPU, the whole design follows: a switched NVLink fabric to flatten seventy-two GPUs into a single all-to-all domain, a copper spine because the reach is short, and liquid cooling because that much silicon in one rack cannot be air-cooled, with ordinary Ethernet left to do the slower job of stitching racks together into a cluster.