infiniband

RDMA (Remote Direct Memory Access) lets one computer's network card place data directly into another computer's memory without involving either machine's CPU or operating-system kernel on the data path. InfiniBand is the high-performance, lossless interconnect built to do this at ultra-low latency and high bandwidth; RoCE (RDMA over Converged Ethernet) offers the same RDMA semantics on Ethernet hardware. Together they are the scale-out fabric that carries collectives between servers once traffic leaves the fast in-node NVLink domain.\n\n**It removes the CPU and kernel from the transfer.** A conventional TCP/IP send copies the payload from application memory into kernel buffers, walks the OS network stack, interrupts the CPU, and copies again at the receiver — latency and CPU overhead that are ruinous for the tight, frequent gradient exchanges in distributed training. RDMA registers memory regions in advance so the NIC can read and write them directly; the transfer is zero-copy and kernel-bypass, so data moves NIC-to-NIC while the CPUs are free to compute.\n\n**InfiniBand is the fabric engineered for it.** InfiniBand provides a lossless, credit-flow-controlled link with very low latency and native RDMA support, plus switches and adapters (HCAs) designed for large clusters. It is why supercomputers and AI training clusters standardize on it for inter-node communication. RoCE brings RDMA to Ethernet for shops that prefer that ecosystem, trading some of InfiniBand's determinism for commodity switching. Both expose the same verbs API and the same core benefit: remote memory access without host software in the loop.\n\n| | TCP/IP over Ethernet | RDMA (InfiniBand / RoCE) |\n|---|---|---|\n| Data path | app→kernel→NIC, copies | NIC↔app memory, zero-copy |\n| CPU involvement | interrupts, stack processing | bypassed on data path |\n| Latency | microseconds-to-ms, variable | ~single-digit µs, stable |\n| Loss behavior | lossy, retransmit | lossless (IB) / PFC (RoCE) |\n| Role in clusters | general networking | scale-out collectives |\n\n```svg\n\n```\n\n**With GPUDirect it reaches all the way to GPU memory.** GPUDirect RDMA lets the NIC read and write GPU memory directly, so a gradient can travel from one GPU's memory across the fabric into a remote GPU's memory without ever staging through host RAM or the CPU. This is the scale-out counterpart to NVLink: NVLink gives enormous bandwidth inside a node, InfiniBand/RDMA carries the collective between nodes with minimal latency, and libraries like NCCL choose which path each part of an all-reduce takes. The two fabrics together define the bandwidth hierarchy that model-parallel layouts must respect.\n\nRead InfiniBand and RDMA through a quant lens rather than a 'faster network' lens: the wins are measured as latency (single-digit microseconds versus a variable OS stack) and CPU cycles saved (zero-copy versus per-packet processing), which is exactly what a latency-bound collective on small messages needs. The design question is where each hop sits in the bandwidth hierarchy — NVLink inside the node, RDMA across it — because the slowest link a collective must cross sets the step time, and RDMA exists to make that inter-node crossing as close to a local memory access as the wire allows.