RDMA (Remote Direct Memory Access) and InfiniBand are the high-performance networking technologies that enable direct memory-to-memory data transfer between machines without involving the CPU or operating system — achieving latencies under 1 microsecond and throughputs over 400 Gbps, making them essential for HPC clusters, distributed training, and low-latency storage systems.

How RDMA Works

• Traditional networking: App → OS kernel → TCP/IP stack → NIC → wire → NIC → kernel → App.
• Each step: System calls, context switches, memory copies — adds latency.
• RDMA: App → NIC → wire → NIC → remote memory (bypasses both CPUs and kernels).
• Zero-copy: Data goes directly from wire to application buffer — no intermediate copies.
• Kernel bypass: NIC handles protocol processing in hardware — no OS involvement.
• CPU offload: CPU freed for computation while NIC handles transfers.

RDMA Operations

RDMA Transports

InfiniBand Generations

RDMA in Distributed ML Training

• NCCL over InfiniBand: Default for multi-node GPU training.
• GPUDirect RDMA: NIC reads directly from GPU memory — no CPU staging buffer.
• 8× H100 DGX pods connected via 8× NDR400 (3.2 Tbps per node).
• Gradient AllReduce: Ring/tree AllReduce over RDMA achieves near-wire-speed.
• Without RDMA: Multi-node training bandwidth drops 5-10x → scaling becomes impractical beyond 2-4 nodes.

Programming RDMA

• libibverbs: Low-level C API for RDMA operations (complex — ~200 LOC for simple send).
• UCX (Unified Communication X): Higher-level library abstracting RDMA transports.
• NCCL / Gloo: ML-specific collective communication over RDMA.

RDMA and InfiniBand are the networking foundation of modern AI supercomputers — the ability to move data between machines at hardware speed without CPU involvement is what makes it possible to train trillion-parameter models across thousands of GPUs with near-linear scaling efficiency.