rdma remote direct memory access

**RDMA (Remote Direct Memory Access)** is the **networking capability that allows a computer to access memory on a remote machine without involving the remote CPU or operating system — the data transfer happens directly between network adapter and application memory, bypassing kernel, protocol stack, and remote CPU, achieving latencies of 1-2 µs and bandwidths of 200+ Gbps that are impossible with conventional TCP/IP socket programming**. **Why RDMA Exists** TCP/IP socket communication: data copies user buffer → kernel socket buffer → NIC → network → NIC → kernel buffer → user buffer. Each copy takes CPU cycles and memory bandwidth. OS overhead (system calls, interrupts, scheduling) adds 10-50 µs latency. For HPC and distributed ML (gradient allreduce, parameter server), this overhead dominates. **RDMA Operation Model** - **Two-sided (send/receive)**: both sides involved. Sender posts a send work request (WR); receiver must pre-post a receive WR. The NIC delivers directly to receiver's pre-registered memory buffer. Similar semantics to MPI messaging. - **One-sided (read/write)**: initiator specifies remote memory address (rkey + virtual address obtained via out-of-band exchange). RDMA Write: push data to remote memory without remote CPU involvement. RDMA Read: pull data from remote memory. Atomic (Compare-and-Swap, Fetch-and-Add) operations. **Verbs API** Low-level RDMA programming interface: - **Protection Domain (PD)**: namespace for memory registrations and queue pairs. - **Memory Registration**: ``ibv_reg_mr()`` pins and registers buffer (virtual → physical mapping given to NIC), returns lkey/rkey. - **Queue Pair (QP)**: pair of send queue (SQ) and receive queue (RQ). Types: RC (Reliable Connected — in-order delivery, acknowledgments), UC (Unreliable Connected), UD (Unreliable Datagram — broadcast/multicast). - **Completion Queue (CQ)**: NIC posts completion events; application polls CQ (busy-poll for low latency vs event-driven interrupt for efficiency). - **Work Request (WR)**: descriptor posted to SQ/RQ specifying operation, buffer, length, remote address. **Transport Technologies** - **InfiniBand**: native RDMA, lossless fabric (credit-based flow control), industry standard in HPC (Frontier, Summit, Aurora). - **RoCE v2 (RDMA over Converged Ethernet)**: RDMA semantics over UDP/IPv4, requires priority-flow control (PFC) or DCQCN for lossless operation. Lower cost than IB, adopted by hyperscalers. - **iWARP**: RDMA over TCP, preserves TCP reliability/NAT traversal but higher latency. **High-Level Abstractions** - **UCX (Unified Communication X)**: portable RDMA API (used by OpenMPI, OpenSHMEM), selects best transport automatically. - **libfabric (OFI)**: OpenFabrics Interface, provider model (verbs, psm2, CXI for Slingshot). - **NCCL, Gloo**: distributed ML collective libraries using RDMA. **Performance** - Latency: ~1.0-1.5 µs (IB HDR) vs 50-200 µs (TCP/IP loopback). - Bandwidth: 200 Gbps per port (IB NDR: 400 Gbps). - CPU offload: near-zero CPU involvement for data path. RDMA is **the networking technology that removes the CPU from the data movement critical path — enabling distributed HPC and AI systems to exchange data at memory-bus speeds across a cluster, making large-scale parallel computing economically and technically feasible by eliminating the latency and CPU overhead of conventional networking**.