rdma remote direct memory access
**Remote Direct Memory Access (RDMA)** is the **high-performance networking technology that allows one computer to read from or write to another computer's memory directly, bypassing the remote CPU and OS kernel entirely — achieving latencies under 2 microseconds and bandwidths exceeding 400 Gbps (50 GB/s) per port, making it the foundation of interconnect fabrics in HPC clusters, AI training systems, and high-frequency trading networks**.
**Why RDMA Is Transformative**
Traditional TCP/IP networking involves multiple software layers: application → socket API → kernel TCP/IP stack → NIC driver → hardware. Each layer adds latency (context switches, buffer copies, protocol processing). A typical TCP round-trip takes 20-50 microseconds. RDMA eliminates all intermediate software — the sending application posts a descriptor to the NIC hardware, which DMA-reads the data from the source memory and sends it directly to the remote NIC, which DMA-writes it into the destination memory. Total latency: 1-2 microseconds.
**RDMA Technologies**
- **InfiniBand (IB)**: Purpose-built RDMA fabric. The dominant interconnect in HPC and AI training clusters. Current generation: NDR (400 Gbps / 50 GB/s per port). Provides full RDMA semantics natively. Mellanox/NVIDIA ConnectX-7 adapters.
- **RoCE (RDMA over Converged Ethernet)**: RDMA over standard Ethernet infrastructure using the InfiniBand transport protocol encapsulated in UDP/IP. Requires lossless Ethernet (PFC flow control, ECN). Lower cost than InfiniBand but more complex network configuration.
- **iWARP**: RDMA over TCP. Works on any IP network without special configuration but has higher latency than IB or RoCE due to TCP processing.
**RDMA Operations (Verbs)**
| Operation | Description | Remote CPU Involved? |
|-----------|-------------|---------------------|
| **RDMA Write** | Write data to remote memory | No |
| **RDMA Read** | Read data from remote memory | No |
| **Send/Receive** | Message passing (two-sided) | Yes (receive posted) |
| **Atomic** | Fetch-and-add, compare-and-swap on remote memory | No |
One-sided operations (Read/Write/Atomic) are the key innovation — the remote CPU is completely uninvolved. This enables millions of operations per second per core because each operation completes in hardware without any remote software execution.
**GPU-Direct RDMA**
NVIDIA GPUDirect RDMA allows the NIC to read/write GPU memory directly, eliminating the CPU staging buffer. Data flows: GPU memory → NIC → network → remote NIC → remote GPU memory. Critical for distributed deep learning where gradient tensors must move between GPUs on different nodes at NVLink-like speeds.
RDMA is **the technology that makes distributed computing feel like shared memory** — providing memory-to-memory data movement so fast that the network interconnect becomes nearly invisible to the application, enabling clusters of machines to cooperate as efficiently as processors on a single motherboard.