infiniband architecture rdma

**InfiniBand Architecture** is **the high-performance networking standard designed for low-latency, high-bandwidth interconnects in HPC and AI clusters — providing hardware-offloaded RDMA operations, reliable transport with sub-microsecond latency, and scalable switched fabric architecture that has become the de facto standard for GPU cluster networking in large-scale machine learning infrastructure**. **InfiniBand Protocol Stack:** - **Physical Layer**: electrical signaling at 25-50 Gb/s per lane (SerDes technology); 4× or 12× lane aggregation produces 100-600 Gb/s links; copper cables (DAC) for <5m, active optical cables (AOC) for 5-100m, fiber optics for longer distances - **Link Layer**: 2KB packet size with 8-bit CRC for error detection; credit-based flow control ensures lossless transmission; virtual lanes (up to 15 data VLs + 1 management VL) enable QoS and deadlock-free routing - **Network Layer**: 128-bit Global Identifier (GID) addressing; subnet-based routing with LID (Local Identifier) for intra-subnet, GID for inter-subnet; supports IPv4/IPv6 encapsulation for WAN connectivity - **Transport Layer**: multiple transport services — Reliable Connection (RC), Unreliable Connection (UC), Reliable Datagram (RD), Unreliable Datagram (UD); RC is most common for RDMA, providing in-order delivery with hardware-level retransmission **Queue Pair (QP) Model:** - **Send/Receive Queues**: each QP consists of a Send Queue (SQ) and Receive Queue (RQ); applications post Work Requests (WRs) to queues; HCA (Host Channel Adapter) processes WRs asynchronously and posts Completion Queue Entries (CQEs) when operations complete - **RDMA Operations**: RDMA Write (write to remote memory without remote CPU involvement), RDMA Read (read from remote memory), RDMA Atomic (atomic compare-and-swap, fetch-and-add); Send/Receive for traditional message passing - **Memory Registration**: applications register memory regions with the HCA, receiving an R_Key (remote key) and L_Key (local key); registration pins physical pages and grants HCA DMA access; remote peers use R_Key to access registered memory via RDMA operations - **Zero-Copy Transfer**: data moves directly from application buffer to NIC to remote NIC to remote application buffer; CPU only posts the operation descriptor — no data copying through kernel buffers, achieving 95%+ of wire bandwidth **Subnet Management:** - **Subnet Manager (SM)**: centralized control plane that discovers topology, assigns LIDs, computes routing tables, and configures switch forwarding; typically runs on a dedicated management node or integrated into a switch - **LID Assignment**: SM assigns 16-bit LIDs to each port; unicast LIDs for point-to-point, multicast LIDs for one-to-many; LID Mask Control (LMC) enables multiple paths between endpoints for load balancing - **Routing Algorithms**: SM computes forwarding tables using algorithms like Min-Hop (shortest path), DFSSSP (Deadlock-Free Single-Source Shortest Path), or Fat-Tree optimized routing; tables downloaded to switches via Subnet Management Packets (SMPs) - **Topology Discovery**: SM sends SMP queries to discover switches, links, and endpoints; builds complete topology graph; reconfigures routing on link failures or topology changes; discovery and reconfiguration complete in seconds for 1000-node clusters **Performance Characteristics:** - **Latency**: RC Send/Receive latency <1μs for small messages (ConnectX-7); RDMA Write latency 0.6-0.8μs; latency dominated by HCA processing and wire time, not software overhead - **Bandwidth**: NDR (400 Gb/s) achieves 48+ GB/s effective bandwidth for large messages; 95%+ efficiency due to hardware offload and zero-copy; multiple QPs enable full link utilization from concurrent operations - **CPU Efficiency**: RDMA operations consume <5% CPU utilization at line rate; CPU freed for computation while network transfers proceed in background; critical for GPU workloads where CPU orchestrates GPU kernels - **Scalability**: single subnet supports 48K endpoints (16-bit LID space); multi-subnet fabrics with routers scale to millions of endpoints; flat address space within subnet simplifies programming model **Programming Interfaces:** - **Verbs API**: low-level C API (libibverbs) for direct HCA access; applications create QPs, post WRs, poll CQs; maximum performance but complex programming model requiring careful resource management - **UCP/UCX**: Unified Communication X library provides high-level abstractions (Active Messages, RMA, Atomics) over Verbs; automatic protocol selection, multi-rail support, and fault tolerance; used by MPI implementations and ML frameworks - **MPI over IB**: MPI libraries (OpenMPI, MVAPICH, Intel MPI) implement MPI semantics using IB Verbs; MPI_Send/Recv map to IB Send/Recv or RDMA operations; collective operations optimized for IB hardware multicast and adaptive routing - **NCCL over IB**: NVIDIA Collective Communications Library detects IB devices and uses RDMA for GPU-to-GPU transfers; implements ring, tree, and collnet algorithms optimized for IB topology; achieves 90%+ of theoretical bandwidth for all-reduce operations InfiniBand architecture is **the networking foundation of modern AI infrastructure — its hardware-offloaded RDMA, sub-microsecond latency, and lossless fabric enable the efficient distributed training of frontier models, making it the interconnect of choice for every major AI lab and cloud provider building GPU supercomputers**.