hpc cluster infiniband networking

**HPC Cluster Networking** enables **extreme-scale distributed computation through high-bandwidth, low-latency interconnects like InfiniBand and RoCE, with RDMA verbs API providing efficient point-to-point and collective communication.** **InfiniBand Generations (HDR, EDR, NDR)** - **InfiniBand Bandwidth Evolution**: SDR (2.5 Gbps) → DDR (5 Gbps) → QDR (40 Gbps) → FDR (54.5 Gbps) → EDR (100 Gbps) → HDR (200 Gbps) → NDR (400 Gbps). - **EDR (Enhanced Data Rate)**: 100 Gbps per link (12x lanes × 12.5 Gbps effective). Dual-port NICs provide 200 Gbps aggregate. Typical for TOP500 clusters <2021. - **HDR (High Data Rate)**: 200 Gbps per link (12x lanes × 16.67 Gbps). Dual-port = 400 Gbps. Emerging in latest supercomputers (Fugaku, Summit). - **Lane Count**: All modern InfiniBand uses 12 lanes (full width). Older variants (1x/4x/8x) available for backward compatibility. **RDMA Verbs API and Queue Pairs** - **RC (Reliable Connected)**: Point-to-point reliable delivery with ordering. Creates connection between two endpoints. Typical for send/recv, small-message optimization. - **UD (Unreliable Datagram)**: Connectionless, datagram semantics. No in-order delivery; lost datagrams not retransmitted. Lower overhead for all-to-all collectives. - **Queue Pair (QP)**: Endpoint consisting of Send Queue (SQ) and Recv Queue (RQ). Application posts work requests (WRs) to queues; hardware executes asynchronously. - **Completion Queue (CQ)**: Collects completed work. Application polls/waits on CQ to detect completion. Decouples WR submission from completion detection. **RoCE (RDMA over Converged Ethernet) v2** - **RoCE Protocol**: RDMA over Ethernet using InfiniBand Transport Layer. UDP/IP encapsulation enables Ethernet deployment without new hardware. - **RoCE v2**: Uses IP, routable across switches (vs RoCE v1 link-local only). Rate-limited per flow via UDP source port hashing. - **Congestion Control (DCQCN)**: Data Center QCN algorithm detects congestion (explicit congestion notification from switches), throttles sender. Reduces packet loss. - **Switch Requirements**: RoCE requires ECN-capable switches. Not all enterprise switches support ECN marking. **IB Queue Pair States and Transitions** - **RESET → INIT → RTR (Ready to Receive)**: Initial connection setup. Exchange queue pair numbers, PSN (packet sequence number). - **RTR → RTS (Ready to Send)**: Sender transitions to RTS. Now both sides can exchange data. Must be coordinated (RST → RTS → RTS both sides). - **RTS → SQERROR**: Send queue error (postdated WR, QP disabled). Application must recover by resetting QP. - **Connection Semantics**: After connection establishment, sender/receiver can exchange messages in-order and reliably (bit-error rate ~1e-15 due to CRC protection). **Adaptive Routing and Switch Topology** - **Deterministic Routing**: Fixed path selection (up*/down* routing). Simple, loop-free but may not use all available bandwidth. - **Adaptive Routing**: Path dynamically selected based on network congestion. Balances load across paths, improves bisection bandwidth. Requires more processing. - **Network Topology Options**: Fat-tree (Clos network) most common. Dragonfly (Cray) alternative offering higher radix, lower hop count for large clusters. **Fat-Tree and Dragonfly Topologies** - **Fat-Tree**: Tree with uniform bandwidth at each level (no bandwidth bottleneck). Level 0 = hosts, Level 1 = edge switches, Level 2+ = core switches. Bisection bandwidth = (number_of_hosts × link_bandwidth) / 2. - **Dragonfly**: Hierarchical ring + full mesh + spine. Groups of hosts connected locally (ring), inter-group via spine (full mesh). Excellent for all-to-all, lower radix than fat-tree. - **Switch Radix**: Fat-tree requires high-radix switches (256+ ports). Dragonfly lower radix (48-128 ports typical) reducing switch cost. - **Scaling**: Fat-tree suitable up to 10,000 nodes; beyond that, Dragonfly preferred. **Performance Characteristics** - **Latency**: RDMA latency ~1-2µs (hardware offload). TCP/IP latency ~10-100µs (kernel processing). 10-100x difference critical for synchronized algorithms. - **Bandwidth**: Link bandwidth fully utilized (>95%) for streaming loads. Point-to-point utilization high (message matching overhead minimal). - **Injection Bandwidth**: Peak injection = (number of NIC ports) × (link bandwidth). Typical HPC node: 2×100Gbps = 200Gbps injection. **MPI over RDMA Performance** - **Rendezvous Protocol**: Small messages sent as eager (buffer preposted). Large messages use rendezvous (sender waits for receiver prepost). Threshold ~64KB-1MB depending on MPI implementation. - **Collective Optimization**: All-reduce implemented via tree (minimize latency) or ring (maximize bandwidth). InfiniBand topology determines optimal algorithm. - **Bandwidth Saturation**: Typical HPC application saturates InfiniBand in parallel regions (synchronous collectives). Asynchronous computation/communication hides latency.