HPC Cluster Networking enables extreme-scale distributed computation through high-bandwidth, low-latency interconnects like InfiniBand and RoCE, with RDMA verbs API providing efficient point-to-point and collective communication.

InfiniBand Generations (HDR, EDR, NDR)

• InfiniBand Bandwidth Evolution: SDR (2.5 Gbps) → DDR (5 Gbps) → QDR (40 Gbps) → FDR (54.5 Gbps) → EDR (100 Gbps) → HDR (200 Gbps) → NDR (400 Gbps).
• EDR (Enhanced Data Rate): 100 Gbps per link (12x lanes × 12.5 Gbps effective). Dual-port NICs provide 200 Gbps aggregate. Typical for TOP500 clusters <2021.
• HDR (High Data Rate): 200 Gbps per link (12x lanes × 16.67 Gbps). Dual-port = 400 Gbps. Emerging in latest supercomputers (Fugaku, Summit).
• Lane Count: All modern InfiniBand uses 12 lanes (full width). Older variants (1x/4x/8x) available for backward compatibility.

RDMA Verbs API and Queue Pairs

• RC (Reliable Connected): Point-to-point reliable delivery with ordering. Creates connection between two endpoints. Typical for send/recv, small-message optimization.
• UD (Unreliable Datagram): Connectionless, datagram semantics. No in-order delivery; lost datagrams not retransmitted. Lower overhead for all-to-all collectives.
• Queue Pair (QP): Endpoint consisting of Send Queue (SQ) and Recv Queue (RQ). Application posts work requests (WRs) to queues; hardware executes asynchronously.
• Completion Queue (CQ): Collects completed work. Application polls/waits on CQ to detect completion. Decouples WR submission from completion detection.

RoCE (RDMA over Converged Ethernet) v2

• RoCE Protocol: RDMA over Ethernet using InfiniBand Transport Layer. UDP/IP encapsulation enables Ethernet deployment without new hardware.
• RoCE v2: Uses IP, routable across switches (vs RoCE v1 link-local only). Rate-limited per flow via UDP source port hashing.
• Congestion Control (DCQCN): Data Center QCN algorithm detects congestion (explicit congestion notification from switches), throttles sender. Reduces packet loss.
• Switch Requirements: RoCE requires ECN-capable switches. Not all enterprise switches support ECN marking.

IB Queue Pair States and Transitions

• RESET → INIT → RTR (Ready to Receive): Initial connection setup. Exchange queue pair numbers, PSN (packet sequence number).
• RTR → RTS (Ready to Send): Sender transitions to RTS. Now both sides can exchange data. Must be coordinated (RST → RTS → RTS both sides).
• RTS → SQERROR: Send queue error (postdated WR, QP disabled). Application must recover by resetting QP.
• Connection Semantics: After connection establishment, sender/receiver can exchange messages in-order and reliably (bit-error rate ~1e-15 due to CRC protection).

Adaptive Routing and Switch Topology

• Deterministic Routing: Fixed path selection (up/down routing). Simple, loop-free but may not use all available bandwidth.
• Adaptive Routing: Path dynamically selected based on network congestion. Balances load across paths, improves bisection bandwidth. Requires more processing.
• Network Topology Options: Fat-tree (Clos network) most common. Dragonfly (Cray) alternative offering higher radix, lower hop count for large clusters.

Fat-Tree and Dragonfly Topologies

• Fat-Tree: Tree with uniform bandwidth at each level (no bandwidth bottleneck). Level 0 = hosts, Level 1 = edge switches, Level 2+ = core switches. Bisection bandwidth = (number_of_hosts × link_bandwidth) / 2.
• Dragonfly: Hierarchical ring + full mesh + spine. Groups of hosts connected locally (ring), inter-group via spine (full mesh). Excellent for all-to-all, lower radix than fat-tree.
• Switch Radix: Fat-tree requires high-radix switches (256+ ports). Dragonfly lower radix (48-128 ports typical) reducing switch cost.
• Scaling: Fat-tree suitable up to 10,000 nodes; beyond that, Dragonfly preferred.

Performance Characteristics

• Latency: RDMA latency ~1-2µs (hardware offload). TCP/IP latency ~10-100µs (kernel processing). 10-100x difference critical for synchronized algorithms.
• Bandwidth: Link bandwidth fully utilized (>95%) for streaming loads. Point-to-point utilization high (message matching overhead minimal).
• Injection Bandwidth: Peak injection = (number of NIC ports) × (link bandwidth). Typical HPC node: 2×100Gbps = 200Gbps injection.

MPI over RDMA Performance

• Rendezvous Protocol: Small messages sent as eager (buffer preposted). Large messages use rendezvous (sender waits for receiver prepost). Threshold ~64KB-1MB depending on MPI implementation.
• Collective Optimization: All-reduce implemented via tree (minimize latency) or ring (maximize bandwidth). InfiniBand topology determines optimal algorithm.
• Bandwidth Saturation: Typical HPC application saturates InfiniBand in parallel regions (synchronous collectives). Asynchronous computation/communication hides latency.