infiniband
**InfiniBand** is a **high-bandwidth, low-latency networking technology using RDMA for GPU cluster communication** — providing 200-400 Gbps per port with microsecond latencies, InfiniBand is the interconnect of choice for large-scale AI training where multi-node communication efficiency determines scaling effectiveness.
**What Is InfiniBand?**
- **Definition**: High-performance networking fabric for clusters.
- **Technology**: RDMA (Remote Direct Memory Access).
- **Vendor**: NVIDIA/Mellanox (dominant).
- **Use Case**: HPC, AI training, storage networks.
**Why InfiniBand for AI**
- **Bandwidth**: 400 Gbps (NDR) vs. 100 Gbps Ethernet.
- **Latency**: ~1 μs vs. ~10-50 μs Ethernet.
- **RDMA**: Bypass CPU for GPU-to-GPU transfers.
- **Scaling**: Efficient all-reduce across thousands of GPUs.
- **Proven**: Used in largest AI training runs.
**InfiniBand Generations**
**Speed Evolution**:
```
Generation | Speed (per port) | Year
-----------|------------------|------
EDR | 100 Gbps | 2014
HDR | 200 Gbps | 2019
NDR | 400 Gbps | 2022
XDR | 800 Gbps | 2024
GDR | 1600 Gbps | Future
```
**Comparison with Ethernet**:
```
Aspect | InfiniBand NDR | 400G Ethernet
--------------|----------------|---------------
Bandwidth | 400 Gbps | 400 Gbps
Latency | ~1 μs | ~10-50 μs
RDMA | Native | RoCE (extra)
Congestion | Credit-based | Drop-based
CPU overhead | Minimal | Higher
AI training | Optimized | Improving
Cost | Higher | Lower
```
**RDMA Explained**
**How RDMA Works**:
```
Traditional Network:
CPU → Copy to buffer → NIC → Network → NIC → Copy to buffer → CPU
RDMA:
GPU Memory → NIC → Network → NIC → GPU Memory
(CPU not involved, zero-copy)
```
**GPU Direct RDMA**:
```svg
```
**AI Training Infrastructure**
**Typical Large Cluster**:
```svg
```
**NCCL with InfiniBand**
```python
import torch
import torch.distributed as dist
import os
# Set NCCL environment for InfiniBand
os.environ["NCCL_IB_DISABLE"] = "0" # Enable InfiniBand
os.environ["NCCL_NET_GDR_LEVEL"] = "5" # Enable GPUDirect
# Initialize distributed
dist.init_process_group(
backend="nccl",
init_method="env://",
)
# Training code - NCCL uses InfiniBand automatically
model = DistributedDataParallel(model)
```
**Checking InfiniBand**
```bash
# List InfiniBand devices
ibstat
# Show port status
ibstatus
# Check link speed
ibstat mlx5_0 | grep Rate
# Performance test
ib_write_bw -d mlx5_0
```
**InfiniBand vs. Alternatives**
```
Use Case | Best Choice
----------------------|------------------
AI training (1000+ GPU) | InfiniBand NDR
Small clusters (<64 GPU)| Either (cost-dependent)
Cloud/flexibility | Ethernet (easier)
Maximum performance | InfiniBand
Budget constrained | 400G Ethernet + RoCE
```
**Cost Considerations**
```
Component | InfiniBand | 400G Ethernet
-------------------|------------|---------------
NIC/HCA | $3-5K | $1-2K
Switch (port) | $500-1K | $200-400
Total system cost | Higher | Lower
Performance/$ | Better at scale | Better for small
```
InfiniBand is **the performance backbone of large-scale AI training** — when training frontier models across thousands of GPUs, the efficiency of collective operations enabled by InfiniBand's low latency and RDMA capabilities directly determines how well training scales.