nvlink
**NVLink** is **NVIDIA's high-bandwidth GPU-to-GPU interconnect** — providing substantially higher bandwidth and lower latency than PCIe for multi-GPU systems, enabling efficient large-scale training and inference across multiple GPUs.
**PCIe vs. NVLink Comparison**
| Feature | PCIe Gen4 x16 | NVLink 4.0 (H100) |
|---------|-------------|-------------------|
| Bandwidth (1 link) | 64 GB/s | 900 GB/s |
| Links per GPU | 1 | 18 |
| Total bi-directional | 128 GB/s | 900 GB/s |
| Latency | ~1.5 μs | ~1 μs |
| Topology | Star (via CPU) | Any (direct GPU-GPU) |
**NVLink Generations**
- **NVLink 1.0 (P100, 2016)**: 160 GB/s.
- **NVLink 2.0 (V100, 2018)**: 300 GB/s total.
- **NVLink 3.0 (A100, 2020)**: 600 GB/s total.
- **NVLink 4.0 (H100, 2022)**: 900 GB/s total + NVSwitch fabric.
**NVSwitch**
- Full all-to-all GPU interconnect fabric: Any GPU → any GPU at full bandwidth.
- NVIDIA DGX A100/H100: 8 GPUs + 6 NVSwitches → 300 GB/s all-to-all.
- NVLink Network (NVL72, 2024): 72 H100 GPUs in one NVLink domain.
**Peer-to-Peer (P2P) Memory Access**
```cuda
// Enable P2P access between GPU 0 and GPU 1
cudaSetDevice(0);
cudaDeviceEnablePeerAccess(1, 0);
// Direct copy GPU0 → GPU1 (bypasses CPU)
cudaMemcpyPeerAsync(dst_on_gpu1, 1, src_on_gpu0, 0, size, stream);
```
**Impact on Distributed Training**
- AllReduce within node: NVLink AllReduce ~10x faster than PCIe AllReduce.
- Tensor parallelism: Sharded matrix multiply requires high-bandwidth all-reduce every layer.
- Without NVLink: PCIe bottleneck limits GPU count for efficient tensor parallelism.
- With NVLink: Can tensor-parallelize across 8 GPUs efficiently.
NVLink is **the critical infrastructure for large-scale LLM training** — without it, inter-GPU communication would bottleneck all forms of model parallelism, and trillion-parameter models would be infeasible to train within reasonable time and cost budgets.