NVLink is NVIDIA's high-bandwidth GPU-to-GPU interconnect — providing substantially higher bandwidth and lower latency than PCIe for multi-GPU systems, enabling efficient large-scale training and inference across multiple GPUs.

PCIe vs. NVLink Comparison

NVLink Generations

• NVLink 1.0 (P100, 2016): 160 GB/s.
• NVLink 2.0 (V100, 2018): 300 GB/s total.
• NVLink 3.0 (A100, 2020): 600 GB/s total.
• NVLink 4.0 (H100, 2022): 900 GB/s total + NVSwitch fabric.

NVSwitch

• Full all-to-all GPU interconnect fabric: Any GPU → any GPU at full bandwidth.
• NVIDIA DGX A100/H100: 8 GPUs + 6 NVSwitches → 300 GB/s all-to-all.
• NVLink Network (NVL72, 2024): 72 H100 GPUs in one NVLink domain.

Peer-to-Peer (P2P) Memory Access

// Enable P2P access between GPU 0 and GPU 1
cudaSetDevice(0);
cudaDeviceEnablePeerAccess(1, 0);

// Direct copy GPU0 → GPU1 (bypasses CPU)
cudaMemcpyPeerAsync(dst_on_gpu1, 1, src_on_gpu0, 0, size, stream);

Impact on Distributed Training

• AllReduce within node: NVLink AllReduce ~10x faster than PCIe AllReduce.
• Tensor parallelism: Sharded matrix multiply requires high-bandwidth all-reduce every layer.
• Without NVLink: PCIe bottleneck limits GPU count for efficient tensor parallelism.
• With NVLink: Can tensor-parallelize across 8 GPUs efficiently.

NVLink is the critical infrastructure for large-scale LLM training — without it, inter-GPU communication would bottleneck all forms of model parallelism, and trillion-parameter models would be infeasible to train within reasonable time and cost budgets.