NVLink and NVSwitch GPU Interconnects: High-Bandwidth All-to-All GPU Networking — specialized interconnect technology enabling 900 GB/s per-GPU communication for tightly-coupled multi-GPU systems

NVLink 4.0 Specifications

• Bandwidth per Direction: 450 GB/s (bidirectional = 900 GB/s total), 5.6× faster than PCIe Gen5 x16 (64 GB/s)
• Scalability: up to 8 GPUs per node (NVLink links 3-4× per GPU, some shared), full bisection bandwidth between pairs
• Latency: sub-microsecond GPU-to-GPU communication (vs 1-2 µs PCIe latency), enables fine-grain synchronization
• Power Efficiency: 900 GB/s with modest power (~20% of GPU compute power), superior to PCIe (higher power for lower bandwidth)
• Protocol: extends PCIe protocol (NVLink 3.0 based on PCIe 4.0, NVLink 4.0 on PCIe 5.0 electrical)

NVSwitch 3.0 Architecture

• All-to-All Connectivity: 8-way crossbar switch (full mesh within node), any GPU pair achieves 900 GB/s simultaneously
• Bisection Bandwidth: 57.6 TB/s total (8 GPUs × 900 GB/s × 8 directions), non-blocking (no contention)
• Scalability: single switch per 8 GPUs (typical), larger clusters cascade switches (rack-level switches for multi-rack)
• Switching Latency: minimal (sub-microsecond), transparent to GPU communication
• Design: custom switch ASIC (not Ethernet switch), optimized for GPU protocols

DGX H100 Superchip Node Architecture

• 8 H100 GPUs: full 8-way NVSwitch 3.0 connectivity, all-to-all GPU communication at 900 GB/s
• CPU: 12-core Intel Xeon (or AMD EPYC), connected to GPU cluster via NVLink-C2C (see below)
• Memory: 141 GB total GPU memory (16 GB HBM3 per GPU, shared via NVSwitch), coherent memory model
• Power: ~10.2 kW for 8 H100s + CPU (8 GPUs × 700 W + 500 W CPU), thermal challenge
• Performance: 141 TFLOPS FP32 aggregate (8 GPUs × 17.5 TFLOPS each), 700+ TFLOPS with sparsity/quantization

NVLink-C2C (Chip-to-Chip)

• Grace-Hopper Superchip: Grace CPU (ARM-based, 144 cores) + Hopper GPU (132 SMs) on single module (not separate dice)
• Integration: CPU + GPU share high-bandwidth interconnect (900 GB/s), coherent memory (CPU accesses GPU HBM, vice versa)
• Use Case: CPU for system services (PCIe control, memory management), GPU for compute, tight coupling enables efficient communication
• Deployment: Frontier compute nodes use Grace-Hopper (experimental, limited volume)

NVLink vs PCIe Comparison

• Bandwidth: NVLink 4.0 (900 GB/s) vs PCIe Gen5 x16 (64 GB/s), 14× advantage
• Latency: NVLink <1 µs vs PCIe 1-2 µs, 2× improvement
• Power: NVLink more power-efficient (lower power per Gbps), benefits multi-GPU workloads
• Cost: NVLink expensive (specialized silicon), justified for HPC/AI (not consumer)
• Industry Support: NVLink proprietary (NVIDIA only), vs PCIe open standard (AMD, Intel)

NVLink over Fiber

• NVLink-f: optical NVLink (fiber-based), enables long-distance GPU communication (100+ meters)
• Use Case: disaggregated GPU clusters (GPUs in separate racks), avoids copper interconnect limitations
• Latency: fiber adds ~10-100 ns per meter, acceptable for across-datacenter links
• Adoption: still experimental (research deployments), future potential for flexible GPU pools

Multi-GPU Scaling in Deep Learning

• Data Parallelism: batch split across GPUs, each GPU gradient computed independently, allreduce synchronizes gradients
• Model Parallelism: model split across GPUs (layers on different GPUs), forward pass traverses GPUs (serial communication)
• Pipeline Parallelism: layers pipelined (GPU 0→1→2→3 stage-by-stage), reduces synchronization overhead
• Gradient Aggregation: allreduce critical bottleneck (all GPUs exchange gradients), NVLink reduces latency/bandwidth penalty

Communication Efficiency

• Gradient Bandwidth: 8 GPUs × 40 GB gradients = 320 GB gradients per step, allreduce requires 2× (reduce + broadcast)
• NVLink Advantage: 900 GB/s enables allreduce in ~700 ns (640 GB / 900 GB/s), negligible vs 100+ ms compute per batch
• Scalability: 100 nodes × 8 GPUs = 800 GPUs, allreduce scales O(log 800) = 10 steps (vs 800 steps if sequential)
• Overhead: allreduce <5% of training time (with NVLink), vs 10-20% without NVLink optimization

NVIDIA GH200 Superchip (Future)

• Integration: Grace CPU + Hopper GPU stacked 3D (face-to-face), higher bandwidth + lower latency than separate chips
• Memory: 141 GB HBM shared (CPU + GPU), coherent access model
• Expected Performance: 4-5× memory bandwidth vs separate Grace+H100 (via 3D stacking)
• Deployment: targeted at AI (training + inference), emerging 2024-2025

Challenges

• Heat Dissipation: 8 H100s in single node = 10+ kW power density (requires liquid cooling), thermal management critical
• Scalability Beyond 8: beyond-8-GPU scaling requires multi-level NVSwitch (rack-level switches), introduces latency hierarchy
• Synchronization: tightly-coupled GPUs require frequent synchronization (allreduce every few steps), latency-sensitive

Future Roadmap: NVLink generation per GPU generation (+50% bandwidth typically), optical interconnect NVLink-f emerging, heterogeneous GPU clusters (mix of CPU+GPU types) requiring flexible interconnects.