Multi-GPU Programming

Multi-GPU Programming is the distributed computing paradigm that coordinates multiple GPUs to solve problems requiring more memory or compute than a single GPU provides — utilizing high-bandwidth interconnects like NVLink (900 GB/s between GPUs), NVSwitch (14.4 TB/s aggregate), and collective communication libraries like NCCL (NVIDIA Collective Communications Library) that implement optimized all-reduce, broadcast, and gather operations achieving 80-95% scaling efficiency for data-parallel training across 8-1024 GPUs, making multi-GPU programming essential for training large language models (70B-175B parameters) and processing datasets that exceed single-GPU memory (80GB) where proper communication optimization and load balancing determine whether applications achieve linear speedup or suffer from communication bottlenecks that limit scaling to 20-40% efficiency.

Multi-GPU Architectures:
- NVLink: direct GPU-to-GPU interconnect; 900 GB/s bidirectional on A100 (12 links × 25 GB/s × 3); 900 GB/s on H100; 5-10× faster than PCIe 4.0 (64 GB/s); enables peer-to-peer memory access
- NVSwitch: full bisection bandwidth switch; connects 8 GPUs in DGX A100; 14.4 TB/s aggregate bandwidth; every GPU can communicate with every other at full NVLink speed
- PCIe: fallback interconnect; PCIe 4.0: 64 GB/s, PCIe 5.0: 128 GB/s; 5-10× slower than NVLink; sufficient for some workloads; limits scaling
- InfiniBand: inter-node communication; 200-400 Gb/s (25-50 GB/s) per link; RDMA for low latency; scales to thousands of GPUs

NCCL (NVIDIA Collective Communications Library):
- Collective Operations: all-reduce (sum gradients across GPUs), broadcast (distribute data), reduce-scatter, all-gather; optimized for GPU topology
- Ring Algorithm: default for all-reduce; each GPU sends to next, receives from previous; bandwidth-optimal; latency O(N) for N GPUs
- Tree Algorithm: hierarchical reduction; lower latency for small messages; used automatically by NCCL based on message size
- Performance: 80-95% of hardware bandwidth for large messages (>1MB); 300-800 GB/s all-reduce on 8×A100 with NVLink; 50-70% efficiency for small messages (<1KB)

Data Parallelism:
- Model Replication: each GPU has full model copy; processes different data batch; gradients averaged across GPUs; most common approach
- Batch Splitting: global batch size = per-GPU batch × num GPUs; 8 GPUs with batch 32 each = effective batch 256; improves throughput 6-8× on 8 GPUs
- Gradient Synchronization: all-reduce after backward pass; averages gradients; synchronized update; NCCL all-reduce costs 5-20ms for 1GB on 8 GPUs
- Scaling Efficiency: 85-95% on 8 GPUs, 70-85% on 64 GPUs, 50-70% on 512 GPUs; communication overhead increases with GPU count

Model Parallelism:
- Tensor Parallelism: split individual layers across GPUs; each GPU computes portion of layer; requires all-reduce for activations; used in Megatron-LM
- Pipeline Parallelism: split model into stages; each GPU handles consecutive layers; micro-batching to hide pipeline bubbles; GPipe, PipeDream
- Hybrid Parallelism: combine data, tensor, and pipeline parallelism; used for largest models (GPT-3, GPT-4); 3D parallelism (data × tensor × pipeline)
- Communication: tensor parallelism requires frequent all-reduce (every layer); pipeline parallelism requires point-to-point (between stages); optimize based on interconnect

Memory Management:
- Unified Memory: automatic migration between GPUs; convenient but slower; 2-5× overhead vs explicit; use for prototyping
- Peer-to-Peer Access: cudaDeviceEnablePeerAccess(); direct memory access between GPUs; requires NVLink or PCIe P2P; 5-10× faster than host staging
- Explicit Copies: cudaMemcpyPeer() or cudaMemcpyPeerAsync(); explicit control; optimal performance; requires careful orchestration
- Memory Pooling: allocate memory once, reuse across iterations; eliminates allocation overhead; critical for performance

Load Balancing:
- Static Partitioning: divide work equally across GPUs; simple but inflexible; assumes uniform work per element
- Dynamic Scheduling: work queue shared across GPUs; GPUs pull work as they finish; handles load imbalance; 10-30% overhead for coordination
- Heterogeneous GPUs: different GPU models (A100 + V100); assign work proportional to capability; requires profiling and tuning
- Straggler Mitigation: detect slow GPUs; redistribute work; speculative execution; 10-20% improvement for imbalanced workloads

Communication Optimization:
- Overlap Communication and Computation: start all-reduce early; compute independent operations while communicating; 20-50% speedup
- Gradient Accumulation: accumulate gradients for multiple micro-batches; single all-reduce for accumulated gradients; reduces communication frequency
- Compression: compress gradients before all-reduce; 10-100× compression with minimal accuracy loss; PowerSGD, 1-bit SGD; 2-5× speedup
- Hierarchical Communication: reduce within node (NVLink), then across nodes (InfiniBand); exploits fast local interconnect; 30-60% improvement

PyTorch Distributed:
- DistributedDataParallel (DDP): standard data parallelism; automatic gradient synchronization; 85-95% scaling efficiency on 8 GPUs
- Backend: NCCL for GPUs (fastest), Gloo for CPU, MPI for HPC; NCCL recommended for all GPU workloads
- Initialization: torch.distributed.init_process_group(); one process per GPU; rank and world_size identify processes
- Launch: torchrun or torch.distributed.launch; handles process spawning and environment setup

Horovod:
- Framework-Agnostic: supports PyTorch, TensorFlow, MXNet; consistent API across frameworks
- Ring All-Reduce: bandwidth-optimal algorithm; 80-95% scaling efficiency; automatic topology detection
- Tensor Fusion: batches small tensors into single all-reduce; reduces overhead; 20-40% speedup for models with many small layers
- Timeline: profiling tool; visualizes communication and computation; identifies bottlenecks

Scaling Patterns:
- Weak Scaling: increase problem size with GPU count; maintain per-GPU work constant; ideal: linear speedup; achievable: 80-95% efficiency
- Strong Scaling: fixed problem size; increase GPU count; communication overhead grows; efficiency drops; 70-85% on 64 GPUs typical
- Batch Size Scaling: increase batch size with GPU count; maintains training time; may require learning rate adjustment; 85-95% efficiency
- Sequence Length Scaling: increase sequence length with GPU count; for transformers; enables longer contexts; 70-85% efficiency

Multi-Node Scaling:
- InfiniBand: 200-400 Gb/s links; RDMA for low latency; GPUDirect RDMA bypasses CPU; 5-10 μs latency
- Ethernet: 100-400 Gb/s; higher latency than InfiniBand; sufficient for some workloads; RoCE (RDMA over Converged Ethernet) improves performance
- Topology: fat-tree, dragonfly, or custom topologies; affects communication patterns; NCCL auto-detects and optimizes
- Scaling Limits: 70-85% efficiency on 64 GPUs (8 nodes), 50-70% on 512 GPUs (64 nodes); communication becomes bottleneck

Fault Tolerance:
- Checkpointing: save model state periodically; resume from checkpoint on failure; overhead 1-5% of training time
- Elastic Training: add/remove GPUs dynamically; handles node failures; PyTorch Elastic, Horovod Elastic
- Redundancy: replicate critical data; detect and recover from errors; 5-10% overhead; critical for long training runs
- Monitoring: track GPU health, temperature, errors; preemptive replacement; reduces unexpected failures

Performance Profiling:
- Nsight Systems: timeline view; shows communication and computation; identifies idle time; visualizes multi-GPU execution
- NCCL Tests: benchmark collective operations; measure bandwidth and latency; verify interconnect performance
- PyTorch Profiler: per-operation timing; identifies bottlenecks; shows communication overhead
- Metrics: scaling efficiency, communication time %, GPU utilization, achieved bandwidth; target 80-95% efficiency

Common Bottlenecks:
- Communication Overhead: all-reduce dominates for small models or large GPU counts; overlap with computation; compress gradients
- Load Imbalance: uneven work distribution; dynamic scheduling; profile to identify; 10-30% efficiency loss
- Memory Bandwidth: limited by slowest GPU; ensure uniform memory access patterns; 20-40% efficiency loss
- Synchronization: frequent barriers reduce efficiency; minimize synchronization points; use asynchronous operations

Best Practices:
- Use NCCL: fastest collective communication library for NVIDIA GPUs; 80-95% of hardware bandwidth
- Overlap Communication: start all-reduce early; compute independent operations while communicating; 20-50% speedup
- Batch Size: scale batch size with GPU count; maintains efficiency; adjust learning rate accordingly
- Profile: use Nsight Systems and PyTorch Profiler; identify bottlenecks; optimize based on data
- Topology-Aware: understand interconnect topology; optimize communication patterns; NCCL handles automatically but manual optimization helps

Advanced Techniques:
- ZeRO (Zero Redundancy Optimizer): partitions optimizer states, gradients, and parameters across GPUs; reduces memory by 4-16×; enables larger models
- Gradient Checkpointing: recompute activations during backward; trades compute for memory; enables 2-4× larger models
- Mixed Precision: FP16 for compute, FP32 for gradients; 2× speedup; reduces communication volume by 2×
- Pipeline Parallelism: split model into stages; micro-batching; reduces memory per GPU; 70-85% efficiency

Real-World Performance:
- GPT-3 Training: 1024 A100 GPUs; 3D parallelism (data × tensor × pipeline); 50-60% scaling efficiency; 34 days training time
- Stable Diffusion: 8 A100 GPUs; data parallelism; 85-90% scaling efficiency; 2-3 days training time
- ResNet-50: 64 V100 GPUs; data parallelism; 90-95% scaling efficiency; 1 hour training time on ImageNet
- BERT-Large: 16 V100 GPUs; data parallelism; 85-90% scaling efficiency; 3 days training time

Multi-GPU Programming is the essential skill for modern AI development — by leveraging high-bandwidth interconnects like NVLink and optimized communication libraries like NCCL, developers achieve 80-95% scaling efficiency across 8-1024 GPUs, enabling training of large language models and processing of massive datasets that would be impossible on single GPUs, making multi-GPU programming the difference between training models in days versus months and the key to pushing the frontiers of AI capabilities.