multi gpu programming nccl

**Multi-GPU Programming** is **the distributed computing paradigm that coordinates multiple GPUs to solve problems requiring more memory or compute than a single GPU provides** — utilizing high-bandwidth interconnects like NVLink (900 GB/s between GPUs), NVSwitch (14.4 TB/s aggregate), and collective communication libraries like NCCL (NVIDIA Collective Communications Library) that implement optimized all-reduce, broadcast, and gather operations achieving 80-95% scaling efficiency for data-parallel training across 8-1024 GPUs, making multi-GPU programming essential for training large language models (70B-175B parameters) and processing datasets that exceed single-GPU memory (80GB) where proper communication optimization and load balancing determine whether applications achieve linear speedup or suffer from communication bottlenecks that limit scaling to 20-40% efficiency. **Multi-GPU Architectures:** - **NVLink**: direct GPU-to-GPU interconnect; 900 GB/s bidirectional on A100 (12 links × 25 GB/s × 3); 900 GB/s on H100; 5-10× faster than PCIe 4.0 (64 GB/s); enables peer-to-peer memory access - **NVSwitch**: full bisection bandwidth switch; connects 8 GPUs in DGX A100; 14.4 TB/s aggregate bandwidth; every GPU can communicate with every other at full NVLink speed - **PCIe**: fallback interconnect; PCIe 4.0: 64 GB/s, PCIe 5.0: 128 GB/s; 5-10× slower than NVLink; sufficient for some workloads; limits scaling - **InfiniBand**: inter-node communication; 200-400 Gb/s (25-50 GB/s) per link; RDMA for low latency; scales to thousands of GPUs **NCCL (NVIDIA Collective Communications Library):** - **Collective Operations**: all-reduce (sum gradients across GPUs), broadcast (distribute data), reduce-scatter, all-gather; optimized for GPU topology - **Ring Algorithm**: default for all-reduce; each GPU sends to next, receives from previous; bandwidth-optimal; latency O(N) for N GPUs - **Tree Algorithm**: hierarchical reduction; lower latency for small messages; used automatically by NCCL based on message size - **Performance**: 80-95% of hardware bandwidth for large messages (>1MB); 300-800 GB/s all-reduce on 8×A100 with NVLink; 50-70% efficiency for small messages (<1KB) **Data Parallelism:** - **Model Replication**: each GPU has full model copy; processes different data batch; gradients averaged across GPUs; most common approach - **Batch Splitting**: global batch size = per-GPU batch × num GPUs; 8 GPUs with batch 32 each = effective batch 256; improves throughput 6-8× on 8 GPUs - **Gradient Synchronization**: all-reduce after backward pass; averages gradients; synchronized update; NCCL all-reduce costs 5-20ms for 1GB on 8 GPUs - **Scaling Efficiency**: 85-95% on 8 GPUs, 70-85% on 64 GPUs, 50-70% on 512 GPUs; communication overhead increases with GPU count **Model Parallelism:** - **Tensor Parallelism**: split individual layers across GPUs; each GPU computes portion of layer; requires all-reduce for activations; used in Megatron-LM - **Pipeline Parallelism**: split model into stages; each GPU handles consecutive layers; micro-batching to hide pipeline bubbles; GPipe, PipeDream - **Hybrid Parallelism**: combine data, tensor, and pipeline parallelism; used for largest models (GPT-3, GPT-4); 3D parallelism (data × tensor × pipeline) - **Communication**: tensor parallelism requires frequent all-reduce (every layer); pipeline parallelism requires point-to-point (between stages); optimize based on interconnect **Memory Management:** - **Unified Memory**: automatic migration between GPUs; convenient but slower; 2-5× overhead vs explicit; use for prototyping - **Peer-to-Peer Access**: cudaDeviceEnablePeerAccess(); direct memory access between GPUs; requires NVLink or PCIe P2P; 5-10× faster than host staging - **Explicit Copies**: cudaMemcpyPeer() or cudaMemcpyPeerAsync(); explicit control; optimal performance; requires careful orchestration - **Memory Pooling**: allocate memory once, reuse across iterations; eliminates allocation overhead; critical for performance **Load Balancing:** - **Static Partitioning**: divide work equally across GPUs; simple but inflexible; assumes uniform work per element - **Dynamic Scheduling**: work queue shared across GPUs; GPUs pull work as they finish; handles load imbalance; 10-30% overhead for coordination - **Heterogeneous GPUs**: different GPU models (A100 + V100); assign work proportional to capability; requires profiling and tuning - **Straggler Mitigation**: detect slow GPUs; redistribute work; speculative execution; 10-20% improvement for imbalanced workloads **Communication Optimization:** - **Overlap Communication and Computation**: start all-reduce early; compute independent operations while communicating; 20-50% speedup - **Gradient Accumulation**: accumulate gradients for multiple micro-batches; single all-reduce for accumulated gradients; reduces communication frequency - **Compression**: compress gradients before all-reduce; 10-100× compression with minimal accuracy loss; PowerSGD, 1-bit SGD; 2-5× speedup - **Hierarchical Communication**: reduce within node (NVLink), then across nodes (InfiniBand); exploits fast local interconnect; 30-60% improvement **PyTorch Distributed:** - **DistributedDataParallel (DDP)**: standard data parallelism; automatic gradient synchronization; 85-95% scaling efficiency on 8 GPUs - **Backend**: NCCL for GPUs (fastest), Gloo for CPU, MPI for HPC; NCCL recommended for all GPU workloads - **Initialization**: torch.distributed.init_process_group(); one process per GPU; rank and world_size identify processes - **Launch**: torchrun or torch.distributed.launch; handles process spawning and environment setup **Horovod:** - **Framework-Agnostic**: supports PyTorch, TensorFlow, MXNet; consistent API across frameworks - **Ring All-Reduce**: bandwidth-optimal algorithm; 80-95% scaling efficiency; automatic topology detection - **Tensor Fusion**: batches small tensors into single all-reduce; reduces overhead; 20-40% speedup for models with many small layers - **Timeline**: profiling tool; visualizes communication and computation; identifies bottlenecks **Scaling Patterns:** - **Weak Scaling**: increase problem size with GPU count; maintain per-GPU work constant; ideal: linear speedup; achievable: 80-95% efficiency - **Strong Scaling**: fixed problem size; increase GPU count; communication overhead grows; efficiency drops; 70-85% on 64 GPUs typical - **Batch Size Scaling**: increase batch size with GPU count; maintains training time; may require learning rate adjustment; 85-95% efficiency - **Sequence Length Scaling**: increase sequence length with GPU count; for transformers; enables longer contexts; 70-85% efficiency **Multi-Node Scaling:** - **InfiniBand**: 200-400 Gb/s links; RDMA for low latency; GPUDirect RDMA bypasses CPU; 5-10 μs latency - **Ethernet**: 100-400 Gb/s; higher latency than InfiniBand; sufficient for some workloads; RoCE (RDMA over Converged Ethernet) improves performance - **Topology**: fat-tree, dragonfly, or custom topologies; affects communication patterns; NCCL auto-detects and optimizes - **Scaling Limits**: 70-85% efficiency on 64 GPUs (8 nodes), 50-70% on 512 GPUs (64 nodes); communication becomes bottleneck **Fault Tolerance:** - **Checkpointing**: save model state periodically; resume from checkpoint on failure; overhead 1-5% of training time - **Elastic Training**: add/remove GPUs dynamically; handles node failures; PyTorch Elastic, Horovod Elastic - **Redundancy**: replicate critical data; detect and recover from errors; 5-10% overhead; critical for long training runs - **Monitoring**: track GPU health, temperature, errors; preemptive replacement; reduces unexpected failures **Performance Profiling:** - **Nsight Systems**: timeline view; shows communication and computation; identifies idle time; visualizes multi-GPU execution - **NCCL Tests**: benchmark collective operations; measure bandwidth and latency; verify interconnect performance - **PyTorch Profiler**: per-operation timing; identifies bottlenecks; shows communication overhead - **Metrics**: scaling efficiency, communication time %, GPU utilization, achieved bandwidth; target 80-95% efficiency **Common Bottlenecks:** - **Communication Overhead**: all-reduce dominates for small models or large GPU counts; overlap with computation; compress gradients - **Load Imbalance**: uneven work distribution; dynamic scheduling; profile to identify; 10-30% efficiency loss - **Memory Bandwidth**: limited by slowest GPU; ensure uniform memory access patterns; 20-40% efficiency loss - **Synchronization**: frequent barriers reduce efficiency; minimize synchronization points; use asynchronous operations **Best Practices:** - **Use NCCL**: fastest collective communication library for NVIDIA GPUs; 80-95% of hardware bandwidth - **Overlap Communication**: start all-reduce early; compute independent operations while communicating; 20-50% speedup - **Batch Size**: scale batch size with GPU count; maintains efficiency; adjust learning rate accordingly - **Profile**: use Nsight Systems and PyTorch Profiler; identify bottlenecks; optimize based on data - **Topology-Aware**: understand interconnect topology; optimize communication patterns; NCCL handles automatically but manual optimization helps **Advanced Techniques:** - **ZeRO (Zero Redundancy Optimizer)**: partitions optimizer states, gradients, and parameters across GPUs; reduces memory by 4-16×; enables larger models - **Gradient Checkpointing**: recompute activations during backward; trades compute for memory; enables 2-4× larger models - **Mixed Precision**: FP16 for compute, FP32 for gradients; 2× speedup; reduces communication volume by 2× - **Pipeline Parallelism**: split model into stages; micro-batching; reduces memory per GPU; 70-85% efficiency **Real-World Performance:** - **GPT-3 Training**: 1024 A100 GPUs; 3D parallelism (data × tensor × pipeline); 50-60% scaling efficiency; 34 days training time - **Stable Diffusion**: 8 A100 GPUs; data parallelism; 85-90% scaling efficiency; 2-3 days training time - **ResNet-50**: 64 V100 GPUs; data parallelism; 90-95% scaling efficiency; 1 hour training time on ImageNet - **BERT-Large**: 16 V100 GPUs; data parallelism; 85-90% scaling efficiency; 3 days training time Multi-GPU Programming is **the essential skill for modern AI development** — by leveraging high-bandwidth interconnects like NVLink and optimized communication libraries like NCCL, developers achieve 80-95% scaling efficiency across 8-1024 GPUs, enabling training of large language models and processing of massive datasets that would be impossible on single GPUs, making multi-GPU programming the difference between training models in days versus months and the key to pushing the frontiers of AI capabilities.