cublas cudnn optimization

**cuBLAS and cuDNN Optimization** is **the systematic tuning of NVIDIA's highly-optimized math libraries to achieve 80-95% of theoretical peak performance** — where cuBLAS (CUDA Basic Linear Algebra Subroutines) delivers 10-20 TFLOPS for matrix multiplication on A100 (80-95% of 19.5 TFLOPS peak) and 60-80 TFLOPS with Tensor Cores (80-95% of 312 TFLOPS FP16 peak), while cuDNN (CUDA Deep Neural Network library) provides optimized convolution (15-30 TFLOPS), batch normalization, activation functions, and RNN operations that are 10-100× faster than naive implementations, making proper library usage and tuning essential for deep learning where cuBLAS/cuDNN handle 80-95% of compute and optimization techniques like algorithm selection, workspace tuning, tensor core enablement, and batching can improve performance by 2-10× over default settings. **cuBLAS Fundamentals:** - **GEMM**: cublasGemmEx() for matrix multiplication; supports FP32, FP16, INT8, TF32; 10-20 TFLOPS FP32, 60-80 TFLOPS FP16 with Tensor Cores on A100 - **GEMV**: matrix-vector multiplication; 500-1000 GB/s; memory-bound; 80-95% of peak bandwidth - **Batched Operations**: cublasGemmStridedBatchedEx() for multiple matrices; amortizes overhead; 90-95% efficiency vs single GEMM - **Math Modes**: CUBLAS_DEFAULT_MATH (CUDA cores), CUBLAS_TENSOR_OP_MATH (Tensor Cores), CUBLAS_TF32_TENSOR_OP_MATH (TF32); explicit control **cuBLAS Optimization:** - **Tensor Cores**: cublasSetMathMode(handle, CUBLAS_TENSOR_OP_MATH); enables Tensor Cores; 10-20× speedup for FP16; 312 TFLOPS on A100 - **TF32**: automatic on A100; 8× faster than FP32 (156 TFLOPS vs 19.5 TFLOPS); no code changes; maintains FP32 range - **Algorithm Selection**: cublasGemmAlgo_t specifies algorithm; CUBLAS_GEMM_DEFAULT_TENSOR_OP for Tensor Cores; auto-tuning available - **Workspace**: provide workspace buffer; enables better algorithms; 10-30% speedup; typical size 32-256MB **cuDNN Fundamentals:** - **Convolution**: cudnnConvolutionForward(); supports 2D, 3D; multiple algorithms; 15-30 TFLOPS on A100; 80-95% of peak with Tensor Cores - **Batch Normalization**: cudnnBatchNormalizationForwardTraining(); fused operations; 2-5× faster than separate kernels - **Activation**: cudnnActivationForward(); ReLU, sigmoid, tanh; fused with convolution; 20-40% speedup - **Pooling**: cudnnPoolingForward(); max, average pooling; 500-1000 GB/s; memory-bound **cuDNN Optimization:** - **Algorithm Selection**: cudnnGetConvolutionForwardAlgorithm(); finds best algorithm; CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM often fastest - **Workspace Tuning**: cudnnGetConvolutionForwardWorkspaceSize(); larger workspace enables faster algorithms; 10-50% speedup; typical 100MB-2GB - **Tensor Cores**: cudnnSetConvolutionMathType(CUDNN_TENSOR_OP_MATH); enables Tensor Cores; 10-20× speedup for FP16 - **Auto-Tuning**: cudnnFindConvolutionForwardAlgorithm(); benchmarks all algorithms; finds optimal; 20-50% speedup over default **Mixed Precision:** - **FP16 Compute**: use FP16 for matrix operations; 2× memory bandwidth, 10-20× compute (Tensor Cores); 312 TFLOPS on A100 - **FP32 Accumulation**: accumulate in FP32; maintains accuracy; prevents overflow; standard practice - **Automatic Mixed Precision (AMP)**: PyTorch/TensorFlow automatic FP16; 2-3× training speedup; minimal code changes - **TF32**: automatic on A100; 8× speedup vs FP32; no code changes; maintains FP32 range; 156 TFLOPS **Batching Strategies:** - **Batched GEMM**: cublasGemmStridedBatchedEx(); processes multiple matrices; 90-95% efficiency; amortizes overhead - **Batch Size**: larger batches improve efficiency; 32-256 typical; 80-95% efficiency; limited by memory - **Micro-Batching**: split large batch into micro-batches; fits in cache; 10-30% speedup for some workloads - **Dynamic Batching**: combine multiple requests; improves throughput; 2-4× for inference serving **Algorithm Selection:** - **cuBLAS Algorithms**: CUBLAS_GEMM_DEFAULT, CUBLAS_GEMM_ALGO0-23; different tile sizes, strategies; auto-tune for workload - **cuDNN Algorithms**: IMPLICIT_GEMM, IMPLICIT_PRECOMP_GEMM, GEMM, DIRECT, FFT, WINOGRAD; different trade-offs; auto-tune critical - **Heuristics**: cudnnGetConvolutionForwardAlgorithm_v7(); uses heuristics; fast but may not be optimal; benchmark for critical paths - **Benchmarking**: cudnnFindConvolutionForwardAlgorithm(); benchmarks all; finds optimal; 20-50% speedup; cache results **Workspace Management:** - **Purpose**: temporary storage for algorithms; larger workspace enables faster algorithms; trade memory for speed - **Size**: query with cudnnGetConvolutionForwardWorkspaceSize(); typical 100MB-2GB; depends on algorithm and tensor size - **Allocation**: pre-allocate workspace; reuse across operations; eliminates allocation overhead (5-50ms) - **Tuning**: try different workspace limits; find optimal trade-off; 10-50% speedup with larger workspace **Fusion Opportunities:** - **Convolution + Activation**: cudnnConvolutionBiasActivationForward(); fuses conv, bias, activation; 20-40% speedup; reduces memory traffic - **Batch Norm + Activation**: fused batch normalization; 2-5× faster than separate; reduces kernel launches - **GEMM + Bias**: cublasGemmEx() with bias; fused operation; 10-20% speedup; reduces memory accesses - **Custom Fusion**: use cuDNN fusion API; define custom fusions; 20-60% speedup for complex patterns **Performance Profiling:** - **Nsight Compute**: profiles cuBLAS/cuDNN kernels; shows Tensor Core utilization, memory bandwidth, compute throughput - **Metrics**: achieved TFLOPS / peak TFLOPS; target 80-95%; memory bandwidth utilization; target 80-100% - **Bottlenecks**: memory-bound (small matrices), compute-bound (large matrices), launch overhead (many small operations) - **Optimization**: increase batch size, use Tensor Cores, fuse operations, tune workspace **Tensor Core Utilization:** - **Requirements**: matrix dimensions multiples of 8 (FP16), 16 (INT8); proper alignment; FP16/BF16 data types - **Verification**: Nsight Compute shows Tensor Core utilization; target 50-80%; low utilization indicates dimension mismatch - **Padding**: pad matrices to multiples of 8/16; enables Tensor Cores; 10-20× speedup outweighs padding overhead - **Performance**: 312 TFLOPS FP16 on A100, 989 TFLOPS on H100; 10-20× faster than CUDA cores **Memory Optimization:** - **Data Layout**: NCHW (channels first) vs NHWC (channels last); NHWC often faster on Tensor Cores; 10-30% speedup - **Alignment**: 128-byte alignment for optimal performance; cudaMalloc provides automatic alignment - **Pinned Memory**: use pinned memory for CPU-GPU transfers; 2-10× faster; cudaMallocHost() - **Prefetching**: overlap data transfer with computation; async transfers; 20-50% throughput improvement **Multi-GPU Scaling:** - **Data Parallelism**: replicate model, split data; cuBLAS/cuDNN on each GPU; 85-95% scaling efficiency on 8 GPUs - **Model Parallelism**: split model across GPUs; requires careful orchestration; 70-85% efficiency - **Gradient Synchronization**: NCCL all-reduce after backward; 5-20ms for 1GB on 8 GPUs with NVLink - **Load Balancing**: ensure equal work per GPU; monitor utilization; 10-30% improvement with proper balancing **Framework Integration:** - **PyTorch**: automatic cuBLAS/cuDNN usage; torch.backends.cudnn.benchmark=True for auto-tuning; 20-50% speedup - **TensorFlow**: automatic library usage; XLA compilation for fusion; 30-60% speedup - **JAX**: automatic with jax.default_matmul_precision('high'); 2-3× speedup - **ONNX Runtime**: cuDNN/cuBLAS backend; TensorRT for optimization; 2-5× inference speedup **Best Practices:** - **Enable Tensor Cores**: use FP16/BF16, set math mode, ensure proper dimensions; 10-20× speedup - **Auto-Tune**: use cudnnFindConvolutionForwardAlgorithm(); benchmark algorithms; 20-50% speedup; cache results - **Batch Operations**: use batched APIs; amortizes overhead; 90-95% efficiency - **Fuse Operations**: use fused APIs; reduces memory traffic and kernel launches; 20-60% speedup - **Profile**: use Nsight Compute; verify Tensor Core utilization, memory bandwidth; optimize based on data **Performance Targets:** - **cuBLAS GEMM**: 10-20 TFLOPS FP32, 60-80 TFLOPS FP16 (Tensor Cores) on A100; 80-95% of peak - **cuDNN Convolution**: 15-30 TFLOPS on A100; 80-95% of peak with Tensor Cores; 70-85% without - **Memory Bandwidth**: 80-100% of peak (1.5-2 TB/s on A100); for memory-bound operations - **Tensor Core Utilization**: 50-80%; indicates proper usage; low utilization means dimension mismatch **Common Pitfalls:** - **Wrong Dimensions**: matrix dimensions not multiples of 8/16; Tensor Cores disabled; 10-20× slowdown - **Default Settings**: not enabling Tensor Cores; using default algorithms; 2-10× slower than optimal - **Small Batches**: batch size too small; low efficiency; increase batch size for 2-5× improvement - **No Auto-Tuning**: using default algorithm; 20-50% slower than optimal; always auto-tune critical paths **Real-World Performance:** - **ResNet-50 Training**: 15-25 TFLOPS on A100; 80-90% of peak; cuDNN convolution dominates; 2-3× speedup with optimization - **BERT Training**: 20-30 TFLOPS on A100; 85-95% of peak; cuBLAS GEMM dominates; 2-4× speedup with Tensor Cores - **GPT-3 Inference**: 30-50 TFLOPS on A100; 80-90% of peak; cuBLAS batched GEMM; 3-5× speedup with batching and Tensor Cores - **Stable Diffusion**: 10-20 TFLOPS on A100; 70-85% of peak; cuDNN convolution; 2-3× speedup with optimization cuBLAS and cuDNN Optimization represent **the foundation of high-performance deep learning** — by properly configuring these highly-optimized libraries to enable Tensor Cores, auto-tune algorithms, batch operations, and fuse computations, developers achieve 80-95% of theoretical peak performance (10-20 TFLOPS FP32, 60-80 TFLOPS FP16 on A100) and 2-10× speedup over default settings, making library optimization essential for deep learning where cuBLAS/cuDNN handle 80-95% of compute and proper tuning determines whether training takes days or weeks and whether inference meets latency requirements.