cublas

**cuBLAS** is the **NVIDIA GPU implementation of BLAS routines for dense linear algebra, especially matrix multiplication** - it powers the GEMM-heavy compute core of modern neural network training and inference. **What Is cuBLAS?** - **Definition**: CUDA library for vector and matrix operations, including highly optimized GEMM kernels. - **Core Relevance**: Linear layers and many attention projections map directly to GEMM workloads. - **Precision Modes**: Supports FP32, FP16, BF16, and tensor-core accelerated math modes. - **Companion Stack**: Often paired with cuBLASLt for advanced epilogue fusion and layout options. **Why cuBLAS Matters** - **Throughput Driver**: GEMM performance strongly determines overall model step time. - **Hardware Utilization**: Tuned kernels maximize occupancy and tensor-core usage on supported devices. - **Numerical Control**: Allows precision-performance tradeoffs through configurable math modes. - **Ecosystem Standard**: Most frameworks rely on cuBLAS as their default dense math backend. - **Optimization Anchor**: Profiling against cuBLAS throughput helps identify true compute bottlenecks. **How It Is Used in Practice** - **Shape Planning**: Align matrix dimensions to tensor-core-friendly sizes when designing model blocks. - **Backend Selection**: Use cuBLASLt interfaces for fused epilogues and layout-specific speed gains. - **Benchmark Discipline**: Track achieved FLOPs versus theoretical peaks to prioritize further tuning. cuBLAS is **a central performance dependency for transformer-scale workloads** - strong GEMM efficiency is mandatory for competitive GPU training speed.