CUDA Tensor Operations and cuBLAS is the NVIDIA GPU library ecosystem for high-performance linear algebra and tensor computation β providing highly optimized implementations of matrix multiplication (GEMM), convolution, and tensor contractions that form the computational backbone of deep learning training and inference, scientific simulation, and numerical computing. cuBLAS and its extensions (cuBLASLt, cuDNN, cuTENSOR) achieve near-theoretical-peak GPU performance by exploiting Tensor Cores, memory hierarchy, and instruction-level parallelism.
Why GEMM Is Central
- Matrix multiplication (GEMM: C = Ξ±ΓAΓB + Ξ²ΓC) accounts for 70β90% of FLOPs in deep learning.
- Fully connected layers: Weight matrix Γ activation matrix β GEMM.
- Attention mechanism: QΓK^T β GEMM; AttentionΓV β GEMM.
- Convolution: Implicit GEMM β convert conv to matrix multiply via im2col.
- Consequence: Optimizing GEMM throughput β optimizing overall model throughput.
NVIDIA Tensor Cores
- Dedicated matrix multiply units introduced in Volta (V100): 4Γ4 matrix Γ 4Γ4 matrix in one instruction.
- Each Tensor Core: 64 FP16 FMAs per clock β 128 FLOPs/clock.
- H100 GPU: 528 Tensor Cores Γ 2 GHz Γ 256 FLOPs = ~270 TFLOPS (FP16 theoretical).
- Tensor Core precision: FP16, BF16, FP8, INT8, INT4 β different datatypes for training vs. inference.
cuBLAS API
``cpp
// Single-precision GEMM: C = A Γ B
cublasHandle_t handle;
cublasCreate(&handle);
cublasSgemm(handle,
CUBLAS_OP_N, CUBLAS_OP_N, // no transpose
M, N, K, // dimensions
&alpha, // scalar
d_A, M, // matrix A (device)
d_B, K, // matrix B (device)
&beta,
d_C, M); // output C (device)
`
cuBLASLt (LightWeight cuBLAS)
- More flexible GEMM interface for Tensor Core operations.
- Supports: Mixed precision (FP16 in, FP32 accumulate), epilogue fusion (ReLU, bias add after GEMM).
- Algorithm search: cublasLtMatmulAlgoGetIds() β enumerate algorithms β benchmark β pick fastest.
- Used by: PyTorch F.linear(), cuDNN attention layers, TensorRT.
cuTENSOR
- General tensor contraction library (beyond 2D matrix multiply).
- C[i,j,k] = A[i,m,n] Γ B[m,n,j,k] β arbitrary tensor index contraction.
- Used for: Tensor network simulation, multi-dimensional convolution, quantum chemistry.
Memory Hierarchy Optimization
- GEMM tiles: Partition A, B into tiles that fit in L1/shared memory β reduce global memory traffic.
- Tiling hierarchy: Thread block tile (fits in shared memory) β Warp tile (fits in registers) β Thread tile.
- Shared memory double buffering: Load next tile while computing current tile β hide memory latency.
- Memory layout: Row-major vs. column-major matters for coalescing β cuBLAS handles transparently.
FP8 GEMM (H100 Feature)
- H100 Hopper: FP8 Tensor Core β 2Γ throughput vs. FP16 at same precision level.
- Training: Use FP8 for forward pass GEMM β FP32 accumulation β FP8 gradient β ~2Γ faster.
- cuBLASLt FP8 GEMM: E4M3 and E5M2 formats supported.
- Scaling: Dynamic loss scaling required to prevent underflow in FP8 gradient range.
Batched GEMM
- Many independent small GEMMs in parallel: Batch of B matrices.
- Example: Attention heads in transformer β B = batch_size Γ num_heads independent QK^T GEMMs.
- cublasSgemmBatched(): Array of matrix pointers β launch B GEMMs in one call.cublasSgemmStridedBatched()` β matrices contiguous in memory β faster.
- Strided batched:
Flash Attention vs. cuBLAS GEMM
- Standard attention: 3 separate GEMM calls β intermediate matrices in global memory β memory bound.
- Flash Attention: Fused kernel β computes QΓK^T + softmax + ΓV in one pass β no global write of attention matrix.
- Flash Attention implementation uses CUDA directly, not cuBLAS β custom tiling for SRAM.
CUDA tensor operations and cuBLAS are the computational engine underneath every major AI framework β when PyTorch, TensorFlow, or JAX run a matrix multiplication, they invoke cuBLAS at the lowest level, making cuBLAS performance optimization directly equivalent to optimizing the throughput of every neural network trained or deployed on NVIDIA hardware, which encompasses the vast majority of AI computation worldwide.