gpu kernel fusion

**GPU Kernel Fusion** is the **performance optimization technique that combines multiple separate GPU kernels into a single fused kernel — eliminating the overhead of multiple kernel launches (5-20 us each), removing intermediate global memory reads and writes between kernels, and increasing the arithmetic intensity of the fused computation by keeping intermediate results in registers or shared memory where they can be reused at 10-100x lower latency**. **Why Fusion Matters** A typical deep learning inference pipeline applies dozens of operations sequentially: GEMM → bias add → LayerNorm → ReLU → GEMM → ... Each operation, when implemented as a separate kernel, writes its output to global memory (~400 cycle latency) and the next kernel reads it back. For element-wise operations (bias, activation, normalization), the compute is trivial but the memory traffic dominates — the kernel is severely memory-bound. **Fusion Types** - **Element-Wise Fusion**: Combine operations that operate on the same elements independently: `y = relu(x + bias)` as one kernel instead of three (add, bias, relu). Each element is loaded once, all operations applied in registers, result stored once. Memory traffic reduction: 3x → 1x. - **Reduction + Element-Wise Fusion**: LayerNorm computes mean and variance (reductions) followed by normalization (element-wise). Fusing avoids materializing intermediate reduction results to global memory. - **GEMM + Epilogue Fusion**: Matrix multiplication followed by bias addition, activation, and residual connection. cuBLAS supports epilogue fusion (bias, ReLU, GELU) directly in the GEMM kernel. The epilogue executes on the GEMM output tile while it's still in registers/shared memory. - **Vertical Fusion (Operator Fusion in DL Compilers)**: Multiple layers of a neural network fused into a single kernel. TVM, Triton, XLA, and TensorRT automatically identify fusion opportunities in the computation graph and generate fused kernels. **Quantifying the Benefit** Consider three element-wise operations on an array of N float32 values: - **Unfused**: 3 kernel launches × (N reads + N writes) × 4 bytes = 24N bytes of memory traffic + 15-60 us launch overhead. - **Fused**: 1 kernel launch × (N reads + N writes) × 4 bytes = 8N bytes of memory traffic + 5-20 us launch overhead. - **Speedup**: 3x memory traffic reduction → 2-3x kernel speedup for memory-bound operations. **Automatic Fusion Frameworks** - **Triton (OpenAI)**: Python DSL for writing fused GPU kernels. Programmers express tile-level operations; Triton compiler handles register allocation, shared memory management, and instruction scheduling. - **torch.compile (PyTorch)**: Traces the computation graph, identifies fusion opportunities, and generates fused kernels via Triton or C++ codegen. - **TensorRT**: NVIDIA's inference optimizer. Layer fusion is a primary optimization: Conv+BN+ReLU, GEMM+bias+GELU, multi-head attention fusion. - **XLA (TensorFlow/JAX)**: Compiler infrastructure that fuses element-wise operations and reduces memory-bound kernel chains to single fused operations. **GPU Kernel Fusion is the compiler optimization that unlocks the GPU's true potential** — because the raw computational throughput of modern GPUs is so high that most individual operations are memory-bound, and only by fusing operations to eliminate intermediate memory traffic can the compute units be kept productively busy.