kernel fusion

Kernel fusion combines multiple sequential GPU operations into a single CUDA kernel, reducing memory bandwidth overhead and kernel launch latency to significantly improve LLM inference and training performance. Problem: standard deep learning frameworks execute operations as separate GPU kernels—each kernel reads inputs from GPU memory (HBM), computes, writes outputs back. Between operations, intermediate results make expensive round-trips through HBM (bandwidth ~3 TB/s on H100, but still the bottleneck). Fusion benefit: combined kernel keeps intermediate results in fast on-chip memory (SRAM/registers, ~30 TB/s bandwidth), avoiding HBM round-trips. This can improve performance 2-10× for memory-bound operations. Common fusion patterns: (1) Attention fusion—combine Q×K, softmax, ×V into single kernel (FlashAttention); (2) Layer norm + activation—fuse normalization with subsequent nonlinearity; (3) Bias + GeLU—combine bias addition with activation function; (4) Fused MLP—combine linear → activation → linear into fewer kernels; (5) Fused softmax—compute softmax without materializing full attention matrix; (6) Rotary embedding fusion—integrate positional encoding into attention kernel. Implementation approaches: (1) Hand-written CUDA—maximum performance, high development effort (FlashAttention); (2) Torch.compile/Inductor—PyTorch JIT compiler automatically fuses eligible operations; (3) Triton—Python-like GPU kernel language enabling custom fused kernels with lower effort; (4) TensorRT—NVIDIA inference optimizer with automatic fusion; (5) XLA—TensorFlow/JAX compiler with fusion passes. FlashAttention: the most impactful fusion—reduces attention from O(N²) memory to O(N) by tiling computation and keeping partial results in SRAM. Kernel fusion is one of the most effective optimization techniques for both LLM training and inference performance.