gpu kernel fusion

Kernel fusion (also called operator fusion) is the optimization of combining several separate GPU operations into a single kernel, so that intermediate results stay in fast on-chip memory instead of being written out to and read back from HBM between every step. It is the single most important trick a deep-learning compiler applies, because the operations that dominate a modern model are limited by memory bandwidth and kernel-launch overhead, not by arithmetic — and fusion attacks exactly those two costs.\n\n**Most deep-learning operators are memory-bound, which is why fusion pays off.** An elementwise add, a GELU, a bias, a layer-norm — each does trivial arithmetic per element but must stream its entire input and output through global memory. Run them as separate kernels and each one pays a full HBM read plus a full HBM write, and the GPU's compute units sit mostly idle waiting on bandwidth. Fuse a chain of them into one kernel and you read the input once, do all the arithmetic while the data sits in registers, and write the result once. The floating-point work is unchanged; what disappears is the traffic to HBM and all but one of the kernel launches.\n\n**Fusion comes in a few distinct shapes.** *Vertical* (producer-consumer) fusion merges a chain where each op consumes the previous op's output — a matmul feeding a bias feeding an activation — and keeps the hand-off in registers or shared memory. *Horizontal* fusion batches independent operations that share inputs, or many tiny operations, into one launch to amortize dispatch overhead and raise occupancy. *Epilogue* fusion folds the cheap elementwise tail (bias, activation, residual add) directly into a compute-bound kernel's writeback stage, as cuBLASLt and CUTLASS do for GEMMs — you get the elementwise work essentially for free while the matmul result is still in registers.\n\n**The roofline is the clean way to see what fusion does.** Every kernel has an arithmetic intensity — FLOPs performed per byte moved — and the roofline model says a kernel is memory-bound until that intensity is high enough to saturate the compute units. A lone elementwise op has terrible intensity (a couple of FLOPs per element read and written) and lives deep in the memory-bound region. Fusing a chain divides the same FLOPs by far fewer bytes, pushing the fused kernel rightward toward the compute-bound ridge. Fusion does not add arithmetic; it deletes the bytes in the denominator.\n\n**Not everything fuses the same way, and some fusions are whole algorithms.** Elementwise chains and reductions fuse readily; compute-bound matmuls and convolutions are already efficient and typically only fuse their epilogues. Operations with a global dependency need more care — a softmax needs a full-row max and sum before it can normalize — which is why the highest-value fusions are redesigned algorithms rather than mechanical merges. FlashAttention is the canonical example: it fuses the entire query-key-softmax-value pipeline into one kernel using an online-softmax recurrence, so the enormous N-by-N score matrix is never written to HBM at all. Compilers such as TorchInductor, XLA, and TensorRT find the easy fusions automatically; the hard ones are still written by hand in Triton or CUDA.\n\n| Fusion type | What it merges | Primary win |\n|---|---|---|\n| **Vertical** (producer→consumer) | a chain like matmul → bias → GELU | intermediates stay on-chip, fewer HBM trips |\n| **Horizontal** | independent ops sharing inputs / many tiny ops | one launch, higher occupancy |\n| **Epilogue** | activation / bias / residual into a GEMM writeback | elementwise tail is nearly free |\n| **Whole-algorithm** (e.g. FlashAttention) | tiled QK·softmax·V via online softmax | the N×N score matrix never touches HBM |\n\n```svg\n \n```\n\nRead fusion through a *how-many-times-does-this-data-cross-HBM* lens rather than a *how-many-FLOPs-does-this-do* lens: the arithmetic in a transformer's pointwise and normalization layers is almost free, so the compiler's job — and yours, when you drop into Triton — is to keep intermediates on-chip and collapse many launches into one, which is why the same math can run several times faster with no change to the numbers it computes.