flash attention

FlashAttention is a breakthrough algorithm that computes exact attention without materializing the full N×N attention matrix, reducing memory from O(N²) to O(N) while achieving 2-4x speedup. Standard attention computation creates massive intermediate matrices: for a 32K context, the attention matrix alone requires 4GB in FP32. FlashAttention works by tiling: it processes query, key, and value matrices in blocks that fit in SRAM, computing partial attention scores using the online softmax trick that incrementally updates normalization factors. The algorithm fuses the entire attention operation into a single GPU kernel, avoiding repeated memory round-trips. This enables training with longer contexts (up to 65K tokens practically) and larger batch sizes on the same hardware. FlashAttention-2 further improved performance through better work partitioning across GPU warps and reduced non-matmul operations. FlashAttention-3 targets Hopper architecture features like TMA and FP8. The technique applies to both training and inference, with particularly dramatic gains for long sequences. Integration is available through PyTorch, xFormers, and direct CUDA implementations. FlashAttention has become standard practice, integrated into frameworks like HuggingFace Transformers and training systems like DeepSpeed.