flash attention algorithm

**FlashAttention** is **the IO-aware attention algorithm that reduces memory access and enables exact attention computation with O(N) memory complexity instead of O(N²) by tiling and recomputation** — achieving 2-4× speedup for attention layers and enabling 4-8× longer sequence lengths within GPU memory limits, making it the standard attention implementation in modern LLMs including GPT-4, Llama 2, and Falcon. **Memory Bottleneck in Standard Attention:** - **Quadratic Memory**: standard attention materializes N×N attention matrix for sequence length N; 16K sequence with 128 heads requires 16K×16K×128×2 bytes = 64GB just for attention scores; exceeds A100 80GB memory at 20K tokens - **Memory Bandwidth Limitation**: modern GPUs are memory-bound for attention; A100 delivers 312 TFLOPS but only 1.5-2 TB/s HBM bandwidth; attention's low arithmetic intensity (FLOPs per byte) means performance limited by memory speed, not compute - **Intermediate Activations**: standard implementation stores Q, K, V matrices (3×N×d), attention scores (N×N), attention weights after softmax (N×N), and output (N×d); total memory: O(N² + Nd) where N² term dominates for long sequences - **Backward Pass**: gradient computation requires storing attention weights from forward pass; doubles memory requirement; prevents training on sequences >4K tokens on single A100 for typical model sizes **FlashAttention Algorithm:** - **Tiling Strategy**: divides Q, K, V into blocks that fit in SRAM (on-chip fast memory); processes attention in tiles without materializing full N×N matrix in HBM (slow off-chip memory); block size typically 64-256 tokens depending on head dimension - **Online Softmax**: computes softmax incrementally using numerically stable online algorithm; maintains running max and sum statistics; eliminates need to store full attention matrix before softmax; enables single-pass computation - **Recomputation in Backward**: instead of storing attention weights for backward pass, recomputes them from Q, K, V during backpropagation; trades compute for memory; on modern GPUs, recomputation is faster than loading from HBM due to memory bandwidth bottleneck - **Kernel Fusion**: fuses attention operations (matmul, softmax, dropout, matmul) into single CUDA kernel; reduces kernel launch overhead and intermediate memory traffic; achieves 70-80% of theoretical peak memory bandwidth vs 20-30% for unfused implementation **Performance Improvements:** - **Speed**: 2-4× faster than PyTorch standard attention for typical sequence lengths (2K-8K); speedup increases with sequence length; at 16K tokens, FlashAttention is 5-7× faster; speedup comes from reduced memory traffic, not more FLOPs - **Memory**: O(N) memory complexity vs O(N²); enables 4-8× longer sequences in same memory; 40GB A100 can handle 32K tokens with FlashAttention vs 4K with standard attention for 7B parameter model - **Training Throughput**: end-to-end training speedup of 15-30% for models where attention is bottleneck (long sequences, many layers); GPT-3 scale models see 20-25% speedup; enables training on longer contexts without sequence packing - **Exact Attention**: unlike approximate attention methods (Linformer, Performer), FlashAttention computes exact attention; no quality degradation; drop-in replacement for standard attention with identical outputs (within numerical precision) **FlashAttention-2 Improvements:** - **Better Parallelism**: FlashAttention-2 improves work partitioning across GPU SMs (streaming multiprocessors); reduces thread block idle time; achieves 2× speedup over FlashAttention-1 on A100/H100 - **Reduced Non-Matmul FLOPs**: optimizes softmax and other non-matmul operations; reduces overhead from 15% to 5% of total time; particularly beneficial for shorter sequences where matmul is less dominant - **Multi-Query Attention Support**: optimized kernel for MQA (multi-query attention) and GQA (grouped-query attention) used in Llama 2, Falcon; achieves near-theoretical speedup from reduced KV cache size - **Sequence Length Flexibility**: removes power-of-2 sequence length restrictions; handles arbitrary lengths efficiently; simplifies integration and eliminates padding overhead **Adoption and Impact:** - **Framework Integration**: native support in PyTorch 2.0+ (torch.nn.functional.scaled_dot_product_attention), Hugging Face Transformers, JAX (via Pallas), TensorFlow (via XLA); automatically used when available - **Model Training**: used in training GPT-4, Llama 2 (65B, 70B), Falcon (40B, 180B), MPT (7B-30B), StableLM; enables longer context windows (8K-32K) that define current model capabilities - **Inference Optimization**: combined with KV caching, enables efficient long-context inference; critical for applications like document QA, code generation, and multi-turn conversations where context exceeds 4K tokens - **Research Enablement**: makes long-context research practical; enables experiments with 32K-100K token contexts on academic hardware; democratizes long-context model development FlashAttention is **the algorithmic innovation that removed the memory wall for attention computation** — transforming attention from a quadratic memory bottleneck into a linear-memory operation through careful algorithm-hardware co-design, enabling the long-context capabilities that distinguish modern LLMs from their predecessors.