flash attention
**FlashAttention** is a **memory-efficient, IO-aware exact attention algorithm that reduces GPU memory usage from O(N²) to O(N) and speeds up attention computation by 2-4x** — enabling training of long-context LLMs without approximation.
**The Standard Attention Problem**
- Standard attention materializes the full N×N attention matrix (N = sequence length).
- For N=4096, this is 4096² × 2 bytes = 32 MB per head.
- For 32 heads × 40 layers = 41 GB just for attention matrices (bottleneck for long contexts).
- High-bandwidth memory (HBM) is much slower than SRAM: unnecessary memory reads/writes dominate runtime.
**FlashAttention Key Insight**
- **Tiling**: Split Q, K, V into blocks that fit in SRAM (fast on-chip memory).
- **Online Softmax**: Compute softmax incrementally — no need to store the full attention matrix.
- **Fused Kernel**: Single CUDA kernel performs Q×K, softmax, and ×V in one pass.
- **Result**: Never materialize the full N×N matrix — only store the O(N) output.
**Performance Gains**
| Metric | Standard Attn | FlashAttention | FlashAttention-2 |
|--------|--------------|----------------|------------------|
| Memory | O(N²) | O(N) | O(N) |
| Speedup vs baseline | 1x | 2-4x | 4-8x |
| Max sequence (A100) | ~8K | ~64K | ~128K |
**FlashAttention-2 Improvements**
- Better thread block partitioning for modern GPUs.
- Parallelism across sequence dimension (not just batch/heads).
- ~2x faster than FlashAttention-1.
**FlashAttention-3 (2024)**
- Designed for Hopper (H100) GPUs.
- Exploits asynchronous execution and FP8 precision.
- ~1.5-2x faster than FlashAttention-2 on H100.
**Adoption**: PyTorch 2.0+ includes `torch.nn.functional.scaled_dot_product_attention` with FlashAttention built in.
FlashAttention is **the critical engineering breakthrough that made 100K+ context LLMs practical** — it enabled GPT-4's 128K context and Gemini's 1M context windows.