flash attention

**FlashAttention** FlashAttention is a memory-efficient attention algorithm that fuses operations into optimized CUDA kernels reducing memory complexity from O of n squared to O of n enabling longer context windows on the same GPU. Standard attention materializes the full attention matrix in GPU memory which becomes prohibitive for long sequences. FlashAttention uses tiling to compute attention in blocks keeping intermediate results in fast SRAM instead of slow HBM memory. It recomputes attention scores during backward pass instead of storing them trading computation for memory. This IO-aware algorithm achieves 2-4x speedup while using less memory. FlashAttention-2 further optimizes parallelism and reduces non-matmul operations. The technique enables training with 64K token contexts on consumer GPUs. It is essential for long-context models like GPT-4 Claude and Llama-2-Long. FlashAttention demonstrates that algorithm design for modern hardware can dramatically improve efficiency without changing model architecture. It is now standard in frameworks like PyTorch and Hugging Face Transformers.