FlashAttention is an IO-aware attention algorithm that computes exact attention while never writing the full N×N score matrix to HBM. Instead of materializing all pairwise scores, it processes attention in tiles that stay in fast on-chip SRAM, folding each tile into a running softmax. The result is identical to standard attention, but with dramatically less memory traffic — which is what actually limits attention on modern accelerators.
Standard attention is memory-bound, not compute-bound. The naive implementation forms S = QKᵀ, an N×N matrix for sequence length N, writes it to HBM, reads it back to apply softmax, writes the probabilities, then reads them again to multiply by V. The multiply-adds are cheap; the killer is shuttling those large intermediate matrices in and out of high-bandwidth memory. Traffic scales as O(N²), so as context length grows, attention spends nearly all its time moving data rather than computing.
Tiling plus an online softmax removes the round trip. FlashAttention loads blocks of Q, K, and V into SRAM, computes each block's partial scores there, and combines them using an online softmax that keeps a running maximum and normalizer — so it never needs the whole row of scores at once. Because each tile is consumed on-chip and discarded, the O(N²) matrix is never written to HBM. The math is exact (not an approximation), but HBM traffic falls to roughly O(N), turning a memory-bound kernel into a far faster one.
| Aspect | Standard attention | FlashAttention |
|---|---|---|
| N×N scores | written to HBM | kept in SRAM tiles |
| HBM traffic | O(N²) | O(N) |
| Softmax | full-row, materialized | online / running |
| Result | exact | exact (identical) |
| Bottleneck | memory bandwidth | much closer to compute |
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="Segoe UI,Arial,sans-serif"><rect width="760" height="470" fill="#0d1117"/><defs><marker id="ar" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#f87171"/></marker><marker id="ac" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#22d3ee"/></marker><marker id="ag" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#34d399"/></marker><marker id="ah" markerWidth="8" markerHeight="8" refX="6.5" refY="3" orient="auto"><path d="M0,0 L7,3 L0,6 Z" fill="#8b949e"/></marker></defs><text x="20" y="30" fill="#e6edf3" font-size="19" font-weight="700">FlashAttention: never write the N×N score matrix to HBM</text><text x="20" y="50" fill="#8b949e" font-size="12.5">Tile Q, K, V into blocks, keep them in fast SRAM, and fuse softmax + weighted-sum so attention is IO-bound no more.</text><rect x="20" y="66" width="226" height="298" rx="7" fill="#0c141d" stroke="#30363d"/><text x="32" y="87" fill="#e6edf3" font-size="13.5" font-weight="600">Standard attention: the N×N wall</text><rect x="267" y="66" width="226" height="298" rx="7" fill="#0c141d" stroke="#30363d"/><text x="279" y="87" fill="#e6edf3" font-size="13.5" font-weight="600">FlashAttention: tile & stream</text><rect x="514" y="66" width="226" height="298" rx="7" fill="#0c141d" stroke="#30363d"/><text x="526" y="87" fill="#e6edf3" font-size="13.5" font-weight="600">Why it wins</text><rect x="104" y="118" width="88" height="16" rx="2" fill="#241d33" stroke="#c4b5fd" stroke-width="1.1"/><text x="148" y="130" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="700">Kᵀ</text><rect x="82" y="140" width="16" height="88" rx="2" fill="#12233a" stroke="#60a5fa" stroke-width="1.1"/><text x="90" y="187" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="700">Q</text><rect x="104" y="140" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="126" y="140" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="148" y="140" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="170" y="140" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="104" y="162" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="126" y="162" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="148" y="162" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="170" y="162" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="104" y="184" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="126" y="184" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="148" y="184" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="170" y="184" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="104" y="206" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="126" y="206" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="148" y="206" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><rect x="170" y="206" width="22" height="22" fill="#3a1717" stroke="#f87171" stroke-width="0.8"/><text x="148" y="243" fill="#fca5a5" font-size="9.5" text-anchor="middle" font-weight="700">S = QKᵀ — N×N scores</text><line x1="148" y1="250" x2="148" y2="252" stroke="#f87171" stroke-width="1.8" marker-end="url(#ar)"/><rect x="102" y="254" width="96" height="30" rx="5" fill="#2a1414" stroke="#f87171" stroke-width="1.2"/><text x="150" y="267" fill="#f87171" font-size="9.5" text-anchor="middle" font-weight="700">HBM (slow)</text><text x="150" y="279" fill="#fca5a5" font-size="8.4" text-anchor="middle">store & reload O(N²)</text><text x="36" y="316" fill="#8b949e" font-size="9">The whole N×N matrix is written out, then</text><text x="36" y="329" fill="#8b949e" font-size="9">read back for softmax and ×V. Memory,</text><text x="36" y="342" fill="#8b949e" font-size="9">not math, is the bottleneck.</text><rect x="300" y="118" width="25" height="14" rx="2" fill="#241d33" stroke="#c4b5fd" stroke-width="0.9"/><rect x="327" y="118" width="25" height="14" rx="2" fill="#241d33" stroke="#c4b5fd" stroke-width="0.9"/><rect x="354" y="118" width="25" height="14" rx="2" fill="#241d33" stroke="#c4b5fd" stroke-width="0.9"/><rect x="381" y="118" width="25" height="14" rx="2" fill="#241d33" stroke="#c4b5fd" stroke-width="0.9"/><text x="354" y="114" fill="#c4b5fd" font-size="8.6" text-anchor="middle">K,V blocks — loop →</text><rect x="280" y="138" width="14" height="25" rx="2" fill="#101820" stroke="#30363d" stroke-width="0.9"/><rect x="280" y="165" width="14" height="25" rx="2" fill="#12233a" stroke="#60a5fa" stroke-width="0.9"/><rect x="280" y="192" width="14" height="25" rx="2" fill="#101820" stroke="#30363d" stroke-width="0.9"/><rect x="280" y="219" width="14" height="25" rx="2" fill="#101820" stroke="#30363d" stroke-width="0.9"/><text x="270" y="192" fill="#60a5fa" font-size="8.6" text-anchor="middle" transform="rotate(-90 270 192)">Q blocks</text><rect x="300" y="138" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="327" y="138" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="354" y="138" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="381" y="138" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="300" y="165" width="25" height="25" fill="#0f1f24" stroke="#155e63" stroke-width="0.8"/><rect x="327" y="165" width="25" height="25" fill="#0c3b3f" stroke="#22d3ee" stroke-width="1.6"/><rect x="354" y="165" width="25" height="25" fill="#0f1f24" stroke="#155e63" stroke-width="0.8"/><rect x="381" y="165" width="25" height="25" fill="#0f1f24" stroke="#155e63" stroke-width="0.8"/><rect x="300" y="192" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="327" y="192" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="354" y="192" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="381" y="192" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="300" y="219" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="327" y="219" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="354" y="219" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><rect x="381" y="219" width="25" height="25" fill="#0e161d" stroke="#30363d" stroke-width="0.8"/><text x="339.5" y="178.5" fill="#67e8f9" font-size="8" text-anchor="middle" font-weight="700">SRAM</text><rect x="280" y="260" width="138" height="46" rx="5" fill="#111a24" stroke="#34d399" stroke-width="1.1"/><text x="288" y="275" fill="#6ee7b7" font-size="9" font-weight="700">running state (in SRAM):</text><text x="288" y="289" fill="#cdd9e5" font-size="8.7">m = max · l = Σexp · O = weighted sum</text><text x="288" y="301" fill="#8b949e" font-size="8.2">rescale m, l, O as each new block arrives</text><text x="280" y="354" fill="#8b949e" font-size="8.6">full S never touches HBM — only the tile does.</text><polygon points="576,106 548,140 604,140" fill="#153524" stroke="#34d399" stroke-width="1.2"/><text x="576" y="133" fill="#6ee7b7" font-size="9" text-anchor="middle" font-weight="700">SRAM</text><text x="614" y="130" fill="#8b949e" font-size="8">~19 TB/s, KB</text><polygon points="548,144 604,144 628,178 524,178" fill="#3a2a17" stroke="#fbbf24" stroke-width="1.2"/><text x="576" y="167" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="700">HBM</text><text x="634" y="166" fill="#8b949e" font-size="8">~2 TB/s, GB</text><text x="526" y="194" fill="#cdd9e5" font-size="9">Flash keeps the working tile in SRAM,</text><text x="526" y="206" fill="#cdd9e5" font-size="9">so it rarely pays the HBM tax.</text><text x="526" y="228" fill="#8b949e" font-size="9">HBM traffic (attention)</text><line x1="554" y1="260" x2="554" y2="316" stroke="#30363d"/><line x1="554" y1="316" x2="720" y2="316" stroke="#30363d"/><rect x="574" y="260" width="46" height="56" fill="#3a1717" stroke="#f87171" stroke-width="1.2"/><text x="597" y="255" fill="#f87171" font-size="9" text-anchor="middle" font-weight="700">O(N²)</text><text x="597" y="329" fill="#8b949e" font-size="8.4" text-anchor="middle">standard</text><rect x="652" y="300" width="46" height="16" fill="#153524" stroke="#34d399" stroke-width="1.2"/><text x="675" y="295" fill="#34d399" font-size="9" text-anchor="middle" font-weight="700">linear</text><text x="675" y="329" fill="#8b949e" font-size="8.4" text-anchor="middle">flash</text><text x="526" y="352" fill="#8b949e" font-size="8.7">Same exact math, far fewer memory trips.</text><rect x="20" y="384" width="226" height="70" rx="7" fill="#111a24" stroke="#30363d"/><text x="32" y="403" fill="#e6edf3" font-size="12.5" font-weight="700">The insight</text><text x="32" y="420" fill="#cdd9e5" font-size="10">Attention is limited by memory movement, not</text><text x="32" y="433" fill="#cdd9e5" font-size="10">FLOPs. The N-by-N scores are huge; writing</text><text x="32" y="446" fill="#cdd9e5" font-size="10">them to HBM and back is what makes long</text><text x="32" y="459" fill="#cdd9e5" font-size="10">context slow.</text><rect x="267" y="384" width="226" height="70" rx="7" fill="#111a24" stroke="#30363d"/><text x="279" y="403" fill="#e6edf3" font-size="12.5" font-weight="700">Online softmax</text><text x="279" y="420" fill="#cdd9e5" font-size="10">Softmax normally needs the whole row. Flash</text><text x="279" y="433" fill="#cdd9e5" font-size="10">keeps a running max and sum so it can process</text><text x="279" y="446" fill="#cdd9e5" font-size="10">blocks one at a time and still get the exact</text><text x="279" y="459" fill="#cdd9e5" font-size="10">result.</text><rect x="514" y="384" width="226" height="70" rx="7" fill="#111a24" stroke="#30363d"/><text x="526" y="403" fill="#e6edf3" font-size="12.5" font-weight="700">What it unlocks</text><text x="526" y="420" fill="#cdd9e5" font-size="10">Exact attention with linear memory: longer</text><text x="526" y="433" fill="#cdd9e5" font-size="10">context, bigger batches, higher GPU</text><text x="526" y="446" fill="#cdd9e5" font-size="10">utilization. Standard in modern LLM training</text><text x="526" y="459" fill="#cdd9e5" font-size="10">and inference kernels.</text></svg>
**It reshaped how long-context models are trained and served.** By cutting attention's memory traffic and footprint, FlashAttention made longer sequences practical without the quadratic memory blowup, and later versions (FlashAttention-2 and -3) pushed GPU utilization higher by improving work partitioning and exploiting newer hardware. It pairs naturally with the KV cache at inference: the cache limits how much context you can hold, while FlashAttention limits how expensive it is to attend over that context. Both are fundamentally about respecting the memory hierarchy.
Read FlashAttention through a quant lens rather than a 'clever kernel' lens: attention's cost is bytes moved through HBM, and per the roofline a memory-bound kernel is capped by bandwidth long before FLOPs. FlashAttention lowers arithmetic intensity's denominator — HBM bytes — by keeping the N×N scores on-chip, so the same math lands higher on the roofline. The design question is how large a tile fits in SRAM and how few HBM bytes per token attention must touch, a measured traffic budget rather than a FLOP count.
flash attentionefficient attentionio-aware attention
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.