<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Attention — Scaled Dot-Product in Detail</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">Attn(Q,K,V) = softmax(QK&#x1D40;/&#x221A;d_k) &#xB7; V — every token queries every other for relevance, then reads their values</text>

  <!-- === STEP-BY-STEP DATA FLOW === -->
  <text x="380" y="74" fill="#e6edf3" font-size="12" font-weight="600" text-anchor="middle">Step-by-Step (one head, seq_len=5, d_k=4)</text>

  <!-- Step 1: Input X → project to Q, K, V -->
  <text x="40" y="95" fill="#6b7684" font-size="9">Step 1: project input to Q, K, V</text>

  <!-- Input X -->
  <rect x="40" y="102" width="50" height="55" rx="3" fill="#14202c" stroke="#8b98a5" stroke-width="0.8"/>
  <text x="65" y="122" fill="#8b98a5" font-size="8" text-anchor="middle">X</text>
  <text x="65" y="134" fill="#6b7684" font-size="7" text-anchor="middle">5×d</text>

  <!-- Three projection arrows -->
  <path d="M92,118 L118,108" fill="none" stroke="#60a5fa" stroke-width="1"/>
  <path d="M92,130 L118,130" fill="none" stroke="#34d399" stroke-width="1"/>
  <path d="M92,142 L118,148" fill="none" stroke="#f59e0b" stroke-width="1"/>

  <!-- Q matrix -->
  <rect x="120" y="95" width="42" height="42" rx="2" fill="#0f1a2a" stroke="#60a5fa" stroke-width="1.1"/>
  <text x="141" y="113" fill="#93c5fd" font-size="9" text-anchor="middle" font-weight="600">Q</text>
  <text x="141" y="128" fill="#6b7684" font-size="7" text-anchor="middle">5×4</text>

  <!-- K matrix -->
  <rect x="120" y="142" width="42" height="42" rx="2" fill="#0f2a1f" stroke="#34d399" stroke-width="1.1"/>
  <text x="141" y="160" fill="#6ee7b7" font-size="9" text-anchor="middle" font-weight="600">K</text>
  <text x="141" y="175" fill="#6b7684" font-size="7" text-anchor="middle">5×4</text>

  <!-- V matrix (below) -->
  <rect x="40" y="165" width="42" height="42" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1.1"/>
  <text x="61" y="183" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">V</text>
  <text x="61" y="198" fill="#6b7684" font-size="7" text-anchor="middle">5×4</text>

  <!-- Step 2: QK^T -->
  <text x="185" y="95" fill="#6b7684" font-size="9">Step 2: QK&#x1D40; scores</text>

  <!-- Matmul Q × K^T -->
  <path d="M164,116 L192,125" fill="none" stroke="#60a5fa" stroke-width="0.8"/>
  <path d="M164,160 L192,145" fill="none" stroke="#34d399" stroke-width="0.8"/>
  
  <!-- Score matrix (5×5) -->
  <rect x="195" y="105" width="60" height="60" rx="3" fill="#080d14" stroke="#a78bfa" stroke-width="1.2"/>
  <!-- Heatmap cells (5x5 with varying brightness) -->
  <g>
    <rect x="198" y="108" width="10" height="10" fill="#a78bfa" opacity="0.8"/><rect x="209" y="108" width="10" height="10" fill="#a78bfa" opacity="0.2"/><rect x="220" y="108" width="10" height="10" fill="#a78bfa" opacity="0.1"/><rect x="231" y="108" width="10" height="10" fill="#a78bfa" opacity="0.3"/><rect x="242" y="108" width="10" height="10" fill="#a78bfa" opacity="0.1"/>
    <rect x="198" y="119" width="10" height="10" fill="#a78bfa" opacity="0.2"/><rect x="209" y="119" width="10" height="10" fill="#a78bfa" opacity="0.7"/><rect x="220" y="119" width="10" height="10" fill="#a78bfa" opacity="0.3"/><rect x="231" y="119" width="10" height="10" fill="#a78bfa" opacity="0.1"/><rect x="242" y="119" width="10" height="10" fill="#a78bfa" opacity="0.6"/>
    <rect x="198" y="130" width="10" height="10" fill="#a78bfa" opacity="0.1"/><rect x="209" y="130" width="10" height="10" fill="#a78bfa" opacity="0.5"/><rect x="220" y="130" width="10" height="10" fill="#a78bfa" opacity="0.2"/><rect x="231" y="130" width="10" height="10" fill="#a78bfa" opacity="0.7"/><rect x="242" y="130" width="10" height="10" fill="#a78bfa" opacity="0.4"/>
    <rect x="198" y="141" width="10" height="10" fill="#a78bfa" opacity="0.3"/><rect x="209" y="141" width="10" height="10" fill="#a78bfa" opacity="0.1"/><rect x="220" y="141" width="10" height="10" fill="#a78bfa" opacity="0.6"/><rect x="231" y="141" width="10" height="10" fill="#a78bfa" opacity="0.8"/><rect x="242" y="141" width="10" height="10" fill="#a78bfa" opacity="0.2"/>
    <rect x="198" y="152" width="10" height="10" fill="#a78bfa" opacity="0.1"/><rect x="209" y="152" width="10" height="10" fill="#a78bfa" opacity="0.6"/><rect x="220" y="152" width="10" height="10" fill="#a78bfa" opacity="0.4"/><rect x="231" y="152" width="10" height="10" fill="#a78bfa" opacity="0.2"/><rect x="242" y="152" width="10" height="10" fill="#a78bfa" opacity="0.9"/>
  </g>
  <text x="225" y="175" fill="#c4b5fd" font-size="7.5" text-anchor="middle">scores (5×5)</text>

  <!-- Step 3: Scale + Softmax -->
  <text x="275" y="95" fill="#6b7684" font-size="9">Step 3: /&#x221A;d + softmax</text>
  <path d="M257,135 L278,135" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="276,132 282,135 276,138" fill="#8b98a5"/>

  <rect x="285" y="105" width="60" height="60" rx="3" fill="#080d14" stroke="#c4b5fd" stroke-width="1.2"/>
  <!-- Softmax weights (rows sum to 1) -->
  <g>
    <rect x="288" y="108" width="10" height="10" fill="#c4b5fd" opacity="0.7"/><rect x="299" y="108" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="310" y="108" width="10" height="10" fill="#c4b5fd" opacity="0.05"/><rect x="321" y="108" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="332" y="108" width="10" height="10" fill="#c4b5fd" opacity="0.05"/>
    <rect x="288" y="119" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="299" y="119" width="10" height="10" fill="#c4b5fd" opacity="0.5"/><rect x="310" y="119" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="321" y="119" width="10" height="10" fill="#c4b5fd" opacity="0.05"/><rect x="332" y="119" width="10" height="10" fill="#c4b5fd" opacity="0.4"/>
    <rect x="288" y="130" width="10" height="10" fill="#c4b5fd" opacity="0.05"/><rect x="299" y="130" width="10" height="10" fill="#c4b5fd" opacity="0.3"/><rect x="310" y="130" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="321" y="130" width="10" height="10" fill="#c4b5fd" opacity="0.5"/><rect x="332" y="130" width="10" height="10" fill="#c4b5fd" opacity="0.2"/>
    <rect x="288" y="141" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="299" y="141" width="10" height="10" fill="#c4b5fd" opacity="0.05"/><rect x="310" y="141" width="10" height="10" fill="#c4b5fd" opacity="0.4"/><rect x="321" y="141" width="10" height="10" fill="#c4b5fd" opacity="0.6"/><rect x="332" y="141" width="10" height="10" fill="#c4b5fd" opacity="0.1"/>
    <rect x="288" y="152" width="10" height="10" fill="#c4b5fd" opacity="0.05"/><rect x="299" y="152" width="10" height="10" fill="#c4b5fd" opacity="0.4"/><rect x="310" y="152" width="10" height="10" fill="#c4b5fd" opacity="0.2"/><rect x="321" y="152" width="10" height="10" fill="#c4b5fd" opacity="0.1"/><rect x="332" y="152" width="10" height="10" fill="#c4b5fd" opacity="0.7"/>
  </g>
  <text x="315" y="175" fill="#c4b5fd" font-size="7.5" text-anchor="middle">weights (rows sum=1)</text>

  <!-- Step 4: × V → output -->
  <text x="365" y="95" fill="#6b7684" font-size="9">Step 4: weights × V</text>
  <path d="M347,135 L368,135" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <polygon points="366,132 372,135 366,138" fill="#8b98a5"/>

  <!-- V reference -->
  <path d="M61,207 C61,218 370,218 370,155" fill="none" stroke="#f59e0b" stroke-width="0.8" stroke-dasharray="3,2"/>

  <!-- Output matrix -->
  <rect x="375" y="105" width="42" height="60" rx="2" fill="#1a1520" stroke="#e6edf3" stroke-width="1.2"/>
  <text x="396" y="132" fill="#e6edf3" font-size="9" text-anchor="middle" font-weight="600">out</text>
  <text x="396" y="150" fill="#8b98a5" font-size="7" text-anchor="middle">5×4</text>
  <text x="396" y="178" fill="#6b7684" font-size="7.5" text-anchor="middle">context-</text>
  <text x="396" y="188" fill="#6b7684" font-size="7.5" text-anchor="middle">aware</text>

  <!-- === RIGHT: Multi-Head Attention === -->
  <rect x="440" y="82" width="290" height="135" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="585" y="100" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Multi-Head Attention (h heads)</text>

  <!-- Parallel heads -->
  <rect x="460" y="110" width="52" height="30" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="486" y="128" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Head 0</text>
  <rect x="518" y="110" width="52" height="30" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="544" y="128" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Head 1</text>
  <rect x="576" y="110" width="52" height="30" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="602" y="128" fill="#c4b5fd" font-size="7.5" text-anchor="middle">Head 2</text>
  <rect x="634" y="110" width="52" height="30" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8" opacity="0.5"/>
  <text x="660" y="128" fill="#8b98a5" font-size="7.5" text-anchor="middle">...</text>
  <rect x="692" y="110" width="28" height="30" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="0.8" opacity="0.5"/>
  <text x="706" y="128" fill="#8b98a5" font-size="7" text-anchor="middle">h-1</text>

  <!-- Concat + project -->
  <path d="M585,142 L585,155" fill="none" stroke="#8b98a5" stroke-width="0.8"/>
  <rect x="530" y="158" width="110" height="22" rx="3" fill="#14202c" stroke="#8b98a5" stroke-width="0.8"/>
  <text x="585" y="172" fill="#e6edf3" font-size="8" text-anchor="middle">Concat → W_O projection</text>

  <!-- Output -->
  <path d="M585,180 L585,195" fill="none" stroke="#e6edf3" stroke-width="0.8"/>
  <text x="585" y="207" fill="#8b98a5" font-size="8" text-anchor="middle">output (seq_len × d_model)</text>

  <!-- Explanation -->
  <text x="585" y="148" fill="#6b7684" font-size="7.5" text-anchor="middle">each head: d_k = d_model / h (e.g., 128 = 12288/96)</text>

  <!-- === BOTTOM LEFT: Causal mask === -->
  <rect x="30" y="220" width="200" height="115" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="130" y="238" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Causal Mask (decoder)</text>

  <!-- Lower-triangular mask visualization -->
  <g>
    <rect x="55" y="248" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="69" y="248" width="12" height="12" fill="#0d1117"/><rect x="83" y="248" width="12" height="12" fill="#0d1117"/><rect x="97" y="248" width="12" height="12" fill="#0d1117"/><rect x="111" y="248" width="12" height="12" fill="#0d1117"/>
    <rect x="55" y="262" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="69" y="262" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="83" y="262" width="12" height="12" fill="#0d1117"/><rect x="97" y="262" width="12" height="12" fill="#0d1117"/><rect x="111" y="262" width="12" height="12" fill="#0d1117"/>
    <rect x="55" y="276" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="69" y="276" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="83" y="276" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="97" y="276" width="12" height="12" fill="#0d1117"/><rect x="111" y="276" width="12" height="12" fill="#0d1117"/>
    <rect x="55" y="290" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="69" y="290" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="83" y="290" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="97" y="290" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="111" y="290" width="12" height="12" fill="#0d1117"/>
    <rect x="55" y="304" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="69" y="304" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="83" y="304" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="97" y="304" width="12" height="12" fill="#34d399" opacity="0.6"/><rect x="111" y="304" width="12" height="12" fill="#34d399" opacity="0.6"/>
  </g>
  <text x="140" y="270" fill="#6ee7b7" font-size="8">green = can attend</text>
  <text x="140" y="284" fill="#6b7684" font-size="8">black = masked (-inf)</text>
  <text x="130" y="324" fill="#8b98a5" font-size="8" text-anchor="middle">token can only see past + self</text>

  <!-- === BOTTOM CENTER: Complexity + optimizations === -->
  <rect x="245" y="220" width="245" height="115" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="367" y="238" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Complexity + Optimizations</text>

  <text x="260" y="258" fill="#f87171" font-size="9">Naive: O(n&#xB2;) time and memory</text>
  <text x="260" y="274" fill="#8b98a5" font-size="8.5">128K context → 128K&#xB2; = 16B elements</text>

  <text x="260" y="296" fill="#34d399" font-size="9" font-weight="600">Optimizations:</text>
  <text x="260" y="312" fill="#8b98a5" font-size="8.5">FlashAttention: tile in SRAM, no n&#xB2; mem</text>
  <text x="260" y="326" fill="#8b98a5" font-size="8.5">KV cache: store past K,V for decode</text>
  <text x="260" y="340" fill="#6b7684" font-size="8">Ring attention: shard seq across GPUs</text>

  <!-- === BOTTOM RIGHT: Attention variants === -->
  <rect x="505" y="220" width="225" height="115" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="617" y="238" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Attention Variants</text>

  <text x="520" y="258" fill="#c4b5fd" font-size="9">Multi-Head (MHA): full K,V per head</text>
  <text x="520" y="274" fill="#60a5fa" font-size="9">Multi-Query (MQA): shared K,V</text>
  <text x="520" y="290" fill="#34d399" font-size="9">Grouped-Query (GQA): groups share</text>
  <text x="520" y="306" fill="#fbbf24" font-size="9">Sliding window: local context only</text>
  <text x="520" y="322" fill="#8b98a5" font-size="9">Cross-attention: Q from one, K/V other</text>
  <text x="520" y="338" fill="#6b7684" font-size="8">GQA used by Llama-3, Gemma, Mistral</text>

  <!-- Bottom key insight -->
  <rect x="30" y="348" width="700" height="40" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="365" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Attention = differentiable database lookup. Q is the query, K is the key, V is the value. Soft match by dot product.</text>
  <text x="380" y="381" fill="#8b98a5" font-size="9" text-anchor="middle">This single operation — weighted average based on learned relevance — is why transformers can model any sequence.</text>

  <!-- Scale context -->
  <text x="380" y="410" fill="#fbbf24" font-size="9.5" text-anchor="middle">GPT-4: 96 heads × 96 layers, d_k=128, context=128K → attention dominates compute and memory</text>
  <text x="380" y="428" fill="#8b98a5" font-size="9" text-anchor="middle">FlashAttention reduced memory from O(n&#xB2;) to O(n) — enabling 100K+ context without OOM</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Attention is how a model decides what matters — every token votes on relevance, and the network learns what to attend to.</text>
</svg>
```ntion is the core operation of the Transformer: it lets every token in a sequence look at every other token and pull in the information most relevant to it. Each token emits three learned vectors — a query, a key, and a value — and attention scores how well one token's query matches every token's key, turns those scores into weights that sum to one, and returns a weighted average of the values. The result is a context-aware representation of each token, computed in a single parallel operation rather than step by step.\n\n**A dot product measures relevance, a softmax turns it into a mixture.** For a given query, the model takes its dot product with every key in the sequence; a large dot product means that token is relevant. Those raw scores are divided by the square root of the head dimension to keep them from growing too large as vectors get wider, then passed through a softmax so they become non-negative weights that add to one. Attention is therefore a soft, differentiable lookup: instead of retrieving one entry from a table, it retrieves a blend of all entries, weighted by learned similarity.\n\n**The output is a weighted sum of values, and that is what makes context flow.** Once the weights are known, the output for each position is the sum of every token's value vector scaled by its weight. A token that needs its subject two words back will place most of its weight there; a token that needs a distant clause will reach across the whole sequence. Because queries, keys, and values are all learned projections, the model discovers on its own which relationships matter — subject-verb agreement, coreference, syntax — without any hard-wired notion of position or distance beyond what the encoding supplies.\n\n| Component | Role | Shape intuition |\n|---|---|---|\n| Query (Q) | what this token is looking for | one vector per token |\n| Key (K) | what each token offers | one vector per token |\n| Value (V) | the content to mix in | one vector per token |\n| QKᵀ | relevance score per pair | sequence × sequence |\n| ÷ √d + softmax | scores → weights (sum to 1) | row-wise distribution |\n| Σ weight·V | context-mixed output | one vector per token |\n\n```svg\n<svg viewBox="0 0 1000 500" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif">\n  <rect x="0" y="0" width="1000" height="500" rx="14" fill="#1d1c1a"/>\n  <text x="36" y="38" fill="#e8e6f7" font-size="20" font-weight="600">Attention &#8212; each token builds a weighted mix of every token&#8217;s value</text>\n\n  <text x="250" y="70" fill="#c9c3f2" font-size="14" text-anchor="middle">Scaled dot-product attention, step by step</text>\n  <rect x="60" y="92" width="64" height="30" rx="6" fill="#2b2456" stroke="#7d70e0" stroke-width="1.4"/><text x="92.0" y="111.0" fill="#c9c3ff" font-size="13" text-anchor="middle">Q</text><rect x="140" y="92" width="64" height="30" rx="6" fill="#2b2456" stroke="#7d70e0" stroke-width="1.4"/><text x="172.0" y="111.0" fill="#c9c3ff" font-size="13" text-anchor="middle">K</text><rect x="220" y="92" width="64" height="30" rx="6" fill="#16352b" stroke="#3f9d6f" stroke-width="1.4"/><text x="252.0" y="111.0" fill="#bff0e4" font-size="13" text-anchor="middle">V</text><text x="316" y="111" fill="#8a8a86" font-size="12">learned projections of the input</text><line x1="92" y1="122" x2="120" y2="150" stroke="#8a8a86" stroke-width="1.2"/><line x1="172" y1="122" x2="140" y2="150" stroke="#8a8a86" stroke-width="1.2"/><rect x="96" y="152" width="110" height="30" rx="6" fill="#252334" stroke="#5b4fc4" stroke-width="1.4"/><text x="151.0" y="171.0" fill="#c9c3ff" font-size="13" text-anchor="middle">scores = QK&#7488;</text><text x="220" y="171" fill="#8a8a86" font-size="12">dot product every query&#215;key</text><line x1="151" y1="182" x2="151" y2="206" stroke="#8a8a86" stroke-width="1.2" marker-end="url(#amar)"/><rect x="96" y="208" width="110" height="28" rx="6" fill="#2a2418" stroke="#c98a2e" stroke-width="1.4"/><text x="151.0" y="226.0" fill="#f0d9b5" font-size="13" text-anchor="middle">&#247; &#8730;d</text><text x="220" y="226" fill="#8a8a86" font-size="12">stabilize the magnitudes</text><line x1="151" y1="236" x2="151" y2="258" stroke="#8a8a86" stroke-width="1.2" marker-end="url(#amar)"/><rect x="96" y="260" width="110" height="28" rx="6" fill="#3a2a18" stroke="#e0913a" stroke-width="1.4"/><text x="151.0" y="278.0" fill="#f0d9b5" font-size="13" text-anchor="middle">softmax</text><text x="220" y="278" fill="#8a8a86" font-size="12">weights, each &#8805;0, sum = 1</text><line x1="151" y1="288" x2="151" y2="310" stroke="#8a8a86" stroke-width="1.2" marker-end="url(#amar)"/><line x1="252" y1="122" x2="200" y2="312" stroke="#3f9d6f" stroke-width="1.2" stroke-dasharray="3 3"/><rect x="96" y="312" width="150" height="32" rx="6" fill="#16352b" stroke="#3f9d6f" stroke-width="1.4"/><text x="171.0" y="332.0" fill="#bff0e4" font-size="13" text-anchor="middle">output = &#931; weight&#183;V</text><text x="300" y="332" fill="#3f9d6f" font-size="12">context-mixed vector</text><defs><marker id="amar" markerWidth="9" markerHeight="9" refX="6" refY="4.5" orient="auto"><path d="M1,1 L7,4.5 L1,8" fill="none" stroke="#8a8a86" stroke-width="1.3"/></marker></defs>\n\n  <line x1="500" y1="58" x2="500" y2="392" stroke="#3a3a37" stroke-width="1.2" stroke-dasharray="4 4"/>\n\n  <text x="750" y="70" fill="#c9c3f2" font-size="14" text-anchor="middle">One query attends: weights sum to 1</text>\n  <rect x="690" y="94" width="120" height="32" rx="6" fill="#2b2456" stroke="#7d70e0" stroke-width="1.4"/><text x="750.0" y="114.0" fill="#c9c3ff" font-size="13" text-anchor="middle">query: &#8220;sat&#8221;</text><rect x="560" y="170" width="150" height="26" rx="5" fill="#232220" stroke="#4a4a47" stroke-width="1.2"/><text x="572" y="187" fill="#d4d4d0" font-size="13">The</text><text x="686" y="187" fill="#8a8a86" font-size="12">0.05</text><line x1="750" y1="126" x2="710" y2="183" stroke="#7d70e0" stroke-width="0.9" opacity="0.40"/><rect x="560" y="204" width="150" height="26" rx="5" fill="#16352b" stroke="#3f9d6f" stroke-width="1.2"/><text x="572" y="221" fill="#d4d4d0" font-size="13">cat</text><text x="686" y="221" fill="#bff0e4" font-size="12">0.42</text><line x1="750" y1="126" x2="710" y2="217" stroke="#7d70e0" stroke-width="3.5" opacity="0.77"/><rect x="560" y="238" width="150" height="26" rx="5" fill="#16352b" stroke="#3f9d6f" stroke-width="1.2"/><text x="572" y="255" fill="#d4d4d0" font-size="13">sat</text><text x="686" y="255" fill="#bff0e4" font-size="12">0.20</text><line x1="750" y1="126" x2="710" y2="251" stroke="#7d70e0" stroke-width="2.0" opacity="0.55"/><rect x="560" y="272" width="150" height="26" rx="5" fill="#232220" stroke="#4a4a47" stroke-width="1.2"/><text x="572" y="289" fill="#d4d4d0" font-size="13">on</text><text x="686" y="289" fill="#8a8a86" font-size="12">0.05</text><line x1="750" y1="126" x2="710" y2="285" stroke="#7d70e0" stroke-width="0.9" opacity="0.40"/><rect x="560" y="306" width="150" height="26" rx="5" fill="#232220" stroke="#4a4a47" stroke-width="1.2"/><text x="572" y="323" fill="#d4d4d0" font-size="13">the</text><text x="686" y="323" fill="#8a8a86" font-size="12">0.06</text><line x1="750" y1="126" x2="710" y2="319" stroke="#7d70e0" stroke-width="1.0" opacity="0.41"/><rect x="560" y="340" width="150" height="26" rx="5" fill="#16352b" stroke="#3f9d6f" stroke-width="1.2"/><text x="572" y="357" fill="#d4d4d0" font-size="13">mat</text><text x="686" y="357" fill="#bff0e4" font-size="12">0.22</text><line x1="750" y1="126" x2="710" y2="353" stroke="#7d70e0" stroke-width="2.1" opacity="0.57"/><text x="635" y="162" fill="#8a8a86" font-size="12" text-anchor="middle">keys / values (all tokens)</text><text x="810" y="312" fill="#3f9d6f" font-size="12" text-anchor="middle">thick arrow = high weight</text><text x="750" y="390" fill="#f0d9b5" font-size="13" text-anchor="middle">output &#8776; 0.42&#183;V(cat) + 0.22&#183;V(mat) + &#8230;</text><text x="750" y="410" fill="#8a8a86" font-size="12" text-anchor="middle">the weights are a probability distribution over context</text>\n\n  <text x="70" y="424" fill="#f0d9b5" font-size="12.5">A token&#8217;s query is compared to every token&#8217;s key by dot product, giving a raw score per pair; dividing by &#8730;d keeps the</text>\n  <text x="70" y="443" fill="#d4d4d0" font-size="12">scores in a stable range, and a softmax turns them into weights that sum to one &#8212; a soft, learned lookup over the sequence.</text>\n  <text x="70" y="468" fill="#8a8a86" font-size="13">The output is the weighted sum of the value vectors, so each position pulls in exactly the context it needs. Cost grows as the</text>\n  <text x="70" y="487" fill="#8a8a86" font-size="13">sequence squared (every query &#215; every key), which is why the KV cache, Flash Attention, and sparse variants all target it.</text>\n</svg>\n```\n\n**Its cost is quadratic in sequence length, and that shapes all of modern LLM hardware.** Because every query compares against every key, both the compute and the score matrix grow with the sequence length squared, and during generation the keys and values of all past tokens must be kept around — the KV cache. This single fact drives a huge amount of systems work: Flash Attention restructures the computation to avoid materializing the full score matrix, multi-query and grouped-query attention shrink the KV cache, and sparse or sliding-window variants skip most of the pairs outright. Self-attention (queries, keys, and values from the same sequence) builds internal context; cross-attention (queries from one sequence, keys and values from another) is how a decoder reads an encoder or a prompt.\n\nRead attention through a quant lens rather than a 'the model focuses' lens: the number that governs it is the sequence-length-squared count of query-key pairs, which sets both the FLOPs and the memory traffic every layer pays, and the KV cache turns that into a per-token memory cost during generation. Almost every attention variant is an attempt to move that number — reduce the pairs scored (sparse, sliding-window), reduce the bytes cached per token (MQA, GQA, MLA), or reduce the passes over memory (Flash Attention) — so the design question is always which part of the quadratic you can drop without losing the context the task actually needs.