<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">KV Cache — Why LLM Inference Eats Memory</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">store past Keys and Values so you don't recompute attention over the full history every token</text>

  <!-- === TOP: Without vs With KV cache === -->
  <!-- Without cache (naive) -->
  <rect x="30" y="65" width="330" height="100" rx="6" fill="#080d14" stroke="#f87171" stroke-width="1.2"/>
  <text x="195" y="83" fill="#fca5a5" font-size="10" text-anchor="middle" font-weight="600">Without KV Cache (naive): recompute all</text>

  <!-- Token sequence processed each time -->
  <g fill="#2a1a1a" stroke="#f87171" stroke-width="0.6">
    <rect x="42" y="92" width="20" height="16" rx="2"/><rect x="65" y="92" width="20" height="16" rx="2"/>
    <rect x="88" y="92" width="20" height="16" rx="2"/><rect x="111" y="92" width="20" height="16" rx="2"/>
    <rect x="134" y="92" width="20" height="16" rx="2"/><rect x="157" y="92" width="20" height="16" rx="2"/>
    <rect x="180" y="92" width="20" height="16" rx="2"/><rect x="203" y="92" width="20" height="16" rx="2"/>
    <rect x="226" y="92" width="20" height="16" rx="2"/>
  </g>
  <rect x="249" y="92" width="20" height="16" rx="2" fill="#2a1a0a" stroke="#fbbf24" stroke-width="1"/>
  <text x="259" y="103" fill="#fbbf24" font-size="7" text-anchor="middle">new</text>

  <text x="195" y="125" fill="#f87171" font-size="8.5" text-anchor="middle">step N: recompute K,V for ALL N tokens (wasteful!)</text>
  <text x="195" y="140" fill="#8b98a5" font-size="8" text-anchor="middle">compute: O(N) per step → O(N&#xB2;) total for N tokens</text>
  <text x="195" y="155" fill="#6b7684" font-size="7.5" text-anchor="middle">all that work thrown away, then redone next step</text>

  <!-- With cache (efficient) -->
  <rect x="380" y="65" width="350" height="100" rx="6" fill="#080d14" stroke="#34d399" stroke-width="1.2"/>
  <text x="555" y="83" fill="#6ee7b7" font-size="10" text-anchor="middle" font-weight="600">With KV Cache: reuse past K,V</text>

  <!-- Cached tokens (green, from memory) -->
  <g fill="#0f2a1f" stroke="#34d399" stroke-width="0.6">
    <rect x="392" y="92" width="20" height="16" rx="2"/><rect x="415" y="92" width="20" height="16" rx="2"/>
    <rect x="438" y="92" width="20" height="16" rx="2"/><rect x="461" y="92" width="20" height="16" rx="2"/>
    <rect x="484" y="92" width="20" height="16" rx="2"/><rect x="507" y="92" width="20" height="16" rx="2"/>
    <rect x="530" y="92" width="20" height="16" rx="2"/><rect x="553" y="92" width="20" height="16" rx="2"/>
    <rect x="576" y="92" width="20" height="16" rx="2"/>
  </g>
  <text x="484" y="80" fill="#34d399" font-size="7" text-anchor="middle">cached in HBM</text>
  <!-- New token (only this one gets computed) -->
  <rect x="599" y="92" width="20" height="16" rx="2" fill="#2a1a0a" stroke="#fbbf24" stroke-width="1"/>
  <text x="609" y="103" fill="#fbbf24" font-size="7" text-anchor="middle">new</text>

  <!-- Only new token arrow -->
  <path d="M609,108 L609,118" fill="none" stroke="#fbbf24" stroke-width="1"/>
  <text x="660" y="115" fill="#fbbf24" font-size="8">only compute</text>
  <text x="660" y="127" fill="#fbbf24" font-size="8">K,V for 1 token</text>

  <text x="555" y="142" fill="#34d399" font-size="8.5" text-anchor="middle">step N: compute K,V for new token only, append to cache</text>
  <text x="555" y="155" fill="#8b98a5" font-size="8" text-anchor="middle">compute: O(1) per step → O(N) total</text>

  <!-- === MIDDLE: Memory anatomy === -->
  <rect x="30" y="175" width="430" height="130" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="245" y="193" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">KV Cache Memory Layout (one layer)</text>

  <!-- K cache (growing bar) -->
  <text x="50" y="215" fill="#34d399" font-size="9">K cache:</text>
  <g fill="#14261f" stroke="#2f6d55" stroke-width="0.5">
    <rect x="110" y="205" width="22" height="16" rx="1"/><rect x="134" y="205" width="22" height="16" rx="1"/>
    <rect x="158" y="205" width="22" height="16" rx="1"/><rect x="182" y="205" width="22" height="16" rx="1"/>
    <rect x="206" y="205" width="22" height="16" rx="1"/><rect x="230" y="205" width="22" height="16" rx="1"/>
    <rect x="254" y="205" width="22" height="16" rx="1"/><rect x="278" y="205" width="22" height="16" rx="1"/>
  </g>
  <rect x="302" y="205" width="22" height="16" rx="1" fill="#2a1a0a" stroke="#fbbf24" stroke-width="0.8"/>
  <path d="M328,213 L345,213" fill="none" stroke="#f87171" stroke-width="1"/>
  <polygon points="343,210 349,213 343,216" fill="#f87171"/>
  <text x="365" y="216" fill="#f87171" font-size="8">grows →</text>

  <!-- V cache -->
  <text x="50" y="242" fill="#f59e0b" font-size="9">V cache:</text>
  <g fill="#2a1a0a" stroke="#8a5a2a" stroke-width="0.5">
    <rect x="110" y="232" width="22" height="16" rx="1"/><rect x="134" y="232" width="22" height="16" rx="1"/>
    <rect x="158" y="232" width="22" height="16" rx="1"/><rect x="182" y="232" width="22" height="16" rx="1"/>
    <rect x="206" y="232" width="22" height="16" rx="1"/><rect x="230" y="232" width="22" height="16" rx="1"/>
    <rect x="254" y="232" width="22" height="16" rx="1"/><rect x="278" y="232" width="22" height="16" rx="1"/>
  </g>
  <rect x="302" y="232" width="22" height="16" rx="1" fill="#2a1a0a" stroke="#fbbf24" stroke-width="0.8"/>

  <!-- Size formula -->
  <text x="245" y="270" fill="#e6edf3" font-size="9" text-anchor="middle" font-weight="600">Size = 2 × layers × seq_len × n_heads × d_head × dtype_bytes</text>
  <text x="245" y="286" fill="#8b98a5" font-size="8.5" text-anchor="middle">Llama-3 70B, 128K ctx, FP16: 2 × 80 × 128K × 64 × 128 × 2B = ~40 GB per request!</text>
  <text x="245" y="300" fill="#6b7684" font-size="8" text-anchor="middle">This is why batch size is limited at long context — KV cache fills all HBM</text>

  <!-- === RIGHT: Solutions === -->
  <rect x="475" y="175" width="255" height="130" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="602" y="193" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">KV Cache Optimizations</text>

  <text x="490" y="213" fill="#c4b5fd" font-size="9" font-weight="600">PagedAttention (vLLM):</text>
  <text x="490" y="227" fill="#8b98a5" font-size="8.5">allocate KV in pages (like OS VM)</text>
  <text x="490" y="239" fill="#6b7684" font-size="8">no fragmentation → 2-4x more batches</text>

  <text x="490" y="257" fill="#34d399" font-size="9" font-weight="600">GQA / MQA:</text>
  <text x="490" y="271" fill="#8b98a5" font-size="8.5">share K,V across head groups</text>
  <text x="490" y="283" fill="#6b7684" font-size="8">Llama-3: 8 KV heads (not 64) → 8x smaller cache</text>

  <text x="490" y="299" fill="#fbbf24" font-size="9" font-weight="600">Quantized KV (FP8/INT4):</text>
  <text x="490" y="311" fill="#8b98a5" font-size="8.5">compress cached K,V to lower precision</text>

  <!-- === BOTTOM: Prefill vs decode relationship === -->
  <rect x="30" y="318" width="700" height="95" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="336" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">KV Cache in the Inference Lifecycle</text>

  <text x="50" y="358" fill="#60a5fa" font-size="9" font-weight="600">Prefill (prompt):</text>
  <text x="180" y="358" fill="#8b98a5" font-size="9">compute K,V for all prompt tokens → fill the cache</text>

  <text x="50" y="376" fill="#fbbf24" font-size="9" font-weight="600">Decode (generation):</text>
  <text x="195" y="376" fill="#8b98a5" font-size="9">append 1 K,V per step, attend to full cache</text>

  <text x="50" y="394" fill="#34d399" font-size="9" font-weight="600">Prefix caching:</text>
  <text x="165" y="394" fill="#8b98a5" font-size="9">reuse system prompt KV across requests (save prefill cost)</text>

  <text x="50" y="412" fill="#c4b5fd" font-size="9" font-weight="600">Speculative decode:</text>
  <text x="185" y="412" fill="#8b98a5" font-size="9">draft model fills cache speculatively, verify in one pass</text>

  <!-- Footer -->
  <text x="380" y="435" fill="#fbbf24" font-size="9.5" text-anchor="middle">The KV cache is the #1 memory bottleneck in LLM serving — it's why H200 (141GB HBM3e) matters more than raw FLOPS</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">KV cache trades memory for compute: store past work so each new token only costs one forward pass, not N.</text>
</svg>
```KV cache stores the attention keys and values computed for every token a language model has already processed, so that generating each new token does not have to recompute them. It is the single largest, fastest-growing consumer of accelerator memory during LLM inference, and it is why decoding is memory-bound rather than compute-bound.\n\n**It turns quadratic recomputation into linear reuse.** In self-attention, each new token's query attends to the keys and values of all previous tokens. Without a cache, every decode step would recompute K and V for the entire sequence — O(n²) work to emit n tokens. By caching each token's K and V once and appending the new token's, the model does only O(n) new work per step and reuses everything prior. The cost of that shortcut is memory: the cache must be held in fast memory (HBM) for the whole generation.\n\n**The cache grows linearly and can dwarf the weights.** Its size is roughly 2 × layers × kv_heads × head_dim × sequence_length × batch × bytes_per_element — the factor of 2 for K and V. Because it scales with both context length and batch size, long-context, high-throughput serving can make the KV cache larger than the model weights themselves, and it slams into the HBM capacity ceiling. Reading it back every step is pure memory traffic, which is exactly why token generation sits low on the roofline, limited by bandwidth.\n\n| Lever | What it does | Effect on cache |\n|---|---|---|\n| MHA (baseline) | full keys/values per head | largest cache |\n| GQA | share KV across head groups | few× smaller |\n| MQA | one KV head for all queries | smallest, some quality cost |\n| KV quantization | store K/V in INT8/FP8 | ~2-4× smaller |\n| PagedAttention | page the cache like virtual memory | less waste, higher batch |\n\n```svg\n<svg viewBox="0 0 1000 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif">\n  <rect x="0" y="0" width="1000" height="470" rx="14" fill="#1d1c1a"/>\n  <text x="36" y="38" fill="#e8e6f7" font-size="20" font-weight="600">KV cache — store each token's keys &amp; values once, reuse them every step</text>\n\n  <!-- ===== decode reuse ===== -->\n  <text x="230" y="76" fill="#c9c3f2" font-size="14" text-anchor="middle">Autoregressive decode: attend to all past tokens</text>\n  <text x="230" y="120" fill="#8a8a86" font-size="12" text-anchor="middle">cached keys/values from earlier tokens</text>\n  <rect x="70" y="150" width="52" height="34" rx="5" fill="#2a2a28" stroke="#5b4fc4" stroke-width="1.6"/><text x="96" y="172" fill="#c9c3f2" font-size="12" text-anchor="middle">t1</text><rect x="74" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="84" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">K</text><rect x="98" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="108" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">V</text><line x1="96" y1="196" x2="392" y2="168" stroke="#3f9d6f" stroke-width="1" opacity="0.55"/><rect x="140" y="150" width="52" height="34" rx="5" fill="#2a2a28" stroke="#5b4fc4" stroke-width="1.6"/><text x="166" y="172" fill="#c9c3f2" font-size="12" text-anchor="middle">t2</text><rect x="144" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="154" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">K</text><rect x="168" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="178" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">V</text><line x1="166" y1="196" x2="392" y2="168" stroke="#3f9d6f" stroke-width="1" opacity="0.55"/><rect x="210" y="150" width="52" height="34" rx="5" fill="#2a2a28" stroke="#5b4fc4" stroke-width="1.6"/><text x="236" y="172" fill="#c9c3f2" font-size="12" text-anchor="middle">t3</text><rect x="214" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="224" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">K</text><rect x="238" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="248" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">V</text><line x1="236" y1="196" x2="392" y2="168" stroke="#3f9d6f" stroke-width="1" opacity="0.55"/><rect x="280" y="150" width="52" height="34" rx="5" fill="#2a2a28" stroke="#5b4fc4" stroke-width="1.6"/><text x="306" y="172" fill="#c9c3f2" font-size="12" text-anchor="middle">t4</text><rect x="284" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="294" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">K</text><rect x="308" y="196" width="20" height="16" rx="2" fill="#0f6b5a" stroke="#1f9e85"/><text x="318" y="208" fill="#bff0e4" font-size="12" text-anchor="middle">V</text><line x1="306" y1="196" x2="392" y2="168" stroke="#3f9d6f" stroke-width="1" opacity="0.55"/>\n  <!-- current token -->\n  <rect x="366" y="150" width="52" height="34" rx="5" fill="#3a2a18" stroke="#e0913a" stroke-width="2"/>\n  <text x="392" y="172" fill="#f0d9b5" font-size="12" text-anchor="middle">t5</text>\n  <text x="392" y="204" fill="#e0913a" font-size="13" text-anchor="middle">new q</text>\n  <text x="392" y="140" fill="#e0913a" font-size="13" text-anchor="middle">now</text>\n\n  <text x="70" y="256" fill="#f0d9b5" font-size="12.5">Each step: compute only the new token's K,V and append.</text>\n  <text x="70" y="276" fill="#d4d4d0" font-size="12">With cache: O(n) work per step. Without: recompute all → O(n²).</text>\n  <text x="70" y="300" fill="#8a8a86" font-size="13">The cache is pure memory traffic — decode is memory-bound, not compute-bound.</text>\n\n  <!-- divider -->\n  <line x1="500" y1="70" x2="500" y2="345" stroke="#3a3a37" stroke-width="1.2" stroke-dasharray="4 4"/>\n\n  <!-- ===== growth chart ===== -->\n  <text x="735" y="76" fill="#c9c3f2" font-size="14" text-anchor="middle">Cache size grows linearly with context (illustrative)</text>\n  <text x="552" y="272" fill="#8a8a86" font-size="12" text-anchor="end">20</text><line x1="560" y1="268" x2="910" y2="268" stroke="#2a2a28" stroke-width="0.7"/><text x="552" y="219" fill="#8a8a86" font-size="12" text-anchor="end">40</text><line x1="560" y1="215" x2="910" y2="215" stroke="#2a2a28" stroke-width="0.7"/><text x="552" y="166" fill="#8a8a86" font-size="12" text-anchor="end">60</text><line x1="560" y1="162" x2="910" y2="162" stroke="#2a2a28" stroke-width="0.7"/><text x="552" y="114" fill="#8a8a86" font-size="12" text-anchor="end">80</text><line x1="560" y1="110" x2="910" y2="110" stroke="#2a2a28" stroke-width="0.7"/><text x="560" y="336" fill="#8a8a86" font-size="12" text-anchor="middle">0k</text><text x="648" y="336" fill="#8a8a86" font-size="12" text-anchor="middle">8k</text><text x="735" y="336" fill="#8a8a86" font-size="12" text-anchor="middle">16k</text><text x="822" y="336" fill="#8a8a86" font-size="12" text-anchor="middle">24k</text><text x="910" y="336" fill="#8a8a86" font-size="12" text-anchor="middle">32k</text>\n  <text x="530" y="110" fill="#8a8a86" font-size="12" text-anchor="end">GB</text>\n  <line x1="560" y1="320" x2="910" y2="320" stroke="#6f6f6a" stroke-width="1.3"/>\n  <line x1="560" y1="320" x2="560" y2="102" stroke="#6f6f6a" stroke-width="1.3"/>\n  <!-- HBM ceiling -->\n  <line x1="560" y1="110" x2="910" y2="110" stroke="#d33f3f" stroke-width="1.6" stroke-dasharray="5 4"/>\n  <text x="906" y="104" fill="#d33f3f" font-size="13" text-anchor="end">HBM capacity ceiling</text>\n  <polyline points="560.0,320.0 581.9,306.9 603.8,293.8 625.6,280.6 647.5,267.5 669.4,254.4 691.2,241.2 713.1,228.1 735.0,215.0 756.9,201.9 778.8,188.8 800.6,175.6 822.5,162.5 844.4,149.4 866.2,136.2 888.1,123.1 910.0,110.0" fill="none" stroke="#d33f3f" stroke-width="2.4"/>\n  <polyline points="560.0,320.0 581.9,316.7 603.8,313.4 625.6,310.1 647.5,306.8 669.4,303.5 691.2,300.2 713.1,296.8 735.0,293.5 756.9,290.2 778.8,286.9 800.6,283.6 822.5,280.3 844.4,277.0 866.2,273.7 888.1,270.4 910.0,267.1" fill="none" stroke="#e0913a" stroke-width="2.4"/>\n  <polyline points="560.0,320.0 581.9,318.4 603.8,316.7 625.6,315.1 647.5,313.5 669.4,311.9 691.2,310.2 713.1,308.6 735.0,307.0 756.9,305.4 778.8,303.7 800.6,302.1 822.5,300.5 844.4,298.8 866.2,297.2 888.1,295.6 910.0,294.0" fill="none" stroke="#1f9e85" stroke-width="2.4"/>\n  <text x="790" y="162" fill="#d33f3f" font-size="13">MHA (all heads)</text>\n  <text x="888" y="265" fill="#e0913a" font-size="13" text-anchor="end">GQA</text>\n  <text x="888" y="296" fill="#1f9e85" font-size="13" text-anchor="end">MQA</text>\n  <text x="560" y="360" fill="#f0d9b5" font-size="13">bytes = 2 · layers · kv_heads · head_dim · seq · batch · dtype</text>\n  <text x="560" y="378" fill="#8a8a86" font-size="13">GQA/MQA cut kv_heads → smaller cache, more context in the same HBM.</text>\n</svg>\n```\n\n**Managing the cache is the core of inference optimization.** Because capacity and bandwidth are the binding constraints, the whole toolbox targets the KV cache: grouped- and multi-query attention (GQA/MQA) cut the number of KV heads; KV quantization shrinks each entry; PagedAttention (vLLM) pages the cache in fixed blocks to eliminate fragmentation and pack more concurrent requests; and prefix caching reuses shared prompts. Each trades a little quality or complexity for more context or higher batch in the same HBM.\n\nRead the KV cache through a quant lens rather than a feature lens: it is bytes-in-HBM and bytes-moved-per-token, and per the roofline those two numbers cap decode throughput before FLOPs ever do. The right question is how many gigabytes the cache costs at a target context and batch, and how much bandwidth each generated token demands — a measured memory budget that decides how long a context and how many users a given accelerator can serve.