<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">LLM Inference — Prefill, Decode, and the Memory Wall</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">two phases, two bottlenecks: compute-bound prefill, then memory-bound autoregressive decode</text>

  <!-- === TOP: Two-phase diagram === -->
  <text x="380" y="72" fill="#e6edf3" font-size="12" font-weight="600" text-anchor="middle">The Two Phases of LLM Inference</text>

  <!-- PREFILL phase (left) -->
  <rect x="30" y="82" width="330" height="100" rx="6" fill="#080d14" stroke="#60a5fa" stroke-width="1.3"/>
  <text x="195" y="100" fill="#93c5fd" font-size="11" text-anchor="middle" font-weight="600">Phase 1: Prefill (prompt processing)</text>

  <!-- Token batch going through in parallel -->
  <g>
    <rect x="45" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
    <rect x="72" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
    <rect x="99" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
    <rect x="126" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
    <rect x="153" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
    <rect x="180" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
    <rect x="207" y="110" width="24" height="16" rx="2" fill="#0f1a2a" stroke="#3b82f6" stroke-width="0.6"/>
  </g>
  <text x="145" y="140" fill="#93c5fd" font-size="8" text-anchor="middle">all prompt tokens processed in parallel</text>

  <!-- Big matmul indication -->
  <rect x="250" y="108" width="95" height="22" rx="3" fill="#122040" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="297" y="122" fill="#93c5fd" font-size="8.5" text-anchor="middle">big GEMM (parallel)</text>

  <text x="195" y="158" fill="#60a5fa" font-size="9" text-anchor="middle" font-weight="600">COMPUTE-BOUND</text>
  <text x="195" y="172" fill="#8b98a5" font-size="8" text-anchor="middle">high arithmetic intensity, GPU fully utilized</text>
  <text x="195" y="182" fill="#6b7684" font-size="7.5" text-anchor="middle">TTFT (time to first token) = prefill latency</text>

  <!-- DECODE phase (right) -->
  <rect x="380" y="82" width="350" height="100" rx="6" fill="#080d14" stroke="#f59e0b" stroke-width="1.3"/>
  <text x="555" y="100" fill="#fbbf24" font-size="11" text-anchor="middle" font-weight="600">Phase 2: Decode (token generation)</text>

  <!-- Single token at a time -->
  <rect x="400" y="112" width="24" height="16" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <path d="M428,120 L445,120" fill="none" stroke="#a1701a" stroke-width="0.8"/>
  <rect x="448" y="108" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="475" y="122" fill="#fbbf24" font-size="8" text-anchor="middle">1 token</text>
  <path d="M506,120 L523,120" fill="none" stroke="#a1701a" stroke-width="0.8"/>
  <rect x="526" y="108" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="553" y="122" fill="#fbbf24" font-size="8" text-anchor="middle">1 token</text>
  <path d="M584,120 L601,120" fill="none" stroke="#a1701a" stroke-width="0.8"/>
  <rect x="604" y="108" width="55" height="22" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="631" y="122" fill="#fbbf24" font-size="8" text-anchor="middle">1 token</text>
  <text x="680" y="122" fill="#6b7684" font-size="9">...</text>

  <text x="555" y="148" fill="#8b98a5" font-size="8" text-anchor="middle">one token per forward pass, sequential</text>
  <text x="555" y="162" fill="#f59e0b" font-size="9" text-anchor="middle" font-weight="600">MEMORY-BOUND</text>
  <text x="555" y="174" fill="#8b98a5" font-size="8" text-anchor="middle">low arithmetic intensity — load all weights for 1 token</text>
  <text x="555" y="184" fill="#6b7684" font-size="7.5" text-anchor="middle">TPS (tokens per second) = decode throughput</text>

  <!-- === MIDDLE: KV Cache (the critical data structure) === -->
  <rect x="30" y="195" width="440" height="110" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="250" y="213" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">KV Cache — Why Memory Grows with Context</text>

  <!-- KV cache growing -->
  <rect x="50" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="82" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="114" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="146" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="178" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="210" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="242" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.6"/>
  <rect x="274" y="225" width="30" height="18" rx="2" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <rect x="306" y="225" width="30" height="18" rx="2" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="321" y="237" fill="#fbbf24" font-size="7" text-anchor="middle">new</text>
  <!-- Arrow showing growth -->
  <path d="M340,234 L365,234" fill="none" stroke="#f87171" stroke-width="1"/>
  <polygon points="363,231 369,234 363,237" fill="#f87171"/>
  <text x="400" y="232" fill="#f87171" font-size="8">grows each step</text>
  <text x="400" y="244" fill="#8b98a5" font-size="8">past K,V cached</text>

  <text x="250" y="265" fill="#8b98a5" font-size="9" text-anchor="middle">Each layer stores K and V for all past tokens → memory = 2 × layers × seq_len × d_model × dtype</text>
  <text x="250" y="280" fill="#6b7684" font-size="8.5" text-anchor="middle">GPT-4 (128K context): ~30 GB KV cache per request in FP16 — why batching is hard at long context</text>
  <text x="250" y="295" fill="#34d399" font-size="8.5" text-anchor="middle">PagedAttention (vLLM): page KV cache like virtual memory → no fragmentation, higher batch size</text>

  <!-- === RIGHT: Optimization stack === -->
  <rect x="485" y="195" width="245" height="110" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="607" y="213" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Optimization Techniques</text>

  <text x="500" y="232" fill="#c4b5fd" font-size="9">Quantization: FP16→INT8→FP4 (2-4x)</text>
  <text x="500" y="248" fill="#60a5fa" font-size="9">Speculative decode: draft+verify (2x)</text>
  <text x="500" y="264" fill="#34d399" font-size="9">Continuous batching: pack requests</text>
  <text x="500" y="280" fill="#fbbf24" font-size="9">Prefix caching: reuse system prompt KV</text>
  <text x="500" y="296" fill="#8b98a5" font-size="9">FlashAttention: fused SRAM tiling</text>

  <!-- === BOTTOM: Serving systems === -->
  <rect x="30" y="315" width="700" height="60" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="333" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Serving Frameworks</text>
  <text x="100" y="353" fill="#c4b5fd" font-size="9.5" text-anchor="middle" font-weight="600">vLLM</text>
  <text x="100" y="367" fill="#6b7684" font-size="8" text-anchor="middle">PagedAttn, best throughput</text>
  <text x="250" y="353" fill="#fbbf24" font-size="9.5" text-anchor="middle" font-weight="600">TensorRT-LLM</text>
  <text x="250" y="367" fill="#6b7684" font-size="8" text-anchor="middle">NVIDIA, FP8, inflight batch</text>
  <text x="400" y="353" fill="#34d399" font-size="9.5" text-anchor="middle" font-weight="600">SGLang</text>
  <text x="400" y="367" fill="#6b7684" font-size="8" text-anchor="middle">RadixAttention, structured</text>
  <text x="550" y="353" fill="#60a5fa" font-size="9.5" text-anchor="middle" font-weight="600">TGI</text>
  <text x="550" y="367" fill="#6b7684" font-size="8" text-anchor="middle">HuggingFace, streaming</text>
  <text x="680" y="353" fill="#8b98a5" font-size="9.5" text-anchor="middle" font-weight="600">Triton</text>
  <text x="680" y="367" fill="#6b7684" font-size="8" text-anchor="middle">multi-model ensemble</text>

  <!-- Key metrics -->
  <rect x="30" y="385" width="700" height="35" rx="4" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="402" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">Key Metrics: TTFT (ms) · TPS (tokens/s/user) · Throughput (tokens/s/GPU) · P99 latency · cost per 1M tokens</text>
  <text x="380" y="416" fill="#8b98a5" font-size="9" text-anchor="middle">GPT-4o: ~200ms TTFT, ~80 TPS | H100 serving Llama-70B: ~4000 tokens/s throughput with vLLM</text>

  <!-- Hardware callout -->
  <text x="380" y="440" fill="#fbbf24" font-size="9.5" text-anchor="middle">Hardware sweet spot: H100/H200 (HBM3e bandwidth) · Groq (SRAM, no HBM) · Cerebras (wafer-scale)</text>

  <text x="380" y="462" fill="#6b7684" font-size="11" text-anchor="middle">Inference is memory-bandwidth limited: the GPU loads 100B+ parameters to produce one token. Bandwidth is king.</text>
</svg>
```inference is the serving side of a large language model: taking a trained model and running its forward pass to answer real user requests, optimized for latency, throughput, and cost. Training happens once; inference happens on every request, forever, which is why it dominates the operational cost of any deployed model and why so much engineering goes into making it faster and cheaper.\n\n```svg\n<svg viewBox="0 0 760 500" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">\n  <rect x="0" y="0" width="760" height="500" fill="#0d1117"/>\n  <text x="380" y="30" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">The Two Phases of LLM Inference</text>\n  <text x="380" y="52" fill="#8b98a5" font-size="12.5" text-anchor="middle">prefill is compute-bound and parallel; decode is memory-bound and sequential — they need opposite optimizations</text>\n\n  <!-- PREFILL panel -->\n  <rect x="34" y="72" width="330" height="372" rx="10" fill="#0e1a15" stroke="#34d399" stroke-width="1.6"/>\n  <text x="50" y="96" fill="#6ee7b7" font-size="13.5" font-weight="700">PREFILL — read the whole prompt at once</text>\n  <rect x="60" y="120" width="58" height="26" rx="4" fill="#134e3a" stroke="#34d399" stroke-width="1"/>\n  <text x="89" y="138" fill="#6ee7b7" font-size="12" text-anchor="middle">The</text>\n  <rect x="126" y="120" width="58" height="26" rx="4" fill="#134e3a" stroke="#34d399" stroke-width="1"/>\n  <text x="155" y="138" fill="#6ee7b7" font-size="12" text-anchor="middle">cat</text>\n  <rect x="192" y="120" width="58" height="26" rx="4" fill="#134e3a" stroke="#34d399" stroke-width="1"/>\n  <text x="221" y="138" fill="#6ee7b7" font-size="12" text-anchor="middle">sat</text>\n  <rect x="258" y="120" width="58" height="26" rx="4" fill="#134e3a" stroke="#34d399" stroke-width="1"/>\n  <text x="287" y="138" fill="#6ee7b7" font-size="12" text-anchor="middle">on</text>\n  <path d="M199 148 L199 168 M193 162 L199 169 L205 162" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="80" y="170" width="238" height="42" rx="6" fill="#0f241b" stroke="#34d399" stroke-width="1.2"/>\n  <text x="199" y="196" fill="#a7f3d0" font-size="12.5" text-anchor="middle">Transformer — all tokens in parallel</text>\n  <path d="M199 212 L199 230 M193 224 L199 231 L205 224" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="80" y="232" width="238" height="34" rx="5" fill="#1c1633" stroke="#a78bfa" stroke-width="1.2"/>\n  <text x="199" y="254" fill="#c4b5fd" font-size="12" text-anchor="middle">KV cache built for every token</text>\n  <text x="199" y="300" fill="#8b98a5" font-size="13" text-anchor="middle">compute-bound: big matmuls</text>\n  <text x="199" y="320" fill="#8b98a5" font-size="13" text-anchor="middle">latency &#8733; prompt length</text>\n  <rect x="120" y="336" width="158" height="30" rx="15" fill="#0d2b22" stroke="#34d399" stroke-width="1.3"/>\n  <text x="199" y="356" fill="#6ee7b7" font-size="12" font-weight="700" text-anchor="middle">sets TTFT</text>\n  <text x="199" y="404" fill="#6b7684" font-size="12" text-anchor="middle">one heavy pass, then the prompt is</text>\n  <text x="199" y="420" fill="#6b7684" font-size="12" text-anchor="middle">done — you pay it once per request</text>\n\n  <!-- hand-off arrow -->\n  <path d="M318 249 C350 249 360 190 396 190" stroke="#a78bfa" stroke-width="2" fill="none" stroke-dasharray="5 4"/>\n  <text x="382" y="180" fill="#a78bfa" font-size="12" text-anchor="middle">KV handed off</text>\n\n  <!-- DECODE panel -->\n  <rect x="396" y="72" width="330" height="372" rx="10" fill="#101a28" stroke="#60a5fa" stroke-width="1.6"/>\n  <text x="412" y="96" fill="#93c5fd" font-size="13.5" font-weight="700">DECODE — one token at a time</text>\n  <rect x="470" y="120" width="182" height="42" rx="6" fill="#0f2036" stroke="#60a5fa" stroke-width="1.2"/>\n  <text x="561" y="146" fill="#bfdbfe" font-size="12.5" text-anchor="middle">Transformer forward pass</text>\n  <path d="M561 162 L561 180 M555 174 L561 181 L567 174" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="500" y="182" width="122" height="32" rx="5" fill="#3b2f0a" stroke="#fbbf24" stroke-width="1.1"/>\n  <text x="561" y="203" fill="#fcd34d" font-size="12" text-anchor="middle">next token</text>\n  <path d="M561 214 L561 230 M555 224 L561 231 L567 224" stroke="#5b6b7f" stroke-width="1.3" fill="none"/>\n  <rect x="470" y="232" width="182" height="32" rx="5" fill="#1c1633" stroke="#a78bfa" stroke-width="1.1"/>\n  <text x="561" y="253" fill="#c4b5fd" font-size="13" text-anchor="middle">append K,V &#8594; grow cache</text>\n  <!-- loop arrow back up -->\n  <path d="M652 248 C692 248 692 141 654 141" stroke="#60a5fa" stroke-width="1.8" fill="none"/>\n  <path d="M660 148 L652 140 L662 134" stroke="#60a5fa" stroke-width="1.8" fill="none"/>\n  <text x="702" y="196" fill="#60a5fa" font-size="12" text-anchor="middle" transform="rotate(90 702 196)">repeat per token</text>\n  <text x="561" y="300" fill="#8b98a5" font-size="13" text-anchor="middle">memory-bound: reads full KV each step</text>\n  <text x="561" y="320" fill="#8b98a5" font-size="13" text-anchor="middle">latency &#8733; output length</text>\n  <rect x="482" y="336" width="158" height="30" rx="15" fill="#0d1f36" stroke="#60a5fa" stroke-width="1.3"/>\n  <text x="561" y="356" fill="#93c5fd" font-size="12" font-weight="700" text-anchor="middle">each token = TPOT</text>\n  <text x="561" y="404" fill="#6b7684" font-size="12" text-anchor="middle">the GPU is starved unless many</text>\n  <text x="561" y="420" fill="#6b7684" font-size="12" text-anchor="middle">requests are batched together</text>\n\n  <text x="380" y="474" fill="#6b7684" font-size="13" text-anchor="middle">Continuous batching packs many requests' decode steps into each pass — how a memory-bound phase is made to fill a compute machine.</text>\n</svg>\n```\n\n**Inference has two phases with opposite characters.** A request is served in a *prefill* phase that reads the entire prompt in one parallel pass, and a *decode* phase that then generates the answer one token at a time. Prefill is compute-bound — it is a batch of large matrix multiplies over all prompt tokens — and it sets the time to first token (TTFT). Decode is memory-bound — every new token requires a full forward pass that reads the growing KV cache — and its per-token cost sets the time per output token (TPOT). Almost every inference optimization is aimed at one of these two phases.\n\n**Autoregression is the reason decode is slow.** The model cannot produce token five until it has produced token four, so decode is inherently sequential. Each step does relatively little arithmetic but must stream the full model weights and the entire KV cache through the GPU, so the bottleneck is memory bandwidth, not compute. This is why a single request leaves most of a GPU's arithmetic units idle, and why serving is fundamentally a batching problem.\n\n**Batching is what makes serving economical.** Because decode underuses the GPU, the way to get throughput is to run many requests together so their token-generation steps share each weight read. *Continuous batching* (also called in-flight batching) adds and removes requests from the running batch every step instead of waiting for a whole batch to finish, keeping the GPU full even as sequences start and end at different times. This single technique is the largest throughput lever in modern serving.\n\n**The KV cache is the central resource.** Every token's keys and values are cached so they are not recomputed, but that cache grows with sequence length and batch size and quickly becomes the memory bottleneck. *PagedAttention* manages it in fixed pages like virtual memory to eliminate fragmentation, quantized KV cache shrinks it to INT8 or INT4, and grouped-query attention (GQA/MQA) reduces how many KV heads must be stored at all. How much cache you can hold directly sets how many requests you can batch.\n\n**The rest of the toolkit attacks one phase or the other.** Quantizing weights to INT8, FP8, or INT4 cuts the memory traffic that throttles decode; FlashAttention fuses the attention kernel to avoid round-trips to slow memory; tensor parallelism splits a model too big for one GPU; speculative decoding lets a small draft model propose several tokens that the big model verifies in one pass, breaking the one-token-per-step limit. Frameworks like vLLM, TensorRT-LLM, TGI, and SGLang package these together behind an API.\n\n| Phase | Bottleneck | Latency scales with | Key metric | Main levers |\n|---|---|---|---|---|\n| Prefill | GPU compute (matmul) | prompt length | TTFT | FlashAttention, tensor parallelism, prefix caching |\n| Decode | memory bandwidth (KV + weight reads) | output length | TPOT | continuous batching, quantization, speculative decoding |\n\n| Metric | Meaning | Who cares |\n|---|---|---|\n| TTFT | time to first token | interactive feel, chat responsiveness |\n| TPOT | time per output token | streaming speed, long-answer latency |\n| Throughput | tokens/sec across all requests | cost per token, GPU utilization |\n\nRead inference through a *prefill-versus-decode* lens rather than a *single-number-latency* lens: the two phases are bound by different resources, so they respond to different optimizations, and the whole discipline of serving is deciding how to trade one against the other. Prefill wants compute and parallelism; decode wants memory bandwidth and big batches — and every framework, from vLLM to TensorRT-LLM, is ultimately a set of choices about how to keep both phases fed while packing enough concurrent requests onto the GPU to make the economics work.\n