what is llm inference
**LLM inference is running an already-trained model forward to answer a live request, and it behaves nothing like training on the hardware underneath it.** Training adjusts a model's weights over millions of examples, is dominated by large batched matrix multiplies, and cares mostly about raw compute throughput. Inference happens one user request at a time (or a modest batch of them), has to respond with low latency, and — critically — spends a large share of its time not doing much math at all, but waiting on memory. Building AI silicon that trains well and silicon that serves well are genuinely different engineering problems.
**Almost everything about LLM inference performance comes down to one split: prefill versus decode.** When a prompt first arrives, the model processes every token of that prompt at once in a single parallel pass — this is prefill, and because it works on many tokens simultaneously, it's compute-heavy and keeps the accelerator's math units busy. Then generation begins, and the model produces exactly one new token at a time, each one depending on everything generated before it — this is decode, and it fundamentally cannot be parallelized across tokens the way prefill can. Decode has to reload and process attention over the whole growing context on every single step, so it moves far less math per unit of data moved. That imbalance is why decode, not prefill, is usually the phase engineers are fighting to speed up.
```svg
```
**The KV cache is what makes decode possible at all, and it's also what makes it expensive.** Every generated token has to "pay attention" to every token that came before it, comparing against stored key and value vectors for each one. Recomputing those vectors from scratch on every single new token would be wasteful, so they're stored — cached — the moment they're first computed, during prefill and each decode step. The catch: that cache grows with every token generated, and for long conversations or long documents it can consume more memory than the model's own weights. This is precisely why high-bandwidth memory matters so much for inference hardware — the same HBM stacking covered in the packaging discussion earlier is what lets an accelerator hold and rapidly re-read a large, constantly growing KV cache without decode grinding to a halt.
| Phase | Bottleneck | Parallelizable? | Batching Benefit |
|---|---|---|---|
| Prefill | Compute (matrix multiply throughput) | Yes — all prompt tokens at once | Moderate — already compute-efficient |
| Decode | Memory bandwidth (re-reading KV cache) | No — strictly sequential per request | Large — batching multiple users amortizes memory reads |
```flowchart
st=>start: User prompt arrives at the inference server
prefill=>operation: Prefill phase processes entire prompt in parallel; populates KV cache
decode=>operation: Decode loop generates one new token using the cached keys/values
append=>operation: New token's key/value vectors appended to the growing KV cache
check=>operation: Check for stop condition (end token or max length reached)
loop=>operation: If not done, repeat decode step for the next token
pass=>end: Final generated response returned to the user
st->prefill->decode->append->check->loop->pass
```
**None of this is abstract for AI accelerator design — it directly explains why raw FLOPs alone don't make a chip good at serving LLMs.** A chip can have enormous peak compute and still serve slowly if its memory bandwidth can't keep decode fed fast enough, which is exactly why HBM capacity and bandwidth, KV-cache-aware scheduling, and request batching get as much engineering attention as raw matrix-multiply throughput. Understanding the prefill/decode split is the single most useful mental model for reasoning about why one AI chip serves a chatbot faster than another with a bigger spec sheet.