kv cache

The KV cache stores the attention keys and values computed for every token a language model has already processed, so that generating each new token does not have to recompute them. It is the single largest, fastest-growing consumer of accelerator memory during LLM inference, and it is why decoding is memory-bound rather than compute-bound.\n\n**It turns quadratic recomputation into linear reuse.** In self-attention, each new token's query attends to the keys and values of all previous tokens. Without a cache, every decode step would recompute K and V for the entire sequence — O(n²) work to emit n tokens. By caching each token's K and V once and appending the new token's, the model does only O(n) new work per step and reuses everything prior. The cost of that shortcut is memory: the cache must be held in fast memory (HBM) for the whole generation.\n\n**The cache grows linearly and can dwarf the weights.** Its size is roughly 2 × layers × kv_heads × head_dim × sequence_length × batch × bytes_per_element — the factor of 2 for K and V. Because it scales with both context length and batch size, long-context, high-throughput serving can make the KV cache larger than the model weights themselves, and it slams into the HBM capacity ceiling. Reading it back every step is pure memory traffic, which is exactly why token generation sits low on the roofline, limited by bandwidth.\n\n| Lever | What it does | Effect on cache |\n|---|---|---|\n| MHA (baseline) | full keys/values per head | largest cache |\n| GQA | share KV across head groups | few× smaller |\n| MQA | one KV head for all queries | smallest, some quality cost |\n| KV quantization | store K/V in INT8/FP8 | ~2-4× smaller |\n| PagedAttention | page the cache like virtual memory | less waste, higher batch |\n\n```svg\n\n```\n\n**Managing the cache is the core of inference optimization.** Because capacity and bandwidth are the binding constraints, the whole toolbox targets the KV cache: grouped- and multi-query attention (GQA/MQA) cut the number of KV heads; KV quantization shrinks each entry; PagedAttention (vLLM) pages the cache in fixed blocks to eliminate fragmentation and pack more concurrent requests; and prefix caching reuses shared prompts. Each trades a little quality or complexity for more context or higher batch in the same HBM.\n\nRead the KV cache through a quant lens rather than a feature lens: it is bytes-in-HBM and bytes-moved-per-token, and per the roofline those two numbers cap decode throughput before FLOPs ever do. The right question is how many gigabytes the cache costs at a target context and batch, and how much bandwidth each generated token demands — a measured memory budget that decides how long a context and how many users a given accelerator can serve.