kv cache management
**KV cache management** is the process of efficiently storing, reusing, and evicting the **key-value pairs** computed during transformer attention in LLM inference. Each time a token is generated, the model computes attention over all previous tokens — storing these KV pairs in a cache avoids redundant recomputation and is essential for efficient autoregressive generation.
**How the KV Cache Works**
- **During Generation**: Each transformer layer computes **key (K)** and **value (V)** vectors for each token. These are stored in the KV cache.
- **Autoregressive Reuse**: When generating the next token, the model only computes K and V for the **new token**, then concatenates them with the cached K and V from all previous tokens to compute attention.
- **Without KV Cache**: The model would need to reprocess the **entire sequence** for every new token — making generation O(n²) instead of O(n).
**Memory Challenge**
The KV cache grows **linearly** with sequence length and **linearly** with model size:
- For a 70B parameter model with 128K context: the KV cache can consume **40+ GB** of GPU memory per request.
- For batch serving, KV cache memory scales with **batch_size × sequence_length**, often becoming the primary memory bottleneck.
**Management Techniques**
- **Paged Attention (vLLM)**: Manages KV cache as virtual memory pages, eliminating fragmentation and enabling efficient memory sharing across requests.
- **Multi-Query Attention (MQA)**: Shares K and V heads across attention heads, reducing KV cache size by the number of heads (e.g., 8× reduction).
- **Grouped-Query Attention (GQA)**: Groups multiple query heads to share K and V heads — a middle ground between MHA and MQA. Used in **Llama 2** and later.
- **KV Cache Compression**: Quantize cached K and V values to lower precision (FP16 → INT8 or INT4) to reduce memory.
- **Sliding Window Attention**: Only cache the last N tokens, limiting memory to a fixed window. Used in **Mistral** models.
- **Token Eviction (H2O, StreamLLM)**: Evict less important KV entries based on attention scores to maintain a fixed cache budget.
Efficient KV cache management is the **single most impactful optimization** for LLM serving throughput and is the core innovation behind high-performance inference engines like vLLM, TensorRT-LLM, and SGLang.