kv cache

The key-value (KV) cache stores precomputed key and value tensors from previous tokens during autoregressive generation, avoiding redundant computation as sequences extend. Without caching, generating token N requires recomputing attention for all N-1 previous tokens—O(N²) complexity per sequence. With KV cache, only the new token's Q×K and attention×V computations are needed—O(N) per token. However, KV cache grows linearly with sequence length and batch size, often becoming the dominant memory consumer. For LLaMA-2-70B with 80 layers, 64 heads, and 128-dimensional heads, each token requires 2 (K+V) × 80 × 64 × 128 × 2 bytes = 2.6MB in FP16. A batch of 8 sequences at 4K context consumes 85GB—exceeding single GPU memory. Optimization techniques include: grouped-query attention (GQA) sharing K/V across heads (8x reduction), KV cache quantization to INT8 or INT4, sliding window attention limiting cache to recent tokens, and PagedAttention for memory-efficient management. The KV cache fundamentally shapes inference system design, determining maximum batch sizes, context lengths, and overall throughput. Efficient KV cache management is essential for production LLM serving.