KV cache optimization
**KV Cache Optimization** is **a critical inference technique that caches key and value tensors for previous tokens to eliminate recomputation — reducing memory bandwidth by 10x and achieving 20-100x speedup in autoregressive generation compared to naïve inference**.
**KV Cache Mechanism:**
- **Tensor Storage**: storing K and V matrices (not Q) for all previous tokens: shape [seq_len, batch, head, dim] — for 7B Llama with 2048 context, requires 8.4GB per batch
- **Reuse Pattern**: current token Q only computes attention scores with all previous K/V cached — eliminates O(n²) redundant computations
- **Incremental Generation**: each new token only needs current attention computation, not recalculate entire sequence — reduces attention FLOPS from 4s² to 4s for sequence length s
- **Batch Processing**: maintaining separate KV caches per batch element enables efficient batching — critical for inference serving throughput
**Memory and Latency Trade-offs:**
- **Memory Footprint**: KV cache dominates memory during inference (typically 80% of peak memory) — dominant cost at batch size >1 with float16 format
- **Memory-Bound Operation**: KV cache access becomes bottleneck with memory bandwidth 100-200GB/s on A100 — achieves only 30-40 TFLOPS vs 312 TFLOPS peak compute
- **Batch Size Limitations**: large batch sizes saturate GPU memory with KV cache faster than compute — typical batch size 32-64 before OOM
- **Latency Reduction**: decoding latency from 50-200ms per token to 5-20ms with KV cache — enables real-time conversational interfaces
**Paged Attention Innovation:**
- **Page-Level Storage**: dividing KV cache into fixed-size pages (16 or 32 tokens) with virtual addressing — enables dynamic allocation and sharing
- **Memory Fragmentation Reduction**: paging reduces external fragmentation from 37% to <5% in typical workloads — 4-8x improvement in memory utilization
- **Efficient Batching**: different sequences with varying lengths share GPU memory pages — server throughput increases from 10 req/s to 40-50 req/s
- **vLLM Implementation**: open-source system using paged attention with 100-200x speedup over HuggingFace Transformers — serves 1000+ concurrent requests
**Advanced Optimization Techniques:**
- **KV Cache Quantization**: compressing cache from float32 to int8 with minimal accuracy loss (<0.5% perplexity) — reduces memory by 4x
- **Selective Caching**: pruning and caching only high-attention tokens from early layers (80-90% reduction) — sparse pattern benefits long documents
- **Recomputation Strategies**: trading computation for memory by recomputing early layer KV instead of caching — useful when memory constraints tight
- **Multi-GPU KV Splitting**: distributing cache across 4-8 GPUs using all-gather for attention computation — enables processing 32K context windows
**KV Cache Optimization is essential for production LLM inference — enabling real-time serving of large models like GPT-3 and Llama on resource-constrained hardware through efficient memory utilization.**