kv cache quantization

**KV cache quantization** reduces the precision of the key-value (KV) cache in transformer models during inference, dramatically reducing memory consumption and enabling longer context lengths or larger batch sizes. **The KV Cache Problem** During autoregressive generation (e.g., GPT, LLaMA), transformers cache the key and value tensors from previous tokens to avoid recomputing them: - **Memory per token**: 2 × num_layers × hidden_dim × 2 bytes (FP16) or 4 bytes (FP32). - **Example**: LLaMA-7B with 32 layers, 4096 hidden dim, FP16 = 2 × 32 × 4096 × 2 = 524KB per token. - **For 2048 tokens**: 1GB of KV cache per sequence. - **Batch size 32**: 32GB just for KV cache — often exceeds available GPU memory. **How KV Cache Quantization Works** - **Quantize**: Convert FP16 key/value tensors to INT8 or INT4 after computation. - **Store**: Cache quantized tensors (2-4× memory reduction). - **Dequantize**: Convert back to FP16 when needed for attention computation. **Quantization Schemes** - **INT8**: 2× memory reduction, minimal accuracy loss (<1% perplexity increase). - **INT4**: 4× memory reduction, moderate accuracy loss (1-3% perplexity increase). - **Per-Token Quantization**: Compute scale/zero-point per token for better accuracy. - **Per-Channel Quantization**: Separate quantization parameters per attention head. **Advantages** - **Memory Savings**: 2-4× reduction in KV cache memory. - **Longer Contexts**: Fit 2-4× more tokens in the same memory budget. - **Larger Batches**: Increase batch size for higher throughput. - **Cost Reduction**: Use smaller GPUs or serve more users per GPU. **Accuracy Impact** - **INT8**: Negligible impact (<0.5% perplexity increase) for most models. - **INT4**: Noticeable but acceptable (1-3% perplexity increase) for many applications. - **Model-Dependent**: Larger models (70B+) are more robust to KV quantization than smaller models. **Frameworks Supporting KV Quantization** - **vLLM**: Production inference server with INT8 KV cache support. - **TensorRT-LLM**: NVIDIA inference library with INT4/INT8 KV quantization. - **llama.cpp**: Supports various KV cache quantization formats. - **Hugging Face Transformers**: Experimental support via BetterTransformer. **Practical Impact** For a 7B parameter model serving 32 concurrent users: - **Without KV quantization**: 32GB KV cache (requires A100 80GB). - **With INT8 KV quantization**: 16GB KV cache (fits on A100 40GB or A6000). - **With INT4 KV quantization**: 8GB KV cache (fits on RTX 4090 24GB). KV cache quantization is **essential for production LLM serving** — it enables longer contexts, higher throughput, and deployment on more affordable hardware.