KV cache quantization reduces the precision of the key-value (KV) cache in transformer models during inference, dramatically reducing memory consumption and enabling longer context lengths or larger batch sizes.

The KV Cache Problem

During autoregressive generation (e.g., GPT, LLaMA), transformers cache the key and value tensors from previous tokens to avoid recomputing them:

• Memory per token: 2 × num_layers × hidden_dim × 2 bytes (FP16) or 4 bytes (FP32).
• Example: LLaMA-7B with 32 layers, 4096 hidden dim, FP16 = 2 × 32 × 4096 × 2 = 524KB per token.
• For 2048 tokens: 1GB of KV cache per sequence.
• Batch size 32: 32GB just for KV cache — often exceeds available GPU memory.

How KV Cache Quantization Works

• Quantize: Convert FP16 key/value tensors to INT8 or INT4 after computation.
• Store: Cache quantized tensors (2-4× memory reduction).
• Dequantize: Convert back to FP16 when needed for attention computation.

Quantization Schemes

• INT8: 2× memory reduction, minimal accuracy loss (<1% perplexity increase).
• INT4: 4× memory reduction, moderate accuracy loss (1-3% perplexity increase).
• Per-Token Quantization: Compute scale/zero-point per token for better accuracy.
• Per-Channel Quantization: Separate quantization parameters per attention head.

Advantages

• Memory Savings: 2-4× reduction in KV cache memory.
• Longer Contexts: Fit 2-4× more tokens in the same memory budget.
• Larger Batches: Increase batch size for higher throughput.
• Cost Reduction: Use smaller GPUs or serve more users per GPU.

Accuracy Impact

• INT8: Negligible impact (<0.5% perplexity increase) for most models.
• INT4: Noticeable but acceptable (1-3% perplexity increase) for many applications.
• Model-Dependent: Larger models (70B+) are more robust to KV quantization than smaller models.

Frameworks Supporting KV Quantization

• vLLM: Production inference server with INT8 KV cache support.
• TensorRT-LLM: NVIDIA inference library with INT4/INT8 KV quantization.
• llama.cpp: Supports various KV cache quantization formats.
• Hugging Face Transformers: Experimental support via BetterTransformer.

Practical Impact

For a 7B parameter model serving 32 concurrent users:

• Without KV quantization: 32GB KV cache (requires A100 80GB).
• With INT8 KV quantization: 16GB KV cache (fits on A100 40GB or A6000).
• With INT4 KV quantization: 8GB KV cache (fits on RTX 4090 24GB).

KV cache quantization is essential for production LLM serving — it enables longer contexts, higher throughput, and deployment on more affordable hardware.