All Topics Glossary - Letter K | Chip Foundry Services

kv cache compression

kv cache quantization, kv cache eviction, attention memory, gqa, mqa, streamingllm, h2o, low rank kv cache

**KV cache compression reduces the memory footprint or bandwidth of stored attention keys and values used during autoregressive inference.** Long contexts and many concurrent requests can make cache capacity the binding limit even when model weights fit, so compression can raise batch size, context length, and serving throughput. For a decoder, cache storage grows with layers, sequence length, batch, key-value heads, head dimension, and bytes per element. Grouped-query and multi-query attention reduce the number of KV heads architecturally; post-training cache techniques quantize, evict, summarize, or encode states after or during generation. A production definition names the model family and release, parameter and active-parameter scale, vocabulary, context window, data cutoff and provenance, objective, precision, adaptation method, decoding policy, serving stack, target hardware, safety controls, evaluation protocol, and known limitations. Labels such as large, frontier, open, multimodal, efficient, or state of the art are not specifications; results must identify the exact artifact, prompt template, sampling settings, software version, hardware, and measurement date. Specify attention architecture, cache tensor layout, precision by key and value, calibration, scale granularity, residual windows, eviction or retention policy, positional treatment, block size, offload tier, maximum context, recomputation, and quality budget. **Architecture, algorithms, and system integration.** New token states enter a cache manager that may retain a recent high-precision window, quantize older blocks, score important tokens, evict low-value positions, project keys and values into a lower-rank form, or move blocks to host memory. Attention reconstructs or selectively reads the required representation. Quantization uses lower-bit values plus scales and sometimes zero points; eviction methods preserve recency, attention sinks, heavy hitters, or task-important tokens; low-rank methods store compressed latent factors; GQA and MQA share KV projections among query heads; paging manages fragmentation but does not itself compress bytes per live token. INT8, INT4, and mixed-precision caches trade fidelity for capacity; sliding-window and StreamingLLM-style policies bound history; H2O-style heavy-hitter retention selects influential positions; low-rank or latent attention compresses dimensions; offload and recomputation trade memory for bandwidth or compute. A modern AI system spans data collection and governance, filtering and deduplication, tokenization, distributed training, checkpointing, post-training, evaluation, model registry, quantization and compilation, inference schedulers, accelerators, memory and interconnect, retrieval or tools, application policy, observability, and incident response. Decisions at one layer change accuracy, latency, memory traffic, energy, safety, and maintainability elsewhere. Evaluation combines task quality with calibration, robustness, subgroup behavior, contamination resistance, factuality, safety, privacy, memorization, latency to first token, inter-token latency, throughput, concurrency, memory capacity and bandwidth, accelerator utilization, energy per useful output, availability, and cost. Means alone conceal tail behavior, prompt sensitivity, evaluator uncertainty, and failures on rare but consequential cases. **Implementation, compute behavior, and failure modes.** Keep a protected recent window, calibrate keys and values separately, fuse quantize and dequantize with attention kernels, align blocks with paged allocation, update importance online without quadratic metadata, preserve positional semantics, and expose a target-only uncompressed fallback. Compression saves HBM and memory traffic but adds conversion, metadata, irregular gathers, host transfers, or recomputation. Benefits depend on whether attention cache bandwidth, weight bandwidth, compute, or scheduling is actually limiting and whether kernels efficiently support the chosen format. Outliers saturate low-bit ranges, eviction removes a crucial early instruction, attention-sink mishandling destabilizes streaming, low-rank error accumulates, page metadata overwhelms savings, offload creates tail-latency spikes, and approximate cache changes output quality in hard-to-detect ways. Implementation uses immutable dataset and model manifests, content-addressed artifacts, deterministic preprocessing where feasible, seeded experiments, versioned prompts and templates, staged rollouts, bounded resource use, typed interfaces, admission control, timeouts, retries with budgets, telemetry, and reversible releases. Training and serving must agree on tokenizer files, special-token IDs, chat formatting, position treatment, numerical precision, and stop conditions. Delivered performance depends on tensor shapes, arithmetic intensity, quantization format, kernel fusion, batch and sequence distributions, HBM capacity and bandwidth, cache hierarchy, host memory, accelerator topology, collective communication, PCIe or fabric links, storage, power caps, cooling, and scheduler placement. Peak FLOPS or a single benchmark number cannot predict end-to-end behavior. Common failures include train-test leakage, duplicated or poisoned data, tokenizer drift, checkpoint incompatibility, unstable optimization, catastrophic forgetting, numerical overflow, router collapse, silent truncation, cache exhaustion, latency cliffs, evaluator bias, benchmark gaming, hallucination, unsafe tool calls, privacy leakage, model extraction, dependency compromise, and dashboards that average away the affected users. **Evaluation, governance, and lifecycle controls.** Compare logits and generated quality against full-precision cache over short and long contexts, needle retrieval, multi-turn instructions, code, multilingual text, repeated tokens, position boundaries, cancellations, mixed batches, paging churn, OOM recovery, and load tests. Bytes per token, compression ratio, maximum resident tokens, HBM and host traffic, dequantization time, attention latency, throughput, concurrency, fragmentation, retrieval accuracy, perplexity or task delta, energy, and p99 latency matter. Cached prompts and generated states can encode sensitive information; isolation, zeroization, tenant boundaries, retention, encryption for offload, crash dumps, and access-controlled telemetry remain required after compression. Validation combines schema and unit tests, small-run training checks, loss and gradient diagnostics, distributed-failure injection, golden-token tests, reference decoding, numerical comparisons, benchmark suites, adversarial and red-team evaluation, human review with calibrated rubrics, subgroup slices, load and soak testing, hardware profiling, canary deployment, rollback drills, and post-release monitoring. Independent test sets and frozen protocols protect the measurement boundary. Dataset snapshots, licenses and consent, filtering rules, tokenizer assets, source revision, configuration, seeds, optimizer state, checkpoints, adapter lineage, compiler and runtime, container, accelerator firmware, evaluation prompts, judge models, human labels, approvals, model cards, incidents, and deprecation remain linked. Reproducibility is a chain of custody rather than a saved weight file. Owners define data rights, privacy and retention, security classification, acceptable use, safety thresholds, model and supply-chain provenance, access control, secrets, export and regional obligations, environmental reporting, human escalation, vulnerability response, audit evidence, and final release authority. Automated scores inform but do not replace accountability for the deployed system. | Technique | What shrinks | Extra work | Strength | Primary risk | |---|---|---|---|---| | KV quantization | Bytes per element | Scale and conversion | Broad capacity gain | Outlier error | | Token eviction | Stored positions | Importance tracking | Bounded context memory | Lost critical history | | GQA or MQA | KV head count | Architectural training | Large native reduction | Possible quality tradeoff | | Low-rank encoding | Head dimension/state | Projection reconstruction | Structured compression | Approximation and kernels | | Host offload | HBM residency | Data transfer | Extends capacity | Tail latency bandwidth | ```svg ``` **Selection and practical application.** Use GQA or MQA when designing the model, quantization for broad capacity gains with calibrated error, bounded eviction for streaming tasks tolerant of forgotten detail, low-rank methods for compatible architectures, and offload only when transfer tails are acceptable. Long-context assistants, retrieval-augmented generation, code agents, multi-turn chat, document analysis, and high-concurrency inference use cache compression to fit more useful work per accelerator. Cache format interacts with attention architecture, tokenizer length, batching, paging, kernels, memory hierarchy, scheduler fairness, model quality, and privacy policy. The useful optimization boundary is the complete model-serving product. Improving loss, benchmark accuracy, tokens per second, compression ratio, or accelerator utilization can move the bottleneck or weaken robustness, fairness, security, recoverability, and user value elsewhere, so qualification follows representative workflows from source data through production outcomes. A production definition names the model family and release, parameter and active-parameter scale, vocabulary, context window, data cutoff and provenance, objective, precision, adaptation method, decoding policy, serving stack, target hardware, safety controls, evaluation protocol, and known limitations. Labels such as large, frontier, open, multimodal, efficient, or state of the art are not specifications; results must identify the exact artifact, prompt template, sampling settings, software version, hardware, and measurement date. Evaluation combines task quality with calibration, robustness, subgroup behavior, contamination resistance, factuality, safety, privacy, memorization, latency to first token, inter-token latency, throughput, concurrency, memory capacity and bandwidth, accelerator utilization, energy per useful output, availability, and cost. Means alone conceal tail behavior, prompt sensitivity, evaluator uncertainty, and failures on rare but consequential cases. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.

KV cache optimization

paged attention, memory efficiency, inference optimization, token latency

**KV Cache Optimization** is **a critical inference technique that caches key and value tensors for previous tokens to eliminate recomputation — reducing memory bandwidth by 10x and achieving 20-100x speedup in autoregressive generation compared to naïve inference**. **KV Cache Mechanism:** - **Tensor Storage**: storing K and V matrices (not Q) for all previous tokens: shape [seq_len, batch, head, dim] — for 7B Llama with 2048 context, requires 8.4GB per batch - **Reuse Pattern**: current token Q only computes attention scores with all previous K/V cached — eliminates O(n²) redundant computations - **Incremental Generation**: each new token only needs current attention computation, not recalculate entire sequence — reduces attention FLOPS from 4s² to 4s for sequence length s - **Batch Processing**: maintaining separate KV caches per batch element enables efficient batching — critical for inference serving throughput **Memory and Latency Trade-offs:** - **Memory Footprint**: KV cache dominates memory during inference (typically 80% of peak memory) — dominant cost at batch size >1 with float16 format - **Memory-Bound Operation**: KV cache access becomes bottleneck with memory bandwidth 100-200GB/s on A100 — achieves only 30-40 TFLOPS vs 312 TFLOPS peak compute - **Batch Size Limitations**: large batch sizes saturate GPU memory with KV cache faster than compute — typical batch size 32-64 before OOM - **Latency Reduction**: decoding latency from 50-200ms per token to 5-20ms with KV cache — enables real-time conversational interfaces **Paged Attention Innovation:** - **Page-Level Storage**: dividing KV cache into fixed-size pages (16 or 32 tokens) with virtual addressing — enables dynamic allocation and sharing - **Memory Fragmentation Reduction**: paging reduces external fragmentation from 37% to <5% in typical workloads — 4-8x improvement in memory utilization - **Efficient Batching**: different sequences with varying lengths share GPU memory pages — server throughput increases from 10 req/s to 40-50 req/s - **vLLM Implementation**: open-source system using paged attention with 100-200x speedup over HuggingFace Transformers — serves 1000+ concurrent requests **Advanced Optimization Techniques:** - **KV Cache Quantization**: compressing cache from float32 to int8 with minimal accuracy loss (<0.5% perplexity) — reduces memory by 4x - **Selective Caching**: pruning and caching only high-attention tokens from early layers (80-90% reduction) — sparse pattern benefits long documents - **Recomputation Strategies**: trading computation for memory by recomputing early layer KV instead of caching — useful when memory constraints tight - **Multi-GPU KV Splitting**: distributing cache across 4-8 GPUs using all-gather for attention computation — enables processing 32K context windows **KV Cache Optimization is essential for production LLM inference — enabling real-time serving of large models like GPT-3 and Llama on resource-constrained hardware through efficient memory utilization.**

kv cache optimization

key value cache management, attention cache compression, kv cache quantization, inference memory reduction

**KV Cache Optimization** is **the set of techniques for reducing memory footprint and bandwidth requirements of cached key-value pairs in autoregressive transformer inference** — including quantization (INT8/INT4), eviction policies, compression, and architectural changes that collectively enable 2-10× memory reduction, allowing larger batch sizes, longer contexts, or deployment on smaller GPUs while maintaining generation quality. **KV Cache Fundamentals:** - **Cache Purpose**: in autoregressive generation, each token attends to all previous tokens; recomputing K and V for all previous tokens at each step costs O(L²) FLOPs; caching K and V reduces to O(L) per token; trades memory for compute - **Memory Scaling**: for model with L layers, H heads, sequence length N, head dimension d, batch size B: cache size = B×L×2×N×H×d×sizeof(dtype); Llama 2 70B at N=4096, B=32, FP16: 70×2×4096×8×128×2 = 58GB just for KV cache - **Bandwidth Bottleneck**: each generated token loads entire KV cache from HBM; at 4K context, loads 1.8GB per token; A100 HBM bandwidth 1.5TB/s limits to ~800 tokens/sec; memory bandwidth, not compute, determines throughput - **Growth During Generation**: cache grows by B×L×2×H×d×sizeof(dtype) per token; for Llama 2 70B, adds 14MB per token; 1000-token generation requires 14GB additional memory; limits maximum batch size or context length ```svg ``` **Quantization Techniques:** - **INT8 KV Cache**: quantize cached K and V from FP16 to INT8; 2× memory reduction; per-tensor or per-channel quantization; dequantize before attention computation; quality loss <0.1 perplexity for most models; supported in TensorRT-LLM, vLLM, HuggingFace TGI - **INT4 KV Cache**: aggressive 4× reduction; requires careful calibration; group-wise quantization (groups of 32-128 elements) maintains quality; perplexity increase 0.1-0.3; enables 4× larger batches or contexts; used in GPTQ-for-LLMs, AWQ - **Mixed Precision**: quantize V cache more aggressively than K; V contributes to output values (less sensitive), K affects attention scores (more sensitive); K in INT8, V in INT4 provides 3× reduction with minimal quality loss - **Dynamic Quantization**: quantize during generation, not ahead of time; adapts to actual value distributions; slightly higher latency (quantization overhead) but better quality; used in production systems with strict quality requirements **Eviction and Compression:** - **H2O (Heavy Hitter Oracle)**: evicts KV pairs with lowest attention scores; keeps "heavy hitter" tokens that receive most attention; maintains 20-30% of cache with <1% quality loss; requires tracking attention scores (overhead); effective for long contexts where most tokens rarely attended - **StreamingLLM**: keeps first few tokens (system prompt) and recent window; evicts middle tokens; exploits observation that attention focuses on recent context and initial tokens; enables infinite context with fixed memory; quality depends on task (good for chat, poor for long-document QA) - **Scissorhands**: learns to predict which KV pairs can be evicted; trains small predictor network on attention patterns; achieves 50-70% cache reduction with <0.5% quality loss; requires model-specific training - **Sparse Attention Patterns**: for models with structured sparsity (sliding window, block-sparse), only cache tokens within attention pattern; Mistral 7B with 4K sliding window caches only 4K tokens regardless of total length; enables unbounded generation **Architectural Optimizations:** - **Multi-Query Attention (MQA)**: shares K and V across all query heads; reduces cache by number of heads (typically 32-64×); used in PaLM, Falcon; 1-2% quality trade-off for massive memory savings - **Grouped Query Attention (GQA)**: shares K and V across groups of heads; 4-8× cache reduction with <0.5% quality loss; used in Llama 2, Mistral; sweet spot between MHA and MQA - **Cross-Layer KV Sharing**: shares KV cache across multiple layers; reduces cache by factor of layers sharing; experimental technique; 2-4 layers can share with acceptable quality loss; total reduction 2-4× - **Low-Rank KV Projection**: projects K and V to lower dimension before caching; cache stores low-rank version; reconstruct full dimension during attention; 2-4× reduction; requires architecture modification and retraining **PagedAttention and Memory Management:** - **PagedAttention (vLLM)**: treats KV cache like virtual memory with paging; divides cache into fixed-size blocks (pages); non-contiguous storage eliminates fragmentation; enables near-optimal memory utilization (90-95% vs 20-40% for naive allocation) - **Block Size**: typical block size 16-64 tokens; smaller blocks reduce internal fragmentation but increase metadata overhead; 32 tokens balances trade-offs for most workloads - **Copy-on-Write**: multiple sequences can share cache blocks (e.g., common prompt); only copy when sequences diverge; critical for beam search and parallel sampling where sequences share prefix - **Memory Pool**: pre-allocates memory pool for cache blocks; eliminates allocation overhead during generation; enables predictable latency; pool size determines maximum concurrent requests **Production Deployment Impact:** - **Throughput**: INT8 quantization + PagedAttention enables 2-3× higher throughput vs naive FP16 caching; serves 100-200 requests/sec vs 30-50 for Llama 2 70B on 8×A100 - **Latency**: reduced memory bandwidth improves time-to-first-token by 20-40%; subsequent tokens 10-20% faster; critical for interactive applications where latency matters - **Cost**: 2-4× memory reduction enables deployment on smaller/fewer GPUs; Llama 2 70B with optimizations fits on 4×A100 vs 8×A100; halves infrastructure cost - **Context Length**: memory savings enable 2-4× longer contexts at same batch size; or maintain context while increasing batch size 2-4×; flexibility to optimize for throughput or capability KV Cache Optimization is **the critical enabler of practical LLM deployment** — by addressing the memory bottleneck that dominates inference costs, these techniques transform LLMs from research artifacts requiring massive GPU clusters into production systems that serve millions of users on reasonable hardware budgets.

kv cache quantization

kv, optimization

**KV cache quantization** reduces the precision of the key-value (KV) cache in transformer models during inference, dramatically reducing memory consumption and enabling longer context lengths or larger batch sizes. **The KV Cache Problem** During autoregressive generation (e.g., GPT, LLaMA), transformers cache the key and value tensors from previous tokens to avoid recomputing them: - **Memory per token**: 2 × num_layers × hidden_dim × 2 bytes (FP16) or 4 bytes (FP32). - **Example**: LLaMA-7B with 32 layers, 4096 hidden dim, FP16 = 2 × 32 × 4096 × 2 = 524KB per token. - **For 2048 tokens**: 1GB of KV cache per sequence. - **Batch size 32**: 32GB just for KV cache — often exceeds available GPU memory. **How KV Cache Quantization Works** - **Quantize**: Convert FP16 key/value tensors to INT8 or INT4 after computation. - **Store**: Cache quantized tensors (2-4× memory reduction). - **Dequantize**: Convert back to FP16 when needed for attention computation. **Quantization Schemes** - **INT8**: 2× memory reduction, minimal accuracy loss (<1% perplexity increase). - **INT4**: 4× memory reduction, moderate accuracy loss (1-3% perplexity increase). - **Per-Token Quantization**: Compute scale/zero-point per token for better accuracy. - **Per-Channel Quantization**: Separate quantization parameters per attention head. **Advantages** - **Memory Savings**: 2-4× reduction in KV cache memory. - **Longer Contexts**: Fit 2-4× more tokens in the same memory budget. - **Larger Batches**: Increase batch size for higher throughput. - **Cost Reduction**: Use smaller GPUs or serve more users per GPU. **Accuracy Impact** - **INT8**: Negligible impact (<0.5% perplexity increase) for most models. - **INT4**: Noticeable but acceptable (1-3% perplexity increase) for many applications. - **Model-Dependent**: Larger models (70B+) are more robust to KV quantization than smaller models. **Frameworks Supporting KV Quantization** - **vLLM**: Production inference server with INT8 KV cache support. - **TensorRT-LLM**: NVIDIA inference library with INT4/INT8 KV quantization. - **llama.cpp**: Supports various KV cache quantization formats. - **Hugging Face Transformers**: Experimental support via BetterTransformer. **Practical Impact** For a 7B parameter model serving 32 concurrent users: - **Without KV quantization**: 32GB KV cache (requires A100 80GB). - **With INT8 KV quantization**: 16GB KV cache (fits on A100 40GB or A6000). - **With INT4 KV quantization**: 8GB KV cache (fits on RTX 4090 24GB). KV cache quantization is **essential for production LLM serving** — it enables longer contexts, higher throughput, and deployment on more affordable hardware.