kv cache eviction

**KV cache eviction** is the **policy for removing or compressing cached attention states when memory pressure grows, while preserving important context for ongoing generation** - eviction strategy directly affects long-context quality and system stability. **What Is KV cache eviction?** - **Definition**: Controlled deletion of KV entries based on recency, importance, or budget limits. - **Trigger Conditions**: Activated when cache occupancy approaches memory thresholds. - **Policy Types**: Includes recency-based, attention-score-based, and hybrid importance eviction. - **Serving Context**: Critical for long sessions, streaming workloads, and high-concurrency environments. **Why KV cache eviction Matters** - **Memory Safety**: Prevents out-of-memory failures during extended decoding workloads. - **Latency Stability**: Managed eviction avoids emergency compaction and runtime stalls. - **Quality Preservation**: Smart policies keep high-value tokens while dropping low-impact history. - **Capacity Scaling**: Efficient eviction enables more simultaneous requests per device. - **Operational Control**: Policy tuning provides explicit tradeoff knobs for quality versus cost. **How It Is Used in Practice** - **Importance Scoring**: Estimate token utility from attention statistics and structural markers. - **Tiered Memory**: Move low-priority states to slower storage before full eviction when possible. - **Guardrail Metrics**: Monitor perplexity drift and factuality changes after eviction adjustments. KV cache eviction is **a required control mechanism for robust long-running inference** - well-designed eviction keeps serving stable without collapsing answer quality.