cache hit rate

**Cache hit rate** is the percentage of requests that are successfully served from the cache (hits) versus the total number of requests (hits + misses). It is the primary metric for evaluating cache effectiveness. **Formula** $$\text{Hit Rate} = \frac{\text{Cache Hits}}{\text{Cache Hits} + \text{Cache Misses}} \times 100\%$$ **Interpreting Hit Rate** - **>90%**: Excellent — the cache is highly effective. The vast majority of requests are served from cache. - **70–90%**: Good — the cache is working well but there may be opportunities to improve. - **50–70%**: Moderate — consider if the cache strategy matches the access patterns. - **<50%**: Poor — the cache may be too small, eviction policy may be wrong, or the workload may not benefit from caching. **Factors That Affect Hit Rate** - **Cache Size**: Larger caches store more entries and have higher hit rates, but cost more memory. - **Eviction Policy**: **LRU** (Least Recently Used), **LFU** (Least Frequently Used), and other policies determine which entries to remove when the cache is full. - **TTL (Time to Live)**: Shorter TTLs cause entries to expire before they can be reused; longer TTLs risk serving stale data. - **Access Pattern**: Workloads with high **temporal locality** (recently accessed items are likely to be accessed again) benefit most from caching. - **Cache Key Design**: Using too-specific keys (exact prompt match) reduces hit rates vs. semantic matching. **Cache Hit Rate for LLM Applications** - **Exact Match Caching**: Typically **5–15%** hit rate for conversational AI (queries vary widely). - **Semantic Caching**: Can achieve **20–40%** hit rate by matching semantically similar queries. - **FAQ/Support Bots**: Often **50–80%** hit rate because users ask the same questions repeatedly. - **KV Cache**: ~100% hit rate within a single generation (each new token reuses all previous KV entries). **Monitoring** - Track hit rate over time — sudden drops may indicate cache invalidation issues, workload changes, or deployment problems. - Monitor by cache tier (L1/L2) and by query type to identify optimization opportunities. Cache hit rate directly translates to **cost savings and latency reduction** — even a 10% improvement can significantly reduce LLM API spending.