Cache hit rate is the percentage of requests that are successfully served from the cache (hits) versus the total number of requests (hits + misses). It is the primary metric for evaluating cache effectiveness.

Formula

$$\text{Hit Rate} = \frac{\text{Cache Hits}}{\text{Cache Hits} + \text{Cache Misses}} \times 100\%$$

Interpreting Hit Rate

• >90%: Excellent — the cache is highly effective. The vast majority of requests are served from cache.
• 70–90%: Good — the cache is working well but there may be opportunities to improve.
• 50–70%: Moderate — consider if the cache strategy matches the access patterns.
• <50%: Poor — the cache may be too small, eviction policy may be wrong, or the workload may not benefit from caching.

Factors That Affect Hit Rate

• Cache Size: Larger caches store more entries and have higher hit rates, but cost more memory.
• Eviction Policy: LRU (Least Recently Used), LFU (Least Frequently Used), and other policies determine which entries to remove when the cache is full.
• TTL (Time to Live): Shorter TTLs cause entries to expire before they can be reused; longer TTLs risk serving stale data.
• Access Pattern: Workloads with high temporal locality (recently accessed items are likely to be accessed again) benefit most from caching.
• Cache Key Design: Using too-specific keys (exact prompt match) reduces hit rates vs. semantic matching.

Cache Hit Rate for LLM Applications

• Exact Match Caching: Typically 5–15% hit rate for conversational AI (queries vary widely).
• Semantic Caching: Can achieve 20–40% hit rate by matching semantically similar queries.
• FAQ/Support Bots: Often 50–80% hit rate because users ask the same questions repeatedly.
• KV Cache: ~100% hit rate within a single generation (each new token reuses all previous KV entries).

Monitoring

• Track hit rate over time — sudden drops may indicate cache invalidation issues, workload changes, or deployment problems.
• Monitor by cache tier (L1/L2) and by query type to identify optimization opportunities.

Cache hit rate directly translates to cost savings and latency reduction — even a 10% improvement can significantly reduce LLM API spending.