Cache invalidation is the process of removing or updating stale entries from a cache when the underlying data changes. It is famously considered one of the two hard problems in computer science (along with naming things and off-by-one errors) because getting it wrong leads to serving outdated, incorrect data.

Why Cache Invalidation is Challenging

• Consistency vs. Performance: Aggressive invalidation keeps data fresh but reduces cache hit rates. Conservative invalidation improves performance but risks stale data.
• Distributed Caches: In distributed systems, ensuring all cache nodes invalidate consistently and simultaneously is difficult.
• Hidden Dependencies: Data changes may ripple through multiple cached entries in non-obvious ways.

Invalidation Strategies

• Time-Based (TTL): Set a Time to Live on each cache entry — it's automatically removed after expiration. Simple and effective for data that can tolerate some staleness. TTL values: seconds for real-time data, hours for relatively stable data, days for static content.
• Event-Based: Invalidate cache entries when the source data changes. Requires an event system (pub/sub, webhooks, database triggers) to notify the cache.
• Write-Through: When data is updated, the cache is updated simultaneously — no stale entries, but adds write latency.
• Manual Invalidation: Explicitly clear or update specific cache entries when you know the data has changed.
• Version-Based: Include a version number in cache keys. When data changes, increment the version — old cache entries naturally become unreferenced.

AI-Specific Considerations

• Model Updates: When a model is updated, all cached responses should be invalidated because the new model may produce different answers.
• RAG Source Updates: When retrieval documents are updated, cached RAG results need invalidation.
• Semantic Cache: Invalidating entries in a semantic cache requires understanding which cached responses are affected by a data change.
• System Prompt Changes: Modifying system prompts should invalidate all response caches.

Best Practice: Use TTL as a safety net (entries eventually expire even if event-based invalidation fails) combined with event-based invalidation for time-sensitive data changes.