cache invalidation
**Cache invalidation** is the process of removing or updating **stale entries** from a cache when the underlying data changes. It is famously considered one of the **two hard problems in computer science** (along with naming things and off-by-one errors) because getting it wrong leads to serving outdated, incorrect data.
**Why Cache Invalidation is Challenging**
- **Consistency vs. Performance**: Aggressive invalidation keeps data fresh but reduces cache hit rates. Conservative invalidation improves performance but risks stale data.
- **Distributed Caches**: In distributed systems, ensuring all cache nodes invalidate consistently and simultaneously is difficult.
- **Hidden Dependencies**: Data changes may ripple through multiple cached entries in non-obvious ways.
**Invalidation Strategies**
- **Time-Based (TTL)**: Set a **Time to Live** on each cache entry — it's automatically removed after expiration. Simple and effective for data that can tolerate some staleness. TTL values: seconds for real-time data, hours for relatively stable data, days for static content.
- **Event-Based**: Invalidate cache entries when the source data changes. Requires an event system (pub/sub, webhooks, database triggers) to notify the cache.
- **Write-Through**: When data is updated, the cache is updated simultaneously — no stale entries, but adds write latency.
- **Manual Invalidation**: Explicitly clear or update specific cache entries when you know the data has changed.
- **Version-Based**: Include a version number in cache keys. When data changes, increment the version — old cache entries naturally become unreferenced.
**AI-Specific Considerations**
- **Model Updates**: When a model is updated, all cached responses should be invalidated because the new model may produce different answers.
- **RAG Source Updates**: When retrieval documents are updated, cached RAG results need invalidation.
- **Semantic Cache**: Invalidating entries in a semantic cache requires understanding which cached responses are affected by a data change.
- **System Prompt Changes**: Modifying system prompts should invalidate all response caches.
**Best Practice**: Use TTL as a **safety net** (entries eventually expire even if event-based invalidation fails) combined with event-based invalidation for time-sensitive data changes.