prompt caching

**Prompt caching** is a technique that stores and reuses the **processed prefix of prompts** (particularly system prompts and common instructions) to avoid redundant computation on repeated or similar requests. It can dramatically reduce latency and cost when many requests share the same prompt prefix. **How Prompt Caching Works** - **Prefix Computation**: The first time a prompt is processed, the system computes the **KV (key-value) cache** for the prompt tokens through the transformer layers — this is the expensive step. - **Cache Storage**: The computed KV cache for the prefix is stored in GPU memory or a fast cache layer. - **Reuse**: Subsequent requests with the **same prefix** skip the prefix computation and directly reuse the cached KV states, only computing the new (unique) portion of the prompt. **Where Prompt Caching Helps Most** - **Long System Prompts**: If every request includes a 2,000-token system prompt, caching avoids reprocessing those tokens for each request. - **Few-Shot Examples**: Prompts with many in-context examples can be cached since the examples don't change between requests. - **Multi-Turn Conversations**: Earlier turns in a conversation don't change — their KV cache can be reused when processing new turns. - **Batch Processing**: When processing many inputs with the same instructions, the instruction prefix is computed once. **Provider Support** - **Anthropic**: Offers explicit **prompt caching** with a cache_control parameter — cached prompts cost 90% less and have lower latency. - **OpenAI**: Automatic prompt caching for repeated prefixes with 50% cost reduction on cached tokens. - **Google Gemini**: Context caching API for reusing long contexts across multiple requests. **Cost Savings** With prompt caching, a 3,000-token system prompt that appears in every request effectively becomes **free** after the first request, saving both compute and API costs. **Limitations** - **Exact Prefix Match**: Most implementations require an **exact match** of the cached prefix — any change invalidates the cache. - **Memory Overhead**: Stored KV caches consume GPU memory proportional to the prefix length and model size. Prompt caching is one of the **highest-impact, lowest-effort** optimizations for production LLM applications with consistent prompt structures.