Prompt caching is a technique that stores and reuses the processed prefix of prompts (particularly system prompts and common instructions) to avoid redundant computation on repeated or similar requests. It can dramatically reduce latency and cost when many requests share the same prompt prefix.

How Prompt Caching Works

• Prefix Computation: The first time a prompt is processed, the system computes the KV (key-value) cache for the prompt tokens through the transformer layers — this is the expensive step.
• Cache Storage: The computed KV cache for the prefix is stored in GPU memory or a fast cache layer.
• Reuse: Subsequent requests with the same prefix skip the prefix computation and directly reuse the cached KV states, only computing the new (unique) portion of the prompt.

Where Prompt Caching Helps Most

• Long System Prompts: If every request includes a 2,000-token system prompt, caching avoids reprocessing those tokens for each request.
• Few-Shot Examples: Prompts with many in-context examples can be cached since the examples don't change between requests.
• Multi-Turn Conversations: Earlier turns in a conversation don't change — their KV cache can be reused when processing new turns.
• Batch Processing: When processing many inputs with the same instructions, the instruction prefix is computed once.

Provider Support

• Anthropic: Offers explicit prompt caching with a cache_control parameter — cached prompts cost 90% less and have lower latency.
• OpenAI: Automatic prompt caching for repeated prefixes with 50% cost reduction on cached tokens.
• Google Gemini: Context caching API for reusing long contexts across multiple requests.

Cost Savings

With prompt caching, a 3,000-token system prompt that appears in every request effectively becomes free after the first request, saving both compute and API costs.

Limitations

• Exact Prefix Match: Most implementations require an exact match of the cached prefix — any change invalidates the cache.
• Memory Overhead: Stored KV caches consume GPU memory proportional to the prefix length and model size.

Prompt caching is one of the highest-impact, lowest-effort optimizations for production LLM applications with consistent prompt structures.