Caching strategies compared
Updated 4 July 2026 · first published 4 July 2026
Caching is one of the highest-leverage cost reduction tactics for LLM workloads. The idea is straightforward: do not pay for inference if you can return a cached result. But "caching" is not one technique - it is four distinct strategies with different trade-offs in cost savings, latency, accuracy, and implementation complexity. Choosing the wrong one wastes engineering effort; choosing the right one can cut 30–70% of your bill.
Strategy comparison
| Strategy | How it works | Cost savings | Latency impact | Implementation effort | Best for | Providers |
|---|---|---|---|---|---|---|
| Prompt cache (provider-side) | Provider caches the KV computation for repeated prompt prefixes. Subsequent requests with the same prefix skip reprocessing. | 50% on cached input tokens | 60–80% faster first-token latency | Zero to low - automatic on most providers | Long system prompts, RAG with shared context, agent loops | OpenAI, Anthropic, Google |
| Semantic cache (embedding-based) | Embed the user query, check a vector store for similar past queries. If similarity exceeds a threshold, return the cached response without calling the LLM. | 20–50% depending on query repetition | 50–90% faster (no LLM call) | Medium - needs embedding model, vector store, similarity threshold tuning | FAQ bots, customer support, repetitive queries | Any (self-hosted via GPTCache, Redis with vector search) |
| Context caching (Google-style) | Cache large context documents (PDFs, codebases) server-side. Pay once for context processing, reuse across many queries at reduced per-query cost. | 70–90% on context-heavy queries | Moderate improvement on context processing | Low to medium - upload context, manage TTL | Document Q&A, code review over large repos, multi-turn with static context | Google (Gemini), Anthropic (extended thinking context) |
| Application-level cache (Redis/CDN) | Deterministic hash of the full request (prompt + params). Store the response in Redis or a CDN. Return cached response on exact match. | Up to 100% on cache hits | 90–99% faster (no LLM call) | Low - standard caching infrastructure | Deterministic queries, classification, extraction with fixed prompts | Any (infrastructure you control) |
Prompt caching in detail
Prompt caching is the easiest win. OpenAI automatically caches prompts longer than 1,024 tokens. Anthropic caches prefixes of 2,048+ tokens. The cache is keyed on the exact prefix - same system prompt, same few-shot examples, same document context. When a cache hit occurs, you pay 50% less for the cached portion of input tokens.
The catch: cache invalidation is provider-controlled. If the prompt changes even slightly - different system message, different variable in the prefix - the cache misses. Design your prompts to front-load stable content and push dynamic content to the end of the prompt.
Semantic caching in detail
Semantic caching skips the LLM entirely when a similar question was asked before. The flow: embed the query, search a vector store (Pinecone, Qdrant, Redis with vector search), check if the top result exceeds a similarity threshold (typically 0.92–0.97), and return the cached answer if it does.
The risk is returning stale or slightly wrong answers. A threshold that is too low returns wrong results; one that is too high means almost no cache hits. Tune it per use case. For factual FAQ retrieval, 0.95+ is safe. For creative generation, semantic caching is rarely appropriate - the whole point is novelty.
Context caching in detail
Google's context caching lets you upload a large document once and pay a reduced rate for subsequent queries against it. This is powerful for document Q&A where you process the same 200-page PDF hundreds of times. The cost model: pay the full input token rate once for the cache write, then a reduced rate (often 25% of normal) for each cache read.
The trade-off is TTL. Cached contexts expire. If your workload has bursty access patterns - 500 queries in one hour, then nothing for a week - the cache may expire between bursts. Size your TTL to match your access pattern.
Application-level caching in detail
Deterministic caching is the most aggressive strategy. Hash the full request payload (prompt, temperature, max tokens, model, tools) and store the response. On exact match, return the cached response instantly - no LLM call, no latency, no cost.
This only works when the same input should always produce the same output. Classification, entity extraction, structured data parsing, and template-based generation are good candidates. Conversational or creative tasks are not - users expect varied responses.
Combining strategies
The best implementations layer multiple strategies. A typical stack: application-level cache as the first check (cheapest, fastest), semantic cache as the second check (catches near-duplicates), then the LLM call with provider-side prompt caching enabled (reduces cost even on cache misses). This layered approach can reduce total LLM spend by 40–70% on workloads with any degree of repetition.
Track cache hit rates separately for each layer. If your semantic cache hit rate drops below 10%, the vector store overhead may not be worth it. If your application cache hit rate is above 40%, you likely have deterministic tasks that should be optimized at the prompt level.
Related
- Prompt caching explained - deeper dive into provider-side caching.
- Semantic cache economics - ROI analysis of semantic caching.
- AI cost optimization - broader optimization strategies.
Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →