Long context vs RAG: where the cost breakeven actually is

Updated 19 July 2026 · first published 19 July 2026

Twelve months ago, 1M-token context windows existed in research papers. Today they are the baseline. The first product question that follows is inevitable: can we drop the RAG stack and just stuff all the context into every request? The honest answer is yes—if your query volume, document corpus, and caching strategy align. But "if" is doing heavy lifting. For teams still running on high-volume search workloads against churning corpora, RAG is still cheaper. For teams with stable documents and moderate request volume, long context with prompt caching can win outright. And for most teams in between, the answer is "it depends on your numbers."

This page is about making that "it depends" concrete. We will build a cost model for each approach, work through the breakeven math, and show when to pick long context, when to pick RAG, and when a hybrid might be the right call. The models are illustrative—your numbers will differ—but the framework and the formula stay the same.

The cost shape of long context

Long context is simple to cost: you pay for input tokens on every single request. The bill grows linearly with context size and request volume. If you are using Claude 3.5 Sonnet and padding 800k tokens of context, that is roughly $2.40 per request in input cost alone (at $3 per 1M input tokens). If you get 1,000 requests per day, that is $2,400 per day in input tokens.

The calculation is straightforward until caching enters the picture. Prompt caching changes the economics by orders of magnitude on the first request that primes a cache and dramatically on every subsequent request that hits the cache. A 800k-token prefix, cached and reused 100 times before invalidation, amortizes to $0.024 per request in input cost. The bulk of your spend moves from input tokens to output tokens.

The trap in the caching model is churn. A corpus that changes every day invalidates the cache on every request. A cache with a one-day TTL against a one-day refresh cycle is not a cache—it is an input token fee that the provider calls a "cache write" at a higher rate. If the corpus churns faster than the cache pays for itself, you have paid the premium for no benefit. The breakeven on cache invalidation cost depends on your document update frequency and the size of the stable prefix you can maintain across updates.

The cost shape of RAG

RAG cost is multi-part and most teams undercount it. The visible costs are embedding API charges and vector database storage and query. The hidden costs are re-ranking models, retrieval latency SLAs you pay for, and—the biggest one—engineering and maintenance labor to keep retrieval quality acceptable.

The per-request cost model for RAG breaks down like this. You pay for an embedding of the query. Most teams embed once and cache the query-embedding for its logical lifetime (hours to days), so query embedding cost is often negligible on repeated queries. You pay for a vector DB query, which is usually a fractional cent on cloud vector stores and free if you self-host. You pay for retrieval, which might be a single semantic search or a multi-hop re-ranking pipeline. You pay for the tokens in the retrieved context that you then feed to the model.

The accounting that breaks RAG teams is conflating the cost of retrieval infrastructure with the cost of RAG as a feature. A single vector DB query costs almost nothing. A vector store that handles 10M queries per month against 10B embeddings, with availability SLA, redundancy, backups, and schema migrations, costs your team something per query even if the provider's API line shows 0.001 cents. Most mature RAG programs allocate infrastructure cost back to RAG as a cost per request delivered. That number is often larger than the API cost.

Building the breakeven formula

The cost comparison only makes sense when you are comparing equivalent workloads. Long context stuffing serves requests against the entire corpus. Optimal RAG performs semantic search, returns top-K results, and serves requests against a subset. If RAG's quality loss is acceptable, RAG is cheaper at moderate to high query volume. If quality matters and long context is the only way to achieve it, then RAG's cost advantage disappears into a quality tradeoff that is not a cost story anymore.

With that scope set: long context cost per request is input token cost for the full context plus output token cost. RAG cost per request is embedding cost, vector store query cost, re-ranking cost if any, plus input and output token cost of the model call against retrieved context.

The breakeven happens at the query volume where the accumulated embedding cost (amortized across the lifetime of a single embedding before re-embedding) plus retrieval infrastructure cost equals the input token cost difference between full context and retrieved subset.

Here is the formula. Long context cost per request:

LC_cost = (full_context_tokens × input_token_price)

RAG cost per request, assuming query embeddings are cached for d days:

RAG_cost = (query_embedding_cost / (queries_per_day × d)) + vector_db_query_cost + (retrieved_context_tokens × input_token_price)

Breakeven is when LC_cost == RAG_cost. Rearranging:

full_context_tokens × price = (query_embedding_cost / (queries_per_day × d)) + vector_db_cost + retrieved_context_tokens × price

If you add re-ranking, add its cost per request. If you use prompt caching, reduce the left side by the cache hit rate times the cache-read rate discount.

Worked examples

Let's ground this with real numbers. Assume we are working with Claude 3.5 Sonnet: $3 per 1M input tokens, $15 per 1M output tokens.

Example 1: Moderate query volume, stable corpus, no caching

Scenario: internal knowledge base (tech docs, runbooks, policies). 500k tokens of context. 100 requests per day. Corpus is stable (updated monthly). Typical output is 500 tokens.

Long context cost per request: (500,000 × $3/1M) + (500 × $15/1M) = $1.50 + $0.0075 = $1.51

RAG cost per request: Assuming query embedding via Claude at $3/1M input, 100-token query: (100 × $3/1M / (100 × 30)) + $0.001 (vector DB) + (50,000 × $3/1M) + (500 × $15/1M) = $0.00001 + $0.001 + $0.15 + $0.0075 = $0.16

RAG is about 9x cheaper. Even accounting for 0.5 FTE of engineering labor to maintain retrieval quality ($50k/year = $27/day / 100 requests = $0.27/request), RAG is still $0.43, under 30% of long context cost.

Example 2: High-volume API, embedded results, prompt caching enabled

Scenario: customer-facing API. 800k token context (large knowledge base). 50,000 requests per day. Context is stable (daily refresh). Cache hit rate 85% (80% of requests hit a cached prefix, 20% trigger cache write at 25% premium). Output average 200 tokens.

Long context cost with caching: Input cost is 800k × $3/1M, but 85% of requests pay the cache-read rate ($1.50/1M = 50% discount), 15% pay cache-write rate and full rate. (800,000 × $1.50/1M × 0.85) + (800,000 × $3.75/1M × 0.15) + (200 × $15/1M) = $1.02 + $0.45 + $0.003 = $1.47

RAG cost per request: Query embedding (cached over 30 queries = 50 req/day / 30): (100 × $3/1M / 30) + $0.002 (higher-load DB) + (100,000 × $3/1M) + (200 × $15/1M) = $0.00001 + $0.002 + $0.30 + $0.003 = $0.31

RAG is still cheaper at $0.31 vs $1.47, but the gap has closed. At this query volume, caching reduces long context cost by ~80%, but the sheer input token volume still dominates. RAG wins unless quality requirements force you to use full context.

Example 3: High-churn corpus with moderate volume

Scenario: dynamic content (support tickets, logs, live data feeds). 200k tokens context. 1,000 requests per day. Corpus churns hourly (docs updated continuously). No stable prefix to cache. Output 300 tokens.

Long context cost per request: (200,000 × $3/1M) + (300 × $15/1M) = $0.60 + $0.0045 = $0.60

RAG cost per request (churn means no embedding cache sustainability): Query embedding is typically re-done: (100 × $3/1M) + $0.005 (DB query under churn) + (30,000 × $3/1M) + (300 × $15/1M) = $0.0003 + $0.005 + $0.09 + $0.0045 = $0.10

RAG is still cheaper at $0.10, but the cost difference is small enough that quality considerations might force a hybrid: BM25 hybrid search (text + semantic) to improve recall on a churning corpus without full context stuffing, or accept long context for its quality guarantees and optimize elsewhere.

Why prompt caching changes the answer

Prompt caching is the most disruptive economic factor in this comparison. A 1M-token cached prefix costs $1.50 per request on the first call, then $0.015 (1% of full price) on every hit. At 100 cache hits before invalidation, the amortized cost is $0.0155 per request—effectively free for input tokens.

The catch is churn. Cache invalidation on a 1M-token corpus updates means you pay the write premium (25% markup) to re-prime the cache. If the corpus refreshes every hour and you get 1,000 requests per day, you are writing the cache 24 times per day, 480 cache writes per month. The cache-write cost can dominate the savings if the hit rate drops below 10-20 hits per write. Most teams find that long-context-with-caching beats RAG only when the corpus is stable for at least several hours and hit rates exceed 20+ requests per cache write cycle.

The second-order caching cost is invalidation risk. A cache coherency bug (a data update that does not invalidate the prefix) results in stale context served to all downstream requests. RAG retrieves fresh data on each query, so staleness is bounded by your embedding freshness. Long context with cache bypasses that protection. The operational cost of a cache staleness incident often exceeds the infrastructure cost of running the corresponding RAG system for a year.

Latency and quality as cost factors

The pure cost model ignores latency and quality, and those are often decisive. Long context adds latency on the LLM call itself—a 1M-token input can take 2-5x longer to process than a 50k-token input, depending on the model's attention mechanism. RAG adds a retrieval hop (50-200ms) but reduces model latency on a small context.

For end-user facing APIs under latency SLA, the calculation includes the cost of meeting that SLA. If long context's model latency pushes you past your 2-second budget and you have to over-provision for tail latency, that is a hidden cost. If RAG's retrieval hop costs you the SLA because of vector DB latency, the cost of RAG infra becomes "whatever it takes to meet the SLA."

Quality is similar. Long context on a 1M-token corpus gives the model more information to reason over. RAG's top-K retrieval can miss relevant context if search precision is low. If RAG's quality loss requires you to run a re-ranking model or a second pass with fallback to full context, that cost should be in the RAG side of the equation. If the quality difference matters for your use case but not your financial viability (e.g., an internal copilot that is nice to have but not mission-critical), quality cost is a product decision, not a FinOps one.

Decision matrix: long context, RAG, or hybrid

Pick long context if:

Pick RAG if:

Pick hybrid if:

The engineering cost often decides it

One variable the formula cannot quantify is engineering complexity. RAG requires building and maintaining retrieval infrastructure, fine-tuning search precision, handling edge cases where retrieval fails, and keeping embedding vectors in sync with your corpus. Long context requires nothing—you call the API with a big input. The effort to build and run each system over five years is not the same, and for some teams, the operational debt of maintaining RAG is the deciding factor to switch to long context, cost advantage or not.

The flip is equally true. Teams that already have mature RAG pipelines with monitoring, chunking strategies, and query optimization in place often find that the cost to migrate away and handle the quality loss is higher than the cost to keep running RAG. Do not flip approaches on a 5–10% cost difference if your team knows one system well and is unfamiliar with the other.

Building your own breakeven model

To use this framework on your actual workload:

  1. Measure your query volume per day and your average context size (full corpus for long-context model, retrieved subset for RAG model).
  2. Get your current embedding cost per query and your vector DB infrastructure cost per query. If you do not track the latter, divide your monthly vector DB bill by your queries per month.
  3. Measure your corpus churn rate: what fraction of your documents change per day? If less than 5%, caching strategies matter; if over 50%, assume churn invalidates any cache.
  4. Plug into the formula. Compute long-context cost and RAG cost per request. Multiply by daily query volume to get daily spend for each approach.
  5. Run the numbers for a 20% and 50% increase in query volume (common for scaling) to see which approach scales better.

Your break-even query volume is where RAG cost per request equals long-context cost per request at your actual numbers. If you are below it, long context is cheaper. If you are above it, RAG is cheaper. The width of the band where neither is obviously better is where hybrid and operational complexity should drive the decision.

Related

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research