Hidden LLM Costs

Updated 12 July 2026

LLM spending isn't just per-token pricing. Infrastructure overhead, redundant calls, retry chains, and cache management add significant cost. Many teams track only API tokens but miss 20–50% of their true LLM spend. Learn what to measure.

• Hidden costs often exceed API costs by 20–50%
• Vector databases, embeddings, and retry logic are major contributors
• Cost-per-successful-response differs from cost-per-token
• Observability itself has a cost footprint

1. Vector Database & Embedding Infrastructure

RAG systems (Retrieval-Augmented Generation) require embedding generation, vector storage, and retrieval. This infrastructure cost is often invisible in per-token API bills.

Real scenario: A customer support chatbot with 100K monthly queries (avg 3K tokens per query). Embedding every query plus document-chunk reindexing runs ~5M embedding tokens/month — at OpenAI's text-embedding-3-small rate of $0.02 per 1M tokens, that's $0.10/month; even text-embedding-3-large ($0.13 per 1M) stays under $1. The embedding API is not the hidden cost here — the always-on vector database is: a managed tier for this workload typically runs $50–$100/month whether or not queries flow. Total: ~$50–$100/month of infrastructure on top of your LLM API bill, almost all of it vector DB hosting.

How to measure: Audit your cloud bill (AWS, GCP, Azure) for vector DB infrastructure. Add explicit embedding API costs from your provider dashboard. Sum these separately from LLM token costs.

2. Retry Chains & Fallback Models

When your primary LLM fails or hits rate limits, fallback logic automatically retries or switches to a cheaper model. Each retry consumes tokens without delivering a result to the user.

Example calculation: Your system makes 1M API calls/month to GPT-5.5 ($5M tokens in, $30M out = $155/month). If your fallback logic retries failed calls, and 5% fail and retry once, you're now making 1.05M calls. That's a 5% cost increase ($7.75/month extra just from retries).

The trap: If your fallback logic has a bug and retries aggressively on transient errors, failure rate can spike to 30–50%, multiplying your bill 2–4×. This often goes unnoticed for weeks.

How to measure: Log every API call with (call_id, model, attempt_number, success, reason_if_failed). Calculate: (total_tokens_across_all_attempts) / (successful_responses). Compare to your base per-token cost.

3. Observability & Logging Overhead

Tracing LLM calls (logging prompts, completions, latency) requires infrastructure: log aggregation, metrics storage, trace backends.

Real scenario: An LLM app logging every prompt + completion to Datadog. Each LLM call generates ~2–5 KB of structured logs, so 10M API calls/month produces 20–50 GB — at an assumed $1/GB ingestion, $20–$50/month. Log the full prompt and completion text (often 10–20 KB per call) and the same traffic is 100–200 GB, or $100–$200/month, before indexing and retention fees.

How to measure: Check your observability platform bill separately. If it's in your shared cloud account, segment it by cost center or tag.

4. Duplicate & Wasted Calls

Inefficient prompt engineering, debug/test calls left in production, and race conditions cause wasted LLM calls that don't deliver user value.

How to measure: Classify every LLM call: (user_facing / internal_debug / redundant / unused_feature). Run a week-long audit. Calculate waste rate = (debug + redundant + unused) / total_calls.

5. Cache Overhead

Prompt caching (Anthropic, OpenAI) offers 80–90% discounts on cache hits, but cache writes have a 1.25–2× write cost plus storage overhead.

Storage cost: The billing models differ, and that's the trap. Anthropic charges per write (1.25× the base input price for the default 5-minute TTL) with no ongoing storage fee — caching a 100K-token PDF on Opus 4.8 costs 100K × $6.25/M ≈ $0.63 to write, then ≈$0.05 per full cached re-read (100K × $0.50/M). Google's Gemini context caching instead bills storage per token-hour on top of discounted reads, so a long retention window accrues cost even with zero hits — check current per-model storage rates before caching large contexts for days. OpenAI's automatic prompt caching applies its cached-input discount with no separate write or storage fee.

How to measure: If using prompt caching, track: total_write_tokens_cached, cache_hit_rate, cache_hit_tokens. Calculate: (write_cost + (hit_tokens × hit_rate × cache_read_price)) vs. (total_tokens × non_cache_price). Most teams see 40–60% savings once hit rate exceeds 20%.

6. Multi-Model Orchestration Overhead

Routing queries to different models (GPT-5.5 for reasoning, Gemini Flash for simple classification) requires routing logic, classifier overhead, and slop.

Real scenario: 1M calls/month, routing 30% to GPT-5.5 and 70% to Gemini 2.5 Flash; a typical call is ~1.5K input / 500 output tokens. Per call that's ~$0.0225 on GPT-5.5 ((1,500 × $5/M) + (500 × $30/M)) vs ~$0.0017 on 2.5 Flash — so misrouting 5% of traffic (50,000 simple calls sent to GPT-5.5) wastes 50,000 × ~$0.021 ≈ $1,040/month. Add the classifier itself (~$70 at $0.00007/call) and the orchestration tax is ~$1,100/month — roughly 14% on top of the ~$7,900 baseline spend, dominated by misroutes rather than the router.

Cost Measurement Framework

Quick Audit Checklist

• Do you log every LLM call? (If not, you can't measure waste)
• Is vector DB cost tracked separately? (Many teams hide this in "database" line items)
• What is your current fallback failure rate? (Should be <2%; >5% indicates a bug)
• Is prompt caching enabled? (If not, you're leaving 40–60% savings on the table for repeated queries)
• Do you classify calls as user-facing vs. internal? (Helps spot debug logging waste)
• What is your observability cost as % of LLM cost? (Should be <5%; >20% suggests over-instrumentation)

Ready to find your hidden costs? FinOps LLM's cost audit digs into infrastructure, waste, and cache efficiency. Book a free audit and we'll quantify the real spend.

Back to research