AI cost optimization: six levers
Updated 1 August 2026
AI cost optimization is not about finding one magic setting. It is about stacking small, repeatable improvements across the lifecycle of a request. The six levers below are the ones we see produce measurable savings in production. Each has a quality tradeoff, a typical impact range, and a set of workloads where it works best.
The six levers at a glance
| Lever | Typical impact | Best for | Quality risk |
|---|---|---|---|
| Model routing | 30–50% | Mixed workloads with simple and complex requests | Low if confidence threshold is tuned |
| Prompt caching | 20–40% | Repeated prefixes, system prompts, documentation | Very low |
| Prompt compression | 15–30% | Long prompts with redundant context | Low if A/B tested |
| Batch & async pricing | 10–50% | Evals, enrichment, reporting, non-urgent jobs | Low; latency increases |
| Provider arbitrage | 20–35% | Tasks where multiple models meet quality bar | Medium; requires benchmarking |
| Fallback controls | 5–15% | Retry storms, outages, model degradation | Low |
1. Model routing
Model routing sends each request to the cheapest model that can handle it. A lightweight classifier scores request complexity and chooses a tier. Simple classification goes to a small model; hard reasoning goes to a frontier model.
The savings come from volume. Most production traffic is not frontier-hard. Routing even half of it to a cheaper model cuts the bill substantially. The key is measuring success rate by tier and adjusting the routing threshold so quality does not regress.
2. Prompt caching
Prompt caching reuses a previously computed prefix. If your prompt starts with system instructions, documentation, or few-shot examples that do not change, caching can reduce input cost by 50–90% on repeated calls.
Caching works best when the same prefix is reused multiple times within a short window. It does not help one-off queries or prompts where the entire context changes every time. Track cache-hit rate as a first-class metric.
3. Prompt compression
Prompt compression removes redundant context, deduplicates examples, and summarizes long inputs before sending them to the model. The goal is to send only the evidence the model needs for the specific task.
This lever is high leverage for applications that pass large documents or conversation history. Every removed input token is a direct cost reduction. Compression must be A/B tested because aggressive summarization can hurt quality.
4. Batch and async pricing
Most providers offer discounted pricing for asynchronous batch workloads. OpenAI's batch API offers up to 50% off for non-urgent jobs. The tradeoff is latency: results arrive hours later, not seconds.
Good candidates include evaluation runs, data enrichment, report generation, and reprocessing jobs. If a workload does not need an immediate response, batch pricing is often the fastest way to cut its cost.
5. Provider arbitrage
Provider arbitrage means sending the same task to whichever provider offers the best capability-per-dollar. For many tasks, models from DeepSeek, Gemini, or Anthropic can match OpenAI quality at a lower price.
This lever requires benchmarking on your own data, not relying on leaderboard scores. The cheapest model that meets your quality bar is the right model. Re-evaluate quarterly because pricing and capability change fast.
6. Fallback controls
Fallback controls limit the cost of failure. They include retry caps, circuit breakers, model-tier ceilings, and agent step limits. Without them, a single outage or buggy prompt can generate thousands of expensive retry calls.
This lever has a smaller direct savings range than routing or caching, but it prevents catastrophic spikes. It also protects the savings from the other five levers by stopping runaway spend.
How to prioritize the levers
Start with attribution. Break spend by feature, owner, model, provider, and workload type. Then rank by total spend and variance. Apply the lever with the highest expected return to the largest spenders first.
Always measure cost per successful outcome, not cost per token. A cheaper model that fails more often can cost more per completed task than an expensive model that succeeds once.
The optimization operating rhythm
Strong teams review AI cost weekly. They compare the largest drivers, pick one or two optimization candidates, test behind feature flags, and reconcile results against provider data. This turns cost reduction from a one-time project into a repeatable habit.
Related
- AI cost optimization — the strategic overview.
- Model routing — building a routing layer.
- Prompt caching explained — when caching pays off.
- Provider arbitrage — benchmarking cheaper alternatives.
- Agent spend guardrails — fallback controls for agents.
Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →
FAQ
What are the main levers for AI cost optimization?
The six main levers are model routing, prompt caching, prompt compression, batch and async pricing, provider arbitrage, and fallback controls. Each has a typical savings range of 5–50%.
Which AI cost optimization lever has the biggest impact?
Model routing usually has the biggest impact, with savings of 30–50%, because it sends each request to the cheapest model that meets the quality bar.
How do I prioritize AI cost optimization work?
Start with attribution. Rank features by total spend and variance. Then apply the lever with the highest return for each feature, measuring cost per successful outcome rather than cost per token.