AI cost optimization checklist

AI cost optimization is not a single project. It is a sequence of repeatable steps. This checklist covers the most reliable levers for 2026, ordered by impact and risk.

Before you optimize

• Reconcile to the invoice. Gateway logs under-count. Use the provider invoice as ground truth.
• Find the top five endpoints. ~80% of cost usually lives on 3–5 endpoints. Ignore the long tail at first.
• Define quality baselines. Cost savings that hurt quality are not savings. Agree on the metrics that matter before testing.

The optimization levers

1. Prompt caching — 50–90% off cached tokens, lowest implementation effort.
2. Model routing — send easy queries to cheaper models, hard queries to frontier models. 30–50% reduction.
3. Semantic caching — reuse similar responses for RAG and support flows. 20–40% reduction on cache-friendly endpoints.
4. Prompt compression — strip redundancy and move static context to system prompts. 10–25% reduction.
5. Batch routing — move non-urgent work to batch APIs. Flat 50% off eligible volume.
6. Provider arbitrage — use equivalent models priced lower on other providers. 30–60% reduction when A/Bs pass.

Governance checks

• Set per-request and per-user cost caps.
• Add alerts for budget variance and anomalous spend.
• Re-run A/B tests monthly; models and prices change.
• Document which optimizations are live and their measured savings.

For a deeper explanation of each lever, see AI cost optimization.

What order should you run these in?

Sequence by effort-to-savings, not by size of the prize. A realistic first 90 days:

1. Weeks 1-2 — reconcile to the invoice, tag the top five endpoints, and set a quality baseline. No optimization yet; you are building the ability to tell whether anything worked.
2. Weeks 3-4 — turn on prompt caching. It is the only lever on this list that needs no quality testing, because the cached tokens are byte-identical to what you were already sending.
3. Weeks 5-8 — move eligible asynchronous work to batch, then A/B model routing on one endpoint. Both are reversible in a deploy.
4. Weeks 9-12 — semantic caching and provider arbitrage, which change what the user sees and therefore need the quality baseline you built in week 1.

How much should you expect to save?

The levers are not additive. Routing 40% of traffic to a cheaper model shrinks the base that caching then applies to. A stack of caching, routing, and batch that looks like 90% on paper typically lands between 40% and 60% in production. Measure each lever against the invoice after it ships rather than summing the vendor claims.

When should you stop optimizing?

When the next lever costs more engineering time than it saves in a year, or when quality regressions start appearing in your evals. Both are worth writing down in advance. Re-open the checklist quarterly instead — provider prices, model tiers, and your own traffic mix all move enough that the answer changes.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research