Benchmarks
FinOps LLM benchmarking starts with cost per successful task, not raw token price. Model choice, cache hit rate, latency, retry behavior, and quality thresholds all change the real unit economics.
- Normalize provider invoices into comparable token buckets.
- Measure quality before and after routing or cache changes.
- Separate synchronous user paths from batchable offline work.
Why cost per token is the wrong benchmark
Per-token pricing is the number providers publish and the number that predicts your bill worst. A model half the price that needs two attempts to complete the task costs the same and adds latency. A cheaper model with a smaller context window forces you to send more retrieval chunks, so the token count rises to meet the discount. The comparable unit is cost per successful task, where "successful" is defined by your own eval before the comparison starts.
What a usable benchmark controls for
- Retries and fallbacks. Count every billable call the task produced, not the one the user saw.
- Cache hit rate. Prompt caching changes effective price by a large factor and varies wildly by workload shape; a benchmark run without it measures a system nobody operates.
- Context length in practice. Measure the p95, not the average — the tail is where the bill lives.
- Quality, held constant. Fix the acceptable quality floor first, then compare cost among the options that clear it. Comparing cost across models at different quality levels is not a benchmark, it is a preference.
How often should you re-run them?
Quarterly, and after any provider price change or new model tier. LLM benchmarks go stale faster than infrastructure benchmarks because all three inputs move independently: providers cut prices, release cheaper tiers, and change rate limits, while your own traffic mix shifts as the product grows. A routing decision made on last quarter's price list is usually still defensible and occasionally expensive.
Synchronous versus batchable work
Split the benchmark before you run it. User-facing paths are latency-bound and cannot use batch pricing; offline enrichment, evals, and index rebuilds can, often at half the cost, and they are frequently the larger share of tokens. Benchmarking them together produces an average that describes neither.