Why does a long conversation cost more than the token count suggests?

Because context is re-sent. Each turn includes the whole conversation so far as input, so a document loaded at the start is billed again on every subsequent turn. Cost grows roughly with the square of conversation length, not linearly.

What is context utilisation?

Tokens sent versus tokens the model actually needed to answer. It is the nearest thing to a waste metric in LLM cost work, and it is what tells you whether you are paying to re-send material that never influences the output.

What is the highest-return fix?

Prompt caching on the stable prefix. If the system prompt and loaded documents do not change during a conversation, caching turns the largest repeated charge into a fraction of itself with no change to output quality.

Quick answer: A context window is advertised as capacity: 200,000 tokens, a million tokens, room for whatever you need. Capacity is not the billing unit. You are charged for what you send, and in a multi-turn...

Paying for the context window twice

Updated September 1, 2026 · first published September 1, 2026

A context window is advertised as capacity: 200,000 tokens, a million tokens, room for whatever you need. Capacity is not the billing unit. You are charged for what you send, and in a multi-turn conversation you send most of it again every single turn.

That is the second payment. A 30,000-token document loaded at the start of a twenty-turn conversation is not 30,000 input tokens. It is closer to 600,000, because turn two re-sends it, and so does turn twenty.

The arithmetic nobody runs

Cost scales with the square of conversation length, not linearly. Doubling the number of turns roughly quadruples the input tokens, assuming context keeps accumulating. Teams that model "cost per message" and multiply by message count under-forecast long sessions badly, and the error grows with exactly the users you most want to keep.

There is often a third charge on top: many providers price requests above a long-context threshold at a premium rate. One extra attachment can push a request across that line and reprice the entire call, input and output, not just the tokens that crossed it.

What to measure

Track context utilisation: tokens sent versus tokens the model actually needed. It is the closest thing to a waste metric this domain has. Track input tokens per conversation, not per request, and watch the p99 — that is where the sessions live that cost twenty times the median.

Alert on requests crossing the long-context price threshold. It is a step change, not a gradient, and it will not look like anything in a daily total.

Three fixes, cheapest first

Cache the stable prefix. If the system prompt and loaded documents do not change during a conversation, prompt caching turns the repeated cost into a fraction of it. This is the single highest-return change for any multi-turn product.

Trim history deliberately. Older turns rarely earn their re-send price. Summarise the earlier part of a long conversation and drop the raw turns; you keep the thread and stop paying full freight for it.

Send the excerpt, not the corpus. Retrieval exists for exactly this. The window being large enough to hold a whole document is not an argument for putting a whole document in it.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research