What is retry tax in LLM systems?

Retry tax is the spend caused by retries and fallbacks divided by total inference spend, measured by workflow and failure reason.

How should teams control agent retry loops?

Set per-request token, per-workflow dollar, attempt-count, and wall-clock limits, with soft routing and hard stop behavior.

Should all retries be disabled?

No. Classify failures first; retry transient provider errors when recovery is valuable, but stop deterministic validation and context errors.

Quick answer: A retry is not free resilience. It is another model call, and an agent can turn one transient error into five paid attempts before anyone sees a failure. The first control is to measure retry tax:...

The retry loop that quietly adds to inference spend

Updated August 27, 2026 · first published August 27, 2026

A retry is not free resilience. It is another model call, and an agent can turn one transient error into five paid attempts before anyone sees a failure. The first control is to measure retry tax: retry and fallback spend divided by total inference spend, segmented by workflow and failure reason.

Count the whole attempt tree

Log the parent request, attempt number, provider, model, tool call, latency, error class, tokens, and final outcome. A request ID alone is insufficient when a tool call triggers a nested model request. Record parent-child relationships so one user task can be reconstructed as a tree.

Separate useful from wasteful retries

Retrying a rate limit may recover a valuable task. Retrying a validation error with the same prompt usually buys another identical failure. A provider timeout, malformed tool input, context overflow, and quality rejection need different policies. Assign each failure class a maximum attempt count and an escalation path.

Set budgets at the loop boundary

Use a per-request token ceiling, a per-workflow dollar ceiling, and a maximum wall-clock duration. A soft ceiling can route the next attempt to a smaller model; a hard ceiling stops the loop and returns a recoverable state. Do not rely on a monthly alert to control a request that can spend its budget in seconds.

Review the tax with quality

A low retry rate is not automatically good if the first attempt fails more often. Track successful-task cost, completion rate, latency, and retry tax together. Fix the largest dollars first: duplicate tool calls, oversized context on retries, fallback chains that repeat the same capability, or validators that reject answers without changing the prompt.

The durable fix is not “disable retries.” It is making every extra attempt visible, bounded, and justified by a measurable improvement in outcome.

Agent spend guardrails
LLM cost monitoring
Cost per successful task

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research