Token budget implementation guide

Updated 12 July 2026 · first published 4 July 2026

Token budgets are the operational layer of LLM cost control. Attribution tells you where money goes; budgets prevent it from going places you do not want. Without budgets, a single misconfigured agent loop or a spike in user traffic can blow through a monthly allocation in hours. This guide covers how to implement four types of token budgets in production, with pseudocode for each pattern.

Budget types

Per-request limits

The simplest budget: cap every request with max_tokens. This is not optional - every production request should have an explicit output token limit. Without it, a verbose model response can consume unexpected tokens and inflate latency.

// Per-request budget enforcement
function callLLM(prompt, config):
    response = provider.complete(
        prompt: prompt,
        model: config.model,
        max_tokens: config.max_output_tokens,    // hard cap
        temperature: config.temperature
    )
    if response.usage.total_tokens > config.warn_threshold:
        log.warn("Request exceeded soft limit",
            tokens: response.usage.total_tokens,
            threshold: config.warn_threshold
        )
    return response

Per-team quotas

Team quotas require a shared counter that persists across requests. The pattern: check remaining budget before each call, reject or downgrade if the budget is exhausted.

// Team quota with Redis-backed counter
function checkTeamBudget(teamId, estimatedTokens):
    key = "budget:" + teamId + ":" + currentPeriod()
    remaining = redis.get(key) or getTeamQuota(teamId)

    if remaining < estimatedTokens:
        if remaining < estimatedTokens * 0.1:
            return DENY    // hard limit: reject request
        else:
            return DOWNGRADE  // soft limit: use cheaper model

    return ALLOW

function recordUsage(teamId, actualTokens):
    key = "budget:" + teamId + ":" + currentPeriod()
    redis.decrby(key, actualTokens)
    remaining = redis.get(key)
    if remaining < getTeamQuota(teamId) * 0.2:
        alert.quotaLow(teamId, remaining)

Per-time-period budgets

Time-period budgets wrap team quotas with a time window. The key difference: you need a grace period mechanism. When a team hits 80% of their monthly budget, send an alert. At 100%, allow a configurable grace period (e.g., 24 hours) before hard enforcement kicks in. This prevents a team from being blocked mid-task on the last day of the month.

// Time-period budget with grace period
function enforceBudget(teamId):
    usage = getUsageForPeriod(teamId, currentMonth())
    limit = getTeamMonthlyLimit(teamId)

    if usage < limit * 0.8:
        return ALLOW

    if usage < limit * 1.0:
        alert.budgetWarning(teamId, usage, limit)
        return ALLOW   // soft warning zone

    if usage < limit * 1.1 and withinGracePeriod(teamId):
        alert.budgetExceeded(teamId, usage, limit)
        return ALLOW   // grace period: allow overage

    return DENY  // hard stop

Per-workflow budgets

Multi-step agents and pipelines need budgets at two levels: per-step (prevent any single step from consuming too much) and per-run (prevent the entire pipeline from exceeding its allocation). Track both in the workflow context.

// Workflow budget tracker
class WorkflowBudget:
    constructor(maxPerStep, maxTotal):
        this.maxPerStep = maxPerStep
        this.maxTotal = maxTotal
        this.spent = 0

    function callStep(stepFn, prompt):
        if this.spent >= this.maxTotal:
            return fallbackResponse("Budget exceeded")

        response = callLLM(prompt, {
            max_tokens: min(this.maxPerStep,
                           this.maxTotal - this.spent)
        })

        this.spent += response.usage.total_tokens
        return response

Graceful degradation when budget is exceeded

Do not just return an error when a budget is hit. Downgrade gracefully. Switch to a cheaper model (GPT-4o-mini instead of GPT-4o), reduce context window size, skip optional processing steps, or return a partial result with a note that full processing requires budget approval. The user experience should degrade, not break.

// Degradation chain
function callWithDegradation(prompt, config):
    if checkBudget(config.teamId, FULL_MODEL):
        return callLLM(prompt, { model: config.primaryModel })

    if checkBudget(config.teamId, CHEAP_MODEL):
        log.info("Downgrading model for budget")
        return callLLM(prompt, { model: config.fallbackModel })

    if checkBudget(config.teamId, MINIMAL_TOKENS):
        truncated = truncateContext(prompt, 50%)
        return callLLM(truncated, {
            model: config.fallbackModel,
            max_tokens: 256
        })

    return cachedOrFallback(prompt)

Monitoring budget health

Track three metrics: burn rate (tokens consumed per hour vs. budget), forecast (at current rate, when will the budget be exhausted), and override count (how many times the grace period was used). If grace period usage exceeds 10% of total requests, the budget is set too low for the workload.

Related

• LLM budget governance - organizational policies behind budgets.
• Agent spend guardrails - budget patterns for autonomous agents.
• How to cap inference costs - provider-side cost controls.

Want this applied to your own LLM spend? FinOps LLM runs a free audit of your AI costs and shows where the savings are. Book free audit →

Back to research