noburn
Log inSign up
blog

How Retry Logic Turns Small LLM Errors Into Large Bills

Exponential backoff is standard practice for HTTP APIs. For LLM APIs it can be a billing disaster. Here is how retry patterns interact with token costs and what to do instead.

by updated 8 min read#llm-cost #retry-logic #error-handling

TL;DR: Backoff changes when you pay for a retry, not whether you pay. Nested retries multiply (three layers of three attempts each is 27 calls), and a streamed response that times out on the client can be billed twice. Classify errors before retrying, cap the total cost of retries rather than their count, and move non-urgent work to the batch API.

Prices updated November 6, 2026.

You deploy a chatbot powered by GPT-6.1 Sol. A user asks a question. The API returns a 429 (rate limit error). Your code retries. The second attempt succeeds. You log it, move on. One logical request turned into two API calls, but the overhead seems small — both are the same prompt, so the cost doubled from $0.10 to $0.20. That's fine, right?

That is fine until rate limits spike at midnight, and 10% of your requests hit 429. One in ten becomes two API calls. A 50K-call night becomes 55K calls, and at $0.10 a call a $5,000 night becomes $5,500. Your LLM bill went up 10% overnight, and your retry logic is invisible — the code looks normal, the error logs show expected 429s, but the cost multiplied. This is the subtle part: retries are not just a reliability pattern. They are a cost amplification pattern.

The problem compounds when retries are nested. An agent calls a tool. The tool makes an LLM call. The LLM call times out and retries. Exponential backoff on both the agent retry and the inner call means the cost of a single agent step can multiply by 5 or 10x before the agent ever sees a final result.

This guide walks through how retry logic interacts with token costs, the most expensive failure modes, and the architectural patterns that prevent runaway bills.

The hidden cost of exponential backoff

Exponential backoff is industry standard: wait 1 second, then 2, then 4, then 8, then 16. The theory is sound: if the API is overloaded, backing off reduces load. The flaw is that it does not change the token cost.

A request retried after 16 seconds costs exactly as much as one retried after 1 second. Backoff delay changes when you pay, not whether you pay. For a $0.10 call retried 5 times, the retries add $0.50 to the original $0.10 whether the backoff is 1 second or 16 seconds.

The real cost problem is frequency. If 10% of your traffic hits a transient error and every one of those calls makes 5 retries, you are not making 55K calls on a 50K night. You are making 75K (45K first-try successes plus 5K × 6 attempts), because each retry is a full round-trip: prompt tokens, completion tokens, full billing.

Consider a realistic scenario:

  • Normal night: 50K calls, all succeed first try. Cost: $5,000 (at $0.10/call average).
  • High-load night with retries: 50K calls, 10% hit rate limits and take 3 attempts each on average (the failed first try plus two retries).
    • 45K calls succeed first try: 45K attempts, $4,500
    • 5K calls hit 429: 5K × 3 attempts = 15K attempts, $1,500 (10K more than the 5K you planned for)
    • Total: 60K billed calls, $6,000 (a 20% increase over $5,000)

That 20% increase is not a mistake or a known risk. It is a silent leak in the cost model.

Why the standard approaches fail

Approach 1: Lower the retry count

The first instinct when the bill arrives is to drop retries from 5 to 3. This helps linearly and misses the structural problem: three attempts at each of three levels (agent, tool, LLM call) means 3 × 3 × 3 = 27 possible LLM calls for one request. Tuning one number in one layer does not address the multiplication.

Approach 2: Add longer backoff delays

The second instinct is to add backoff: if the API is overloaded, wait longer before retrying. Backoff delay reduces request rate, which is good for API health. But it does not reduce cost. A request retried after 16 seconds costs the same as one retried after 1 second. For rate-limit storms, longer backoff can actually help because fewer concurrent retries hit the API at the same time. But the per-request token bill is unchanged.

Approach 3: Retry only on timeout

The third instinct is to catch the timeout specifically and retry only on timeout. This is closer to correct, but it ignores the most expensive failure mode: a streaming request that completes generation on the server and then times out on the client. The server has generated and billed every output token. Your client saw a timeout, treated it as failure, and retried. The server generated and billed the full output a second time. Without an idempotency key, you are billed twice for output you used once.

Approach 4: Observability and alerts

The fourth instinct is to add a dashboard and alert on cost. Log the cost of each API call, visualize retries, and alert when cost per call exceeds a threshold. This is genuinely useful for debugging and every team should do it. But it is detection, not prevention. The dashboard tells you about the $1,000 overrun after the tokens are spent. The next-morning Slack alert is the system working as designed. The money is already gone.

The architecture that prevents runaway costs

Four changes, in order of impact.

1. Classify errors before retrying

The single most important rule: never retry a request whose failure is a property of the input rather than the transport. A 400 (malformed request), a 422 (validation error), a content-policy refusal, and a downstream validation failure will all reproduce on an identical prompt. Retrying them is wasted spend.

Retry only on:

  • Connection errors — network timeouts, socket resets
  • Proven incomplete requests — timeouts where you can prove the API did not complete the request
  • 429 — rate limit
  • 5xx — server error

Do not retry on:

  • 4xx (except 429) — client errors; retrying will not fix them
  • Content policy violations — retrying the exact same prompt will hit the same violation
  • Validation errors — downstream service rejected the input; retrying will not fix it
def should_retry(error_code: str, error_message: str) -> bool: """Classify errors; retry only transient failures.""" # Do not retry client errors if error_code in ['400', '401', '403', '404', '422']: return False # Do not retry policy violations or validation errors if any(phrase in error_message.lower() for phrase in ['content_policy', 'violat', 'invalid', 'malformed']): return False # Retry transient errors only if error_code in ['429', '500', '502', '503', '504']: return True # Retry timeout only if we can prove no tokens were consumed if error_code == 'timeout' and not was_billed(request_id): return True return False
# Usagefor attempt in range(max_retries): try: response = client.chat.completions.create(...) break except Exception as e: if not should_retry(e.status_code, str(e)): raise # Do not retry; propagate immediately if attempt < max_retries - 1: time.sleep(2 ** attempt) # Exponential backoff for transient errors

This single change — classify errors before retrying — can cut retry costs in half.

2. Use idempotency keys to prevent double-billing

The most expensive retry scenario is a streaming request that completes generation on the server but times out on the client before the full response arrives. Here is the timeline:

  1. Client sends request to API
  2. API generates full response (1000 tokens) and bills $0.10
  3. API starts streaming response back to client
  4. Network issue or client timeout occurs mid-stream (client received 600 tokens)
  5. Client sees timeout, assumes failure, and retries
  6. API generates the same response again (1000 tokens) and bills $0.10 again
  7. You have paid $0.20 for output you received once

Without an idempotency key, this happens silently. OpenAI, Anthropic, and other major providers support idempotency keys. Send the same key with a retry and the API returns the cached result without re-running the model.

import uuidfrom openai import OpenAI
client = OpenAI()
def call_with_retry(user_prompt: str, max_retries: int = 3) -> str: idempotency_key = str(uuid.uuid4()) # Generate once per logical request
 for attempt in range(max_retries): try: # Send idempotency key with every attempt response = client.chat.completions.create( model="gpt-6.1-sol", messages=[{"role": "user", "content": user_prompt}], extra_headers={"Idempotency-Key": idempotency_key} ) return response.choices[0].message.content except TimeoutError: if attempt < max_retries - 1: time.sleep(2 ** attempt) continue # Retry with same key raise
result = call_with_retry("Explain quantum computing")

With idempotency keys, retries return the cached first result. One request, one bill, regardless of retries.

3. Cap total retry cost, not retry count

Instead of retrying a fixed number of times (e.g., 5 retries), set a maximum total cost for all attempts combined. If a single call costs $0.50 and you allow 5 retries, one request can spend $3.00 (6 attempts). With a $1.50 budget for that request, you stop after the third attempt, however many retries the count would allow.

def call_with_cost_limit( call_fn, user_id: str, remaining_budget_cents: float, initial_call_cost_cents: float, max_retries: int = 5) -> Any: """Retry a call, but stop if total retry cost exceeds budget.""" spent = initial_call_cost_cents
 for attempt in range(max_retries): try: result, _cost = call_fn() # Call returns (result, cost_cents) return result except Exception as e: if not should_retry(e): raise
 # Estimate cost of next retry (usually similar to first call) estimated_next_cost = initial_call_cost_cents * 1.2 # Conservative estimate
 if spent + estimated_next_cost > remaining_budget_cents: raise BudgetExceeded( f"User {user_id} has ${spent/100:.2f} spent on retries; " f"next retry would cost ${estimated_next_cost/100:.2f}" )
 spent += estimated_next_cost time.sleep(2 ** attempt)
# Usagebudget_remaining_cents = 500 # User has $5 remaining this monthcall_with_cost_limit( call_fn=lambda: llm_api_call(), user_id="user_123", remaining_budget_cents=budget_remaining_cents, initial_call_cost_cents=50 # First attempt cost $0.50)

This ensures retries never spend more than the user has left.

4. Prefer batch API for non-urgent work

If the request does not need a response within seconds, batch API reduces cost by 50% and is retry-friendly because you are not polling a connection. You submit a batch, the API processes it at off-peak times, and you check for results later. If a job fails, retrying is cheap because you retry the entire batch, not individual calls.

Batch API is ideal for:

  • Document summarization (100s of documents)
  • Classification (large label sets)
  • Embedding generation (millions of vectors)
  • Scheduled report generation
from openai import OpenAI
client = OpenAI()
# Prepare batch requestsbatch_requests = [ { "custom_id": f"request-{i}", "method": "POST", "url": "/v1/chat/completions", "body": { "model": "gpt-6.1-sol", "messages": [{"role": "user", "content": f"Summarize: {doc}"}], "max_tokens": 500, } } for i, doc in enumerate(documents[:10000]) # Batch of 10K documents]
# Submit batch (50% discount on input/output tokens)batch = client.batches.create( input_file=batch_requests, # File format varies by provider)
# Check results laterstatus = client.batches.retrieve(batch.id)if status.status == "completed": results = client.batches.retrieve(batch.id).output_file_id

Batch API costs 50% of on-demand pricing. On GPT-6.1 Sol, a request with 50K input tokens costs $0.05 instead of $0.10 (50,000 × $1 per million batch vs. 50,000 × $2 per million standard). For bulk work, this is the cheapest option.

Worked example: the retry cost that goes unnoticed

Take a LangChain agent for customer support automation (an illustrative composite, not a customer). The agent calls a knowledge base tool, which calls the OpenAI API. Both have retry logic.

The knowledge base was slow one morning (database replication lag). The agent's tool call timed out. The agent retried. The tool was still slow. The tool's internal LLM call timed out. The LLM retried. Both with exponential backoff.

A normal query makes one LLM call at about $0.10. During the slow window, one customer query resulted in:

  • Agent attempt 1 → tool call → LLM call, 3 attempts = 3 LLM calls
  • Agent attempt 2 → tool call → LLM call, 3 attempts = 3 LLM calls
  • Agent attempt 3 → tool call → LLM call, 3 attempts = 3 LLM calls
  • Total: 9 LLM calls for one query

A $0.10 query cost $0.90. The customer got a helpful response. The cost was invisible in logs.

By end of the day: 5,000 customer queries, 1,000 of them during the slow window. That is 1,000 × 9 = 9,000 calls plus 4,000 × 1 = 4,000 calls, so 13,000 LLM calls instead of 5,000. Cost: $1,300 instead of $500. The retry logic cost an extra $800 that day, more than the rest of the traffic combined.

The fix was three-part:

  1. Classify errors — only retry timeouts and 429s, not all errors
  2. Add idempotency keys — prevent double-billing on retries
  3. Cap retry budget — stop retrying if cost exceeds remaining budget

After the fix, retries were rare and cheap.

Tools and integrations

Built-in retry handling

Cost-aware retry libraries

Where noburn fits

noburn prevents retry cost blowups by blocking retries before they fire. If a user has $10 remaining in their monthly budget, and a retry would cost $15, noburn blocks it. No retry, no cost.

It works by wrapping your LLM client:

from noburn import with_noburn_budgetfrom langchain.chat_models import ChatOpenAI
model = with_noburn_budget( ChatOpenAI(model="gpt-6.1-sol"), user_id="user_123", monthly_budget_cents=50000 # $500/month)
agent = create_react_agent(model, tools)# Retries that would exceed budget are blocked before the API call fires

The Free plan covers 10,000 recorded calls a month (current plans: pricing). Paid plans start at $19 a month. Documentation and SDKs are at noburn.dev/docs.

Key takeaways

  1. Exponential backoff reduces request rate but not token cost. Three attempts cost three times one, however long the backoff delays are.
  2. Nested retries (agent + tool + LLM) multiply costs exponentially. Three layers of three attempts can cause 27x amplification in the worst case.
  3. Classify errors before retrying. Retrying a 400 or content-policy error is wasted spend — it will fail the same way.
  4. Use idempotency keys to prevent double-billing on timeouts. One request = one bill, even if it retries 10 times.
  5. Cap retry cost, not retry count. Set a maximum total budget for all retries, not a fixed number of attempts.
  6. For bulk work, use batch API. 50% cost reduction with no retries needed.
Builds noburn. github.com/orvi2014