When to Use a Cheaper Model: A Decision Framework for Production Teams

The message lands in Slack on a Tuesday morning. Your data team ran their weekly cost analysis. The bill is 3x what it was two months ago. The feature that shipped last sprint—the one that runs Claude Sonnet 3.5 on every user request—is responsible for most of it.

You have two choices: yank the feature, or figure out which requests actually need the expensive model.

Most teams pick wrong. They either run everything on GPT-4o Mini because it's cheap, watch their quality degrade, and add it back to the incident log. Or they run everything on Sonnet because it's reliable, watch the bill grow indefinitely, and accept cost as the price of production stability.

The actual solution is neither. It's a decision framework—specific rules about which tasks need which models, written into your code, enforced before the API call fires.

The root cause: model choice is a business decision disguised as a technical one

Here's what happened to your architecture. In the prototype phase, you picked Sonnet 3.5 because it's strong across everything: reasoning, retrieval, code generation, summarization. A single model worked. Engineering velocity mattered more than cost.

Then the feature shipped. One thousand users. Ten thousand. The request volume stayed flat in your local testing and exploded in production because you didn't test at scale. Or you did test it, but the cost model was wrong—someone said "it'll be $0.50 per user per month" and nobody re-verified after launch.

Now here's the trap: you can't just switch half your traffic to GPT-4o Mini. Your reasoning tasks will fail more often. Your retrieved context will be garbled. Your error rate spikes, users complain, and you revert. You're back where you started, but now you've lost a week.

The mistake was treating "which model to use" as a single binary decision at the feature level. It's not. Each task within that feature has different requirements. Some need Sonnet. Most don't.

Why standard approaches fail

Load-shedding with latency or cost limits. You set a 10-second timeout or a $0.01-per-request budget, and if the call would exceed it, you return a degraded response. This works until your threshold is wrong—and it always is. A user hits the timeout. They get a blank result. They churn. Or you set the budget so loose that it still blows through your cost target.

Uniform model selection by feature. You run the entire chat feature on Sonnet, the entire search feature on Mini. The first time a Mini search fails on a complex query (nested compound subjects, ambiguous references), your ranking algorithm gets garbage results. You can't measure this easily because the user just gets worse search quality—they don't see an error. Your click-through rate drops 2%, your business metric goes down, and you don't know why.

Benchmarking on your test set, not your prod traffic. You run GPT-4o Mini and Sonnet 3.5 on your 100-question eval set. Mini scores 82%, Sonnet scores 94%. You assume the gap matters. But 92% of your prod requests are classification tasks (the gap is 1%), and 8% are reasoning tasks (the gap is 15%). The aggregate impact is much smaller than your eval suggested.

No per-request routing. You make the model choice at configuration time. The code looks like:

const MODEL = "claude-3-5-sonnet-20241022"; // set once, used everywhere
const response = await anthropic.messages.create({ model: MODEL, messages });

Every request pays Sonnet pricing whether it's a simple classification task or a genuinely hard reasoning problem, because the model choice was never a decision the code could make — it was a decision a person made once, months ago, before the traffic mix existed.

The correct architecture: route by task, not by feature

The fix is to make model selection a per-request decision based on what the request actually needs, not a static config value. That requires classifying the task before the call fires and mapping the classification to a model tier:

function selectModel(task) {
  switch (task.type) {
    case "classification":
    case "extraction":
    case "simple_summarization":
      return "claude-3-5-haiku-20241022";
    case "multi_step_reasoning":
    case "code_generation":
    case "complex_synthesis":
      return "claude-3-5-sonnet-20241022";
    default:
      return DEFAULT_MODEL; // fail toward the expensive, reliable option
  }
}

const model = selectModel(task);
const response = await anthropic.messages.create({ model, messages });

The classification step doesn't need to be another LLM call — for most features it's a rule based on which code path is calling the model (search re-ranking is always cheap; open-ended chat is always expensive), or a lightweight heuristic (input length, presence of a specific tool call, whether the task has a bounded output schema). Reserve an actual classifier call for cases where the task type genuinely can't be determined structurally, since that adds its own latency and cost.

Building the decision table from your own data

The framework in the previous section only works if the task-to-model mapping reflects your actual traffic, not a guess. Pull a sample of production requests, tag each with its task type, and run both models against the same sample to measure the real quality gap per category — not the aggregate gap from a generic eval set. A task category with a 1% quality gap between Haiku and Sonnet is a clear downgrade candidate regardless of what the aggregate eval says; a category with a 15% gap should stay on the expensive model even if it's a small share of total volume. This is the same audit that surfaces the routing rule from "no per-request routing" above — most teams find that 70-90% of requests fall into task categories where the cheaper model is statistically indistinguishable from the expensive one, and the remaining 10-30% is where the cost is actually justified.

Where noburn fits

The tools compared in this article handle observability, routing, or evaluation — all of which operate after the LLM call completes. noburn operates before it. It wraps your existing OpenAI, Anthropic, LangChain, and the Vercel AI SDK client, estimates the token cost of each call, and blocks it if the calling user or project has exceeded their budget. Nothing in this comparison does that at a self-serve price point.

Per-user metering lets you enforce separate limits per end-customer, and Stripe passthrough lets you bill them for their LLM usage without writing a billing layer yourself. The free tier covers 50,000 requests per month. Documentation and SDKs are at noburn.dev/docs.