The Real Cost of RAG Applications in Production
RAG architectures have two cost centers that naive estimates miss: embedding generation and large context windows at retrieval time. Here is what production RAG actually costs per query.
by Orviupdated 3 min read#rag #retrieval-augmented-generation #llm-cost
Prices updated November 6, 2026.
RAG cost estimates that only account for the generation call miss the two line items that actually drive the bill: embedding every chunk at ingestion time, and stuffing retrieved context into the prompt at query time. Both scale with data volume and query pattern in ways a "$X per 1K tokens" napkin estimate doesn't capture.
Cost center 1: embedding generation
Every document chunk gets embedded once at ingestion, and re-embedded any time you change chunking strategy or switch embedding models. For a modest knowledge base — 10,000 documents split into 500-token chunks, roughly 40,000 chunks — a full re-embed pushes about 20 million tokens through your embedding model (check your provider's current embedding rate), which sounds trivial until you're iterating on chunk size or re-indexing after every model upgrade. The real cost driver here isn't the per-call price, it's how often re-embedding happens: teams that treat their chunking strategy as a solved problem re-embed rarely; teams still tuning retrieval quality can trigger a full re-embed weekly, and the cost compounds with corpus growth.
Cost center 2: context window at retrieval time
This is the line item that dominates in production. A query that retrieves the top 8 chunks at 750 tokens each puts 6,000 tokens of retrieved context into the prompt before the model has generated a single token of answer. Add a 400-token system prompt and a 600-token few-shot block, and a single query with a 350-token answer on mid-tier pricing (GPT-6.1 Sol at $2/M input, $10/M output) runs 7,000 input tokens ($0.014) plus 350 output tokens ($0.0035), around $0.0175 — and that's per query, multiplied by every user, every session, every follow-up question in a conversation that re-sends the same retrieved context each turn.
The naive fix — retrieve more chunks to improve recall — directly inflates this number. The three levers that actually control it:
1. Add a reranking step. A reranker lets you retrieve broadly for recall, then keep only what earns its place in the prompt. Cutting context from 6,000 to 2,500 tokens on the earlier example drops input to 3,500 tokens and takes the query from $0.0175 to about $0.0105, a 40% reduction, with retrieval quality often improving because the model sees less noise.
2. Trim the fixed overhead.
The 600-token few-shot block and the 400-token system prompt are paid on every single query, forever. Move stable instructions into the model's prompt-caching mechanism where available, which discounts repeated prefix tokens: cached input on GPT-6.1 Sol and Claude Sonnet 5.5 costs $0.10/M instead of $2/M. Writing the cache isn't free, though: it costs 1.25x the input rate for the default TTL on both Claude and OpenAI's GPT-5.6-and-later models, and OpenAI's caching minimum on those models is 1,024 tokens. Then audit whether the few-shot examples are still earning their cost after you have real traffic to fine-tune against.
3. Put a budget gate in front of the generation call.
This is the part the architecture diagrams leave out. Before the expensive generation call fires, estimate its token cost and check it against a budget for that user, tenant, or project. If the request would push them over, block it and return a graceful error instead of an answer. This is the only mechanism that bounds the tail.
Where noburn fits
The tools discussed here handle observability, routing or evaluation, and all of them act once the LLM call has been made. noburn acts before it: one check() in your own code, placed before the provider call, returns allow or block against a project, per-user or per-run cap, so an over-budget call is never sent. It is not a proxy: your OpenAI or Anthropic client and its connection stay as they are. How the caps resolve, and the reason each one returns, is on the homepage.
Per-user caps and usage quotas keep one customer from spending another's budget, and Pro adds passthrough billing through Stripe. The Free plan covers 10,000 recorded calls a month with no card (current plans: pricing). Setup takes one install and one check: the quickstart, then per-user budgets and agent runs in the SDK reference.