RAG cost estimates that only account for the generation call miss the two line items that actually drive the bill: embedding every chunk at ingestion time, and stuffing retrieved context into the prompt at query time. Both scale with data volume and query pattern in ways a "$X per 1K tokens" napkin estimate doesn't capture.
Cost center 1: embedding generation
Every document chunk gets embedded once at ingestion, and re-embedded any time you change chunking strategy or switch embedding models. For a modest knowledge base — 10,000 documents split into 500-token chunks, roughly 40,000 chunks — a full re-embed at typical embedding pricing runs a few dollars, which sounds trivial until you're iterating on chunk size or re-indexing after every model upgrade. The real cost driver here isn't the per-call price, it's how often re-embedding happens: teams that treat their chunking strategy as a solved problem re-embed rarely; teams still tuning retrieval quality can trigger a full re-embed weekly, and the cost compounds with corpus growth.
Cost center 2: context window at retrieval time
This is the line item that dominates in production. A query that retrieves the top 8 chunks at 750 tokens each puts 6,000 tokens of retrieved context into the prompt before the model has generated a single token of answer. Add a system prompt and a few-shot block and a single query on GPT-4o-class pricing runs around $0.021 — and that's per query, multiplied by every user, every session, every follow-up question in a conversation that re-sends the same retrieved context each turn.
The naive fix — retrieve more chunks to improve recall — directly inflates this number. The three levers that actually control it:
1. Add a reranking step. A reranker lets you retrieve broadly for recall, then keep only what earns its place in the prompt. Cutting context from 6,000 to 2,500 tokens on the earlier example takes the query from $0.021 to about $0.010, a 52% reduction, with retrieval quality often improving because the model sees less noise.
2. Trim the fixed overhead.
The 600-token few-shot block and the 400-token system prompt are paid on every single query, forever. Move stable instructions into the model's prompt-caching mechanism where available, which discounts repeated prefix tokens, and audit whether the few-shot examples are still earning their cost after you have real traffic to fine-tune against.
3. Put a budget gate in front of the generation call.
This is the part the architecture diagrams leave out. Before the expensive generation call fires, estimate its token cost and check it against a budget for that user, tenant, or project. If the request would push them over, block it and return a graceful error instead of an answer. This is the only mechanism that bounds the tail.
Where noburn fits
The tools compared in this article handle observability, routing, or evaluation — all of which operate after the LLM call completes. noburn operates before it. It wraps your existing OpenAI, Anthropic, LangChain, and the Vercel AI SDK client, estimates the token cost of each call, and blocks it if the calling user or project has exceeded their budget. Nothing in this comparison does that at a self-serve price point.
Per-user metering lets you enforce separate limits per end-customer, and Stripe passthrough lets you bill them for their LLM usage without writing a billing layer yourself. The free tier covers 100 requests per month. Documentation and SDKs are at noburn.dev/docs.