Anthropic and OpenAI both cache the parts of your prompt that don't change between calls — a system prompt, a large document you're repeatedly asking questions about, a set of few-shot examples — and charge a fraction of the standard rate for the cached portion on every subsequent call within the cache's lifetime. For workloads with a large, stable prefix and a small, changing suffix (RAG over a fixed document set, an agent with a long system prompt, a chatbot with persistent context), the savings are substantial and require no model or quality tradeoff.
How caching actually works
Both providers cache at the token level based on a shared prefix: the beginning portion of your prompt, up to the point where it starts to differ from a previous call, gets served from cache instead of reprocessed. This means cache hits depend entirely on prompt structure — static content (system instructions, reference documents, tool definitions) needs to come first, and the part that changes per-request (the user's actual message) needs to come last. Put the changing content first and the cache never matches, because the "shared prefix" is empty from the first token.
On the first call you see cache_creation_input_tokens populated (you paid the write surcharge — caching costs slightly more than a normal call the first time, since the provider has to actually build the cache). On every call within the TTL you see cache_read_input_tokens, billed at roughly a tenth of the standard input rate. If cache_read_input_tokens stays at zero across repeated calls, your prefix is changing between requests and the cache never matches — this is the most common implementation bug, usually caused by including a timestamp, request ID, or other per-call variable near the start of the prompt instead of the end.
Structuring prompts to maximize cache hits
messages = [
{"role": "system", "content": LONG_STATIC_SYSTEM_PROMPT}, # cached
{"role": "user", "content": [
{"type": "text", "text": REFERENCE_DOCUMENT, "cache_control": {"type": "ephemeral"}}, # cached
{"type": "text", "text": user_question}, # not cached — changes every call
]},
]The pattern is the same across both providers even though the API mechanics differ slightly: static content declared and ordered first, variable content last, with the cache boundary marked explicitly (Anthropic's cache_control breakpoints; OpenAI's automatic prefix caching, which requires no explicit marker but still depends on prefix stability).
Extending cache lifetime
To extend the lifetime from the default 5 minutes to 1 hour, set the TTL explicitly:
{"type": "text", "text": REFERENCE_DOCUMENT, "cache_control": {"type": "ephemeral", "ttl": "1h"}}The tradeoff is cost: the 1-hour cache costs more to write than the 5-minute default, so it only pays off if your call pattern has gaps longer than 5 minutes between requests that share the same prefix. For a chatbot with continuous traffic, the default TTL refreshes on every call and never expires in practice; for a background job that runs every 20 minutes against the same reference document, the 1-hour TTL is the difference between a cache hit and a full-price rebuild each run.
Where the savings actually land
Caching only pays off proportional to how much of your token cost is in the static prefix versus the variable suffix. A RAG pipeline pasting a 10,000-token document ahead of a 50-token question sees most of its cost disappear on repeat calls. A chatbot with a 200-token system prompt and a 2,000-token user message sees almost nothing, because the expensive part of the request was never cacheable in the first place. Measure the actual token split in your traffic before assuming caching will meaningfully move your bill — it's a structural fit for some workloads and close to irrelevant for others.
Where noburn fits cache_creation_input_tokens populated (you paid the write surcharge). On every call within the TTL you see cache_read_input_tokens, billed at the 0.1x rate. If cache_read_input_tokens stays at zero across repeated calls, your prefix is changing and the cache never matches.
To extend the lifetime from the default 5 minutes to 1 hour, set the TTL explicitly:
Where noburn fits
The tools compared in this article handle observability, routing, or evaluation — all of which operate after the LLM call completes. noburn operates before it. It wraps your existing OpenAI, Anthropic, LangChain, and the Vercel AI SDK client, estimates the token cost of each call, and blocks it if the calling user or project has exceeded their budget. Nothing in this comparison does that at a self-serve price point.
Per-user metering lets you enforce separate limits per end-customer, and Stripe passthrough lets you bill them for their LLM usage without writing a billing layer yourself. The free tier covers 100 requests per month. Documentation and SDKs are at noburn.dev/docs.