In This Article
Key Takeaways
- On most workloads the bill is dominated by input tokens you resend every call — system prompts, tool definitions, retrieved context — not by the model's output.
- Prompt caching lets a reused prefix be read back at roughly 10% of the normal input price on all three major providers, but a cache write costs more than an ordinary read.
- The lever that decides whether caching pays off is the time-to-live (TTL). Anthropic's default dropped to 5 minutes in early 2026; OpenAI now retains caches up to 24 hours; Gemini charges an hourly storage fee for its explicit cache.
- A 1M-token window is available but not free or lossless — you pay per token every call, and accuracy degrades as the window fills. Retrieve small, don't stuff.
Every team that ships something on top of a large language model eventually has the same uncomfortable meeting: the demo worked, users came, and the API bill is now a real line item. The good news is that LLM spend is unusually predictable once you understand where it comes from. The bad news is that most of it hides in a place people don't look — the tokens you send into the model on every single call.
This guide covers the three levers that decide that bill in 2026: prompt caching, the reality of the 1M-token context window, and the cost math that ties them together. Every price here is drawn from the providers' own documentation, linked at the end, because caching pricing moved several times in the first half of 2026 and older write-ups are now wrong.
Why the bill is mostly input tokens
A chat request has two priced halves: the input (everything you send — system prompt, tool schemas, conversation history, retrieved documents, and the user's message) and the output (what the model writes back). Output tokens are priced higher per token, which fools people into optimizing them. But in most production apps the input is 10 to 50 times larger than the output, so the input usually dominates the total.
Consider an agent with a 15,000-token system prompt and tool set, plus a few thousand tokens of retrieved context, that answers with a 400-token reply. You are billing roughly 18,000 input tokens against 400 output tokens on every turn. Cut the output in half and you save 200 tokens; cache the input and you save thousands. That asymmetry is the whole reason prompt caching exists — and the reason it can cut a bill by 50 to 90 percent. If you have never measured your own split, run your prompt through our prompt token counter before you optimize anything; you cannot manage what you have not counted.
How prompt caching actually works
Caching is a prefix match. The provider hashes the exact bytes of your prompt from the start up to a marked breakpoint, and if a later request begins with those identical bytes, the already-computed portion is served from cache instead of re-processed. The single most important consequence: any change anywhere in the prefix — a timestamp in the system prompt, a reordered JSON key, a different tool in the list — invalidates everything after it. Stable content goes first; volatile content goes last.
The economics have two parts. A cache write (the first time a prefix is stored) costs more than a normal input token, because the provider does extra work to store it. A cache read (every subsequent hit) costs far less — about a tenth of the normal input price. So caching is a bet: you pay a small premium once to make many later requests cheap. If the later requests never come, you lose the bet.
Caching is not a free discount you flip on. It's a prefix contract: keep the front of your prompt byte-for-byte identical, reuse it often, and the provider rewards you. Break the prefix, or reuse it rarely, and you pay to store a cache nobody reads.
What the three providers charge
The numbers below are current provider pricing as of mid-2026. They are structurally similar — roughly a 90% discount on cached reads — but the write cost and cache lifetime differ in ways that change your architecture.
Prompt caching, compared (mid-2026)
| Provider | How you enable it | Cache write | Cache read | Lifetime (TTL) |
|---|---|---|---|---|
| Anthropic (Claude) | Explicit breakpoint, or automatic | 1.25× input (5-min) / 2× (1-hour) | ~0.1× input | 5 min or 1 hour |
| OpenAI | Automatic on prefixes ≥ 1,024 tokens | 1.25× on newer models | ~0.1× input | Up to 24 hours (default since May 2026) |
| Google (Gemini) | Implicit (default) or explicit cache | Small write fee (explicit) | ~0.1× input | You set a TTL + hourly storage fee |
Three practical differences stand out. First, Anthropic is explicit and short-lived by default — you mark a breakpoint, and the default cache lives five minutes unless you pay extra for the one-hour tier. Second, OpenAI is automatic and now long-lived — prefixes over 1,024 tokens are cached with no code change, and since May 2026 the default retention runs up to 24 hours. Third, Gemini splits the difference with an implicit cache that just works and an explicit cache you control, the latter carrying an hourly storage fee whether or not anyone reads it. Match the tool to your traffic shape, and to compare real dollar figures across models use our LLM price calculator.
A worked example
Put numbers on it. Take a support assistant with a 20,000-token cached prefix (system prompt plus a knowledge pack), a 2,000-token user turn, and a 500-token answer, serving 10,000 requests a day. Use an illustrative rate of $5.00 per million input tokens, $0.50 per million for cached reads (the ~10% figure above), and $15.00 per million output.
The 20,000-token prefix, with and without caching
Uncached: each request sends 22,000 input tokens. That's 220 million input tokens/day × $5/M = $1,100/day in input alone.
Cached: the 20,000-token prefix reads back at $0.50/M and only the 2,000-token turn pays full price. That's 200M cached tokens × $0.50/M ($100) + 20M fresh tokens × $5/M ($100) = ~$200/day.
Same model, same answers, roughly an 80% cut to the input bill — because the expensive part was the 20,000 tokens you were resending 10,000 times a day for no new information.
The output cost (500 tokens × 10,000 × $15/M = $75/day) is unchanged and, notice, small next to the input. That is the shape of nearly every real workload: the win is on the input, and caching is the biggest single lever you have on it.
The TTL trap
The worked example assumed continuous traffic, so the cache almost never expired and write costs were negligible. Real traffic is bursty, and that is where TTL bites. If your requests arrive further apart than the cache's lifetime, the prefix expires between calls and you pay the write premium again on the next request. On a 5-minute TTL, a workload with a request every ten minutes never hits the cache at all — it only ever writes.
This is not hypothetical. Anthropic's effective default cache lifetime shortened to five minutes in early 2026, and teams that had budgeted around the old one-hour behavior saw effective costs rise 30 to 60 percent until they adjusted — either by pre-warming the cache, moving to the paid one-hour tier, or batching requests to arrive within the window. The rule of thumb: with a five-minute TTL you break even at roughly two reads per write; with a one-hour TTL the doubled write premium means you need at least three. Below that, caching costs you money. For the mechanics of keeping a prefix warm, see our companion guide, prompt caching explained.
The 1M-token context reality
The flagship models from all three providers now advertise a 1M-token context window, and it is tempting to treat that as "just send everything." Two facts push back. First, you pay for every token you send, on every call — a 1M-token prompt is a 1M-token bill each time, cached or not. Second, bigger is not more accurate. Independent testing by Chroma across 18 frontier models found that all of them degrade as the input grows — a phenomenon now widely called "context rot" — and an older Stanford study found accuracy on a retrieval task falling from roughly 70–75% down to 55–60% once about 20 documents were packed in, with the model losing information stranded in the middle.
The takeaway is not "avoid large windows" — they are genuinely useful for a single big document you must reason over whole. It is that the window is a budget, not a target. Retrieving the smallest high-signal slice for the question at hand usually beats stuffing the window, on both cost and quality. If you want to see how different models' windows compare token-for-token, our guide to 1M-token context windows lays out where the big window earns its keep and where it quietly wastes money.
A practical checklist
- Measure your input/output split first. If input dominates — it usually does — caching is your highest-leverage move.
- Freeze the front of your prompt. No timestamps, no per-request IDs, no unsorted JSON, no shuffling tools. The prefix must be byte-identical to cache.
- Match TTL to traffic. Continuous traffic loves the short, cheap cache; bursty traffic needs a longer TTL or a pre-warm, or caching will cost more than it saves.
- Verify cache hits. Every provider reports cached-token counts in the response usage. If that number is zero across repeated calls, a silent invalidator is breaking your prefix.
- Retrieve, don't stuff. Treat the context window as a budget. A smaller, better-curated prompt is usually cheaper and more accurate.
None of this requires a rewrite. It requires knowing which tokens repeat, keeping them stable, and letting the provider's cache do the rest. Do that and the uncomfortable billing meeting turns into a footnote.
Sources: Anthropic prompt caching docs; OpenAI prompt caching and OpenAI caching guide; Google Gemini context caching overview; Chroma "context rot" evaluation; Liu et al., "Lost in the Middle" (Stanford). Illustrative dollar figures use round rates for clarity and vary by model and plan — confirm current pricing on each provider's page before you budget.
Common questions
Does prompt caching make my API calls cheaper? Yes, when you reuse a large, stable prefix across many requests within the cache's lifetime. Cached tokens read back at roughly 10% of the normal input price. The two things that can flip it against you are the write premium and a short TTL — if requests arrive further apart than the cache lives, you keep paying to re-write it.
Is a 1M-token context window worth using? For a single large document you must reason over whole, yes. As a default "send everything" habit, usually not — you pay for every token each call, and accuracy degrades as the window fills. Retrieve the smallest high-signal slice instead.
What's the difference between OpenAI's and Anthropic's caching? OpenAI's is automatic on prefixes over ~1,024 tokens and now retains caches up to 24 hours by default. Anthropic's is explicit (you mark a breakpoint) and defaults to a 5-minute lifetime, with a paid 1-hour tier. Gemini offers both an implicit cache and an explicit one with an hourly storage fee.