Prompt Caching §

Pattern: A named solution to a recurring problem.

Pin the unchanging part of your prompt at the front so the provider can reuse its computed state, and pay a fraction of the cost on every reuse.

Also known as: Context Caching (Google), Implicit Caching, Explicit Caching, Automatic Caching

Understand This First §

Context §

Agentic workloads have a peculiar shape. The same long preamble (an instruction file, a tool catalog, retrieved documents, a conversation transcript) gets sent to the model again and again, with only the last few turns changing between calls. The first call to the provider sees a 50,000-token prompt. The next call sees the same 50,000 tokens plus 200 new ones. Without help, the provider charges full price for the whole thing every time.

Prompt caching is the help. The provider remembers the model’s internal state for a prefix it has seen before, and on the next call, if the new prompt starts with that same prefix byte-for-byte, the provider skips the recomputation and bills the cached portion at a steep discount. OpenAI, Anthropic, Google, and the cross-provider routing layers that wrap them all offer it, though each exposes different controls.

For agent builders, prompt caching is the lever that turns long-context workflows from “we cannot afford to ship this” into “this is fine.” As of October 2026, a cache hit on the major providers bills the reused prefix at about a tenth of the normal input price, and at a twentieth or less on some current models. ProjectDiscovery’s published case study cut their LLM bill 59% by adopting it across their pipeline.

Problem §

How do you afford to send a long, mostly-stable prompt on every turn of an agent loop, when the per-token cost of input scales linearly with prompt length?

The naive math is brutal. A 50,000-token system prompt at $3 per million input tokens costs 15 cents per call. A Ralph Wiggum Loop calling 50 times a day at that price costs $7.50 per day per agent, before any output tokens, before any tool turns, before any retries. Multiply by a fleet of agents and the bill is real money. The whole prompt is also redundant: the first 49,800 tokens are identical to last call’s first 49,800. Paying full price to recompute the same KV-cache (the key/value tensors a transformer holds in memory while it attends to the prompt) that the provider just discarded is an avoidable tax.

Forces §

Solution §

Architect the prompt as stable-prefix-first, variable-suffix-last, and let the provider cache the prefix. The prefix should be everything that does not change between calls in this session: the system prompt, the instruction file, the tool catalog, retrieved documents that survive across turns, and the part of the conversation transcript that is now fixed history. The suffix is whatever changed since last call: the new user turn, the new tool result, the latest streaming output.

Three implementation styles are common. They are defined by mechanism, not by vendor: one provider can offer more than one style, and which one you get often depends on the model.

Implicit caching. The provider hashes the prompt prefix and matches it against recent requests with no annotation from you. OpenAI’s models before GPT-5.6 work only this way. Gemini 2.5 and newer cache implicitly by default, and Gemini 4 Argon offers nothing else. If the prompt clears the provider’s minimum length and starts with a prefix the provider has seen recently, you get the discount. OpenAI’s prompt_cache_key lets you hint which requests share a prefix, which raises hit rates on these models without guaranteeing them. The price of zero configuration is zero control: the cache is opaque, and you cannot force a hit.

Explicit caching. You mark where the cacheable prefix ends, and the provider caches at exactly that point. Anthropic takes cache_control on up to four content blocks per request. OpenAI added the same idea for GPT-5.6 and later as prompt_cache_breakpoint, with an option to cache only at the breakpoints you choose. Google’s version is a named CachedContent resource you create and reference, available on Gemini 2.5 and 3.x models but not on Gemini 4 Argon. Explicit caching is the right choice when you know which prefix is hot and want to decide where the cache boundary falls.

Anthropic also offers an automatic middle ground. Put one cache_control field at the top level of the request, and the API places the breakpoint on the last cacheable block and moves it forward as the conversation grows. That fits an agent loop whose history grows only at the tail.

The economics are easiest to compare as multipliers of the normal input price. As of October 2026, a cache read costs about 0.1x on most current models from all three vendors. It drops to 0.05x on some flagship models (Claude Opus 5.5 and Sonnet 5.5, GPT-6.1 Sol, Gemini 4 Argon) and to 0.025x on Claude Fable 5.1 and Mythos 5.1. Writes cost extra where you control the cache: 1.25x for Anthropic’s five-minute cache and OpenAI’s default 30-minute one, 2x for Anthropic’s one-hour cache, and 2.5x for OpenAI’s 12-hour option. OpenAI’s implicit-only models charge nothing for the write, and Google bills explicit caches an hourly storage fee instead.

Minimum cacheable lengths run from 512 tokens on current Claude models to 4,096 on current Gemini models, and on OpenAI’s older models the minimum shifts with tools, images, and reasoning settings. Every one of these figures moves, so check the vendor’s pricing page before you build a cost model on them.

Cross-provider abstractions (LiteLLM, OpenRouter). Both expose a single caching surface that maps to whichever underlying provider is in use. The semantics flatten to the lowest common denominator (you give up some Anthropic-specific TTL controls when going through OpenRouter, for example), but you get to write the agent once and switch providers without rewriting the cache integration.

The cross-cutting discipline is the same in all three: stable parts first, never reorder, never edit in place. If a fact in the instruction file becomes wrong, the temptation is to fix it in place. Don’t, mid-session. That change invalidates every byte downstream, and you’ll pay full price on the next call. Either let the session finish on the stale fact, or accept the cache miss as the cost of correctness. The same rule covers configuration: choose the tool set and reasoning effort at session start and leave them alone, and push exploratory side work into a subagent, whose churning context never touches the parent’s prefix.

Tip: Order your prompt by stability. Put the stuff that never changes (system prompt, role description) first. Then the stuff that changes per session (project instruction file, retrieved documents). Then the stuff that changes per turn (conversation history, current user message). The earlier in the prompt a token is, the more cache hits it earns over the session’s lifetime.

How It Plays Out §

A team runs a coding agent with a 30,000-token CLAUDE.md and a tool catalog of about 8,000 tokens. Every turn ships those 38,000 tokens plus the conversation so far. Without caching, the bill works out to about $0.11 per turn just on input, and a typical session has 40 turns. They switch on Anthropic’s cache_control with a breakpoint after the tool catalog. The first turn pays the 25% write premium on the prefix. Every subsequent turn within the 5-minute TTL bills the prefix at 10% of normal, around $0.011 instead of $0.114. The session that used to cost $4.56 now costs around $0.50. They extend the TTL to 1 hour for the long-running agents that idle between user turns, and the savings compound.

A multi-tenant RAG system runs hundreds of concurrent users, each with a different retrieved document set. The naive shape (system prompt, then user-specific documents, then user query) gets a cache hit only on the system prompt because every user’s document set differs. The team restructures: system prompt first, then a stable user-tier description (“free” / “pro” / “enterprise”), then the documents, then the query. The first two segments cache cleanly across all users in a tier. The documents cache per-user within their session. The query never caches. Total cost drops 40%, and latency on cached prefixes drops more than half.

A developer building a long-form research agent notices that every turn of the agent’s reflection loop is sending the same 60-page paper as context. The paper hasn’t changed; only the agent’s question about it has. Switching to explicit caching with a breakpoint after the paper ends turns the per-turn cost from prohibitive to nearly free. The 1-hour TTL covers a typical research session end-to-end, so the only full-price call is the first one.

Warning: The most expensive cache is the one that never hits. Sources of silent invalidation include a timestamp injected into the system prompt (“today is 2026-04-27”), a non-deterministic tool catalog ordering, and any pass that rewrites earlier turns of the conversation to “clean up” history. Each new assistant turn appended to the end is fine; rewriting old turns is what kills the cache. Settings changes do the same damage while leaving the text untouched: a harness that adds a tool or raises reasoning effort halfway through a session can discard the cached history. The arXiv paper “Don’t Break the Cache” documents how easy it is to inadvertently miss the cache by reordering equivalent content, especially in long-horizon agentic loops.

Consequences §

The wins are real and measured. Long-context agentic workflows that would otherwise be uneconomic become routine. Latency drops because the provider skips recomputation. Anthropic’s current documentation describes the speedup without quoting a percentage; on conversational agents it shows up as the response starting visibly faster after the first turn warms the cache. Costs drop in lockstep with hit rates: an agent that consistently hits the cache pays a tenth or less of the normal rate on each read of its prefix, plus the occasional write premium.

The cost is architectural discipline. Once a prompt is cached, every byte upstream of any change is a sunk cost. This forces a clear separation between what’s stable and what’s variable, and it punishes mid-session edits to anything early in the prompt. Some teams find this a useful constraint: it nudges them toward keeping configuration in stable files and accumulating volatile state outside the prompt entirely, which is the externalized state discipline. Other teams find it a footgun: every refactor of the prompt template invalidates the cache for every running agent.

A second cost is operational: you have to monitor hit rates. Cached input is billed and reported separately from uncached input, and the ratio is the lever you actually care about. The counters are in every response: Anthropic reports cache_read_input_tokens and cache_creation_input_tokens, OpenAI reports cached_tokens and cache_write_tokens under usage.input_tokens_details, and Google’s SDKs report total_cached_tokens. A hit rate above 80% on a long prefix means the architecture is working. A hit rate near zero means something is invalidating silently (a timestamp, a non-deterministic ordering, an over-eager template change), and you’ll see it on the bill before you see it in code review.

A third is provider lock-in pressure. Each provider’s exact semantics differ: retention windows, breakpoint placement rules, minimum cacheable lengths, write charges, and discount tiers all vary, sometimes between two models from the same vendor. A workload tuned around Anthropic’s one-hour cache and hand-placed breakpoints won’t see the same economics on an implicit-only model with a different minimum length and retention window, and switching providers without recalibrating the prompt structure costs more than it saves. Cross-provider abstractions help but always at the cost of the lowest-common-denominator feature set.

Above all, remember that caching is a cost lever, not a quality lever. It does not extend the context window, does not slow context rot, and does not improve the model’s reasoning on the cached content. A long, stale, repetitive context cached is exactly as bad for output quality as the same context uncached, just much cheaper to keep shipping. Use compaction when the prompt is too long for the model to reason over, and use prompt caching when the prompt is the right length but is repeated across many calls. They solve different problems and compose well.

Related Articles §

Sources §

The mechanism is a direct application of decades of work on transformer KV-cache reuse, applied to the inference-API setting. What’s new is the commercial productization: providers exposing the cache as a first-class billing tier with developer-controlled breakpoints.

Anthropic’s prompt caching feature, launched in 2024 and stabilized through 2025, established the explicit cache_control breakpoint model that other providers have since adopted variants of. The Anthropic documentation is the canonical reference for the explicit-caching shape, and it now also documents the automatic top-level mode and a table of which request-setting changes invalidate which parts of the cache.

OpenAI introduced automatic prompt caching in 2024, prioritizing zero-configuration adoption over caller control, and its pre-GPT-5.6 models remain the reference case for implicit caching. The GPT-5.6 generation added explicit breakpoints and a cache-write charge, one of the variants of Anthropic’s breakpoint model mentioned above. OpenAI’s current documentation quotes cached-input discounts of up to 95%.

Google’s context caching, available through Gemini, splits the surface into implicit and explicit modes. The explicit CachedContent resource model is closer to a named cache entry than to inline breakpoints, which is a different ergonomic choice from Anthropic’s but the same underlying mechanism. Google’s documentation limits explicit caching to models before Gemini 4 Argon, which caches implicitly only.

The vendor modes, multipliers, minimum lengths, and retention windows in this article were checked against the Anthropic, OpenAI, and Google caching documentation and Google’s Gemini API pricing page in October 2026.

The arXiv paper Don’t Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks provides an academic evaluation of cache stability under exactly the workload pattern this article addresses. Its central finding, that small changes in prompt construction order produce large swings in cache hit rate, is the failure mode every production team rediscovers.

The cross-provider abstractions LiteLLM and OpenRouter document the lowest-common-denominator caching surface across providers and are the most concise inventory of which provider supports which feature.

Further Reading §