Prompt Caching: How to Reduce Your Token Costs by Up to 90%

Prompt caching stores the computation already performed on the stable part of a prompt (system prompt, documents, instructions) to avoid reprocessing it with every API call.
Observed gains range from 50% (OpenAI, automatic discount) to 90% on cached tokens (Anthropic), with latency reductions of up to 80%.
The mechanism relies on exact prefix matching: any slight change at the start of the prompt (extra space, different JSON key order, modified tool definition) invalidates the cache.
OpenAI enables it automatically for prompts over 1024 tokens; Anthropic requires an explicit cache_control parameter; Google Gemini charges for cache storage in addition to reads.
The universal structuring rule: static content at the top of the prompt, variable content (user query) at the end.

Prompt caching is a feature offered by model providers (OpenAI, Anthropic, Google) that allows the reuse of internal computations performed on the unchanged portion of a prompt, rather than reprocessing it for every request. In practical terms, if your application sends the same system prompt, the same instructions, or the same reference document with every API call, the model can “remember” that portion and skip directly to processing the variable part. The result: API bills cut by up to half, latency reduced by 50% to 80%, and zero loss in response quality. For a startup making thousands of calls per day with a 2,000-token system prompt, this single optimization can save hundreds of dollars every month without changing the product’s behavior.

The Technical Mechanism Behind Prompt Caching

To understand why prompt caching works, we have to look at how an LLM processes a request. Before generating a single response token, the model must first “read” the entire input prompt and build an internal representation for each token called a key-value (KV) cache, used by the transformer’s attention layers. This stage, known as prefill, is computationally expensive and proportional to the length of the prompt.

Prompt caching leverages a simple observation: in the vast majority of real-world applications, a large portion of the prompt never changes from one call to another. A chatbot’s system prompt, a coding agent’s instructions, tool-calling schemas, or a reference document injected via RAG remain identical for hours, sometimes days. The provider can therefore calculate the KV representation of this stable portion once, store it temporarily, and reuse it directly for all subsequent calls that begin with the exact same prefix.

Technically, providers typically hash the first tokens of the incoming prompt (approximately the first 256 for OpenAI) and compare this hash to their recent cache. If there is a match (a “cache hit”), only the new part of the prompt (typically the user message) is processed with full compute power; the rest is read directly from the cache. This compute saving translates into lower billed tokens and reduced latency, as skipping the prefill also eliminates the associated processing time.

The critical point to remember: this match must be an exact prefix match. One extra space, a different order of JSON keys, or a modified tool definition between two calls, and the cache no longer applies—the model reprocesses the entire prompt at the standard rate.

OpenAI, Anthropic, Google — Three Different Approaches

The three major providers implement prompt caching with different logic and pricing, which directly impacts how you should design your integration.

OpenAI applies caching automatically with no code changes. Once a prompt exceeds 1,024 tokens, the API checks if its prefix matches a recent call processed on the same server, using matching tiers in 128-token increments. In the event of a cache hit, a 50% discount is applied to the cached tokens compared to the standard rate, and latency can drop by up to 80%. For applications sending more than fifteen requests per minute with the same prefix, OpenAI recommends adding a prompt_cache_key parameter to consistently route related calls to the same server and maximize the cache hit rate.

Anthropic takes the opposite approach: caching is explicit and controlled by the developer via the cache_control parameter, inserted at the precise point in the prompt where a “breakpoint” is desired. Up to four breakpoints are allowed per prompt, enabling independent caching of tools, the system prompt, and specific messages. In exchange for this granular control, Anthropic offers some of the most aggressive discounts on the market for tokens read from the cache, while charging a slightly higher rate for the initial cache write. By default, an Anthropic cache is kept for five minutes, with an option to extend it to one hour for an additional fee.

Google Gemini stands out with a different economic model: beyond the reduced read cost, Gemini also charges for the storage of the cache during its lifetime (one hour by default, customizable). This detail changes the ROI calculation—a Gemini cache is only beneficial if it is reused enough times before expiration to offset its storage cost.

Use Cases Where Prompt Caching is a Game Changer

Three categories of applications benefit disproportionately from prompt caching, and it is worth identifying them before investing implementation time.

Chatbots and conversational assistants with large, static system prompts are the simplest and most profitable use case. A 2,000-token system prompt describing a persona, business rules, and output constraints, sent with every turn of the conversation, is an ideal candidate: it never changes between two messages from the same user, and often doesn’t change between users. This is usually the first lever teams pull, as it offers the best effort-to-gain ratio.

Coding assistants leverage caching differently: the repository context (file structure, coding conventions, project instructions) remains stable throughout a work session, while only the user’s specific query varies. Placing this context at the top of the prompt keeps the cache active across dozens of consecutive exchanges.

RAG pipelines and document caching represent the third major use case. When an application repeatedly queries the same reference corpus (product documentation, knowledge base, legal contract) with different questions each time, caching the document rather than the question allows you to pay the full cost only once for the entire corpus, then benefit from the reduced rate for every new question asked about it.

Conversely, an application that generates entirely different prompts for every call, with no stable portion at the top, will see no benefit from prompt caching—there is simply no prefix to reuse.

Structuring Prompts to Maximize Cache Hit Rate

The financial gain from prompt caching depends entirely on the structure of the prompt sent. One single rule governs all providers: place static content first, variable content last. In practice, this means systematically ordering an API call as follows: tool definitions, system prompt and general instructions, stable reference documents, and finally, only at the very end, the specific user message or query.

This constraint has practical implications that often trip up teams early in implementation. Injecting the current date, a session ID, or any other dynamic variable in the middle or at the beginning of the system prompt breaks the cache for the entire prompt that follows, even if 95% of the content remains identical. The solution is to systematically move these variable elements to the very end of the prompt, after the static part.

Similarly, data serialization must remain strictly identical from one call to the next: a JSON object whose key order changes, a randomly reordered list of tools, or even a simple change in indentation is enough to generate a different prefix hash and cause a complete cache miss. Teams achieving the highest cache hit rates lock their prompt generation using deterministic templates rather than letting an ORM or JSON serializer reorder fields freely.

Finally, minimum token volume matters: below approximately 1,024 tokens for OpenAI, the cache does not activate at all. This makes the optimization particularly relevant for system prompts rich in business context, few-shot examples, or documentation, and much less so for short micro-prompts.

Calculating the ROI of Prompt Caching

Prompt caching isn’t free to implement, even if it often requires only a few lines of code. It is therefore useful to quantify the expected gain before prioritizing it on a technical roadmap.

Consider a representative example: an application sends a static 2,000-token system prompt, accompanied by a variable 100-token user query, for 500,000 calls per month. Without caching, every call bills the full 2,100 tokens at the standard input rate. With caching enabled, only the 100 variable tokens are billed at full price; the 2,000 system prompt tokens switch to the reduced cache rate, which is up to ten times cheaper than the standard rate depending on the provider. In this scenario, total monthly costs can drop by roughly 89%, representing hundreds of dollars saved per month on a single feature, with no change in response quality or behavior.

However, this calculation depends on three variables to verify for your own case: the proportion of truly static content in your prompts, the effective cache hit rate (which depends on call frequency and cache TTL), and the initial cache write cost, which is slightly higher than the standard rate for some providers. Low-volume applications or those whose prefixes change frequently will rarely reach the maximum theoretical gains announced by providers.

Limitations and Pitfalls to Know Before Activating

Prompt caching is not an optimization to be blindly enabled across an entire application. Several limits should be anticipated.

The first concerns cache lifespan (TTL), which varies greatly by provider: five minutes by default for Anthropic, one hour for Gemini (with paid extensions). An application where calls using the same prefix are too far apart in time will never benefit from the cache, regardless of how well the prompt is structured.

The second is monitoring and observability: classic usage metrics (input tokens, output tokens) are no longer enough to understand the real cost of an application using caching. It becomes necessary to separately track tokens read from cache, tokens written to cache, and tokens processed at the standard rate; otherwise, the cost-per-request calculation will be incorrect.

The third limit is specific to Anthropic: the initial cache write cost is higher than the standard input rate. In an application with a very low reuse rate (prompts rarely identical from one call to another), caching can actually increase the bill rather than reduce it, as each write costs more than normal processing without ever being offset by a subsequent read.

Finally, prompt caching reuses computation, not the final result: it does not replace a classic application caching strategy that would store the model’s response directly for strictly identical queries. The two approaches are complementary and serve different needs.

Does prompt caching reduce the quality of generated responses?

No. Prompt caching only reuses the already calculated internal representation of the prefix without modifying the model’s behavior or how it generates the rest of the response. The output quality remains identical; only the cost and latency change.

Do I need to activate prompt caching manually?

It depends on the provider. OpenAI enables it automatically for prompts exceeding 1,024 tokens without code changes. Anthropic requires you to explicitly insert a cache_control parameter in the prompt to define the breakpoints to be cached.

Why is my prompt caching not working despite having an identical system prompt?

The most common cause is an invisible difference in the prefix: changing JSON key order, a dynamic variable injected at the start of the prompt, or an extra space. Cache matching requires a strictly identical prefix, character for character.

How long does a prompt cache stay active?

This varies by provider: five minutes by default for Anthropic (extensible to one hour), one hour by default for Google Gemini (customizable). For OpenAI, the duration depends on the load of the server that processed the prompt.

Does prompt caching work for images and tool definitions?

Yes, provided they are strictly identical between two calls. Tool definitions (tool calling) and images embedded in the prompt contribute to the prefix calculation just like text, and any modification to them will also invalidate the cache.

Can prompt caching increase my costs instead of reducing them?

Yes, in one specific case: with Anthropic, the initial cache write rate is higher than the standard rate. If your application rarely generates identical prefixes between calls, each cache write may cost more without ever being offset by a later read.

What is the minimum prompt size required to benefit from caching?

For OpenAI, caching only activates starting at 1,024 tokens, with additional tiers in 128-token increments. Below this threshold, the optimization has no effect, making it most relevant for context-rich system prompts.