What Is Prompt Caching and How Does It Cut AI API Costs?

I am prompt caching. In one sentence: I let a model call reuse a previously processed prefix of a prompt -- system instructions, standing context, reference material that doesn't change between calls -- instead of reprocessing that same text from scratch every single time, which cuts both the latency and the cost of every call that shares that prefix.

What I actually do, in order

A model call doesn't start from nothing. Before it generates a single token of a reply, it has to process the entire prompt it was given -- system instructions, loaded rules, reference documents, prior conversation. When that prefix is large and mostly identical call after call, reprocessing all of it every time is wasted work: the same tokens, read the same way, paid for again and again, for content that hasn't changed since the last call.

I remove that waste on the calls that qualify:

1. Identify a stable prefix. A block of prompt content -- system instructions, a large reference document, standing context -- that's identical across multiple calls in a short window is a caching candidate. 2. Process it once, then hold the result. The first call that includes that prefix pays the full processing cost. What follows is cached for reuse, keyed to that exact prefix content. 3. Reuse the cached result on matching subsequent calls. A later call that starts with the identical prefix skips reprocessing it and picks up from the cached state, only doing full work on what's actually new -- the part after the shared prefix. 4. Expire under the provider's contract, not forever. The cache is not permanent. Its lifetime, refresh behavior, and storage cost vary by provider, model, and cache mode. A call outside that contract falls back to full processing. 5. Treat a change as a cache boundary until telemetry proves otherwise. A changed or reordered reusable prefix, model/provider change, or a cache-affecting setting can stop reuse. The next call may process the block in full again, often without an error.

How I actually work here

In this operation, I'm the reason a session that reloads the same large rule set, standing context, or reference bundle on every call doesn't pay full price for that content every single time. Bootstrap chains, standing instructions, and repeatedly-referenced documents are exactly the kind of stable prefix I exist to make cheap on the second and subsequent read.

I'm also the reason latency drops on repeated calls sharing that prefix: less to process before generation starts means a faster first token, which matters most in any interactive or agent-loop context where the same large context is reloaded call after call.

What I depend on: the prefix actually staying identical, and being placed where it can be recognized as shared. If standing content is reordered, lightly reworded, or interleaved with call-specific material instead of kept as a clean, consistent prefix, I can't recognize it as the same block I already processed -- and every apparent "cache hit" I might have provided quietly becomes a full-cost miss instead.

Where I fail

This is the part that matters more than the part above, so I'm not going to bury it.

I only help if the prefix is genuinely stable and placed first. If the call-specific, always-changing part of the prompt comes before the shared context instead of after it, there's no stable prefix for me to match against -- I need the identical part at the start, not scattered through the call.

A small change can turn reuse into a miss. Reordering a rule file, adding a timestamp, injecting a per-session value into static content, or changing model/provider/configuration may prevent reuse. The failure is often silent: the call still succeeds, but quietly reverts to full-cost processing.

I expire under provider-specific rules. A cache that is not reused inside its configured or provider-managed window can be gone, and the next call pays full price again. Caching only pays off for prefixes reused frequently enough to land inside that window—one-hour rotation across many sessions can turn each session cold.

I don't reduce what a model actually reasons about. I cut reprocessing cost for content the model has already seen. I don't shrink the effective context the model has to weigh when generating a response, and I don't fix a prompt that's bloated with content the call doesn't actually need. A stable-but-unnecessary prefix is still a prefix that shouldn't be there.

I can create a false sense of "it's already cheap" that discourages trimming. Once a large prefix is cached, the visible cost drops enough that nobody feels pressure to ask whether all of that content still needs to be there. Caching hides bloat's cost without removing the bloat itself.

Tech-Tips

- Put stable content first, call-specific content last. I can only match a prefix that's actually a clean, unbroken prefix -- interleaving breaks the match even if most of the content is unchanged. - Keep standing context stable across calls that should share a cache. Put it first, keep dynamic values at the end, and treat a reordered rule, timestamp, model/provider/configuration change as a cache boundary unless the provider's metrics show a hit. - Know your provider's cache window and work inside it. An hour is a practical rotation warning, not a universal TTL; a prefix reused once a day may never actually benefit if the provider window is shorter. - Don't treat a cached prefix as free to keep growing. Caching lowers the cost of reprocessing bloat, but it doesn't lower the reasoning cost of a model actually weighing that much context on every call -- prune what doesn't need to be there regardless of caching. - Instrument cache reads, writes, hits, misses, TTL/expiry, model/version, configuration, and prefix version. A silent miss looks identical to a hit from the outside unless you're checking; without that visibility, broken reuse can go unnoticed for a long time.

Evidence: This piece describes the prompt-caching pattern used throughout our internal repeated agent/model call operation -- stable-prefix reuse, exact- match invalidation, and bounded time-window expiry -- no vendor or specific product is named, per the moat-protection and STORY-FORMAT conventions. Evidence class: internal operating record and Owner attestation, 2026-08-25.