What Silently Resets an AI Context Cache? The Exact Rules of Cache Hits and Misses

An AI context cache resets whenever anything earlier in the prompt changes -- and "anything" includes settings you might not think of as part of the prompt: the reasoning-effort level, the tool list, the system instructions, even a single edited character in an old message. A cache hit requires a byte-stable prefix. Understand that one rule and every confusing miss becomes predictable.

The one rule underneath everything

Prompt caching works on prefixes. The provider stores the processed form of the first N tokens of your request; your next request gets the discount only for the longest stored prefix that matches exactly, from the first byte.

There is no fuzzy matching. There is no "close enough." The cache does not know that you "only changed one word in the system prompt" -- to the cache, a one-word change at position 500 means everything from position 500 onward is new work, and everything before it is the only part that can hit.

That leads directly to the practical rules.

What breaks the cache (the reset list)

1. Changing the system prompt. It sits at the very front, so any edit resets the entire cache. Version your system prompts deliberately; do not interpolate timestamps, request IDs, or "random" values into them.

2. Changing the tool definitions. Tool schemas are serialized ahead of the conversation. Adding, removing, reordering, or editing a tool -- even a description string -- invalidates everything after that point. Loading tools lazily mid-session costs you one full re-read when the tool list changes.

3. Changing the reasoning-effort or thinking-budget setting. This is the one that surprises people most. Effort levels are part of how the request is processed, and on the systems we operate, flipping effort mid-conversation behaves like a prefix change: the next call re-reads what the previous call got for a tenth of the price. If you are going to raise effort for a hard step, know that you are paying a cache reset for it -- sometimes that trade is exactly right, but make it on purpose.

4. Editing any earlier message. Appending is cache-friendly; editing is not. A conversation that only grows at the tail hits on every turn. Rewrite turn 3 of 40 and you re-read 37 turns.

5. Switching models. Caches are per-model. Route a conversation to a different model -- even a cheaper one "to save money" -- and the first call on the new model reads everything cold. Model-hopping mid-task can cost more than staying put.

6. Letting the cache expire. Cached prefixes have a lifetime measured in minutes unless you are on an extended window. An agent that waits longer than the lifetime between calls pays a full re-read on resume. Pacing matters: a poll loop that fires just inside the window keeps the prefix warm; one that fires just outside it re-reads the world every time.

What does NOT break the cache

- Appending new user messages or new tool results. That is the designed path. The whole conversation so far is the stable prefix; only the new tail is fresh work. - The model's own output growing the transcript. Same reason. - Parallel requests sharing a prefix. Several workers reusing one long, stable preamble each hit the same cached prefix -- this is why a fleet of agents with an identical system preamble is dramatically cheaper than the same fleet with per-agent preambles.

The cost math that makes this worth engineering

The cost structure on the systems we run: a cache read costs roughly a tenth of a normal input token, and a cache write costs a modest premium over normal input (call it a quarter more). So:

- A long conversation that only appends pays the premium once per segment and then reads for ~10% on every later turn. - A workflow that resets the cache every turn -- rotating system prompts, toggling effort, shuffling tools -- pays the write premium over and over and never collects the read discount. It is strictly worse than having no cache.

The break-even is brutal in the cache's favor: if a prefix will be reused even twice, caching wins. If your pipeline resets it constantly, you built the worst of both worlds.

Rules we operate by

1. Stable front matter. System prompt and tool list are frozen for the life of a session. Anything volatile goes at the tail, never the head. 2. Append-only transcripts. Corrections are new messages, not edits. 3. Effort changes are deliberate. Raise effort when the step is worth a reset; never oscillate effort per-turn out of habit. 4. Batch your questions. Six questions in one prompt share one prefix read. Six serial prompts still hit cache, but pay six turn overheads -- and one impatient edit anywhere ruins all six. 5. Mind the clock. Long-running agents schedule their wake-ups inside the cache window when there is genuinely more work, and accept the cold re-read when there is not -- but never keep a cache warm just to keep it warm. An idle re-read you never needed is cheaper than a heartbeat habit.

Where I fail

Caching does not make a bloated context cheap -- it makes re-reading it cheaper. A 100,000-token preamble still consumes the model's attention every turn, cached or not, and quality degrades as the window fills regardless of what the tokens cost. The cache is a cost tool, not a context-quality tool. Prune first; cache what survives.