Why Appending to Your Agent's System Prompt Breaks Prompt Caching

Introduction: The $21 Task That Billed $200

Appending a timestamp or a status line to your agent's system prompt looks harmless, and it quietly invalidates the cached prompt prefix on every turn, forcing the whole conversation to be reprocessed at full price. This post shows why that happens, reproduces the failure in a local agent loop you can run yourself, works the cost arithmetic for a 20-turn session, and lays out the cache-stable prompt ordering that keeps the cache hitting.

What the Artificial Analysis claim actually says, and what it does not

George Cameron of Artificial Analysis argued in an AI Engineer talk that caching failures can inflate agent workloads by roughly 10x. The secondary write-ups that reached me pair this with a concrete anecdote: a task that should have cost about $21 billed about $200.

That establishes less than the number implies, and the missing details are exactly the ones that decide whether the claim applies to your stack:

• Reported: a real agent workload where broken caching produced roughly an order-of-magnitude overrun, illustrated by $21 → $200.
• Reported: the mechanism is prompt caching failing to hit, not model pricing or output-token volume.
• Not established in what I could read: the prompt layout, provider, TTL tier, turn count, prefix size, or the split between input cost and retries. I never saw the raw numbers, and I'm not going to invent them.

So the useful question isn't whether $200 is off by a few dollars. It's whether the stated mechanism produces a ~10x gap when the arithmetic is done on a layout I control. That I can check.

The position this post takes: prompt ordering is an architecture decision, not a prompt-engineering tip

The position, stated up front: anything that changes per turn belongs at the bottom of the prompt, not the top. Where you place a timestamp, a scratchpad, a status line, or a retrieved chunk decides which token ranges are reusable — and it is worth more money than most wording choices.

Appending a 20-token status line to the system prompt each iteration does not cost 20 tokens. On a long agent run it can cost the entire cached prefix, every turn, including the accumulated conversation behind it. That's the bug here, and it is a one-line change to fix — but only for someone who knows it exists and can measure it.

What follows: the mechanics, a deterministic reproduction you can run without an API key, the cost arithmetic written out, and the layering scheme I use now.

What a Prompt Cache Actually Caches

Prefix hashing, KV state reuse, and why the unit of reuse is a token range

Transformer inference runs a prefill phase where the model computes attention key/value tensors for every input token, and prefill is the expensive part of a large prompt. Caching persists the KV state for a token range so a later request with an identical leading range skips the recomputation.

In practice:

1. The provider tokenizes the request — tool definitions, system blocks, messages — into one flat sequence.
2. It hashes a prefix of that sequence, up to a boundary you specify or one the provider picks.
3. On a later request it looks for a stored entry whose hashed range is a prefix of the current sequence.
4. A match means those tokens are billed at the cached-input rate and their prefill is skipped; everything after the match processes normally.

The unit of reuse is therefore a token range starting at position 0 — not a block, message, or section. A prefix. No mechanism says "this system block is unchanged, reuse that one"; the match is a single contiguous run from the beginning.

The consequence people consistently underestimate: the first byte that differs terminates the match. Everything after it, however stable, is not reused.

Why the cache key ignores message roles and ignores what you meant

The key is computed over the serialized token stream, so it doesn't know that you renamed a role, reformatted whitespace, reordered JSON keys, or paraphrased a sentence into something semantically identical. Two prompts that mean the same thing are different prompts.

The "ignores roles" point deserves a direct statement because that's where intuition breaks. Renaming a role or moving equivalent content from a system block into a leading user message earns nothing — message boundaries serialize into the token stream, so the tokens differ and the match dies at the boundary. What pays is leaving the bytes alone.

Which means these cosmetics all invalidate the cache:

• {"a":1,"b":2} versus {"b":2,"a":1} in a tool schema.
• A trailing space left by a template literal.
• A tool definition re-serialized via JSON.stringify where key order follows object construction order.
• Moving "role": "system" content into an equivalent "role": "user" prelude.

And these do not:

• Adding tokens at the end of the sequence — the prefix doesn't move.
• Reusing the cached range across turns with different query text after it.

The three costs that matter: cache write premium, cache read discount, and full-price misses

Every token lands in one of three buckets, and the multipliers differ:

The multipliers used here are Anthropic's published ones — public and stable: 0.1x reads, 1.25x writes under the default 5-minute TTL, 2x for the 1-hour TTL. Confirm them on the current pricing page before modelling; the shape transfers, the exact figure does not.

The counterintuitive part: a cache that always misses costs more than no cache at all. Put cache_control on a block that changes every turn and you pay the 1.25x write premium each turn while never collecting the 0.1x read discount — a 25% tax on input tokens sold as an optimisation.

The Ordering Bug: Why Appending to the System Prompt Breaks the Cache

Appending to the system prompt shifts every downstream token, so nothing after it can match

Here is what most agent loops start with, and it looks entirely reasonable:

// The bug, in one line.
const system = `${BASE_SYSTEM}\n\nCurrent time: ${new Date().toISOString()}\nTurn: ${turn}`;

The reasoning: "the dynamic part is at the end of the system prompt, so the stable part in front of it is still a shared prefix." Half right — and the wrong half is the expensive one.

The stable system text is still a shared prefix up to the point where the timestamp diverges. But caching only helps if a stored cache entry ends inside that shared region. With the breakpoint sitting after the timestamp, the entry stored last turn covers ...base + timestamp(t-1), while this turn's prefix is ...base + timestamp(t). Different prefixes, no match.

Worse, if you also breakpoint the tail of the message list — the standard configuration for long runs — that breakpoint's prefix also starts with the changed system block. The cached conversation behind it becomes unreachable, so you lose the system block and the fifty thousand tokens of tool results and history you built up.

The same text appended to the tail of the message list costs you one block, not the run

Move the identical text to the newest user or tool message and the arithmetic flips:

// Same information, different position.
messages.push({
  role: "user",
  content: `<status time="${new Date().toISOString()}" turn="${turn}" />\n\n${userInput}`,
});

The volatile text now sits after every breakpoint you defined. Turn n's prefix equals turn n−1's full request plus new tokens appended; the previous request is a strict prefix of the current one, so every stored breakpoint still matches.

The delta isn't "20 tokens versus 20 tokens" — it's "one lost block versus the whole run".

A worked token-position diagram showing match length before and after the append

Layout: 1,200 tokens of tool definitions, 2,800 tokens of stable system rules, a 20-token status line, and 2,000 tokens of new material per turn.

Turn 1 — cached span ends at the breakpoint ▼

  token  0          1,200              4,000   4,020           6,020
         |-- tools --|-- system base ---|-status-|-- user turn 1 --|
                                              ▼ breakpoint
  cache entry stored: [0, 6020)

Turn 2 — status line changed, everything after it shifted

  token  0          1,200              4,000   4,020            8,020
         |-- tools --|-- system base ---|-status'|-- history + turn 2 --|
                                              ▼ breakpoint (still at 4,020, offers 8,020 too)

  bytes identical to turn 1:  [0, 4000)        <- 4,000 tokens, all wasted
  breakpoints offered:        {4,020, 8,020}
  any breakpoint inside [0, 4000)?   no
  cache_read_input_tokens   = 0
  cache_creation_input_tokens = 8,020

Versus the tail placement:

Turn 2 — volatile text moved to the newest message

  |-- tools --|-- system base --|-- turn 1 --|-- turn 2 + status --|
                                                             ▼ breakpoint
  bytes identical to turn 1:  [0, 6,020)       <- the entire previous request
  cache_read_input_tokens   = 6,020
  cache_creation_input_tokens = 2,000

Four thousand identical tokens on the left, none billed at the read rate. The match length was fine; the breakpoint placement was not.

Reproducing the Invalidation in a Local Agent Loop

A minimal 20-turn agent that appends a status line to the system prompt each iteration

The script below calls a real Messages API. Prompts are kept small so the numbers can be checked by hand, but the shape is the one that breaks in production.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const BASE_SYSTEM = `You are a repository triage agent. Follow these rules...`.repeat(40);
const TOOLS = [
{ name: "list_files", description: "List files in a path",
  input_schema: { type: "object", properties: { path: { type: "string" } }, required: ["path"] } },
{ name: "read_file", description: "Read a file",
  input_schema: { type: "object", properties: { path: { type: "string" } }, required: ["path"] } },
];

const history = [];

for (let turn = 1; turn <= 20; turn++) {
// ⚠️ volatile text at the FRONT of the prompt
const system = `${BASE_SYSTEM}

Current time: ${new Date().toISOString()}
Turn: ${turn}`;

history.push({ role: "user", content: `step ${turn}: inspect the next module` });

const res = await client.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 256,
  tools: TOOLS,
  system: [{ type: "text", text: system, cache_control: { type: "ephemeral" } }],
  messages: history,
});

log(turn, res.usage);
history.push({ role: "assistant", content: res.content });
}

Logging cache_read_input_tokens and cache_creation_input_tokens per request

The three usage fields are not interchangeable; total input is their sum.

function log(turn, usage) {
const read = usage.cache_read_input_tokens ?? 0;
const write = usage.cache_creation_input_tokens ?? 0;
const fresh = usage.input_tokens ?? 0;
const prefix = read + write + fresh;
const hitRate = prefix ? read / prefix : 0;

console.log(
  [
    String(turn).padStart(3),
    String(prefix).padStart(7),
    String(read).padStart(7),
    String(write).padStart(7),
    (hitRate * 100).toFixed(1).padStart(6) + "%",
  ].join(" ")
);
}

cache_read_input_tokens above ~0 means the prefix matched. cache_creation_input_tokens above the size of your new content means you're rewriting a prefix you already paid for. input_tokens above zero on turn 12 of a long run means something upstream of you isn't cached at all.

The same loop rewritten with a static prefix and a dynamic tail

const STATIC_SYSTEM = [
BASE_SYSTEM,
// session-scoped, set once, never re-interpolated
`repo: ${repoName}`,
`language: ${primaryLanguage}`,
].join("\n\n");

for (let turn = 1; turn <= 20; turn++) {
// volatile state goes in the newest message, AFTER every breakpoint
history.push({
  role: "user",
  content: `<status turn="${turn}" at="${new Date().toISOString()}" />
step ${turn}: inspect the next module`,
});

const res = await client.messages.create({
  model: "claude-sonnet-4-5",
  max_tokens: 256,
  tools: TOOLS,
  system: [{ type: "text", text: STATIC_SYSTEM, cache_control: { type: "ephemeral" } }],
  messages: withTailBreakpoint(history),
});

log(turn, res.usage);
history.push({ role: "assistant", content: res.content });
}

withTailBreakpoint marks the second-to-last block so the growing conversation caches incrementally. Two breakpoints cover most loops; Anthropic caps you at four, and spending them on the system block and the tail is the best value available.

Observed per-turn token accounting for both variants, with the sample-size caveat stated plainly

Honest disclosure: I did not run a live 20-turn benchmark against a provider for this post. The table below is arithmetic, computed by the deterministic simulator in the next section from a layout I specified, using Anthropic's documented 0.1x / 1.25x multipliers and $3 per million input tokens.

What that means: the ratios are reproducible and checkable; the absolute dollars depend on your prefix size, TTL tier, and provider's current prices. Treat the shape as the finding and the dollars as a template.

Two things stand out beyond the 6.2x total. First, the append variant's per-turn cost grows without bound — every turn re-writes the whole accumulated conversation. Second, read is exactly zero for all twenty turns while write is enormous. That signature is unambiguous: you're paying the premium and getting nothing back.

Sample-size caveat: one synthetic layout, twenty turns, one provider's multiplier schedule. Enough to demonstrate the mechanism. Not a benchmark, and it says nothing about how the same code behaves on a provider with automatic caching.

Building the Cost Model Before You Build the Agent

Tokens x price x cache multiplier, written out as an explicit formula

Write the model down before the code exists. For a session of K turns where turn k sends prefix P_k and adds N_k new tokens:

cost = Σ_k [ (read_k  × price × r) + (write_k × price × w) + (fresh_k × price × 1) ]

where  read_k  + write_k + fresh_k = P_k
       r = cache read multiplier   (≈ 0.1)
       w = cache write multiplier  (1.25 for 5-minute TTL, 2.0 for 1-hour)

Two closed forms worth memorising:

Perfect cache:  cost ≈ price × ( r × Σ P_k  +  (w − r) × Σ N_k )
Broken cache:   cost ≈ price × w × Σ P_k

The broken form has no N_k term at all. That is the tell: when the cache never hits, cost tracks how much you re-send, not how much you add.

Deriving the 10x gap from a 0% hit rate versus a 90% hit rate on a long prefix

Using the 4,000-token static prefix and 2,000 new tokens per turn over 20 turns from the previous table.

Broken cache:

Σ P_k = 20×4,020 + 2,000×(1+2+…+20)
      = 80,400 + 420,000
      = 500,400 tokens
× 1.25 write premium  = 625,500 token-equivalents
× $3/M                ≈ $1.88

Healthy cache:

turn 1 write:   6,000 × 1.25            =   7,500
turns 2–20 read: 456,000 × 0.1          =  45,600
turns 2–20 write:  38,000 × 1.25        =  47,500
                                    total ≈ 100,600 token-equivalents
                                          ≈ $0.30

Ratio: 6.2x.

Switch the cache-write multiplier from the 5-minute tier (1.25x) to the 1-hour tier (2.0x) — what you actually need when turns include slow tool calls and the default TTL expires between them — and the broken variant's numerator doubles to 1,000,800 token-equivalents, while the healthy variant barely moves because most of its tokens are reads:

1,000,800 / 100,600 ≈ 9.9x

That lands almost exactly where the reported 10x sits once you're on the long-TTL write premium. The reported $21 → $200 is a 9.5x multiple, consistent with this model and not with an exotic explanation. I'm not claiming their workload matches mine — I can't see it — but the mechanism produces the right order of magnitude from documented multipliers alone.

Secondary multipliers: retries, tool-call loops, and re-sent conversation history

The input-token ratio is the floor, not the ceiling. Three things push it further:

• Retries. A transient 429 or 500 that you retry re-sends the full prefix. With a broken cache every retry is a full-price full-prefix write; with a working one a retry re-reads at 0.1x.
• Tool-call loops. An agent making five tool calls per user turn issues five requests with essentially the same prefix. That's where caching pays most — and where the append-to-system bug costs five times per user turn instead of once.
• History re-sends. The prefix grows quadratically in a broken loop (Σ P_k is O(K²)) and linearly in a healthy one. Twenty turns at 2k new tokens each is 420k tokens of re-send; forty turns is 1.64M. The cost scales with session length, which is why the bug is invisible in a two-turn smoke test.

A spreadsheet-style checklist of the inputs you need from your provider's current pricing page

Don't hardcode my numbers. Collect these:

Ten cells. Without them you can't reason about agent cost, and you'll find out the way the $200 anecdote did.

Provider Caching Mechanics That Change the Math

Explicit breakpoints with TTL and write premiums versus automatic prefix matching

Two families of design, failing differently.

Explicit breakpoints (Anthropic's cache_control): you declare where cached spans end, up to a small number of markers. Reads cost around 0.1x, writes around 1.25x for the default short TTL, and an optional longer TTL at roughly 2x write cost. Because you pick the boundaries, you can get it wrong — this is the family where a breakpoint after a timestamp is a self-inflicted wound. The upside is that a working explicit config is a very large discount.

Automatic prefix matching (OpenAI's prompt caching): the provider detects shared prefixes and discounts them with no markers; hits show up in usage as a cached-token count nested in prompt-token details. The failure mode differs — you can't shoot yourself in the foot with a marker, but you also can't pin an expensive stable block, and small invisible front-of-prompt edits still terminate the match.

The architectural lesson transfers to both, because it's about prefix ordering, not markers. On automatic matching you have less instrumentation about why you missed, so the client-side diff in the measurement section matters more.

Minimum cacheable prefix lengths and block granularity, and why they matter below 1k tokens

Caching doesn't engage below a minimum prefix length, and matches round to a granularity step. Two implications:

1. Small prompts aren't cache problems. If the whole request is 600 tokens, nothing is cached and no layout choice matters. Optimising there is noise.
2. Granularity silently drops the tail of your match. If matches round down to a fixed step, a match landing 40 tokens short of the previous prefix can round well below it. Don't design a layering scheme that only works when match length equals prefix length exactly — leave slack.

What happens to the cache when you change model, temperature, or tool schema mid-session

• Model change: cache entries are model-specific, so switching mid-session is a cold start every time. Untested by me, but it follows from the entries being model artefacts.
• Tool schema change: the one people miss. Tool definitions typically serialize at position 0, so adding, removing, or reordering a tool — or a runtime that conditionally registers tools off feature flags — invalidates the entire prefix, including all history, on the turn it changes. I've seen this from a tool list built out of Object.keys() on an object populated in nondeterministic order.
• Temperature: not tested, and I've seen contradictory claims. Don't assume either way; measure it on your provider before building a model on it.

Comparison table of caching models, with a note to verify against current provider docs rather than this post

Check the multipliers, minimums, and TTLs against current provider documentation. Pricing changes faster than blog posts.

Cache-Stable Prompt Architecture

Generalise the fix into three layers ordered by volatility, increasing as you move down the prompt.

Layer 1 — immutable instructions, tool schemas, and few-shot examples

Position 0 gets the things that never change within a session: tool definitions, core system rules, few-shot examples, output format contracts. This block must be byte-identical across every request in the session, including key order in serialized schemas.

The practical rule: if a value comes from Date.now(), crypto.randomUUID(), process.env, or a user record, it doesn't belong in layer 1.

Layer 2 — session-scoped context that changes once, placed above the conversation

Repository maps, user profiles, retrieved documentation sets, tenant config — things that change at session start and then stop. Put them right after layer 1, before the history begins, with a breakpoint at the end.

Changing layer 2 mid-session is legitimate (a user edits their profile, retrieval re-runs), but treat it as an event with a cost, not a per-turn operation.

Layer 3 — per-turn state in the newest user or tool message, never in the system block

Timestamps, turn counters, current file contents, scratchpads, tool results, status. All of it goes in the newest message, after every breakpoint, so appending extends the matchable prefix instead of destroying it.

The reframe that made this click for me: your system prompt isn't "the non-conversation part of the prompt"; it's "the prefix." If it changes per turn, it is conversation, whatever the API field is called.

Breakpoint placement for long runs, including where to re-anchor after a compaction step

Four breakpoints available. A workable allocation for a long agent run:

1. End of layer 1 + layer 2 (tools, rules, session context). Written once, read forever.
2. Tail of the conversation, moved forward each turn — your incremental cache.
3. Spare — for a large retrieved chunk or big tool result you expect to reuse.
4. Spare.

Compaction is the one that bites. When you summarise old turns to fit the context window, the summary replaces the history, so the prefix now diverges where the original history began. Accept the full rewrite once. Don't try to make compaction incremental; you'll build something fragile. After compaction, re-anchor breakpoint 1 at the new end of the static block and breakpoint 2 at the new tail, then let it rebuild.

Untested caveat: I haven't measured how compaction interacts with the provider's lookback when scanning for matches. My expectation is that it behaves like any other mid-prefix divergence, but I'd want to verify before relying on a specific cost.

The Hidden Cache Invalidators

Everything above is structural. These sneak in.

Timestamps, UUIDs, request IDs, and usernames interpolated into the system prompt

The classic. A template injecting an ISO timestamp into the system block, a trace ID for observability, the current user's display name. Each terminates the match at the injection point, every request.

The fix is mechanical: move them to the newest message. If you need them for observability, log them — don't put them in the prompt.

Non-deterministic JSON serialization of tool definitions and reordered keys

JSON.stringify preserves insertion order. If your tool registry is a plain object assembled from async results, or built by iterating a Map populated in completion order, the serialized schema can differ between processes — or between turns, if a tool is conditionally added.

Canonicalise once and hash it:

export function canonicalJson(value) {
if (Array.isArray(value)) {
  return "[" + value.map(canonicalJson).join(",") + "]";
}
if (value && typeof value === "object") {
  const keys = Object.keys(value).sort();
  return "{" + keys.map((k) => JSON.stringify(k) + ":" + canonicalJson(value[k])).join(",") + "}";
}
return JSON.stringify(value);
}

// In CI: assert the canonical form never changes unexpectedly.
// expect(sha256(canonicalJson(TOOLS))).toBe(KNOWN_TOOL_SCHEMA_HASH);

A pinned hash in a test is the cheapest cache-stability guard you can buy.

Retrieval chunks injected above the conversation history

Re-run retrieval each turn and inject top-k chunks into the system block or a pre-conversation message, and you invalidate everything below on every turn — even when the retrieved set is identical, because the ordering may differ.

Two options: pin retrieval per session (retrieve once, place at layer 2, breakpoint after), or put re-retrieved chunks in the newest message where they extend the prefix instead of forking it. Pinning is cheaper; re-retrieving is fresher. Pick deliberately.

Feature flags and A/B prompt variants that silently fork the prefix

A flag swapping one paragraph of the system prompt forks the prefix for the treatment group. Expected and fine — but if the flag is evaluated per request and can flip mid-session, you get a miss at the flip and a mystery in your cost dashboard.

Evaluate prompt-affecting flags once at session start and store the resolved prompt. Then a flag flip is a new session, which is a cost you can predict.

Measuring Cache Hits and Keeping Them Fixed

The per-request metric set to log: prefix length, match length, read tokens, write tokens, miss reason

The provider tells you read and write tokens. It usually doesn't tell you why you missed. Compute that yourself by diffing serialized prefixes client-side:

let previousPrefix = null;

export function diffPrefix(nextSerialized) {
if (previousPrefix === null) {
  previousPrefix = nextSerialized;
  return { reason: "cold_start", at: 0 };
}
const n = Math.min(previousPrefix.length, nextSerialized.length);
let i = 0;
while (i < n && previousPrefix[i] === nextSerialized[i]) i++;

const result =
  i === n && previousPrefix.length === nextSerialized.length
    ? { reason: "identical", at: i }
    : {
        reason: "diverged",
        at: i,
        // the 40 chars before and after divergence usually name the culprit:
        // a timestamp, a UUID, a turn number, a tool name
        window: nextSerialized.slice(Math.max(0, i - 40), i + 40),
      };

previousPrefix = nextSerialized;
return result;
}

Log five fields per request: prefix_tokens, match_tokens, cache_read_input_tokens, cache_creation_input_tokens, miss_reason. The window on a diverged result is worth more than any dashboard — it prints the timestamp eating your budget.

A regression test that asserts hit rate on a fixed 10-turn script

Two tests, different costs.

Fast, offline, runs on every commit. Assert byte-identity of the static prefix across simulated turns. Catches the bug before it ships; needs no API key:

import assert from "node:assert/strict";

test("layer 1 + layer 2 are byte-identical across turns", () => {
const turn1 = buildRequest({ turn: 1, history: [] });
const turn7 = buildRequest({ turn: 7, history: fixtureHistory(6) });

assert.equal(turn1.system.map((b) => b.text).join(""), turn7.system.map((b) => b.text).join(""));
assert.equal(canonicalJson(turn1.tools), canonicalJson(turn7.tools));
});

test("the previous request is a strict prefix of the next", () => {
const a = serialize(buildRequest({ turn: 4, history: fixtureHistory(3) }));
const b = serialize(buildRequest({ turn: 5, history: fixtureHistory(4) }));
assert.ok(b.startsWith(a), "prompt layout is not prefix-monotonic");
});

Slow, nightly, hits the API. Run a fixed 10-turn script and assert the hit rate, not exact token counts:

turn  1: read=0      write=6000   hit=0.0%
turn  2: read=6000   write=2000   hit=75.0%
...
turn 10: read=24000  write=2000   hit=92.3%

Assert read / (read + write + fresh) > 0.8 from turn 3 onward. A threshold, not an exact value — token counts shift when a provider updates its tokenizer, and you don't want a false alarm from that.

Alerting on cache-write spikes and on hit-rate drops after a deploy

Two alerts, both cheap:

• Cache-write spike. If cache_creation_input_tokens per request exceeds the p95 of your new-content size by more than a small factor, you're rewriting a prefix you already paid for. On the broken loop above, this fires on turn 2.
• Hit-rate drop after deploy. Compare a rolling 1-hour read-hit rate against the same window yesterday, keyed by deploy SHA. A feature flag swapping system text, a conditionally added tool, or someone "improving" the prompt shows up here within minutes.

The second one saves you. Most of these bugs are introduced by people who have never heard of prefix caching, in a PR about something else.

When Appending to the System Prompt Is Fine

Specific exceptions, because "never put dynamic content in the system prompt" is too strong and will get ignored.

Short sessions that never reach the minimum cacheable length

Below the provider's minimum cacheable prefix, caching doesn't engage. Nothing written, nothing read, no premium to waste. A two-turn classification call with an 800-token prompt can put whatever it likes in the system block — the layout costs exactly zero.

Check your actual prompt size before refactoring. Plenty of internal tooling lives permanently below the threshold.

Genuinely static additions applied once at session start

A date resolved once and reused for the session, a tenant ID looked up on connection, a resolved feature-flag set: static for the session, so they belong in layer 2, after layer 1, with a breakpoint behind them. They're "dynamic" relative to your codebase but not relative to the cache. The distinction that matters is per-session versus per-turn.

Cases where instruction salience beats caching economics, stated as a tradeoff rather than a rule

There's a real argument on the other side: some models follow constraints more reliably when they appear in the system block, and moving an instruction into a user message can measurably degrade compliance. If a system-block-only instruction prevents a failure class that costs more than the cache does, put it in the system block.

That's a tradeoff, not a rule, and it should be made with numbers. Compute the delta: multiply your measured hit-rate loss by your per-session input cost, and compare it to the measured quality gain. In most agent workloads the cache delta is larger than people expect and the salience delta is smaller than they assume — but "most" isn't "all", so check rather than take my word for it.

Limitations and What I Did Not Test

Confirmed versus inferred: what the public reporting establishes and what this post extrapolates

Sample size, single-provider bias, and untested multi-turn compaction behaviour

Three honest limitations:

1. One synthetic layout, twenty turns, one provider's multiplier schedule. A demonstration of mechanism, not a benchmark. Real sessions have variable turn sizes, wildly different tool-result lengths, and retries.
2. Provider bias. The arithmetic uses Anthropic's explicit-breakpoint model because its numbers are public and its failure mode is the one the anecdote describes. Automatic-caching providers share the prefix-ordering constraint but differ in instrumentation, and I didn't run the equivalent experiment there.
3. Compaction is untested. What happens to match length when summarised history replaces real history is something I reasoned about, not measured. If you run very long sessions, measure it before trusting the re-anchor advice.

Conclusion: Move the Dynamic Part of the Prompt Down

Small finding, outsized consequences. Prompt caching matches a prefix from token zero. Anything that changes per turn and sits at the front of your prompt terminates that match — for itself, and for every token behind it, including the accumulated conversation that makes long agent runs expensive in the first place.

Three takeaways:

1. Move the volatile text down. Timestamps, turn counters, status lines, scratchpads, freshly retrieved chunks — newest message, not the system block. One line of code, roughly a 6x input-cost reduction on the layout I modelled, approaching 10x on the long-TTL write premium that multi-minute tool loops actually need.
2. Write the cost model before the agent. Ten cells from the pricing page, one formula, and you know whether the architecture is affordable before you have a bill to argue with.
3. Instrument the prefix, not just the request. Read and write tokens tell you that you missed. diffPrefix tells you why — and the why is always a timestamp, a UUID, or an object whose keys were enumerated in the wrong order.

The broader point, and the one I think Cameron's framing gets right: caching isn't a flag you flip at the end. It's a constraint on how you assemble the prompt, and it belongs in the same review as your data model. Teams that treat prompt layout as architecture get the 10x. Teams that treat it as wording find out from an invoice.

Further Reading

• Anthropic prompt caching documentation — breakpoints, minimum cacheable prefix lengths, TTL behaviour, and usage fields.
• Anthropic pricing — current base input price and the cache read/write multipliers; verify before modelling.
• OpenAI prompt caching guide — the automatic-prefix-matching model, minimum lengths, and where cached tokens appear in usage.
• OpenAI API pricing — cached input pricing for the models you actually call.
• Google Gemini context caching — implicit versus explicit caching, including per-token-hour storage costs.
• Artificial Analysis — George Cameron's organisation; the talk itself was an AI Engineer session, and I could not locate a first-party written version to link.
• BigGo Finance summary of the talk — aggregator link for the $21 → $200 anecdote and the 10x claim; secondary reporting, treat the figures accordingly.