Cost per Useful Agent Task: Why Cached Tokens Break Naive Inference Pricing Math

Why Naive Token Pricing Breaks Down for Agent Workloads

Most inference cost estimates start with the same multiplication: total tokens times the headline price per million. For a single-shot chat call, that is roughly right. For an agent loop it is wrong, and wrong in a way that changes build decisions, because an agent re-sends the same system prompt, tool schemas, retrieved context and conversation history on every turn.

This post works through why cached tokens break that naive arithmetic, how to derive the cache-hit rate at which caching actually pays for itself, and how to define and measure a cost per useful agent task that survives contact with an invoice.

On a cache-aware provider, the re-sent prefix is not billed at the input rate. It is billed at a cache-read rate, typically a small fraction of input, plus a one-off write premium on the first turn. So tokens × price overstates agent cost, sometimes by 4x or more.

It also understates the number that actually matters, because it ignores the runs that failed. The metric I want on the dashboard is cost per useful task: total spend, retries and human cleanup included, divided by the count of tasks that passed a criterion I wrote down before running anything. That second number is the one compared against the value of the work. The token count is an implementation detail.

The Naive Formula and Where It Breaks

What per-token billing actually charges

A cache-aware bill is not one number. It is four line items with different multipliers:

The exact multipliers are provider-, model- and sometimes date-specific, and they move. On the rate cards I have priced, a write runs somewhere around 1.25x to 2x base input depending on cache lifetime, and a read lands near 0.1x base or a flat 50% discount. Some providers charge no write premium at all: caching is automatic and a miss is simply billed at base input. Others charge storage per hour for an explicitly created cache.

Agents repeat prefixes by construction

A tool-calling loop has a stable head and a growing tail. The head is the system prompt, the tool schemas, the injected retrieved context, any policy text. The tail is the conversation so far. Every turn re-sends both.

If the head is 12,000 tokens and the loop runs 40 turns, the naive formula charges 480,000 input tokens for work that needed to transmit that head once. Turn count multiplies the head at full price. That is why long agent loops produce bills that look impossible next to the token counts in the logs.

A 40-turn tool loop, costed both ways

Take a 12,000-token stable prefix, 40 turns, 300 output tokens per turn. The rates below are placeholders chosen to keep the arithmetic legible: input $3.00/M, output $15.00/M, cache write $3.75/M, cache read $0.30/M.

Naive, no caching:

input   480,000 tok x $3.00/M  = $1.4400
output   12,000 tok x $15.00/M = $0.1800
total                          = $1.6200

Cache-aware, one write then 39 reads:

write    12,000 tok x $3.75/M  = $0.0450
read    468,000 tok x $0.30/M  = $0.1404
output   12,000 tok x $15.00/M = $0.1800
total                          = $0.3654

Same task, same visible output, 4.4x apart.

Now the case that gets missed. Caching enabled, prefix never reused:

write    12,000 tok x $3.75/M  = $0.0450   <- paid anyway
input   468,000 tok x $3.00/M  = $1.4040
output   12,000 tok x $15.00/M = $0.1800
total                          = $1.6290

That is 0.6% more than not caching at all. Enabling caching is not free; it is a bet on reuse.

One simplification worth stating: I charged only the fixed head as reads. A real loop re-sends the growing history too, and on providers that match the longest prefix, the read line item grows with turn count. Both columns in a real bill are larger. I am after the ratio, not the absolute.

How Caching Changes the Arithmetic

Cache write is a premium, cache read is a discount

One-time premium to store the prefix, discount on every reuse. That is the whole model. Everything else — chunking granularity, minimum cacheable length, TTL tiers — only decides whether you actually collect the discount.

Deriving the break-even cache-hit rate

Let base be the input rate, write the write rate, read the read rate, and h the fraction of prompt tokens served as reads. Caching beats no caching when:

h > (write - base) / (write - read)

With my placeholders: (1.25 - 1.00) / (1.25 - 0.10) = 0.25 / 1.15 ≈ 0.217, so a 22% hit rate. Below 22% you are paying more than you would with caching off. Above it, savings scale roughly linearly with h for the constant part of the prefix.

There is a second framing, and operationally it is the more useful one. How many reuses does the write premium need to pay for itself?

r > (write - base) / (base - read) = 0.25 / 0.90 ≈ 0.28

Less than one. On most published rate cards the break-even is so low that the arithmetic is uninteresting. Hit rate is the interesting part, because the failure mode is not "caching was a bad bet" — it is "the prefix quietly stopped matching in production and nobody noticed."

TTL and eviction are the invisible cost

This is where the money actually goes.

• Sparse traffic. A cache TTL of minutes against a workload that runs hourly means every request pays the write premium and gets zero reads.
• Per-user prefixes. Injecting a user ID into the system prompt gives you N prefixes with 1/N the reuse each, and N times the writes.
• Timestamps and nonces in the system prompt. Current time: <now> at the top of the prompt guarantees a miss on every call. It is the single most common cache killer I have seen in review.
• Dynamically assembled tool lists. Sort order changes, a feature flag flips, one extra tool appears, and the prefix hash changes.

Any one of these converts a 0.3x read into a 1.25x write. On the input line item that is a silent 4x regression.

Defining Cost per Useful Task

The denominator: only tasks that pass a pre-written criterion count

A task counts only if it passed a criterion you stated in advance. "It looked fine" is not a criterion. Write it down: exact match on K fields, schema-valid and semantically equivalent, test suite green.

The numerator is broader than the token bill. It includes every attempt, including the ones you abandoned, plus human cleanup priced at a loaded hourly rate. If a reviewer spends 25 minutes fixing 41 invoices, that is real cost and it belongs in the numerator next to the tokens.

A worked measurement of cost per useful task

Here is the shape I use, filled with numbers from the previous section so the arithmetic is checkable:

Task:      extract 20 fields from 50 invoice PDFs, one 40-turn tool loop each.
Criterion: exact match on all 20 fields.
Result:    41/50 fully correct; all 9 failures were in one field, tax ID,
           where the model copied the customer number.
Tokens:    50 x $0.4918 (90% observed cache-hit rate)   = $24.59
Cleanup:   25 min review x $90/h loaded                  = $37.50
Total:                                                   = $62.09
Cost per passing task:  $62.09 / 41                      = $1.51

The naive quote for the same work is $1.62 per invoice, or $81.00 for the batch. The cache-aware token bill is $24.59. Neither is the number to report. $1.51 per passing task is, because that is what a passing extraction costs end to end.

The pass rate and cleanup time here are placeholders; the token rates come from the arithmetic above. Substitute your own measured values. What matters is the shape, and the fact that the largest single line item in this example is a human, not a GPU.

Why the failed attempts are the story

Compare two models on the same task class:

B is 40% cheaper per call and 22% more expensive per passing task, and that is before counting the human time its failures generate. The inversion is common and invisible if you only track cost per call. Which side of it you are on is unknowable without measuring the pass rate on your own task class.

A Cache-Aware Cost Calculator

A per-call cost function with rates passed in

// Rates are USD per million tokens. Pass them in; never hard-code them.
export function perCallCost(tokens, rates) {
  return (
    (tokens.input / 1e6) * rates.input +
    (tokens.output / 1e6) * rates.output +
    (tokens.cacheWrite / 1e6) * rates.cacheWrite +
    (tokens.cacheRead / 1e6) * rates.cacheRead
  );
}

export function costPerUsefulTask({ perCallUsd, attempts, passes, cleanupMinutes, loadedRatePerMinute }) {
  if (passes === 0) return { totalUsd: Infinity, costPerPassingTask: Infinity };
  const totalUsd = perCallUsd * attempts + cleanupMinutes * loadedRatePerMinute;
  return {
    totalUsd,
    costPerPassingTask: totalUsd / passes,
    passRate: passes / attempts,
  };
}

Sensitivity sweep across cache-hit rates

Same task as before — 12,000-token prefix, 40 turns, 300 output tokens per turn — across hit rates. Pass rate fixed at 41/50 so the columns stay comparable.

const rates = { input: 3.0, output: 15.0, cacheWrite: 3.75, cacheRead: 0.3 };
const PREFIX = 12000, TURNS = 40, OUTPUT = 300 * TURNS;
const PASS_RATE = 41 / 50;
const rows = [];

for (const h of [0, 0.25, 0.5, 0.75, 0.9, 1]) {
  const slots = PREFIX * TURNS;
  const writes = PREFIX;                       // first turn writes the head
  const reads = Math.round((slots - writes) * h);
  const misses = slots - writes - reads;
  const perCallUsd = perCallCost(
    { input: misses, output: OUTPUT, cacheWrite: writes, cacheRead: reads },
    rates
  );
  rows.push({
    hitRate: (h * 100).toFixed(0) + "%",
    perCallUsd: perCallUsd.toFixed(4),
    perPassingTask: (perCallUsd / PASS_RATE).toFixed(4),
  });
}
console.table(rows);

The values it produces:

Two things to read off it. First, the 0% row is worse than the naive estimate of $1.6200, because the write premium is still paid. "Hit rate 0% with caching on" and "caching off" are different scenarios, and only one of them is the naive case. Second, the curve above 50% is where the money is, and it is exactly the region workload instability pushes you out of.

Where the rates come from

Pull them from the provider's pricing page at the moment you build the report, store them with an effective date, and fail loudly when that date is older than the provider's usual announcement cadence. Hard-coding rates from a blog post — including this one — is how you end up with a dashboard that is confidently wrong for six months.

Where Caching Silently Stops Working

Prefix instability checklist

Every item below changes the prefix and guarantees a miss on every call:

• per-request UUIDs or correlation IDs injected into the system prompt
• timestamps, Date.now(), or "today's date" at the top of the prompt
• retrieved context assembled in a non-deterministic order (Object.keys on a map, Set iteration after async insertion, parallel retrieval with Promise.race)
• JSON serialized, parsed and re-serialized so key order drifts
• user IDs, tenant IDs, session IDs, or feature-flag buckets embedded in the head
• A/B variant prompts that differ in one sentence near the top
• tool lists built from runtime capability checks, so the schema array length changes between deployments

Move anything volatile to the tail of the prompt, after the last cacheable block. That single refactor is usually worth more than a model downgrade.

Instrumenting cache-hit rate

Log the provider-reported cache read and write token counts on every request, then compute reads / (reads + writes + input). Alert on the low percentile, not the mean: a p50 of 0.85 can hide a p05 of 0.0, and the p05 is the traffic that costs you.

Set the alert line at your computed break-even h, and treat a drop below it the way you would treat a latency regression.

Testing cache hits with a three-call probe

I have not run this against a live key for this post, so treat the expected outputs as expected, not observed. The probe is three calls:

const BASE = process.env.BASE_URL, KEY = process.env.API_KEY;
const STABLE = ["You are a log triage assistant.", "Return JSON only.",
  ...Array.from({ length: 40 }, (_, i) => "Rule " + i + ": ignore noise class " + i + ".")].join("\n");

async function call(system) {
  const res = await fetch(BASE + "/chat/completions", {
    method: "POST",
    headers: { "content-type": "application/json", authorization: "Bearer " + KEY },
    body: JSON.stringify({
      model: process.env.MODEL,
      messages: [{ role: "system", content: system }, { role: "user", content: "Reply OK." }],
      max_tokens: 5,
    }),
  });
  const json = await res.json();
  console.log(JSON.stringify(json.usage));
}

await call(STABLE);                                      // 1: cold
await call(STABLE);                                      // 2: should hit
await call("run-id: " + Date.now() + "\n" + STABLE);     // 3: head changed

Expected, on a provider that reports cache counters: call 1 shows zero or near-zero cached tokens, call 2 shows cached tokens close to the prefix length, call 3 shows them back at zero because the counter sits at the front of the system prompt. If call 2 does not show a hit, the prefix is not cacheable as written — check the minimum cacheable length and whether the cache TTL outlives the gap between your calls.

Field names differ per provider: OpenAI-compatible APIs report usage.prompt_tokens_details.cached_tokens; Anthropic reports usage.cache_read_input_tokens and usage.cache_creation_input_tokens separately. Read the response schema for whatever you call. Public documentation for both is linked below.

Cheaper Models and Routing Move the Cost, They Do Not Remove It

Downgrade cost shifting

A cheaper per-token model that needs more turns, more retries, more tool-call corrections, or a longer reasoning trace can raise cost per passing task while lowering cost per call. Verify that per task class; do not assume it. The failure mode is asymmetric, too, because extra turns also multiply your prefix re-sends. If a downgrade takes a 40-turn loop to 70 turns, you have added 75% to the head-token volume before quality enters the picture.

Routing by task value

Route on measured pass rate per task class, refreshed on a schedule, not on which model feels good enough. A routing table that earns its keep looks like this:

The cheap model wins where the task is narrow and the criterion is mechanical, and loses where the criterion is fuzzy and failures need a human.

What the recent coverage claims versus what you can verify locally

Recent coverage in this cluster — thelec.net on prompt caching and compression as cost levers, iPhone in Canada and adgully on OpenAI's GPT-6 Sol/Luna releases, TechnoSports on prompt caching for high-volume apps, all dated 2026-09-22 or 2026-09-23 — reports price cuts in the region of 50% and positions caching, compression and cheaper models as the three main levers on inference spend. Those are press claims about vendor pricing. I have not verified any of them against a live pricing page, and compression in particular cuts the numerator (fewer tokens per request) while leaving the denominator problem untouched: fewer tokens does not make a failed task useful.

What you can verify locally, today: the cached-token counters on your own responses, your own pass/fail counts against a criterion you wrote, and your own invoice. Everything else is a projection until the invoice agrees.

What To Measure Before You Trust the Savings

The rule I use: if a number in a cost deck cannot be traced to one of the three confirmed rows, it is a hypothesis wearing a currency symbol.

Conclusion: Report Cost per Useful Task, Not Tokens

Stop quoting tokens times price. It overstates cache-aware agent cost, it says nothing about whether the work succeeded, and it compares cleanly against nothing a stakeholder cares about. Quote cost per passing task, with the criterion written down and the human cleanup priced in.

Treat cache-hit rate as a first-class production metric, on the same dashboard as p95 latency, with the break-even h as its alert line. Compute that line from the rates on the provider's page rather than from this post.

And keep the two failure modes separate in your head. A hit rate that collapses costs you a multiple on input tokens and shows up on the invoice. A pass rate that collapses costs you a multiple on everything and shows up as a human spending their afternoon on tax ID fields. The second one is the bigger number, and the naive formula cannot see it at all.

Further Reading

• Prompt caching for high-volume apps (coverage) — TechnoSports, 2026-09-22
• Prompt caching and compression as cost levers (coverage) — thelec.net, 2026-09-22
• Reported 50% price cut on GPT-6 Sol/Luna (coverage) — iPhone in Canada, 2026-09-22
• Prompt caching guide (docs) — OpenAI
• Prompt caching with cache breakpoints (docs) — Anthropic
• Context caching (docs) — Google
• API pricing (pricing page) — read the current rates for the model you actually call
• Model pricing (pricing page) — same caveat