Instrumenting Agent Loops for Cost per Successful Task

Agent systems usually get their cost story wrong in the same way: teams instrument what the provider bills — tokens, requests — then make decisions about the thing that actually matters, which is completed work. This post shows how to instrument an agent loop so the number on the dashboard is cost per successful task rather than cost per token. You will build, in TypeScript and running entirely locally, one span per model call and one per tool call, a three-level token budget with reserve/settle semantics, a cache key scheme that survives multi-tenancy, a model-routing ladder with an explicit stop condition, and a harness that turns a synthetic workload into a cost-per-success figure with p50/p95 spread. Nothing in it depends on a specific model provider staying still.

A BigGo Finance report dated 9 November 2026, headlined "Nvidia Unveils 'Agentic AI Stack' — 'Token Usage Doubling Every Month, Inference Cost Is the Key'," gives the problem a number-shaped hook: it says Nvidia used the launch of an agentic AI stack to argue that token consumption is doubling monthly and that inference cost is now the binding constraint. That is worth taking seriously, and worth reading precisely, because it is a claim about aggregate workload, not about your unit economics.

The Position This Post Takes: Cost per Successful Task

Cost per token is a vendor metric. Cost per request is an infrastructure metric. Neither survives first contact with a loop that re-sends its own context on every step and fails 15–30% of the time. The only number that supports a decision is cost per successful task, and most teams can't compute it: they never defined "successful", and they discard the spend attached to failed runs. I'll argue that with arithmetic you can re-derive, not with a benchmark.

What You Will Have Running by the End

A budget guard and a routing ladder are only useful if you can see what they did. By the end you'll have the span schema, the reserve/settle budget class, the cache-key function, the escalation stop condition, and the NDJSON harness that produce the tables below — plus the jq and PromQL queries that turn those spans into a cost-per-success figure and an alert that fires when it regresses.

What Nvidia's Token-Doubling Claim Says — And What It Doesn't

What the Public Reporting Contains

The item I had access to is a headline and a one-line summary. It states that Nvidia unveiled an "Agentic AI Stack" and argued token usage is doubling every month, making inference cost the key constraint for agentic systems. The summary doesn't include the measurement methodology, the time window, the token mix, or whether the doubling refers to a specific Nvidia platform, a customer cohort, or the market as a whole.

I could not retrieve the underlying Nvidia presentation or a primary Nvidia blog post behind the figure at the time of writing. So the honest classification is:

Reported (secondary source): Nvidia launched an agentic AI stack and characterized inference cost as the key constraint.
Reported but unverified (secondary source): token usage is doubling every month.
Unknown: how that figure was measured, over what population, and whether it is a trailing average, a peak, or a forward projection.

That is thin sourcing, and I'd rather say so than dress it up. Everything after this section is my own engineering work and is labelled as such.

Separating the Vendor Claim From Your Measured Baseline

There is a structural reason to be careful here. A company that sells compute benefits when token volume rises, because volume times price is its revenue. "Tokens are doubling monthly" is, from that vantage point, a demand forecast and a piece of positioning at the same time. That doesn't make it false — agent loops genuinely do consume far more tokens per user action than single-turn chat, as the arithmetic below shows — but it does mean the number should be read as a claim about direction, not as a measured input to your capacity plan.

The measurable part is whatever you can reproduce on your own workloads. A three-step loop and a nine-step loop on the same task differ by roughly 5x in input tokens under a very simple model. That's the number to act on, and you can compute it in an afternoon.

Why 'Tokens Doubling Monthly' Is a Workload Claim, Not a Price Claim

Total inference spend is the product of three terms:

spend = tasks × (tokens per attempt / success rate) × price per token

Doubling the middle term doubles spend only if the third term holds constant. Per-token prices in this market have moved in the opposite direction repeatedly over the last two years, and that alone can swamp a 2x volume increase. So the report's framing — volume up, therefore cost is the constraint — is only true in a world where price per token is flat or rising. It isn't.

The defensible version of the claim is narrower and more useful: for a given price sheet, agentic loops raise cost per unit of user-visible work by an order of magnitude over chat, and the multiplier is driven by loop structure, not by model choice. That's a statement you can test. The rest is market commentary.

Why Agent Loops Break Per-Token Accounting

Context Resend Is the Dominant Cost Term

Every stateless completion API re-sends the entire transcript. That's inherent to the request/response shape: the provider holds no session state, so the conversation is reconstructed on each call. The consequence is that input tokens grow roughly quadratically with step count, because each step pays for the prefix plus everything appended so far.

Define:

P = fixed prefix tokens (system prompt + tool schemas) = 1,200
T = task tokens = 150
g = mean tokens added per step (assistant output + tool result) = 400

Input tokens at step k is P + T + (k−1)·g. Total across n steps:

input(n) = n·(P + T) + g·n·(n−1)/2

That n²/2 term is the whole story. It is also why "we'll just add a reflection step" is a surprisingly expensive sentence.

Tool Output Is Unbounded Input You Did Not Write

The g term is not under your control in the way you think. Assistant output you can cap with max_tokens. Tool results you cannot, unless you explicitly do so. A search_docs call that returns 12 KB of markdown is 3,000 tokens that get re-sent on every subsequent step. A database query that returns 400 rows is worse, and it will land in the transcript five more times.

This is the single most common source of silent cost regression I've seen in agent code: someone improves a tool's output format to be more helpful to the model, and per-task cost triples. The tool span needs a resultTokens attribute for exactly this reason.

Retries and Reflection Steps Multiply the Cheapest Calls

The cheapest calls in the loop are the ones people add for quality: a self-critique step, a "did I answer correctly?" check, a reformat pass. Each one is a full context resend at step n+1, n+2, n+3. Three reflection steps appended to a nine-step loop don't add 33% — they add roughly 3·(P+T+9g) tokens against a baseline whose total is already dominated by the tail.

Retries are worse because they're invisible. A tool that times out and is retried internally is one span with doubled latency; a tool whose failure makes the model try a different approach is two more full-context model calls. Instrument the retry as a first-class span or you will never find it.

A Hand-Computed Token Ledger for a 3-Step vs 9-Step Loop

Using the parameters above and a placeholder price sheet of $3.00 per million input tokens and $15.00 per million output tokens (substitute your own — the ratios are what matter, and prices drift):

Steps	Input tokens	Output tokens	Total	Input cost	Output cost	Cost/run
3	5,250	330	5,580	$0.01575	$0.00495	$0.02070
5	10,750	550	11,300	$0.03225	$0.00825	$0.04050
7	17,850	770	18,620	$0.05355	$0.01155	$0.06510
9	26,550	990	27,540	$0.07965	$0.01485	$0.09450

Tripling the step count from 3 to 9 raises cost per run by 4.6x. Success probability does not rise by 4.6x over the same interval; in most real workloads it falls as steps increase, because longer loops have more places to go wrong. That divergence is the entire argument for measuring cost per success.

The prefix share is worth noting too: at 3 steps the fixed 1,200-token prefix is 68.6% of all input tokens; at 9 steps it's 40.7%. Prefix caching is therefore a massive lever on short loops and a modest one on long ones — the opposite of the intuition most people have.

The Metric That Matters: Cost per Successful Task

Defining 'Successful' Before You Can Measure It

"Success" has to be a predicate the calling system evaluates, not the model. The model saying "I have completed your request" is not evidence. Three workable definitions, in increasing order of cost:

Schema-validated output. The response parsed against a schema and all required fields present. Cheap, automatable, doesn't prove correctness.
Deterministic ground truth. The task has a checkable answer — a test suite passes, a SQL query returns the expected row count, a file hash matches. Strongest option when available.
Downstream acceptance. A human accepted the draft, the ticket was not reopened within 14 days, the invoice reconciled. Ground truth for business value, but the label arrives late.

Pick one, write it down, version it. A success-rate dashboard that silently changed its definition last quarter is not a trend line.

The Formula, and Why Failed Runs Belong in the Numerator

cost_per_success = Σ(cost of every attempt, including failed ones) / count(attempts where success predicate == true)

The numerator includes failures. This is not pedantry — it's the difference between the two numbers you'll be tempted to report. If 30% of nine-step runs fail, the cost per success is $0.09450 / 0.70 = $0.135, nearly 6x the three-step loop's $0.02070 / 0.90 = $0.023. Reporting "$0.0945 per run" hides the fact that you're paying for the failures too.

The correct denominator is successful attempts, not attempts started. Dividing by attempts started gives you cost per attempt, which is a legitimate capacity-planning number and a completely misleading unit-economics number.

Why Cost-per-Token and Cost-per-Request Hide the Real Number

Cost per token is constant by construction within a pricing tier, so it can only move when you change providers or models. It's a vendor comparison metric, useless for loop tuning: an optimisation that cuts tokens 40% leaves cost per token unchanged at exactly $3.00/M.

Cost per request is worse in agentic systems because "request" is ambiguous in the one place it matters. A single user action fans out into nine model calls and four tool calls. Average them together and you get a number that changes when your tool mix changes for reasons unrelated to economics.

Four Cost Metrics and What Each One Conceals

Metric	What it answers	What it conceals
Cost per token	What the provider charges	Everything about your loop; flat by design
Cost per request	Spend ÷ HTTP calls	That one task is 9 requests; tool mix shifts the denominator
Cost per task attempt	Spend ÷ runs started	Treats failed runs as if they produced work
Cost per successful task	Spend ÷ runs meeting the success predicate	Needs a fixed predicate and a late-arriving outcome join
Cost per successful task, p95	Tail economics of the slow, retry-heavy path	Nothing — this is the number that should page you

Instrumenting the Agent Loop

Span Schema: One Span per Model Call, One per Tool Call

Model it as a three-level trace: one task span per user-visible request, one model_call span per completion, one tool_call span per tool invocation. Tool spans matter as much as model spans — a tool result becomes input tokens on the next model call, so the cost is realised in the model span but the cause lives in the tool span.

export type SpanKind = "task" | "model_call" | "tool_call";

export interface Usage {
  inputTokens: number;
  outputTokens: number;
  cachedInputTokens: number;
  estimated: boolean;
}

export interface Span {
  spanId: string;
  runId: string;
  parentId: string | null;
  kind: SpanKind;
  name: string;
  startedAt: string;
  durationMs: number;
  usage?: Usage;
  attributes: Record<string, string | number | boolean>;
}

export function emit(span: Span): void {
  // One JSON object per line; ship it to your log pipeline.
  process.stdout.write(JSON.stringify(span) + "\n");
}

Code: A Minimal Instrumented Agent Loop in TypeScript

type Msg =
  | { role: "user"; content: string }
  | { role: "assistant"; content: string; toolCalls: ToolCall[] }
  | { role: "tool"; content: string; toolCallId: string };

export async function runTask(opts: {
  runId: string;
  tenantId: string;
  task: string;
  client: ModelClient;
  tools: ToolRegistry;
  budget: Budget;
  maxSteps: number;
}) {
  const taskSpanId = newId();
  const t0 = Date.now();
  const transcript: Msg[] = [{ role: "user", content: opts.task }];
  let costUsd = 0;
  let steps = 0;
  let outcome: "success" | "failed" | "budget_exhausted" = "failed";

  while (steps < opts.maxSteps) {
    steps += 1;

    const estimate = opts.client.estimateInputTokens(transcript);
    const hold = opts.budget.reserve(estimate);
    if (!hold.ok) {
      outcome = "budget_exhausted";
      break;
    }

    const callStart = Date.now();
    const res = await opts.client.complete({
      model: opts.client.tierFor(steps),
      messages: transcript,
      tools: opts.tools.schemas(),
    });
    const usage = res.usage ?? opts.client.estimateUsage(transcript, res);
    opts.budget.settle(hold.id, usage.inputTokens + usage.outputTokens);
    costUsd += opts.client.costOf(usage);

    emit({
      spanId: newId(),
      runId: opts.runId,
      parentId: taskSpanId,
      kind: "model_call",
      name: res.model,
      startedAt: new Date(callStart).toISOString(),
      durationMs: Date.now() - callStart,
      usage,
      attributes: { step: steps, stepCostUsd: round6(opts.client.costOf(usage)) },
    });

    if (res.toolCalls.length === 0) {
      outcome = opts.tools.verify(res.text) ? "success" : "failed";
      transcript.push({ role: "assistant", content: res.text, toolCalls: [] });
      break;
    }

    transcript.push({ role: "assistant", content: res.text, toolCalls: res.toolCalls });

    for (const call of res.toolCalls) {
      const toolStart = Date.now();
      const tr = await opts.tools.invoke(call.name, call.args, {
        tenantId: opts.tenantId,
      });
      transcript.push({ role: "tool", content: tr.content, toolCallId: call.id });

      emit({
        spanId: newId(),
        runId: opts.runId,
        parentId: taskSpanId,
        kind: "tool_call",
        name: call.name,
        startedAt: new Date(toolStart).toISOString(),
        durationMs: Date.now() - toolStart,
        attributes: {
          step: steps,
          toolCallId: call.id,
          // The cost cause lives here, not in the model span.
          resultTokens: opts.client.countTokens(tr.content),
          truncated: tr.truncated,
          cacheHit: tr.cacheHit,
        },
      });
    }
  }

  emit({
    spanId: taskSpanId,
    runId: opts.runId,
    parentId: null,
    kind: "task",
    name: "task",
    startedAt: new Date(t0).toISOString(),
    durationMs: Date.now() - t0,
    attributes: { outcome, steps, costUsd: round6(costUsd), tenantId: opts.tenantId },
  });

  return { outcome, steps, costUsd };
}

Reading Usage From Provider Responses, and the Fallback When Usage Is Missing

Never estimate when the provider tells you. Both major APIs return per-call usage; use it verbatim and record estimated: false so you can filter later.

function usageFromAnthropic(raw: any): Usage | null {
  const u = raw?.usage;
  if (!u || typeof u.input_tokens !== "number") return null;
  const cacheWrite = u.cache_creation_input_tokens ?? 0;
  const cacheRead = u.cache_read_input_tokens ?? 0;
  return {
    // input_tokens excludes cached portions; total prompt is the sum.
    inputTokens: u.input_tokens + cacheWrite + cacheRead,
    outputTokens: u.output_tokens ?? 0,
    cachedInputTokens: cacheRead,
    estimated: false,
  };
}

function usageFromOpenAI(raw: any): Usage | null {
  const u = raw?.usage;
  if (!u || typeof u.prompt_tokens !== "number") return null;
  return {
    inputTokens: u.prompt_tokens,
    outputTokens: u.completion_tokens ?? 0,
    cachedInputTokens: u.prompt_tokens_details?.cached_tokens ?? 0,
    estimated: false,
  };
}

The fallback matters for streaming, for local models, and for providers that omit usage on errors. Use a real tokenizer where you have one and a conservative ratio where you don't:

function estimateUsage(transcript: Msg[], res: ModelResponse): Usage {
  const input = transcript.reduce((n, m) => n + countTokens(m.content), 0);
  return {
    inputTokens: input,
    outputTokens: countTokens(res.text),
    cachedInputTokens: 0,
    estimated: true,
  };
}

Two rules for the fallback. Mark every such span estimated: true and keep estimated spans out of the headline dashboard — they bias low, and once they're mixed in you can't separate them again. Then verify the estimator against provider-reported usage on a sample. In a 200-call check I'd expect a tokenizer-based counter to land within a couple of percent of provider numbers, and a chars/4 heuristic to be off by 15–20% on code-heavy transcripts; if you're using chars/4, treat it as a budget guard, never as a billing record.

Joining Runs to Outcomes Without Leaking Prompt Content Into Telemetry

The outcome label usually arrives from a different system than the agent — a test runner, a review queue, a reconciliation job. Join on runId, and emit the outcome as a separate span rather than rewriting the task span, so late-arriving labels don't require mutable telemetry.

For prompt content: hash it, don't ship it. Record promptSha256 (first 16 hex chars) for dedupe and promptTokens for accounting, and keep the raw text in your own store keyed by the same runId. Prompts contain user data, credentials in tool results, and occasionally things you'd rather not have sitting in a log aggregator with a 30-day retention policy.

attributes: {
  promptSha256: sha256(canonical(transcript)).slice(0, 16),
  promptTokens: usage.inputTokens,
  outcome: "success",
  successRuleVersion: "ticket-triage-v3",
}

Enforcing Token Budgets at Step, Task, and Tenant Level

Why One Global Budget Is Not Enough

A single global budget is too coarse — it lets one runaway task eat the month, and it can't answer "which tenant is expensive." Three levels do the work:

Level	Limit answers	Typical shape	On exhaustion
Step	Did this single call exceed sanity?	32k input tokens	Hard fail, log full transcript hash
Task	Is this run converging?	120k tokens or `maxSteps`	Terminate, emit `budget_exhausted`
Tenant	Is this customer's usage sustainable?	Daily/monthly quota	Reject at submission, not mid-loop

The task-level limit is the one that catches loops. The tenant-level limit is the one that catches a customer pointing a crawler at you.

Reserve Before the Call, Settle After It

You cannot know a call's exact token cost before making it, so reserve an upper-bound estimate and settle against actual. Get this wrong and you have a check-then-act race in which concurrent tasks all pass the same budget check and collectively overshoot by however many are in flight.

export class Budget {
  private spent = 0;
  private reserved = 0;
  private seq = 0;
  private holds = new Map<number, number>();

  constructor(
    private readonly limit: number,
    private readonly name: string,
  ) {}

  reserve(estimate: number):
    | { ok: true; id: number }
    | { ok: false; reason: string } {
    if (this.spent + this.reserved + estimate > this.limit) {
      return {
        ok: false,
        reason:
          this.name + ": spent=" + this.spent +
          " reserved=" + this.reserved +
          " limit=" + this.limit,
      };
    }
    const id = ++this.seq;
    this.holds.set(id, estimate);
    this.reserved += estimate;
    return { ok: true, id };
  }

  settle(id: number, actual: number): void {
    const held = this.holds.get(id) ?? 0;
    this.holds.delete(id);
    this.reserved -= held;
    this.spent += actual;
  }

  child(name: string): Budget {
    return this; // or a real sub-budget with its own limit
  }
}

The reservation should over-estimate input plus max_tokens. That's deliberate: under-reserving defeats the purpose, and settling refunds the difference immediately, so the only cost of over-reserving is slightly reduced concurrency at the ceiling.

Fail-Open Versus Fail-Closed When a Budget Is Exhausted

Fail-closed (terminate the run) is correct for the task budget. A loop that has burned 120k tokens without converging is not going to converge on step 14.

Fail-open is sometimes correct for the tenant budget, and it is the more dangerous decision. If exhausting a tenant's monthly budget returns an error to a production workflow, you've turned a cost control into an availability incident. The realistic options are a degraded path (smaller model, shorter context, no reflection steps) or explicit rejection at submission time with a clear message. What you must not do is silently continue while logging a warning — that's a cost control that doesn't control cost.

Code: A Budget Guard Wrapping the Model Client

export function withBudgetGuard(client: ModelClient): ModelClient {
  return {
    ...client,
    async complete(req) {
      // Over-estimate: full prompt + the largest output we allow.
      const reserveFor =
        client.estimateInputTokens(req.messages) + (req.maxTokens ?? 4096);

      // Step-level guard, independent of the caller's own reservation.
      if (reserveFor > client.stepLimit) {
        throw new Error(
          "step budget exceeded: " + reserveFor + " > " + client.stepLimit,
        );
      }

      try {
        return await client.complete(req);
      } catch (err: any) {
        if (err?.status === 429 || err?.status >= 500) {
          // Count the failed attempt; providers may still bill partial work.
          throw Object.assign(err, { billableAttempt: true });
        }
        throw err;
      }
    },
  };
}

Note the billableAttempt flag. Failed calls aren't always free — a truncated stream, a mid-generation error, or a timeout after the prompt was processed may still be billed. Attributing failed attempts to the run's cost is a habit worth forming early, because the alternative is a dashboard that quietly under-reports exactly when things are going badly.

The Truncation Trap: Budgets That Silently Change the Prompt

The most damaging budget implementation is the one that "helps" by shortening the prompt. Truncate tool results to fit and two things happen:

You invalidate the prefix cache. Any change to the prefix means a cache miss on the whole conversation, and cache writes typically cost more than ordinary input tokens. A guard that saves 2,000 input tokens while forcing a full re-write of a 5,000-token prefix is a net loss.
The model re-fetches what you removed. Drop the third tool result and a competent model calls the tool again — adding a full-context round trip that costs more than the truncation saved.

If a prompt must shrink, do it at composition time, deterministically, per tool, before the first model call — with the tool itself returning a summarized form and recording truncated: true. Never do it inside the budget guard.

Model Routing as a Cost Control

Escalation Ladders With an Explicit Stop Condition

The pattern: start at the cheapest tier that historically solves this task class, verify, escalate only on verification failure. The cheap tier is often 10–30x cheaper per token than the frontier tier, so even a 60% first-tier solve rate wins.

The stop condition must be explicit and include cost, not just quality:

escalate(tier n -> n+1) iff
  verifier(tier n output) == fail
  AND steps_remaining >= 2
  AND projected_remaining_cost <= expected_savings_from_success

That last clause is the one people omit. Escalating a task whose remaining budget can't cover the escalated call plus a verifier produces a guaranteed loss: you pay the escalation and still fail.

The Verifier Step and Its Own Cost

The verifier is a model call. It has a full prompt, and it has to be instrumented like any other. Two failure modes follow from forgetting that:

The verifier costs more than the delta it protects. If the verifier runs on every run and only 20% of runs would have failed, its cost is amortised across 100% of traffic. At $0.008 per verification on a $0.021 run, you've added 38% overhead to catch failures worth less than that.
The verifier is not calibrated. A verifier that passes everything is free and useless; one that fails everything doubles your spend. Measure its precision and recall on a labelled set before deploying it, and re-measure when the model version changes.

A cheap workaround: make the verifier a checklist prompt with max_tokens: 64 and a forced JSON response. The verdict token count should be in the tens, not the thousands.

Measuring Router Quality: Success Delta per Routed Tier

The metric is the difference in success probability attributable to the routing decision, not the tier's standalone success rate — tiers see different task distributions, so raw rates aren't comparable.

Segment	Runs	Success at entry tier	Success after escalation	Δ success	Δ cost/run	Δ cost/success
Short lookups	6,200	0.94	0.96	+0.02	+$0.031	+$0.006
Multi-doc synthesis	1,450	0.61	0.88	+0.27	+$0.042	−$0.029
Structured extraction	3,100	0.89	0.91	+0.02	+$0.038	+$0.019

Read the table by the last column. For synthesis, escalation pays for itself — the higher cost per run is more than offset by the failure rate it removes. For short lookups and structured extraction, escalation is a net loss: you're paying 30–40% more per run to move success by two points, and cost per success goes up. Those numbers illustrate the shape to expect; run your own before acting on them, because the crossover point is workload-specific.

Caching That Actually Pays for Agent Loops

Prefix Caching and the Ordering Constraints It Imposes

Prefix caching is the highest-leverage optimisation available, because the fixed prefix is re-sent on every single step and it's byte-identical each time. Providers implement it by caching a prefix of the request; if any byte in the prefix changes, everything after it misses.

That imposes a strict ordering rule:

[ system prompt ]            <- never changes
[ tool schemas ]             <- changes only on deploy
[ tenant policy / persona ]  <- changes rarely, bounded set
[ few-shot examples ]        <- changes rarely
[ ---- cache boundary ---- ]
[ task input ]               <- changes per task
[ tool results ]             <- changes per step
[ timestamps, request ids ]  <- MUST be last

Putting a timestamp, a random request ID, or a per-user greeting at the top of the system prompt destroys the cache for every request, globally, and costs nothing visible until the bill arrives. This is the most common prompt-caching bug in the wild.

With a 94% cache-read rate on the 2.52M prefix tokens in the harness workload below, and cached reads billed at a tenth of the input rate, spend falls from $16.89 to $10.49 and cost per successful task from $0.0387 to $0.0241 — a 38% reduction for a prompt reordering. Note also that providers impose a minimum cacheable prefix length, so short prompts won't cache at all; check the documentation for your provider and model.

Deterministic Tool-Result Caching and Key Design

Cache tool results, not model outputs. Tool results are deterministic functions of their inputs (when they aren't, they shouldn't be cached), and they drive the largest unbounded cost term.

function toolCacheKey(
  tool: string,
  args: Record<string, unknown>,
  scope: { tenantId: string; snapshotId: string },
): string {
  return createHash("sha256")
    .update(
      JSON.stringify({
        v: TOOL_CACHE_VERSION,      // bump on any schema change
        tool,
        args: canonicalize(args),   // sorted keys, normalized types
        tenant: scope.tenantId,     // never cross tenants
        snapshot: scope.snapshotId, // pin to a data version
      }),
    )
    .digest("hex");
}

Four elements, each load-bearing:

v — without a version, a change to a tool's output shape silently serves stale results in a new format.
canonicalize — {a:1,b:2} and {b:2,a:1} are the same request and must produce the same key. Unsorted JSON gives you a zero hit rate and no error.
tenant — omitting this is a cross-tenant data leak, not a cache miss.
snapshot — a cache with no data-version pin serves yesterday's answer to today's question, which is fine for some tools and unacceptable for others.

Where Semantic Caching Quietly Breaks Correctness

Embedding-similarity caching over model outputs is where I'd draw a hard line. Two prompts with 0.94 cosine similarity can differ in the only important syllable: the account number, the date range, the negation. A cache that returns run A's answer to run B's question doesn't throw an error — it produces a plausible, wrong, confidently-worded result, and it does it faster than the correct path would have. That's the worst failure mode a system can have.

Where it's defensible: read-only lookups against a static corpus where the answer doesn't depend on user identity or entitlement. Where it is not: anything that touches per-tenant data, anything with a numeric or date parameter, anything where a wrong answer incurs a real cost. If you must use it, gate it behind a similarity threshold you validated against labelled near-misses, and never let it serve a response that bypasses your normal verification step.

Verifying Cache Gains Instead of Assuming Them

Don't assert caching works — measure it. The span schema already gives you the inputs: cachedInputTokens from usage, plus a cacheHit attribute on tool spans.

jq -s '
  [ .[] | select(.kind=="model_call") ]
  | { calls: length,
      totalIn: (map(.usage.inputTokens) | add),
      cachedIn: (map(.usage.cachedInputTokens) | add) }
  | . + { hitRate: (.cachedIn / .totalIn) }
' runs.ndjson

Two checks catch the common failures. If hitRate is below roughly 0.8 on a workload with a large fixed prefix, something in your prefix is varying per request — diff two rendered prefixes byte-for-byte and you'll find it in minutes. If hitRate is high but cost per success didn't drop, your cached tokens weren't the expensive ones: you cached a prefix that was small relative to tool results, and the win is somewhere else.

Reading the Dashboard Without Fooling Yourself

p50 Versus p95 Cost per Task

Means hide the tail, and the tail is where agents spend money. In the harness below the mean cost per run is $0.0338, the p50 is $0.0207, and the p95 is $0.0651 — the p95 run costs 3.1x the median. That gap is the loop-length distribution showing through.

Report p50 and p95 side by side. A rising p95 with a flat p50 means a minority of runs are taking more steps, which usually means a tool is failing intermittently or a task class is drifting out of the router's competence. Neither shows up in a mean until the tail is large enough to matter, by which point you've been paying for it for a week.

The Step-Count Histogram as an Early Warning

The step histogram moves before the success rate does. When a tool starts returning 429s, the first signal isn't a success-rate drop — it's the mode shifting from 3 steps to 6, because the model is retrying. Cost per task rises immediately; success rate falls later, once retries exhaust the step budget.

Track the histogram as a first-class chart, segmented by task class, with the p95 step count on the same axis. A p95 step count that moves from 6 to 9 while p50 stays at 3 is a tool problem, not a model problem — and you'll find it a day earlier than you would from cost alone.

Alerting on Economics Rather Than Latency

Latency alerts fire when the network wobbles. Economic alerts fire when something structural changed. The alert I want, roughly:

alert: CostPerSuccessRegression
expr: |
  (
    sum(rate(agent_cost_usd_total[1h]))
    / sum(rate(agent_success_total[1h]))
  ) > 1.5 * (
    sum(rate(agent_cost_usd_total[24h] offset 7d))
    / sum(rate(agent_success_total[24h] offset 7d))
  )
for: 30m
annotations:
  summary: "cost per successful task is 1.5x the trailing weekly baseline"

Two things make this alert useful rather than noisy. It compares against the same workload from a week ago instead of a fixed threshold, so organic mix shifts don't page anyone. And it fires on cost per success, so a spend increase that comes with a proportional quality increase stays quiet — the correct behaviour for a system you're actively improving.

Pair it with a guard on the step histogram so you get a leading indicator:

alert: AgentStepCountTailFat
expr: |
  histogram_quantile(0.95, sum(rate(agent_steps_bucket[30m])) by (le, task_class))
  > 1.5 * histogram_quantile(0.95, sum(rate(agent_steps_bucket[30m] offset 1d)) by (le, task_class))
for: 20m

A Reproducible Harness for Cost per Successful Task

Synthetic Workload and Tokenizer-Based Accounting

The harness generates tasks with a controlled step-count distribution and computes tokens from the ledger formula, so results are reproducible and attributable to structure rather than to a model's mood.

## Generate 500 synthetic tasks with a fixed seed and record every span.
node --experimental-strip-types harness/run.ts \
  --tasks 500 \
  --seed 42 \
  --prefix-tokens 1200 \
  --task-tokens 150 \
  --step-growth 400 \
  --price-in 3.00 \
  --price-out 15.00 \
  | tee runs.ndjson

The generator draws a step count per task from {3: 0.60, 5: 0.25, 7: 0.10, 9: 0.05} and a per-class success probability {3: 0.90, 5: 0.86, 7: 0.80, 9: 0.70}, then applies the ledger formula. Every number below is arithmetic over those parameters — not a measurement of any provider, and you can re-derive it by hand from the table earlier in this post.

What to Record per Run

Per run, the minimum viable record is: runId, tenantId, taskClass, steps, outcome, costUsd, inputTokens, outputTokens, cachedInputTokens, estimatedCalls, successRuleVersion, and modelTiers[]. Per span, everything in the schema above plus resultTokens on tool calls. The estimatedCalls count lets you exclude runs whose cost is partly guessed.

jq -s '
  [ .[] | select(.kind=="task") ] as $t
  | ($t | map(.attributes.costUsd) | add) as $spend
  | ($t | map(select(.attributes.outcome=="success")) | length) as $ok
  | {
      runs: ($t | length),
      successes: $ok,
      spendUsd: (($spend * 10000) | round / 10000),
      costPerSuccess: ((($spend / (if $ok == 0 then 1 else $ok end)) * 10000) | round / 10000)
    }
' runs.ndjson

Observed Output From the Harness

$ node --experimental-strip-types harness/run.ts --tasks 500 --seed 42
workload: 500 synthetic tasks, seed 42
step distribution: 3:300  5:125  7:50  9:25
tokens in : 4,475,000
tokens out:   231,000
tokens total: 4,706,000
prefix tokens (resendable): 2,520,000  (56.3% of input)

runs: 500   successes: 436   failed: 64
cost/request       : $0.0338
cost/task-attempt  : $0.0338
cost/success       : $0.0387
tokens/success     : 10,794
p50 cost/run       : $0.0207
p95 cost/run       : $0.0651
p99 cost/run       : $0.0945

with prefix cache @ 94% hit rate, cached reads at 10% of input price:
cost/success       : $0.0241   (-37.7%)

Three things to take from this. The gap between cost per attempt ($0.0338) and cost per success ($0.0387) is 14.5% — that's the price of the 12.8% failure rate, and it grows fast as the failure rate does. The p95/p50 ratio of 3.1x says the tail is driven by loop length, not price variance. And the cache win is real but bounded: 38%, not the 90% that "cached tokens are 10% of price" would suggest, because the prefix is only 56% of input tokens in this workload.

What This Analysis Does Not Tell You

Vendor Growth Claims and Where Their Numbers Come From

Nothing in this post validates or refutes "token usage doubling every month." The harness runs a synthetic workload I designed; it says nothing about market growth, and it should not be cited as if it did. Whether the doubling figure is a trailing measurement, a cohort number, or a projection is unknown from the public reporting I had access to. Treat it as a directional claim from an interested party until the underlying methodology is public.

Sample Size, Pricing Drift, and Other Honest Limits

The arithmetic is exact; the workload is invented. Cost figures are deterministic given the parameters. The parameter values — 1,200-token prefix, 400-token step growth, 0.90/0.70 success rates — are plausible for a tool-using text agent and are not measured from a real system. Substitute your own; the ratios shift, and the direction of the conclusion is what transfers.
Prices used are $3.00/M input and $15.00/M output. Placeholders. Per-token prices have changed repeatedly in both directions, and cached-read discounts are provider-specific. Recompute against your current sheet before making a decision.
The routing table is illustrative. The crossover point at which escalation pays for itself is workload-specific and moves when a model version changes. Measure it on your own task classes.
Verification is the weakest link. Every cost-per-success number inherits the quality of the success predicate. A verifier with 85% precision will produce a cost-per-success figure with roughly that much noise in the denominator — and the noise isn't random, it's biased toward whichever failure mode your verifier misses.
Untested here: the behaviour of provider-side caching across model version changes, the billing treatment of mid-stream failures, and the effect of context-window pressure on long-loop success rates. Each of these could move the numbers materially.

The defensible conclusion is narrow: loop structure dominates unit economics, failed runs belong in the numerator, and cost per successful task is the only metric that supports a build-versus-route-versus-degrade decision. That holds whether or not token volume is doubling.

Introduction