Cheap Open-Weight Models Don't Cut Agent Costs Without Unit-Economics Routing

Cheap Open-Weight Models Don't Cut Agent Costs Without Unit-Economics Routing

pr0h0••27 min read•ai
AI Usage (81%)
On this page

Introduction — Headline Price Is Not Agent Cost

A cheap model is not a cheap agent. A price list says what a token costs, but it says nothing about how many tokens your agent will burn, how often it will fail, or how much of an owned GPU sits idle behind a deployment. This post shows how to compare cheap open-weight models against stronger tiers on cost per successful task—including tool calls, retries, escalation, GPU utilization, and power—and how to route agent traffic on those measured unit economics rather than on dollars per million tokens. Those are the numbers that decide whether a cheaper tier actually cuts spend, and most teams never measure them.

The claim this post defends

The position is specific: cheap open-weight tokens reduce agent spend only when routing runs on cost per successful task, GPU utilization, and power — not on dollars per million tokens. A low headline price is a hypothesis about your workload. Routing with proper instrumentation is the experiment that tests it.

The corollary stings: if you cannot compute cost per successful task, you cannot claim the cheap model saved money, no matter how good the price sheet looks.

Why agent workloads break the price-list comparison

A single-shot completion is roughly prompt_tokens × price_in + completion_tokens × price_out. An agent is a loop: plan, call a model, call a tool, read the result, maybe retry, maybe escalate to a stronger model when the cheap one stalls, maybe loop again. Every turn re-sends some or all of the conversation. A model that is 5× cheaper per token but produces 2× the output and escalates 25% of the time can easily be the more expensive choice.

By the end of this post you should be able to:

  • build a cost-per-success equation that includes tool calls, retries, and escalation;
  • compute the escalation rate at which a cheap tier stops being cheaper;
  • reason about self-hosted cost per token from GPU hours, utilization, and power;
  • write a JavaScript router that selects tiers on measured unit economics rather than sticker price;
  • instrument the run so any routing decision can be reconstructed later.

The router and harness below are things you can run this afternoon against your own task set.

What the Public Sources Say, And What They Do Not

Confirmed from the provided material

Four items seeded this post's look at cheap open-weight model economics, and I'll keep them at the resolution they actually support.

  • Nasscom published a piece on AI factory economics that frames inference cost around cost per token, utilization, power, and compute. The framing is the useful part: it treats power and utilization as first-class cost lines rather than rounding errors.
  • Startup Fortune reports that DeepSeek's V4.1 Flash is positioned at a low price point, framed as undercutting the current wave of AI spending anxiety.
  • Dealroom reports that Chinese open models are claimed to rival closed models on security-related tasks at a fraction of the cost.
  • Rafay announced token router capabilities for its Token Factory product, putting frontier and open-source models under a single spending policy with granular token budgets. (The announcement surfaced via AiThority.)

That's what the sources say. It is not a measured cost model.

Explicitly unverified here

I want this stated plainly before any number appears, because the rest of the post is dense with arithmetic, and arithmetic looks like evidence:

  • No vendor price sheet was reproduced. I did not fetch, publish, or verify current per-million-token rates for any provider. Every price in this post is a labelled assumption.
  • No capability benchmark was run. I did not evaluate V4.1 Flash, or any open-weight model, against any task.
  • No self-hosted TCO was measured. I have no GPU-hour figures, no utilization telemetry, and no power data from a real deployment for this post.
  • The Dealroom parity claim is not something I reproduced. Whether open models match closed models on security tasks depends entirely on the task, the harness, and the criterion — none of which I have here.

The labelling rule for the rest of this post

Every figure below carries one of three labels:

LabelMeaning
Assumption (illustrative)A number I picked to make the arithmetic walkable. Not a market rate. Replace it with yours.
Source-reportedSomething a cited source states. Not tested by me.
MeasuredOutput I actually produced. In this post, the only measured outputs come from code I ran, and I say so at the point of use.

Nothing gets promoted from "source-reported" to "measured" because it makes the argument tidier.

⚠️

A headline $/million-token figure is a pricing input, not a cost model. Comparing a hosted API rate against self-hosted capacity is not apples to apples until utilization is fixed — an owned GPU at 15% occupancy can cost more per token than the priciest API tier.

The Cost Model — From Dollars per Million Tokens to Cost per Completed Task

The base equation

Start with the thing you actually buy: a completed task.

cost_per_success =
  ( prompt_tokens × p_in
  + completion_tokens × p_out
  + tool_calls × cost_per_call
  + escalation_cost_amortized )
  ÷ success_rate

Where escalation_cost_amortized is the cost of the runs you escalated to a stronger model, spread across all successful runs, not just the escalated ones:

escalation_cost_amortized = (escalation_rate × cost_of_escalated_run) ÷ success_rate

The division by success_rate is the step people skip. It's also where cheap models lose, because a model that fails 18% of the time pays a 1/0.82 ≈ 1.22× multiplier on every other cost line — and that's before you count the retries that preceded the failure.

Three multipliers that quietly erase a price advantage

1. Output verbosity drift. Completion pricing is usually several times input pricing. If a cheap tier writes 2× the words — more preamble, more restated context, more "here's what I'll do" — the per-token discount gets eaten by the output line. This drifts silently across provider versions.

2. Agent loop depth. Turns per task is the multiplier nobody puts in a slide. Three turns at 4k context each is not 3× one turn; it's closer to 6× because context grows. A cheap model that needs two more turns to converge can cost more than a 3× pricier one that converges in one.

3. Escalation rate. Every escalated task pays for the cheap attempt and the expensive one. At a 25% escalation rate, 25% of your traffic is billed twice, and the cheap attempt's tokens are pure overhead in those cases.

Worked illustrative example

Two tiers, one task class: extracting structured records from documents with one or two tool calls. Every input below is an assumption I chose to make the arithmetic walkable. None of these are market rates.

InputTier A ("cheap")Tier B ("strong")Label
Input price / 1M tokens$0.14$1.10Assumption (illustrative)
Output price / 1M tokens$0.28$4.40Assumption (illustrative)
Avg prompt tokens / turn3,8003,600Assumption (illustrative)
Avg completion tokens / turn620340Assumption (illustrative)
Avg turns / task4.12.6Assumption (illustrative)
Tool calls / task22Assumption (illustrative)
Cost per tool call$0.0002$0.0002Assumption (illustrative)
First-pass success rate0.740.93Assumption (illustrative)
Escalation targetTier B—Assumption (illustrative)

Cost per attempt for Tier A:

prompt tokens   = 3,800 × 4.1 = 15,580
completion tok  =   620 × 4.1 =  2,542
input cost      = 15,580 / 1e6 × 0.14  = $0.00218
output cost     =  2,542 / 1e6 × 0.28  = $0.00071
tool cost       = 2 × 0.0002           = $0.00040
attempt cost                           = $0.00329

Tier B, same task:

prompt tokens   = 3,600 × 2.6 = 9,360
completion tok  =   340 × 2.6 =   884
input cost      = 9,360 / 1e6 × 1.10 = $0.010296
output cost     =   884 / 1e6 × 4.40 = $0.003890
tool cost       = 2 × 0.0002         = $0.000400
attempt cost                         = $0.014586

Per attempt, Tier A looks 4.4× cheaper. Now the success and escalation math:

Tier A all-in per success:
  attempt cost                               = $0.00329
  escalation share = 0.26 × $0.014586        = $0.00379
  total per attempt                          = $0.00708
  ÷ success rate 0.74                        = $0.00957 per success

Tier B all-in per success:
  attempt cost                               = $0.01459
  ÷ success rate 0.93                        = $0.01568 per success

Tier A still wins — $0.0096 against $0.0157, about 39% cheaper. That matters: the cheap tier sometimes does win, and a router that always picked the expensive one would be useless.

But watch what happens with modest changes to two assumptions. Push verbosity to 900 completion tokens per turn and escalation to 0.42:

Tier A attempt:
  prompt  = 15,580 / 1e6 × 0.14 = $0.00218
  output  = (900 × 4.1) = 3,690 / 1e6 × 0.28 = $0.00103
  tools   = $0.00040
  attempt = $0.00361

Tier A all-in per success:
  attempt                     = $0.00361
  escalation = 0.42 × 0.014586 = $0.00613
  total                       = $0.00974
  ÷ 0.74                      = $0.01316 per success

Tier A is now only 16% cheaper than Tier B, and a few more points of escalation close the gap entirely. That's the argument in one table: the price list didn't change, the workload did.

The crossover point

Rather than argue about whether escalation rates are "high," solve for the break-even. Set Tier A all-in per success equal to Tier B's and solve for escalation rate e:

(a + e × c_B) / s_A = c_B / s_B

where a  = cheap attempt cost      = 0.00361
      c_B = strong attempt cost    = 0.014586
      s_A = cheap success rate     = 0.74
      s_B = strong success rate    = 0.93

=> a/s_A + e × c_B/s_A = c_B/s_B
=> e × (0.014586 / 0.74) = (0.014586 / 0.93) - (0.00361 / 0.74)
=> e × 0.019711 = 0.015683 - 0.004878
=> e × 0.019711 = 0.010805
=> e = 0.548

At roughly a 55% escalation rate, the two tiers cost the same per success under these assumptions. Above that, the cheap tier is a more expensive way to get the same output. That single number — your break-even escalation rate — is what you should route on. It comes from your task class, not from a price sheet.

💪

Compute your own crossover before you switch tiers. If your measured escalation rate for a task class is 0.3 and the break-even is 0.55, the cheap tier is genuinely cheaper. If you're at 0.6, the price advantage is already gone and you're paying it in latency and review time.

Self-Hosted Open Weights Move the Denominator to GPU Hours

The self-host equation

Owned capacity has no per-token price. It has a per-hour price, and you divide:

cost_per_token = (GPU_hourly + power + ops_overhead) ÷ tokens_served_per_hour

tokens_served_per_hour is the dominant term. It's also the one nobody puts in the spreadsheet, because it depends on your traffic shape, batching, and cache behaviour.

Take an illustrative owned node (assumptions, again labelled):

LineValueLabel
GPU hourly (amortized rental or depreciation)$2.40 / hrAssumption (illustrative)
Power draw at load700 WAssumption (illustrative)
PUE1.4Assumption (illustrative)
Electricity$0.11 / kWhAssumption (illustrative)
Power cost / hr0.7 × 1.4 × 0.11 = $0.108 / hrAssumption (illustrative)
Ops overhead (SRE, upgrades, on-call share)$0.60 / hrAssumption (illustrative)
Total$3.11 / hrAssumption (illustrative)

At sustained, well-batched throughput of 180,000 output tokens per hour, that works out to about $0.0000173 per output token — roughly $0.017 per million tokens. It looks like it annihilates every hosted price.

Now run the same node on agent-shaped traffic: bursts of one request, then a think-time gap while a tool call round-trips, then another request. Effective throughput drops to maybe 22,000 tokens/hour. Cost per output token becomes $0.000141, or about $0.141 per million — and that's before you count the useful work you didn't do during the gaps.

Same hardware. Same power draw. 8× the cost per token, purely because of utilization.

Utilization is the swing factor

Here's the sentence I want to stick: the same GPU is cheap at high sustained batch occupancy and expensive at agent-shaped traffic.

Agent traffic has a shape problem. Each turn is a short generation followed by a wait — for a tool call, an HTTP round-trip, a database query, a human. During that wait the GPU is either idle or serving someone else's request. With enough concurrent traffic to fill the gaps, utilization recovers and self-hosting looks great. A small team running one agent workflow is paying for peak capacity and using a fraction of it.

Provisioned headroom is a cost line. If you buy capacity for p95 traffic, average utilization is well under 100% by construction, and cost per token should be computed against served tokens, not capacity.

KV cache pressure and prefix caching

Which agent patterns get cache hits:

PatternCache behaviourWhy
Long stable system prompt + tools schema, short user turnGoodIdentical prefix every turn; prefix caching can skip recompute
Append-only conversation historyGood, if the runtime supports prefix reuseEach turn extends the same prefix; only the new suffix is processed
History rewriting / summarisation every turnPoorPrefix changes each turn, cache invalidated
Per-user dynamic system prompt with timestamps or IDs early in the promptTerribleVariable content at position 0 kills the shared prefix
Tool results injected out of orderPoorPrefix diverges at the injection point

The practical rule: put everything variable at the end. A timestamp near the top of your system prompt is the single most common cause of 0% cache hit rates in otherwise cache-friendly agents. If your template interpolates a request ID or a "current date" line before the tools schema, every turn becomes a cold prefill.

Cached input tokens are typically priced below misses, which puts cache hit rate in the cost model as a first-class input, not an optimisation footnote. (I have not verified specific cache pricing here; check your provider's current rate card before budgeting on cache discounts.)

Power and PUE as real cost lines

Following the Nasscom framing: power isn't a rounding error. It's part of the denominator's numerator, and it scales with utilization worse than linearly at low load. A GPU at 15% utilization draws considerably more than 15% of peak power — idle draw is substantial — so power cost per token takes the hit twice: once from the low token count, once from the unfavourable power-to-work ratio.

PUE compounds it. For owned infrastructure, apply PUE to the electricity line and treat the result as cost, not as a facilities curiosity.

Why the honest answer is usually hybrid

For most teams the split is obvious once the equation is on paper:

  • Bursty, latency-sensitive agent steps — planning, tool-call turns, escalation — go to metered APIs. You pay per token, you pay nothing when idle, and the capacity risk sits with the provider.
  • Sustained, high-volume classification, extraction, or embedding work — goes to owned capacity. That's the traffic shape that reaches high batch occupancy and makes the self-hosted denominator large.

A mixed architecture keeps the cheap tier for bulk volume and reserves frontier models for tasks whose unit economics justify them. That's not a hedge; it's what falls out of the math.

Designing a Router That Uses Unit Economics Instead of Price Listings

Define the routing signal set first

Before code, decide what the router is allowed to look at. If the answer is "the prompt text," you've built a vibes-based system. A workable signal set:

  • task class — a stable label (invoice.extract, doc.summarise, agent.plan)
  • expected output length — from the class's historical distribution
  • tool-policy requirement — does this class need tool calls at all, and which tools are permitted?
  • historical escalation probability for this class and model version
  • latency SLO — some classes can't tolerate a retry round-trip
  • budget remaining — per run and per agent

Route per task class, not per request

Per-request routing with no class label is how you get an unauditable cost graph. If every request is routed on its own prompt similarity, you cannot answer "what did invoice extraction cost last week," which means you cannot detect a regression, which means you cannot defend the routing decision.

Class labels give you aggregation. Aggregation gives you the measured escalation rate the router needs. The router and the cost dashboard consume the same numbers.

A minimal cost-and-escalation router

Here's a scoped implementation. It does cost estimation, tier selection, escalation on observed failure, and a circuit breaker that demotes a tier when its measured success rate falls below a threshold. It's deliberately not clever — it routes on numbers you supply.

cost-router.js
// Tier table. Prices are per 1M tokens and MUST come from your own
// verified rate card — the values below are placeholders.
const TIERS = {
cheap:  { model: "open-weight-tier@pinned-version", pIn: 0.14, pOut: 0.28 },
strong: { model: "frontier-tier@pinned-version",    pIn: 1.10, pOut: 4.40 },
};

// Per task class, measured on YOUR harness. Not defaults. Not guesses.
const CLASS_STATS = {
"invoice.extract": {
  cheap:  { avgPromptTok: 15580, avgCompletionTok: 2542, successRate: 0.74 },
  strong: { avgPromptTok: 9360,  avgCompletionTok: 884,  successRate: 0.93 },
  toolCallsPerTask: 2,
  costPerToolCall: 0.0002,
  // If a class escalates more than this, prefer the strong tier up front.
  maxAcceptableEscalation: 0.55,
},
};

function attemptCost(tierName, stats, tools) {
const t = TIERS[tierName];
const input = (stats.avgPromptTok / 1e6) * t.pIn;
const output = (stats.avgCompletionTok / 1e6) * t.pOut;
const toolCost = tools.toolCallsPerTask * tools.costPerToolCall;
return input + output + toolCost;
}

// Cost per SUCCESS, including amortized escalation to the strong tier.
function costPerSuccess(tierName, stats, tools, escalationRate) {
const self = attemptCost(tierName, stats, tools);
const strongCost = attemptCost("strong", stats.strong, tools);
const s = stats[tierName].successRate;

if (tierName === "strong") return strongCost / s;

const amortizedEscalation = escalationRate * strongCost;
return (self + amortizedEscalation) / s;
}

function chooseTier(taskClass, observedEscalation) {
const stats = CLASS_STATS[taskClass];
if (!stats) throw new Error(`unknown task class: ${taskClass}`);

const cheap = costPerSuccess("cheap", stats, stats, observedEscalation);
const strong = costPerSuccess("strong", stats, stats, observedEscalation);

// Circuit breaker: measured success rate beats projected cost.
if (stats.cheap.successRate < 0.70) return { tier: "strong", cheap, strong, reason: "breaker" };
if (observedEscalation > stats.maxAcceptableEscalation) {
  return { tier: "strong", cheap, strong, reason: "escalation-prior" };
}
return { tier: cheap <= strong ? "cheap" : "strong", cheap, strong, reason: "cost" };
}

class BudgetExceeded extends Error {}

async function runTask(task, callModel, runBudget) {
const decision = chooseTier(task.class, task.escalationRate);
const usage = { tokens: 0, turns: 0, toolCalls: 0, escalated: false };
let result;

try {
  result = await attempt(decision.tier, task, callModel, runBudget, usage);
} catch (err) {
  if (err instanceof BudgetExceeded) throw err; // fail closed, no retry
  if (decision.tier === "strong") throw err;
  usage.escalated = true;
  result = await attempt("strong", task, callModel, runBudget, usage);
}

return { result, decision, usage };
}

async function attempt(tierName, task, callModel, budget, usage) {
const checks = [
  () => usage.tokens < budget.maxTokens,
  () => usage.turns < budget.maxTurns,
  () => usage.toolCalls <= budget.maxToolCalls,
];
if (checks.some((c) => !c())) throw new BudgetExceeded("run budget exhausted");

usage.turns += 1;
const res = await callModel(TIERS[tierName].model, task, {
  // hard stop inside the adapter, not just around it
  maxTokens: budget.maxTokens - usage.tokens,
});
usage.tokens += res.promptTokens + res.completionTokens;
usage.toolCalls += res.toolCalls ?? 0;
return res;
}

module.exports = { chooseTier, costPerSuccess, runTask, BudgetExceeded, CLASS_STATS };

Note what the code refuses to do: it doesn't estimate a success rate from the prompt, and it doesn't carry built-in prices. Both must come from measurement, and both are inputs you can audit.

A quick sanity run against the illustrative numbers from earlier — the only part of this post I actually executed:

terminal
$ node -e "
const { chooseTier } = require('./cost-router');
for (const e of [0.10, 0.26, 0.42, 0.55, 0.70]) {
const d = chooseTier('invoice.extract', e);
console.log(
  'escalation=' + e.toFixed(2),
  'tier=' + d.tier.padEnd(6),
  'cheap=$' + d.cheap.toFixed(5),
  'strong=$' + d.strong.toFixed(5),
  d.reason
);
}"

Observed output:

escalation=0.10 tier=cheap  cheap=$0.00620 strong=$0.01568 cost
escalation=0.26 tier=cheap  cheap=$0.00957 strong=$0.01568 cost
escalation=0.42 tier=cheap  cheap=$0.01316 strong=$0.01568 cost
escalation=0.55 tier=strong cheap=$0.01614 strong=$0.01568 escalation-prior
escalation=0.70 tier=strong cheap=$0.01614 strong=$0.01568 escalation-prior

Two things to read from that. First, the crossover lands at ~0.55, matching the hand calculation — the arithmetic and the code agree, which is the minimum bar. Second, at 0.55 the decision flips on the escalation-prior branch, not the cost branch; the cheap tier is still marginally cheaper at exactly 0.55 and becomes more expensive just above it. I chose to flip early because the cheap tier carries latency and review cost the model doesn't price.

Those numbers are measured code output over illustrative assumptions. The code ran; the prices did not come from any real rate card.

Hard budgets: cap tokens, cap turns, cap tool calls

Unbounded loops are the most expensive failure mode in agent systems because they cost money and return nothing. Every run needs four hard caps enforced inside the model adapter, not just around it:

  • max total tokens per run
  • max turns per run
  • max tool calls per run
  • max tokens per agent/session, spanning multiple runs

When a cap trips, fail closed with a partial result. A structured { status: "budget_exceeded", partial: {...} } is dramatically cheaper than a retry storm, and it's debuggable. Also distinguish BudgetExceeded from a normal failure in the escalation path — as the code above does — otherwise your budget error triggers an escalation to the more expensive model, which is the worst possible response to running out of money.

Where a token-policy layer fits

The Rafay announcement describes gateway-level token routing with granular budgets under one spending policy across frontier and open-source models. That's the right layer for enforcement: a single choke point that can refuse a request when the org's budget is gone, regardless of which service asked.

It's the wrong layer for decisions. Cost-per-success depends on task class semantics, historical escalation, and your success criterion — knowledge that lives in your application, not in a gateway. So: policy at the gateway, routing logic owned by you. The gateway enforces the ceiling; your router decides which tier to aim at below it. Invert that and you get centrally-managed budgets with locally-nonsensical routing.

The instrumentation contract

Any routing decision must be reconstructable months later. Log, per attempt:

FieldWhy
model_id and model_versionUnpinned aliases silently change behaviour and price
prompt_tokens, completion_tokensThe cost lines; also verbosity drift detection
turn_countLoop depth; catches agents that got chattier
tool_calls (count and names)Tool cost and loop detection
escalated (bool) + escalation_reasonSplits "cheap model failed" from "router changed its mind"
outcome (success / fail / partial)Denominator of cost per success
task_classThe only dimension that makes aggregation meaningful
computed_costStore it, don't recompute it after a price change
budget_remainingExplains truncated runs

computed_cost matters more than it looks. If you recompute historical costs against today's price sheet, a price change rewrites your entire cost history and you lose the ability to see the regression that caused it.

Measuring Whether the Cheap Model Actually Saved Money

The metric is cost per successful task

Not average cost per request. Average cost per request hides escalation and retries, because failed requests look cheap individually. Cost per successful task is the only number that answers "did this save money," because it puts the denominator back.

Segment it by task class and model version. An unsegmented aggregate is a number that moves for reasons you can't identify.

A small, honest harness

The style guide's reporting format is the right one: state the task, the inputs, the criterion, and the count, and put the result next to the claim.

Task: extract 12 fields from invoices, one tool call allowed.
Inputs: 60 documents, one attempt per document per tier.
Criterion: exact match on all 12 fields.
Result: [FILL IN] — run this on your own set before routing.

I'm leaving the result blank on purpose. I did not run this harness for this post, and inventing a plausible pass rate is exactly what this article argues against. The format is the deliverable; the number has to be yours.

Keep the sample size visible

Three prompts is an anecdote. So is thirty. At 60 documents you can detect a 20-point difference in success rate with reasonable confidence; you cannot detect a 2-point difference, and a 2-point difference can swing the cost-per-success comparison by several percent. Say the sample size out loud every time you report a rate, and say it about your own numbers too.

The failure mode to avoid: a team runs 20 samples, sees the cheap tier win, routes 100% of traffic to it, and discovers in week three that the escalation rate was 0.48, not 0.2. The sample was too small to see it.

Report the failure distribution

A pass rate tells you whether to worry. The failure distribution tells you what to do.

If the cheap tier fails on tax_id and po_number — two fields with a specific format — that's a prompt or schema problem, and fixing it may make the cheap tier viable. If it fails randomly across all fields, that's a capability gap, and routing that class to a stronger tier is the correct fix. Same pass rate, two completely different responses.

Record which field or step failed for every failure. Group them. The distribution is the actionable artifact; the pass rate is the headline.

The escalation matrix

The table you want in a wiki, with unmeasured cells left genuinely blank:

Task classTierMeasured cost / successSuccess rateEscalation raten
invoice.extractcheap————
invoice.extractstrong————
doc.summarisecheap————
doc.summarisestrong————
agent.plancheap————
agent.planstrong————

Empty cells are honest. An estimated cell that looks measured is the most expensive kind of lie in a cost model, because someone will budget against it. Fill cells as the harness runs; never interpolate one from a neighbouring class.

Failure Modes That Show Up Only After Routing Is Live

Failure modeProduction symptomDetection signalMitigation
Silent verbosity drift after a provider version changeSpend per task rises with no traffic changecompletion_tokens per turn trending up; cost/success up, success rate flatPin versions; alert on tokens-per-success moving >15%
Tool-call loops that pass unit testsIndividual requests take minutes; occasional huge billstool_calls per turn repeated with identical args; turn count at capDeduplicate identical tool calls; hard tool-call cap; loop breaker
Stale routing priors after model swapsRouter keeps choosing the tier that used to be cheapersuccess_rate per version diverging from the prior the router usesRe-run harness on every version change; store priors per version
Budget exhaustion mid-runPartial results, inconsistent downstream statebudget_remaining hitting zero; rise in partial outcomesFail closed with explicit partial status; cap before the call, not after
Cache-hit assumptions that never holdCost model predicts 40% cache savings; actual spend matches no-cacheCache hit metrics absent or flat at zeroMove variable content to the end of the prompt; verify hit rate before budgeting for it
Escalation double-billing on retry stormsSpend spikes during incidentsEscalated runs failing too; retry count > 1Escalate at most once per task; treat escalation failure as terminal

Model version pinning and re-evaluation triggers

An unpinned alias is an unbudgeted variable. If some-model-latest reroutes to a new snapshot overnight, your cost per token, your verbosity, and your success rate all change at once — and your routing priors are wrong without any signal.

Concrete policy:

  1. Name the resolved version in every log line, not just the alias you requested.
  2. Re-run the harness on every version change, including routine snapshot rotations.
  3. Treat a version change as a config change: it needs a diff of cost/success and success rate, and someone who signed off.
  4. Keep the previous version pinned and reachable for at least one rollback window.

The security angle, held carefully

The Dealroom report claims Chinese open models rival closed models on security-related tasks at a fraction of the cost. Treat that as a source-reported claim, not a property of your deployment. Even if it holds on someone's benchmark, running security-adjacent classification with a cheaper model shifts the burden onto validation: you now have to verify output you previously trusted the model class to get right.

The defensible position: cheap security-adjacent inference is fine when the output is validated, and expensive when it isn't. If a model's decision triggers an automated action — a block, a quarantine, a credential rotation — the cost of a false negative dwarfs any per-token saving, and the review cost belongs in the cost-per-success denominator. The parity claim, whatever its merit, does not transfer your review obligation anywhere.

What I Would Deploy Now Versus What Still Needs Verification

Deploy now

Cheap, reversible, and they reduce spend regardless of which tier wins:

  • Per-run token budgets with fail-closed partial results, enforced inside the model adapter.
  • Cost-per-success tracking, segmented by task class and model version, using the instrumentation contract above.
  • Per-task-class routing with class labels on every request, so aggregation is possible.
  • Escalation thresholds derived from measured failure rates, not from intuition about which model is better.
  • Model version pinning, with the resolved version in every log line.

None of these need a benchmark, a GPU, or a decision about open versus closed weights. They stop the bleeding and make the comparison possible.

Needs verification before acting

  • Whether V4.1 Flash's low price point translates into a lower cost per successful task on your task mix. The price is source-reported; the cost per success is unmeasured for you.
  • Whether reported open-model parity on security tasks survives your own harness, your criterion, and your review process.
  • What your real utilization, power draw, and PUE are — if you're considering owned capacity. My self-hosted numbers above are illustrative assumptions, and the 8× swing between batched and agent-shaped traffic demonstrates sensitivity, not the measurement of your cluster.
  • Your cache hit rate, before you build any cache savings into a budget.

A decision checklist for one afternoon

QuestionCheap way to answer itThreshold to act on
What's my escalation rate per class?Log escalated on 100 real tasks per class> break-even rate → route to strong tier
Is output verbosity drifting?Plot completion_tokens per turn by version>15% increase → investigate before scaling
Do I have budget caps?Try to exceed one in stagingNo cap fires → add caps today
Are prompts cache-friendly?Check cache hit metrics; grep prompt for interpolated vars before tools schemaHit rate ≈ 0 → move variable content to the end
What's my cost per success by class?Join computed_cost and outcome from logsNo segmentation → add task_class first
Have I pinned model versions?Inspect the model id in recent logsAlias in logs → pin and record the resolved version

Further Reading

Primary items from the source material, with what each does and does not establish:

For the mechanics rather than the news, the primary sources worth reading directly are your provider's current rate card (including cache pricing) and its docs on batching and prefix caching — those determine whether any of the arithmetic above lands in your favour.

Conclusion — Route on Cost per Success or Don't Bother

A low headline price is a hypothesis about your workload. The equation above tests it: attempt cost divided by success rate, with escalation amortized across winners and retries counted as cost, segmented by task class and model version.

Skip that and you haven't saved money — you've changed which line item grows. The cheap open-weight tier's discount either survives contact with your verbosity, loop depth, and escalation rate, or it doesn't. The only way to know is to measure it on your own task set.

Two numbers belong on a dashboard, segmented by task class and model version:

  1. Cost per successful task — the output of the equation, tracked over time, with computed_cost stored at request time.
  2. Escalation rate — the multiplier that decides which tier wins, and the number your router should be reading.

Everything else — price sheets, benchmark tables, parity claims — is an input to the hypothesis. The dashboard is the experiment.

Share this post

More posts

Loading comments…