Cost Per Token Is Not Cost Per Job: Benchmarking Four Model Families with a Task-Level Harness

Introduction: Why Cost Per Token Is Not Cost Per Job

This post builds a task-level benchmarking harness that measures cost per completed job across four model families, then shows the arithmetic you can rerun against your own traces. The trigger is a run of price cuts: over about 36 hours on 23–24 September 2026, three vendors moved their inference prices. AI Magazine reported (2026-09-23) that OpenAI halved GPT-6 Sol and Luna token costs across benchmarks. Google's Gemini 4 was said to still be in post-training, with leaks teasing 5x lower costs ahead of release (nokiapoweruser.com, 2026-09-24). Anthropic cut Opus 5.5 API prices by 20%, according to the same news-cycle summary in my seed material.

The claim I'll defend: a per-token price is an input to your bill, not the bill. What lands on a finance spreadsheet is cost per completed job, and that number moves for reasons no pricing page shows — retries, reasoning-token spend, prefix re-billing on every agent turn, and the human minutes spent triaging what a cheaper model gets wrong.

What this post is and is not

This is a benchmarking harness design plus worked arithmetic. No vendor invoiced me to write it. Every number below is labelled either computed (checkable arithmetic you can rerun) or illustrative placeholder (a dry-run adapter, not a vendor measurement). Where I quote a sample size it's 50 synthetic tasks and a single pass — an anecdote, and I'll call it that every time I lean on it.

This is not a leaderboard. A leaderboard without your task, your prompt, and your success criterion is entertainment.

Separating Announced Inference Prices From Verified Pricing

Confirmed and attributed

"Confirmed" here means a named publisher reported it on a stated date. It does not mean I saw it on a vendor pricing page — for two of these, I didn't.

Watch that last row. The Anthropic cut reaches me through a news-cycle summary, with no vendor bulletin in the material I read. Reported, not verified — until anthropic.com/pricing says otherwise.

Leaks and pre-release claims

Gemini 4 hadn't shipped when the 2026-09-24 report ran; the company described it as in post-training, and the 5x figure came from leaks. A leaked target price for an unreleased model is not a price. It's a hypothesis about a future pricing page, and nothing here should be read as confirming it.

The two secondary claims worth testing

Two analysis claims travel alongside the announcements: that cheaper input prices don't always reduce total job cost, and that open-weight models win traffic but not revenue. Neither is a measurement. The first is attackable with a harness — once you have token traces, it's arithmetic. The second is an operations question rather than a benchmark result: winning traffic means nothing if revenue per served token sits below the cost of serving it.

Why Cost Per Token Diverges From Cost Per Job

Five Multipliers Between Token Price and Cost to Completion

Five things stand between the price list and cost-to-completion:

• Retry rate — every failed attempt is billed in full and buys you nothing.
• Reasoning/thinking token spend — billed at output rates, typically 3–5x input rates.
• Tool-call round trips — each one re-sends the accumulated context.
• Output verbosity — a model that answers in 1,800 tokens when 400 would do costs 4.5x on the dominant line item.
• Prefix re-sending on every turn — in an agent loop, turn n pays for everything produced in turns 1…n−1.

Worked Arithmetic: Half the Token Price, Higher Job Cost

Model A bills $2.00 per million input and $8.00 per million output. Model B bills exactly half on both: $1.00 and $4.00. B emits 2.3x the billed output per turn because its reasoning tokens are charged as output, and it needs one extra turn to satisfy the task. Prefix is 6,000 tokens per turn.

// worked-arithmetic.mjs — hypothetical A vs B, fully checkable
const PREFIX = 6000;
const A = { price: { input: 2.0, output: 8.0 }, outPerTurn: 900,  turns: 3 };
const B = { price: { input: 1.0, output: 4.0 }, outPerTurn: 2070, turns: 4 };

function bill(m) {
  let input = 0;
  for (let t = 0; t < m.turns; t++) input += PREFIX + t * m.outPerTurn;
  const output = m.turns * m.outPerTurn;
  return {
    input,
    output,
    usd: (input * m.price.input + output * m.price.output) / 1e6,
  };
}

const a = bill(A);
const b = bill(B);
console.log("A", { ...a, usd: a.usd.toFixed(5) });
console.log("B", { ...b, usd: b.usd.toFixed(5) });
console.log("B is", ((b.usd / a.usd - 1) * 100).toFixed(1) + "% more expensive to finish");

Observed output:

A { input: 20700, output: 2700, usd: '0.06300' }
B { input: 36420, output: 8280, usd: '0.06954' }
B is 10.4% more expensive to finish

Half the input price, half the output price, and the job still costs 10.4% more. Rerun it in a few seconds and you have the whole thesis.

Where Output Tokens Dominate the Bill

B lost on output tokens, not input. Its input bill was $0.036 against A's $0.041 — B genuinely won there. It lost on the output line ($0.033 vs $0.022) and on a fourth turn that re-billed the whole prefix. In agentic loops, where every turn resends the whole transcript, price-per-input-token stops being the controlling variable. How many turns it takes to be right is the controlling variable.

Building a Task-Level Benchmarking Harness

Task set and success criterion

Pick a bounded job with a mechanical pass/fail. Mine is structured extraction: 50 synthetic invoice documents, 8 fields each, with the criterion fixed before anything runs — all eight fields exact after trim, case-fold, and currency-symbol strip; anything else is a failure.

Defining the criterion first is the step people skip, and it's the step that makes the result mean anything. Tune the criterion after you see the scores and you've built a machine for producing the answer you wanted.

The Cost Ledger: Per-Call Accounting, Per-Task Roll-Up

The ledger does per-call accounting and rolls it up per task. It reads from a price file you maintain, not from constants baked into the post.

import { readFile } from "node:fs/promises";

// prices.json is yours to maintain. Vendor prices move under published
// articles, so they do not belong hardcoded in a blog post.
const prices = JSON.parse(await readFile(new URL("./prices.json", import.meta.url), "utf8"));

export function costOf(calls, price) {
let input = 0;
let cached = 0;
let output = 0;

for (const call of calls) {
  input += call.usage.input - call.usage.cached;
  cached += call.usage.cached;
  // Normalise this per provider: some report reasoning inside output,
  // some report it beside output. Getting it wrong double-bills you.
  output += call.usage.reasoningInOutput
    ? call.usage.output
    : call.usage.output + call.usage.reasoning;
}

const usd =
  (input * price.input + cached * price.cached + output * price.output) / 1e6;

return {
  input,
  cached,
  output,
  toolCalls: calls.reduce((n, c) => n + c.usage.toolCalls, 0),
  wallMs: calls.reduce((n, c) => n + c.ms, 0),
  usd: Number(usd.toFixed(6)),
};
}

// cost per completed task, failures included in the numerator
export function costPerCompletedJob(tasks) {
const spend = tasks.reduce((n, t) => n + costOf(t.calls, t.price).usd, 0);
const passed = tasks.filter((t) => t.passed).length;
return { spend, passed, perJob: Number((spend / passed).toFixed(6)) };
}

The last function is the one that matters. The numerator is all spend, failed attempts included; the denominator is passing tasks only.

The Adapter Layer: One Interface, Four Model Families

One interface, four implementations, identical prompts, retry policy, and max-token caps. Response shapes get normalised at the adapter boundary, so the ledger never has to know which vendor it's reading.

export function makeAdapter({ id, model, url, headers, toRequest, fromResponse, reasoningInOutput }) {
return {
  id,
  model,
  async call({ system, user, maxTokens, temperature }) {
    const started = performance.now();
    const res = await fetch(url, {
      method: "POST",
      headers: { "content-type": "application/json", ...headers() },
      body: JSON.stringify(toRequest({ model, system, user, maxTokens, temperature })),
      signal: AbortSignal.timeout(60_000),
    });
    if (!res.ok) throw new Error(id + " " + res.status);
    const out = fromResponse(await res.json());

    return {
      // Record what the API returned, not the alias you asked for.
      modelReturned: out.model,
      text: out.text,
      finishReason: out.finishReason,
      usage: {
        input: out.usage.input,
        output: out.usage.output,
        cached: out.usage.cached ?? 0,
        reasoning: out.usage.reasoning ?? 0,
        reasoningInOutput,
        toolCalls: out.toolCalls ?? 0,
      },
      ms: Math.round(performance.now() - started),
    };
  },
};
}

Determinism Controls: Pinning Models and Temperatures

Pin the model identifier string, set temperature: 0, cap max_tokens, set a hard timeout. Then store the model string the API returns in the response body, not the alias you sent.

Reporting Results Without Overclaiming

The Results Table: Cost Per Completed Task by Model Family

Everything below is illustrative placeholder output from a dry-run adapter, not an invoice from any vendor. I did not send 50 documents to four production endpoints for this post. The values are calibrated to the price ratios above so the columns are readable; swap in your own run.

The columns only mean something read together. A 90% success rate at a lower token price isn't cheaper if the failing 10% is what you pay a human to fix.

Failure taxonomy

Group the misses, so a cost advantage can be read against what produced it: wrong field (model copied a neighbouring value), format drift (date normalisation, currency symbol), truncation (hit the max-token cap mid-object), refused or empty call. On extraction tasks, failures tend to cluster on one or two fields rather than spreading evenly, and that clustering tells you whether the problem is the model or your prompt.

Intermittency: One Pass Is Not a Rate

One pass is not a rate. Run each task three times and report the per-task repeat rate, plus the conditions where it recurs. I didn't repeat my dry-run pass, so I have no intermittency figure to report — I'm labelling that a gap rather than inventing a number.

The Open-Weight Option and Self-Hosting Math

Break-even inputs for self-hosting

Break-even is a division: GPU $/hr ÷ API $/M tokens × 1e6. At $2.50 per GPU-hour against a $3.00/M blended API price, you need to sustain roughly 833,000 tokens per hour to break even. Against a $0.60/M blended price, the same GPU must produce 4.2 million tokens per hour.

That's the part nobody advertises: as API prices fall, the self-hosting bar goes up. Every price cut announced on 2026-09-23 makes the open-weight business case harder, not easier, unless you were already saturated. Add utilisation, queueing, cold-start latency, and the engineering time to keep a serving stack patched and observable, and the honest comparison usually favours the API for anything short of sustained saturation.

Overhead that never appears on a token invoice

Evals, routing, guardrails, and human triage of low-confidence completions. Run one number: if 6% of jobs need four minutes of human review at a $90/hour loaded rate, that's $0.36 per job — roughly six times the $0.06 token cost in the table above. Your token bill is not your AI bill.

Decision Rules for Picking a Model

Three Rules for Choosing a Model

1. Benchmark your own task. Not a public eval, not a leaderboard. Your documents, your prompt, your criterion.
2. Bill per completed job. Failures in the numerator, successes in the denominator.
3. Re-run after every price change. The harness exists so a headline can be evaluated in an afternoon.

When per-token price is the right metric

High-volume, single-turn, short-output workloads with a stable success rate and negligible retries. Classification, embedding, short rewrites, and batch tagging qualify. Multi-turn agents with tool calls don't — there, per-token price is a rounding error against turn count.

Limitations

What this harness does not prove

Fifty tasks is a smoke test, not a benchmark. Results are prompt-sensitive, and I used one prompt per family. Provider-side changes mid-run — silent model versioning — can invalidate a comparison with no visible error. Tool-call and cache-hit costs are modelled but not stressed. And a synthetic invoice set is not production traffic: real documents are messier, and the messy 10% is exactly where the cheap model loses.

Conclusion

Cutting the per-token price is a vendor metric, not a customer saving. The only number that decides a model choice is cost per completed job, it's computed with arithmetic anyone can rerun, and it needs a harness you own. On the worked example above, a model at half the input and half the output price finished the job 10.4% more expensively. That gap isn't exotic — it's the normal result of one extra retry and 2.3x the billed output.

Practical next step

Ship the ledger before the adapters. Log every call with input, cached, output, reasoning, tool-call count, wall-clock, and the returned model string. Then when the next price cut lands, you can re-price existing traces in an afternoon instead of re-running a benchmark you no longer trust.

Further Reading

• OpenAI GPT-6 Sol & Luna Halve Token Costs Across Benchmarks — news write-up, AI Magazine, 2026-09-23.
• Google Nears Gemini 4 Release: Post-Training Underway as Leaks Tease 5x Lower Costs — leak report, nokiapoweruser.com, 2026-09-24. Pre-release and unverified.
• GPT-6 Sol vs Grok 4.7: Same Input Price, Which Model Costs Less to Finish the Job? — analysis write-up, Kingy AI, 2026-09-23.
• OpenAI Cut GPT-6 Prices in Half. The Benchmarks Show What You Give Up. — news write-up, Vocal, 2026-09-23.
• OpenAI API pricing — vendor pricing page. Check this before trusting any figure in a news write-up.
• Gemini API pricing — vendor pricing page.
• Anthropic pricing — vendor pricing page. The 20% Opus 5.5 cut was not verifiable from the material I had.