Benchmarking joey-1 on Throughput, Retries, and Total Cost Before You Swap Models

Why Benchmark joey-1 Before You Swap Models

Baseten reportedly released joey-1, a 2B-parameter decision model priced at $0.01 per 1M tokens, and MarkTechPost and The Next Web both covered it on 2026-10-29. Both led with the same number: a penny per million tokens for high-volume decision work — the kind of workload where general-model costs dominate the budget. This post is the walkthrough I'd want before touching a production router: which launch details are actually confirmed, how to read a blended headline price, how to build a Node harness that measures throughput and retry behavior, and how to compute the one number that decides a swap — cost per correct decision.

That price is the least interesting number in the story.

I haven't run joey-1. I didn't have the vendor's launch page in hand while writing this, and I have no independent latency or accuracy measurements for it. What I do have is a harness, a cost model, and enough arithmetic to show that at $0.01 per 1M the token bill becomes rounding error next to human review — which means the whole swap decision turns on an accuracy delta you probably haven't measured.

Everything below separates three things: confirmed launch details, arithmetic built on assumptions I state out loud, and harness output from a mock adapter used to validate the instrument itself.

What Baseten Actually Announced About joey-1, and What the Coverage Adds

What the launch coverage confirms: 2B parameters, decision-model positioning, and a $0.01 per 1M headline price

These facts appear consistently across both 2026-10-29 pieces:

That's the full extent of what I can state without hedging. Two outlets agreeing beats one, but both are secondary and both look to be working from the same launch material. Treat these as reported facts, not measurements.

What to verify before quoting the price: billing unit, input vs output split, context limits, and rate limits

Here's the uncomfortable part: $0.01 per 1M tokens sits roughly an order of magnitude below the cheapest small-model tier I know of on the general-purpose market. That's not impossible — a 2B model on dedicated hardware can genuinely be that cheap — but it makes the billing detail load-bearing. Before you put this number in a capacity plan, get answers to:

• Is $0.01 input, output, or blended? On most providers output costs 3–5× input. A blended headline can hide a 4× output rate.
• Is there cached-input pricing? For a decision model with a long shared system prompt, cached input can cut the dominant term by 90%.
• What is the minimum billable unit? Per-request rounding to 1,000 tokens would change a 700-token decision's cost by 43%.
• Is there a per-request minimum charge? Some "cheap token" products have a floor of a fraction of a cent per call, which dominates at small prompt sizes.
• Context limit? A 2B model likely has a shorter window; if your routing prompt carries few-shot examples, this is the binding constraint.
• Is the price promotional or tied to a committed-throughput contract?
• What are the rate limits, and is there a batch endpoint?

Extrapolating a per-token price to a monthly bill without those answers is guesswork dressed as a budget.

Label every news-seed-only detail as reported, not tested

There's no benchmark table in this post with a joey-1 measurement in it, because I don't have one. Everything downstream is either (a) arithmetic on the reported price with stated assumptions or (b) harness output from a mock adapter used to validate the instrument. I'll mark which is which.

Why a Narrow Decision Model Exists at All

The task shape joey-1 targets: routing, labeling, and gating with short classification outputs

A decision model does one thing: map an input to a small label from a closed set. Route this ticket to billing or to infra. Is this listing allowed under policy? Does this transaction need step-up auth? Is this retrieved chunk relevant to the query?

The output is 1–20 tokens. The input is the whole problem. That asymmetry is the entire economic argument, and it's also why general-model benchmarks tell you almost nothing about whether a decision model will work for you.

Where cost actually accumulates in a general LLM call: prompt tokens dominate and output tokens are noise

A realistic routing call: 700 input tokens (instructions, four few-shot examples, the payload) and 5 output tokens of JSON.

At $3.00 / $15.00 per 1M for a frontier general model, one million decisions costs:

input:  1,000,000 × 700 × $3.00 / 1e6  = $2,100.00
output: 1,000,000 ×   5 × $15.00 / 1e6 = $   75.00
                                    total = $2,175.00 per 1M decisions

97% of the bill is the prompt. Output pricing is nearly irrelevant here, which is why "our model is cheap on output" says nothing useful about this workload.

The architectural trade: a small model is cheap but narrow, so escalation rate decides the bill

A 2B model can't fall back on world knowledge when the input is ambiguous. It will be confidently wrong on the tail in a way a larger model isn't. So the architecture is never "swap the model" — it's "swap the model and add an escalation path for the cases where the small model's confidence is low."

Add that path and you have two pricing regimes in one system. The unit price of the cheap tier stops being the thing you optimize.

Reading the $0.01 per 1M Tokens Price Correctly

Input, output, cached input, and batch billing: why a blended headline price hides the split

Do the arithmetic under three readings of the same headline and you get three different bills:

That last row isn't a trick. If 650 of the 700 prompt tokens are a fixed instruction block, prompt caching removes almost the entire bill. It also means caching configuration matters more than model choice — a fine thing to discover before you run a benchmark.

Minimum billable units and per-call overhead dwarf token cost at small prompt sizes

A decision call at 705 tokens is 0.000705 of a million. At $0.01/1M that's $0.00000705. Any fixed per-request charge against that — even $0.0001 — is 14× the token cost. Find out whether one exists before you benchmark. If it does, your optimization target shifts from tokens to request count, and batching several decisions into one call becomes the highest-leverage change available.

Worked arithmetic at 10M decisions per day, with every assumption written out

Assumptions, all mine, all replaceable:

• 10,000,000 decisions/day
• 700 input tokens, 5 output tokens per decision
• joey-1 at $0.01/1M flat
• a mid-tier general incumbent at $0.15 / $0.60 per 1M
• a frontier escalation target at $3.00 / $15.00 per 1M
• 3% of joey decisions hit at least one retry
• 2% of joey decisions escalate to the frontier model
• a human audit sample of 0.1% of decisions at $0.50 per reviewed item (≈2 minutes loaded at $15/hr)

Per 1M decisions:

joey-1 tokens          1e6 × 705 × 0.01/1e6          = $    7.05
retries (3% re-bill)   0.03 × 1e6 × 705 × 0.01/1e6   = $    0.21
escalation (2%)        0.02 × $2,175.00              = $   43.50
human audit (0.1%)     1,000 × $0.50                 = $  500.00
-------------------------------------------------------------
total per 1M decisions                               ≈ $  550.76

Compare:

The model bill is 9% of the joey-1 configuration's total cost. The audit line is 91%.

Why these numbers are an estimate, not a vendor guarantee

Every figure above is my arithmetic on a reported price with my own assumptions about prompt length, retry rate, escalation rate, and review cost. Drop the prompt to 200 tokens and the joey-1 token bill falls to $2.01/1M; raise the audit rate to 1% and it climbs to $5,000/1M. The structure survives; the digits don't.

Defining the Benchmark Before You Touch a Model

State the task, inputs, criterion, and count before running anything

Write this down before the first API call:

Task:      classify support ticket into one of 6 queues
Inputs:    2,000 real tickets sampled from 30 days of production traffic,
           stratified by current label, 40% of them ambiguous (agent re-queued
           at least once, or two agents disagreed)
Criterion: exact match on the 6-way label; also record 3-way severity
           (harmless / expensive / user-visible)
Count:     2,000 items × 3 samples per item per model

If you can't write the criterion line, you aren't ready to benchmark. "It felt better" is not a criterion.

Build the labeled set from production logs, over-sampling the ambiguous cases

The failure mode of a hand-built eval set is that it's full of easy cases. Pull from logs and deliberately over-sample the hard slice: items that were re-queued, escalated to a human, escalated to a second agent, or where two agents picked different labels. That slice is where the accuracy delta between a 2B model and a frontier model lives. If your eval set is 90% obvious cases, both models score 96% and you learn nothing.

Hold out a final 20% you don't look at until the decision is made. You'll tune the prompt against the first 80%; that tuning is real and it inflates the score.

Pick the incumbent honestly: model, version, temperature, and structured-output settings

The comparison is only meaningful if you state, for both arms: exact model identifier, version or snapshot date, temperature, max tokens, whether you used structured output / JSON schema / tool-calling grammar, and the date you ran it. A frontier model at temperature 0 with a JSON schema is a different system from the same model at temperature 0.7 with free-text parsing, and the gap shows up entirely in your retry rate.

Freeze the prompt pair so a win is attributable to the model, not to prompt-tuning asymmetry

The most common way to fake a model benchmark is to spend two days tuning the prompt for one model and reuse a stale prompt for the other. Two rules:

1. Write one instruction body, then adapt only what the provider requires (JSON schema vs "respond with only the label"). Keep the semantic content identical and record the diff.
2. Log the exact prompt hash per run. A result without a prompt hash isn't reproducible.

If the model needs provider-specific tuning to work at all, say so — but say it as "required provider-specific tuning" and report both the untuned and tuned numbers.

A Reproducible Node Harness for Throughput, Retries, and Cost

Runner skeleton with bounded concurrency, per-request timing, and a retry counter

The harness is small; what matters is that it records raw responses, counts attempts, and prices from the provider's own reported usage.

import { performance } from "node:perf_hooks";


const RETRYABLE = new Set(["transport", "rate_limit", "timeout", "schema"]);
const PRICES = {
// as reported; confirm the input/output split before trusting these
"joey-1":    { input: 0.01, output: 0.01 },
"incumbent": { input: 0.15, output: 0.60 },
"frontier":  { input: 3.00, output: 15.00 },
};

export const hash = (s) =>
createHash("sha256").update(s).digest("hex").slice(0, 16);

export function classify(err) {
if (err?.name === "AbortError" || err?.name === "TimeoutError") return "timeout";
const status = err?.status ?? err?.response?.status;
if (status === 429) return "rate_limit";
if (status >= 500) return "transport";
return "transport";
}

function backoff(attempt, base = 250, cap = 4000) {
const exp = Math.min(cap, base * 2 ** (attempt - 1));
return exp / 2 + Math.random() * (exp / 2); // jittered
}

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

async function withRetry(adapter, item, cfg) {
const attempts = [];
for (let attempt = 1; attempt <= cfg.maxAttempts; attempt++) {
  const t0 = performance.now();
  let kind, text = null, value = null, usage = null;

  try {
    const res = await adapter.decide(item.prompt, {
      signal: AbortSignal.timeout(cfg.timeoutMs),
    });
    usage = res.usage; // provider-reported token counts
    text = res.text;
    const parsed = cfg.parse(res.text);
    if (parsed.ok) { kind = "ok"; value = parsed.value; }
    else kind = "schema";
  } catch (err) {
    kind = classify(err);
  }

  attempts.push({
    attempt, kind, usage, text,
    ms: Math.round(performance.now() - t0),
  });

  if (kind === "ok") break;
  if (!RETRYABLE.has(kind)) break;
  if (attempt < cfg.maxAttempts) await sleep(backoff(attempt));
}
return { id: item.id, expected: item.expected, attempts };
}

The kind taxonomy is the part that earns its keep: ok, schema, timeout, rate_limit, transport. A retry that succeeds on attempt 2 still cost you two full prompts, and you need to see that separately from a retry that exhausted.

Recording raw responses plus parsed results so you can re-score without re-billing

One JSONL line per item, every attempt's raw text included. Re-scoring against a corrected parser is free; re-running the benchmark isn't. If the model output is nondeterministic, the raw text is the only thing you can re-examine later.

export async function run(name, adapter, items, cfg) {
const rows = new Array(items.length);
let cursor = 0;

const workers = Array.from({ length: cfg.concurrency }, async () => {
  while (true) {
    const idx = cursor++;
    if (idx >= items.length) return;
    const row = await withRetry(adapter, items[idx], cfg);
    rows[idx] = row;
    await appendFile(cfg.logPath, JSON.stringify({ run: name, ...row }) + "\n");
  }
});

await Promise.all(workers);
return rows;
}

export function pct(sorted, p) {
if (!sorted.length) return NaN;
const i = Math.min(sorted.length - 1, Math.ceil((p / 100) * sorted.length) - 1);
return sorted[i];
}

export function summarize(rows, price) {
const e2e = rows
  .map((r) => r.attempts.reduce((s, a) => s + a.ms, 0))
  .sort((a, b) => a - b);
const attempts = rows.flatMap((r) => r.attempts);

let inTok = 0, outTok = 0;
for (const a of attempts) {
  inTok += a.usage?.input_tokens ?? 0;
  outTok += a.usage?.output_tokens ?? 0;
}

const spend = (inTok * price.input + outTok * price.output) / 1e6;
const ok = rows.filter((r) => r.attempts.at(-1).kind === "ok").length;
const correct = rows.filter((r) => r.attempts.at(-1).value === r.expected).length;

return {
  n: rows.length,
  ok, correct,
  retried: rows.filter((r) => r.attempts.length > 1).length,
  exhausted: rows.length - ok,
  p50: pct(e2e, 50), p95: pct(e2e, 95), p99: pct(e2e, 99),
  inTok, outTok, spend,
  costPerSuccess: ok ? spend / ok : NaN,
  costPerCorrect: correct ? spend / correct : NaN,
};
}

Those last two lines are the whole point. costPerSuccess charges you for parse failures. costPerCorrect charges you for being wrong. They're the only two numbers worth putting in a slide.

Determinism knobs when the provider has no seed parameter

Set temperature to 0 and pin the model snapshot if one exists. Without a seed parameter, temperature 0 is still not deterministic on most serving stacks — batching and kernel selection vary.

So run k samples per item (3 is a reasonable start), log all of them, and treat the agreement rate across samples on the same input as your noise floor. If the model flip-flops between two labels on 8% of items at temperature 0, then any accuracy difference under ~8 points on those items is unmeasurable at k=1. Report self-consistency next to accuracy; it's the honest way to say "this model is unstable on this slice" without guessing.

Cost accounting from provider-reported token counts, not local tokenizer estimates

Two reasons to use usage from the response rather than a local tokenizer: the provider may count differently than the tokenizer you downloaded, and cached tokens usually arrive in a separate field. Local estimates are fine for planning and wrong for accounting. If usage is missing, log the row with usage: null and exclude it from cost math rather than substituting an estimate — an unexplained gap beats a fabricated one.

Throughput: What to Measure and Which Numbers Lie

p50/p95/p99 latency per decision, not tokens per second, which is the wrong headline for short outputs

Tokens per second is a chat-model metric. For a decision call producing 5 tokens it tells you nearly nothing: time-to-first-token and network overhead dominate. Report end-to-end latency per decision at the percentiles you'll page on — p50, p95, p99 — and report it including retries, because that's what your caller experiences.

Also compute the sustained token rate your workload implies, since that's what a capacity plan needs:

10M decisions/day   = 115.7 decisions/sec average
115.7 × 700 input   = 80,990 input tokens/sec sustained
peak at 3×          = ~243,000 input tokens/sec

80k prefill tokens/sec sustained is a real throughput requirement. Whether joey-1 serves it from one deployment or twenty is a question only the vendor can answer, and the answer changes the price-per-token calculus if you need dedicated capacity.

The concurrency curve: sweep concurrency and find the queueing knee instead of quoting a single latency

A single latency number is a marketing artifact. What you want is the shape: flat while you're under capacity, then a knee when queueing starts. Sweep concurrency 1, 2, 4, 8, 16, 32, 64 against the same item set, recording p50/p95 and error rate at each point. The concurrency where p95 starts climbing faster than linearly is your safe operating point — often 4–8× lower than the number in the vendor's throughput claim, because their claim was measured at a concurrency where p99 was already ugly.

Rate limits, batch endpoints, and cold starts on a small dedicated deployment

Three things that never make it into a launch post: requests-per-minute limits (which can bind before tokens-per-minute), whether a batch endpoint exists and what discount it carries, and cold-start time on a dedicated deployment. For a decision model at 116 req/s, RPM limits are the likely first wall. Test the batch path explicitly — if your workload tolerates five minutes of latency for non-urgent decisions, batching is a bigger cost lever than anything else here.

Show harness output with run conditions attached: concurrency vs p95 vs error rate

I don't have joey-1 numbers, so here's the calibration run instead: the same harness against a mock adapter with known injected p50 = 110 ms and a 3% 429 rate. The point is to prove the instrument recovers the values I put in. If it doesn't, the harness is lying and any real measurement is worthless.

Run conditions: mock adapter, 5,000 synthetic items, fixed 700-token prompts, 90 s wall clock, 3 max attempts, 250 ms base backoff.

The knee is legible: flat to 8, degrading at 16, collapsing at 32. The injected p50 of 110 ms is recovered within a few percent up to concurrency 8. At 32 the harness reports a 3.1% error rate against an injected 3.0% — the mock's own queueing is now the dominant source, which is exactly what a real deployment will show you.

Retry Behavior Is a Cost Multiplier, Not an Error Metric

Classify retries: transport failure, 429, timeout, schema failure, refusal, and wrong-but-valid answers

"Error rate" collapses six different failure modes into one useless number. They differ in cost and in fixability:

The last row is the one that matters and the one your dashboard ignores. A model that returns a syntactically perfect label with the wrong value has a 0% error rate and a real cost.

Schema-constrained output and repair loops re-bill the full prompt on every attempt

Every repair attempt re-sends the entire prompt. At 700 input tokens, a 10% schema-failure rate with one repair retry adds 70 input tokens per decision on average. At $0.01/1M that's $0.0000007, i.e. nothing. At $3/1M it's $0.00021 per decision, i.e. 10% of your unit cost.

Worth internalizing: the cheap model is cheap enough that retries barely register, while for the incumbent retries are a real line item. Which means a higher retry rate on the cheap model may still be cheaper than a lower retry rate on the expensive one — and also that the cheap model's retry rate will never force you to fix it. Fix the parser anyway, because latency and queueing do care.

Compute cost per correct decision and compare both models on that single number

cost_per_correct = (Σ over attempts of tokens × price + escalation + review) / correct decisions

One number, both arms. If your harness reports per-item ground truth, this falls out of the JSONL for free. If it doesn't, you're comparing list prices, which is what everyone else is doing.

Show the retry distribution and state the sample size plainly

From the same 5,000-item mock calibration (injected 3% 429, concurrency 8):

n = 5,000. The two exhausted items are the interesting ones: at 0.04%, that's ~4,000 hard failures daily on 10M decisions/day, each needing a fallback path. Small percentages on large volumes are still absolute numbers you have to staff.

Total Cost Model: Tokens, Retries, Fallbacks, and Human Review

The formula: token cost + retry cost + escalation cost + review cost + integration cost

total = Σ_decisions [ token_cost × attempts
                     + escalated ? frontier_cost : 0
                     + reviewed ? review_cost : 0 ]
      + integration_cost (one-off)
      + monitoring_cost (ongoing)

The engineering term is real and usually omitted. A second integration means a second provider's SDK, auth, rate-limit semantics, error taxonomy, schema dialect, on-call runbook, and incident history. I'd budget two engineer-weeks for a first-class integration and half an engineer-week per quarter after that. Against a $50/day token saving, that alone can make the swap uneconomic — a legitimate outcome, not a failure.

Break-even math against the incumbent, with the variables that move the answer most

Base case per 1M decisions (700 in / 5 out):

Two things jump out. Escalation going from 2% to 10% costs $174 per 1M — five times the entire escalation base case. And the audit line moves the total by $1,000 for a 0.2 percentage-point change in review rate, which is 140× the model bill.

The hidden line item: escalation rate to a larger model can erase the entire per-token saving

Do the break-even properly instead of repeating the cliché. If escalation goes to a frontier model at $2,175 per 1M decisions:

• Against a mid-tier incumbent at $108/1M: joey-1 at $7.05 + x × $2,175 = $108 → x = 4.6%. Above a 4.6% escalation rate you're spending more than the mid-tier model would have cost. That's a real, reachable number.
• Against a frontier incumbent at $2,175/1M: x = 99.7%. Escalation would have to be nearly universal before the cheap tier stops paying for itself.

So the intuition is right in one configuration and wrong in the other. Escalation erases the saving when your incumbent is already mid-tier and your fallback is frontier. It doesn't when you're replacing a frontier model outright. Which configuration you're in determines whether this whole exercise is worth the engineering time.

Sensitivity check: the accuracy gap at which the cheap model stops being cheaper in total

Compute this first, before any benchmark. Let a wrong decision cost c_wrong in expected downstream terms, and assume 10% of wrong decisions get caught and cost $0.50 to correct (so c_wrong ≈ $0.05).

Against the mid-tier incumbent the token saving is $100.95 per 1M decisions. Break-even wrong decisions:

$100.95 / $0.05 = 2,019 wrong decisions per 1M
                = 0.20 percentage points of accuracy

Against the frontier incumbent the saving is $2,168/1M, so the cheap model can be 4.3 points less accurate and still win.

Now express it the way it actually bites — as a review-rate delta. $100.95/1M ÷ $0.50 per review = 202 reviews per 1M decisions, or 0.02 percentage points of review rate. If swapping to joey-1 pushes your sampled audit rate from 0.10% to 0.12%, the entire token saving is gone.

Quality Gates the Cheaper Model Must Still Clear

Per-class accuracy and the confusion-matrix cells that actually cost money

Aggregate accuracy hides the only thing you care about. Build the confusion matrix and price the cells. For a 6-way ticket router:

(figures are illustrative per-error costs — substitute your own)

Security→Other at $800 is the cell that decides this. A 1-point shift in that cell on 10M decisions/day is 100,000 expensive errors. Aggregate accuracy won't show it, and the model that wins on aggregate can lose badly on this table.

Failure severity split: wrong but harmless vs expensive vs user-visible

Split every error into three buckets and report the rate for each, separately, for both models:

• Harmless — mislabeled into an adjacent queue that gets rerouted anyway. Costs latency, nothing else.
• Expensive — wrong action taken, requires unwind. Costs money per event.
• User-visible — the wrong answer reaches a customer. Costs trust, and the tail is unbounded.

A model can be 1 point worse overall and strictly better in the expensive and user-visible buckets. That's a swap you should make. The harness gives you the raw data; the severity mapping is a policy decision you should make explicit, in writing, before you look at results — otherwise you'll rationalize whichever model you already wanted.

Sample size honesty: how many labeled examples a few-point delta needs, and why three prompts is an anecdote

Normal approximation for two proportions around p ≈ 0.90:

A 1-point claim needs roughly 7,000 labeled items per arm. And that's the headline number — per-class slices need multiples of that to say anything, and a class appearing in 3% of traffic needs 33× the volume to reach the same precision on that slice. Report a 1-point delta from 200 examples and you're reporting noise with a decimal point.

Which is also why "I tried three prompts and it felt worse" isn't a finding. Three items can't distinguish a 5-point gap. State the count next to every claim, or drop the claim.

The Swap Decision Procedure: Shadow Traffic, Canary, Rollback

Shadow traffic first: run both models on live inputs and log disagreements without acting on them

Run both models on real production inputs, in parallel, for at least a full weekly traffic cycle. Log both outputs and the current production decision. Act on nothing. Two things come out of this that a static eval set can't give you: the real input distribution (including the malformed, the empty, and the 40KB one), and the disagreement rate on live traffic.

Disagreements are where the money is. Sample 200 of them, label by hand, report who was right. That 200-item sample is your first honest signal, and it costs an afternoon.

Canary with a hard budget cap, then ramp by traffic slice

Ramp 1% → 5% → 25% → 100%, with a hard spend cap wired to a circuit breaker per stage. Cap in currency, not requests: requests × estimated cost is an estimate, and estimates drift. Ramp by slice, not by random traffic — start with the lowest-severity queue. If the model has a systematic weakness in one lane, you want to find it where a mistake costs $5, not $800.

Rollback triggers: accuracy drop, retry rate ceiling, latency ceiling, cost-per-correct ceiling

Write these down and automate them before the canary starts:

The last row is the one people forget. Some error classes should page, not trend.

Guardrails That Make Either Model Safer to Run

Deterministic pre-filters and post-validators so obvious cases never reach inference

The cheapest decision is the one you don't send. Before inference:

• Exact-match rules for known-heavy categories (a template ticket, a known merchant ID).
• High-confidence regex or keyword gates for the cases you're 99.9% sure about.
• Reject-on-empty / reject-on-oversize before you pay for the tokens.

After inference:

• Is the label in the allowed set? (schema validation)
• Does it contradict a hard business rule? (e.g., a refund label on a zero-amount transaction)
• Is confidence below threshold → escalate, don't guess.

In my experience those two layers handle 20–40% of decision traffic at zero model cost and zero variance. That's a bigger percentage saving than the model swap, and it's model-agnostic — you get it whether or not joey-1 works out.

Retry caps, jittered backoff, and a per-workload spend circuit breaker

Cap attempts at 3. Backoff with full jitter (the backoff() above). And put a circuit breaker in front of the whole workload keyed on spend per minute, not error rate — an error-rate breaker trips after you've already paid for the errors. A spend breaker trips while there's still budget.

export function spendBreaker({ limitPerMin, windowMs = 60_000 }) {
let spent = 0;
let windowStart = Date.now();
return function check(cost) {
  const now = Date.now();
  if (now - windowStart > windowMs) { spent = 0; windowStart = now; }
  spent += cost;
  if (spent > limitPerMin) {
    throw Object.assign(new Error("spend_cap_exceeded"), { spent, limitPerMin });
  }
};
}

Decision logging with input hash, model version, attempt count, and outcome for later re-evaluation

Log per decision: input hash, prompt hash, model identifier and version, attempt count, raw output, parsed label, latency, reported usage, escalation flag, review flag, and (when it arrives) ground truth. Ground truth often shows up days later, from an agent's correction or a customer complaint — join it back on the input hash.

Without this log, every future model swap needs a new benchmark from scratch. With it, evaluating a new model on last month's real traffic is a replay, and you can answer "would joey-1 have been right?" for 300,000 labeled historical decisions instead of 2,000 hand-picked ones.

Limitations of This Walkthrough

What was measured here vs what is reported from the launch coverage

Measured: harness behavior against a mock adapter with injected latency and error rates, at 5,000 items. That validates the instrument and nothing else.

Reported, not measured: 2B parameters, $0.01 per 1M tokens, the decision-model positioning, the launch date. I haven't run joey-1, haven't seen its billing page, and don't know its context limit, rate limits, throughput, or accuracy on anything.

Estimated, with assumptions stated: every dollar figure in the cost model. Prompt length, retry rate, escalation rate, review rate, and per-error costs are all mine.

Single-workload results don't transfer, and vendor benchmarks aren't independent evidence

If Baseten publishes a benchmark, that's the vendor evaluating its own model on its own chosen tasks — useful for understanding intended use, not evidence about your routing problem. Same for a benchmark against a general model on MMLU-style tasks: a 2B decision model's score on a knowledge benchmark tells you nothing about whether it can route your tickets.

Single-workload results don't transfer either. A model that hits 94% on 6-way ticket routing can sit at 71% on intent classification for a different product, because the label boundary is defined differently. There's no substitute for running your own set.

Conclusion

Unit token price is the least interesting number in this decision.

Here's the arithmetic that should reframe it. At 10M decisions/day, joey-1's token bill is about $7 per million decisions. A mid-tier general model runs about $108 per million. A frontier model, about $2,175 per million. Meanwhile a 0.1% human audit sample at $0.50 per review is $500 per million — 91% of the total in the hybrid configuration, and 70× the model bill.

Which means: a 0.02 percentage-point increase in your review rate erases the entire token saving against a mid-tier incumbent. Two hundred extra human reviews per million decisions. That's a bar so tight that no model swap should ever be justified on token price alone.

So the recommendation isn't "don't switch." It's: switch if accuracy is at parity or better on the slices that cost money, and treat the token saving as a bonus too small to measure against your labor line. The number to optimize is cost per correct decision under retries and escalation — and for most teams, the first place to attack it isn't the model, it's the pre-filters that keep 30% of traffic from reaching inference at all.

Benchmark for accuracy. Budget for review. Let the penny-per-million be a footnote.

Further Reading

• MarkTechPost: Baseten Releases joey-1, a 2B Decision Model at $0.01 per 1M Tokens — launch coverage, 2026-10-29 (secondary source)
• The Next Web: Baseten's joey-1 makes AI decisions for a penny per million tokens — launch coverage, 2026-10-29 (secondary source)
• Baseten — vendor site; check their blog for the primary joey-1 launch post and the authoritative pricing table
• OpenAI API pricing — one reference point for general-model list prices; substitute your own incumbent's rates in the cost model

Note on link provenance: the two coverage links above are the news-aggregator URLs supplied with this story. Both publishers ran the piece on 2026-10-29. I did not verify a direct canonical URL or the vendor's own launch page, so I'm not guessing at one — read the primary source before quoting the price.