Benchmarking joey-1: throughput, retries, and the real cost of a swap
joey-1 is reported as a 2B decision model from Baseten at $0.01 per 1M tokens, covered by MarkTechPost and The Next Web on 2026-10-29. This video works through the arithmetic that decides a swap, and why the token price isn't it. At 700 input and 5 output tokens per decision, a million routing calls cost about $7.05 in tokens, $43.50 in escalation at 2%, and $500 for a 0.1% human audit — roughly $550.76 all in, where the model bill is 9% of the total and the audit line is 91%. The post also covers what the launch coverage confirms versus what you still have to verify (billing unit, cached input, minimum billable unit, rate limits), a Node harness with bounded concurrency that times each request, counts retryable attempts and prices calls from the provider's reported usage, and the eval-set rules that make an accuracy delta trustworthy: sample from production logs, over-sample the ambiguous cases, hold out a final fifth, and freeze the prompt pair. Important: the author has not run joey-1. Every figure is stated arithmetic on a reported price with assumptions written out, and the harness output comes from a mock adapter used to validate the instrument. Full write-up: https://hackyjs.com/posts/benchmarking-joey-1-on-throughput-retries-and-total-cost-before-you-swap-models