Testing a Multi-Model Router: Fallback Chains, Quality-Cliff Detection, and Per-Request Cost Telemetry

Introduction

A multi-model router is easy to build and hard to trust: it moves work onto cheaper capacity, and cheaper capacity fails differently. This walkthrough covers testing a multi-model inference router end to end — an ordered fallback chain with per-attempt budgets, quality-cliff detection that compares per-tier pass rates against a floor instead of an aggregate, and per-request cost telemetry that rolls token usage up into cost per successful task. The harness runs against a deterministic fake provider, so the chain logic, the failure taxonomy, and the decision rule are all reproducible.

The multi-model routing claim that started this test

What I have is a syndicated news summary dated 2026-11-07 with one usable sentence in it: Fireworks AI says routing requests across many models can cut AI bills. There is no methodology attached, no task mix, no statement about whether quality was held constant, no latency numbers. I could not find the underlying technical write-up, so I am treating the claim as stated but unverified.

None of that makes the claim wrong. It just means we should be specific about what would have to be true for it to hold.

My position, up front: the cost half of that claim is almost certainly right, and the half that matters is the half nobody measures. Routing cuts spend the same way caching does — you shift work onto cheaper capacity. The difference is that a cache preserves the output and a model swap changes its distribution. Without a per-tier pass-rate floor, the whole thing is a quality regression wearing a nice dashboard.

What this router walkthrough builds and measures

I built a multi-model router and the harness that feeds it. The pieces:

• a model registry carrying capabilities, price, context limit, and eval scores per task tier
• a failure classifier that separates retryable, terminal, and capability failures
• an ordered chain executor with per-attempt budget, telemetry, and a decision record
• an eval harness with strict scoring, versioned eval sets, and Wilson intervals
• a sweep script that produces pass count, cost per correct answer, and p95 latency
• per-request cost telemetry with a versioned price table
• a deterministic fake provider plus a fault-injection matrix that tests the router itself
• shadow mode, budget caps, and a kill switch

Two honesty notes before we go further.

First, I ran the harness end to end against a deterministic fake provider, not against live vendor APIs. Every number in the result tables below comes from that fake provider, which I scripted with fixed pass rates and a scripted latency distribution. Those numbers exist to show the table shape and the decision rule. They are not measurements of any real model, and the model names are aliases (small-a, mid-b, large-c) so nobody mistakes them for a ranking.

Second, the router mechanics, the failure classification, the cost arithmetic, and the CI behaviour are all things I actually exercised and can show output for. Those are the confirmed parts.

The claim, separated from the evidence

The three numbers a router must move: pass rate, cost per request, tail latency

A router that reports one of these and not the other two is measuring a side effect, not routing.

• Pass rate, per task tier. Not aggregate. A tier-level floor is a constraint, not a score.
• Cost per successful task, not cost per request. Those diverge exactly when quality drops.
• Tail latency of the chain, not of the winning model. The winning model's p95 is the latency your happy path sees; the chain's p95 is the latency your users see.

The mental model I settled on is a constrained optimization: minimize cost per successful task, subject to pass_rate(tier) >= floor(tier) and chain_p95 <= deadline. The cheapest model that returns a 200 is what you get when you don't state the problem.

The Router Contract: Envelope Fields, Decision Records, Ownership

Request envelope: which fields the router owns and which the caller owns

Most router bugs I have seen come from ownership confusion: the caller sets max_tokens, the router also sets it, and the effective value depends on merge order. Split it explicitly.

The maxTokens rule needs to be hard, not a convention: the router clamps the caller's value to the model's maxOutput and never raises it. A fallback model that supports 32k output is not permission to quietly bump max_tokens to use it. The invoice will notice before you do.

The routing decision record: model, reason code, chain position, attempt count

One record per logical request. It is the only artifact that explains why a request cost what it cost.

export function newDecision(req, chain) {
return {
  requestId: req.requestId,
  idempotencyKey: req.idempotencyKey,
  tenantId: req.tenantId,
  taskTier: req.taskTier,
  chainId: chain.id,
  chainVersion: chain.version,
  evalSetVersion: chain.evalSetVersion,
  route: chain.id + "/" + req.taskTier,
  attempts: [],          // one entry per provider call
  finalModel: null,
  reasonCode: null,      // primary-ok | fallback-ok | attempt-cap | deadline | cost-cap | terminal:*
  outcome: "pending",    // pending | ok | failed | capped
  totalCostUsd: 0,
  wallClockMs: null,
};
}

Each attempts[] entry carries: index, modelId, startedAt, ttftMs, durationMs, errorClass, errorCode, usage, costUsd.

Why the decision record must be logged even on success

Because the escalations are the forecast.

If 4% of extraction traffic escalates to the mid-tier model this week and 9% next week, nothing looks broken — error rate flat, pass rate flat, bill up 30%. What moved is the reason code. That is your earliest signal that a cheap model's behaviour has shifted under you: a silent version update, a prompt change upstream, a distribution change in your own traffic.

A router that only logs failures cannot see any of that. A router that logs the decision on success gets a free leading indicator, and the query is trivial:

SELECT task_tier,
       countIf(reason_code != 'primary-ok') / count() AS escalation_rate
FROM router_decisions
WHERE day = today() - 1
GROUP BY task_tier
ORDER BY escalation_rate DESC;

I would alarm on escalation_rate moving more than a few points week-over-week, before touching anything else.

Classifying Failures Before Writing a Fallback Chain

You cannot write a sane chain before you have a taxonomy. Most chains I have reviewed are "try the next model on any catch", which is how a content-filter rejection turns into three billed calls.

Failure classes: retryable, capability, terminal, and degrade

Three classes, plus a fourth that people always merge into the wrong bucket.

The capability bucket is where the savings come from. A small model returning syntactically valid JSON with a field missing isn't worth retrying — it is telling you the tier needs a stronger model, and a retry buys the same wrong answer for the same money.

Provider error codes versus client-side schema failures

A provider 429 and a schema failure look similar inside a try/catch and mean opposite things.

• Provider error = the request never produced a usable completion. Cost may be zero or partial. Retrying is safe (modulo budget).
• Schema failure = you paid for tokens, the model produced output, and the output does not satisfy your contract. Cost is fully incurred. Retrying is a coin flip.

Treat schema failures as data, not exceptions. Validate every response and record the failing field. In my harness the validator returns { ok: false, detail: "missing:tax_id" } and the router escalates rather than retrying.

const RETRYABLE_STATUS = new Set([408, 409, 425, 429, 500, 502, 503, 504]);
const TERMINAL_STATUS = new Set([400, 401, 403, 404, 413, 415, 422]);

export function classify(err) {
// Client-side contract failures: the call succeeded, the output did not.
if (err.code === "schema_invalid") return "capability";
if (err.code === "empty_completion") return "capability";

// Transport failures.
if (err.name === "ConnectTimeout") return "retryable";
if (err.name === "FirstTokenTimeout") return "retryable";

// Mid-stream stalls are dangerous: bytes may already be at the client.
if (err.name === "InterTokenStall") return "terminal";

const s = err.status;
if (RETRYABLE_STATUS.has(s)) return "retryable";
if (TERMINAL_STATUS.has(s)) return "terminal";

return "terminal"; // unknown => do not spend money guessing
}

The default matters. An unclassified error should be terminal, because "unknown" plus "retry across three models" is an unbounded bill.

Timeouts are not one class: connect, first token, and inter-token stalls

This is the distinction that most quietly breaks production routers.

• Connect timeout — DNS, TLS, or TCP never completed. No tokens billed, no output produced. Safe to fall back. Classic retryable.
• First-token timeout (TTFT) — connection is up, the model is queueing or overloaded. No output yet, so still safe to fall back, and this is the single best signal that a hedge is warranted.
• Inter-token stall — the stream started and then went quiet for N seconds. You may have already flushed partial output to the client. There is no correct fallback here; the client has seen a prefix that a different model will not continue coherently.

The stall case is why I size the TTFT timeout separately from the total timeout, and why I refuse to fall back after the first flush.

Context-length and content-filter rejections that no fallback will fix

Context length is deterministic and pre-computable. Calculate tokens before the call and pick a model whose limit fits. A 413 discovered at attempt two is a router bug that already cost you an attempt, and if the input exceeds every model in the chain, fail fast with a clear error rather than burning the chain to discover it.

Content filter rejections are terminal per provider policy, and I treat them as terminal for the request. There is one legitimate nuance: providers differ in policy, and a request rejected by one may be accepted by another. I do not implement that as a fallback, because "route around the safety filter until someone answers" is the wrong default for a router. If a product genuinely needs a different provider's policy, that is a policy decision made deliberately and logged as such — not a silent entry in a fallback array.

Building Ordered Fallback Chains

Chain shape: cheap-first escalation versus strong-first degradation

Two shapes, two different bets.

Cheap-first escalation — start small, escalate on capability or retryable. Wins on cost, loses on latency for the escalated fraction, and only works if you can cheaply verify the cheap model's output. That is exactly why extraction and classification fit and long-context reasoning does not: a JSON schema gives you an automatic verifier, and a 900-word analysis gives you nothing.

Strong-first degradation — start strong, fall back only on retryable. Wins on latency and quality stability, loses on cost. Correct when the failure cost exceeds the model delta.

My rule: cheap-first only for tiers where the output is machine-checkable against a contract, and the cheap model clears the tier's floor with margin. Everything else is strong-first with retry-only fallback.

Idempotency and the duplicate-execution risk in streaming calls

Two distinct duplication risks, often conflated.

Duplicate model execution is the cheap one: you pay twice for the same prompt. Annoying, bounded, and visible in telemetry.

Duplicate side effects are the expensive one. If the model emits a tool call and your executor runs it, then a fallback replays the conversation and the second model emits the same tool call again, you have just sent the same email twice. The router must replay the recorded tool result, not re-execute the tool, and the tool call must be keyed on the request's idempotencyKey — not on the attempt.

Rules I apply:

1. One idempotencyKey per logical request, generated by the caller, never rewritten by the router.
2. Tool execution happens outside the attempt loop and is memoized on (idempotencyKey, toolCallIndex).
3. A fallback attempt receives prior tool results as data, so the model can continue rather than re-request.
4. No fallback after any bytes have been flushed to the client.

The IETF draft for the Idempotency-Key header is the closest thing to a standard here; provider-side dedupe support varies, so assume the router owns enforcement.

Per-attempt budget: max attempts, wall-clock ceiling, and cost ceiling

Three independent caps, all enforced before each attempt. Whichever trips first ends the chain.

export function makeBudget({ deadlineAt, maxCostUsd, maxAttempts = 3 }) {
return {
  deadlineAt,
  maxCostUsd,
  maxAttempts,
  spentUsd: 0,
  attempt: 0,
};
}

export function admit(budget, now) {
if (budget.attempt >= budget.maxAttempts) return "attempt-cap";
if (now >= budget.deadlineAt) return "deadline";
if (budget.spentUsd >= budget.maxCostUsd) return "cost-cap";
return null; // admitted
}

Most implementations forget the cost cap. That omission is what turns a provider incident into a five-figure invoice — when 200s start coming back with empty bodies, every attempt is billable, every attempt fails validation, and nothing in an uncapped chain stops the bleed.

When to hedge in parallel instead of falling back serially

Serial fallback adds the failed attempt's latency to the request. When your deadline is tight and the primary's TTFT distribution is the problem, hedging is the right tool — fire the second call before the first fails, take the first usable answer, cancel the loser.

Hedge only when all three hold:

• the expected output is small (a hedge that generates 4k tokens twice is a cost problem, not a latency fix)
• the hedge model's price is within a small multiple of the primary's
• the primary's TTFT p95 has been exceeding your soft deadline for a sustained window

Otherwise serial fallback is fine and much cheaper. A hedge on 100% of traffic is just "run two models and pay for both", which is the opposite of the original claim.

Quality Cliffs: Definition and Detection

Defining a quality cliff as a per-tier pass-rate drop past a threshold

A quality cliff is not "the cheaper model is worse". It is: the pass rate for a tier falls below that tier's floor, or falls by more than a configured delta when you swap the model.

Two parameters, both config, both versioned:

• floor(tier) — the minimum acceptable pass rate. Below this, the model is not eligible for the tier at any price.
• cliff_delta — the maximum tolerable week-over-week drop before an alarm fires.

The floor is a hard constraint. The delta is a drift detector. Conflating them is how teams end up arguing about whether 0.78 is "fine" — it is fine if the floor is 0.75 and not fine if it is 0.80, and that should not be a discussion.

Task tiers: extraction, classification, multi-step tool use, long-context reasoning

The verifier column is the whole story. Extraction and classification have cheap, deterministic checkers, which is what makes cheap-first escalation safe — you find out the small model failed without asking a human. long_context has no checker, so a cheap-first chain there is a bet you cannot settle, and the failures show up as user complaints instead of alert metrics.

Why aggregate pass rate hides the cliff that matters

Here is the illustrative sweep output from my harness, aggregated naively:

model      aggregate pass   n
small-a    55/80   (0.688)  80
mid-b      67/80   (0.838)  80
large-c    75/80   (0.938)  80

small-a looks like "12 points worse" — a mild degradation. Break it out:

The aggregate says "small-a is a bit worse". The per-tier view says: small-a is at the floor for extraction, fails classification, is 25 points under the floor for tool use, and is 30 points under for long context. mid-b is also under the floor on long context. Only large-c clears every floor.

That is the cliff: 0.90 on one tier and 0.45 on another, averaged into 0.69. The aggregate number is arithmetically correct and operationally useless.

Building the Eval Harness That Feeds the Router

Task, input, criterion, count: stating the evaluation plainly

Every tier's score has to be describable in four lines. If you cannot write these four lines, the score is not a measurement.

Task: extract 8 fixed fields from support-ticket bodies.
Input: 20 archived tickets, PII redacted, one language per case.
Criterion: strict exact match on all 8 fields, after whitespace trim, case-sensitive.
Count: 20 cases, temperature 0, seed 7, max_tokens 512.

Three things are made explicit there: the decoding parameters, the normalization rule, and the count. Changing any of them changes the score, so they belong in the eval set metadata, not in a comment.

Strict scoring rules and why partial credit masks degradation

Partial credit is how models look better than they are.

An extraction that gets 7 of 8 fields right is, for a downstream writer, a failure — the record cannot be inserted. A classification scored on cosine similarity to the "right" label vector is not a classifier. Under strict scoring, 7/8 is a zero; under field-level partial credit it is 0.875, and the difference between a model at 0.90 and one at 0.75 disappears into the fuzz.

// Strict: every required field must match exactly. No partial credit.
export function scoreExtraction(evalCase, completion) {
let parsed;
try { parsed = JSON.parse(completion.text); }
catch { return { ok: false, why: "unparseable" }; }

for (const field of evalCase.requiredFields) {
  const want = evalCase.expected[field];
  const got = parsed[field];
  if (typeof got !== "string") return { ok: false, why: "type:" + field };
  if (got.trim() !== want) return { ok: false, why: "mismatch:" + field };
}
return { ok: true };
}

Note that the failure reason is returned, not just a boolean. The mismatch:tax_id distribution across a 200-case set is what tells you which capability is missing — and, in my experience, extraction failures cluster on one or two fields rather than spreading evenly.

Sample size: what 20 eval cases can and cannot tell you

At n=20, a score is a rumour.

Wilson 95% intervals, computed with a four-line script:

Concretely: at n=20 with 18 correct, the interval 0.70–0.97 contains 0.75. So does 0.80. Your 20-case suite cannot distinguish a model that clears a 0.90 floor from one that misses a 0.80 floor. If you select a model on a 20-case result, you are selecting on noise.

What 20 cases can do: catch gross breakage. A model scoring 9/20 is genuinely different from one scoring 20/20 — but even these intervals overlap at the edges.

Practical rule I use: n=20 per tier is a smoke test, n≥200 per tier is a selection test, and any model swap decision needs the second one.

Keeping the eval set versioned and separate from the routing config

The chain config references an eval set version; it does not embed scores. Scores live in the eval output, keyed by (modelId, tier, evalSetVersion).

## routing.yaml — references, never inlines, eval results
version: 7
chains:
  extraction:
    evalSetVersion: eval-2026-10-14
    models: [small-a, mid-b, large-c]
  long_context:
    evalSetVersion: eval-2026-10-14
    models: [large-c]

This matters because if scores live in the config, someone will edit a score to unblock a deploy, and the router will then select on a number that no longer corresponds to any test run. If the config only holds a pointer, CI can assert that evalSetVersion exists, is recent, and produced scores for every model in every chain — and fail the build otherwise.

Running the Sweep and Reading the Results

The sweep script: same prompts, N models, fixed decoding parameters

Same inputs, same decoding parameters, one model varied at a time.

import { REGISTRY } from "./registry.js";

const evalSet = await loadEvalSet(process.env.EVAL_SET_VERSION);
const models = process.argv.slice(2);

for (const modelId of models) {
const rows = [];
for (const c of evalSet.cases) {
  const t0 = performance.now();
  const res = await callModel(modelId, c.input, {
    temperature: 0,
    seed: 7,
    maxTokens: c.maxTokens ?? 512,
  });
  const ms = performance.now() - t0;
  rows.push({
    tier: c.tier,
    ok: score(c, res).ok,
    costUsd: costOf(modelId, res.usage),
    ms,
  });
}

for (const tier of evalSet.tiers) {
  const t = rows.filter(r => r.tier === tier);
  const passes = t.filter(r => r.ok).length;
  const spend = t.reduce((s, r) => s + r.costUsd, 0);
  console.log(
    modelId.padEnd(10), tier.padEnd(16),
    String(passes + "/" + t.length).padEnd(7),
    ("$" + (spend / Math.max(passes, 1)).toFixed(4)).padEnd(15),
    p95(t.map(r => r.ms)).toFixed(0)
  );
}
}

EVAL_SET_VERSION=eval-2026-10-14 node sweep.mjs small-a mid-b large-c

Result table format with pass count, cost per correct answer, and p95 latency

The three columns must sit on the same row, per tier, or you will compare across tiers by accident.

model      tier            pass    $/correct   p95 ms
small-a    extraction      18/20   $0.0004     410
small-a    classification  17/20   $0.0004     380
small-a    tool_use        11/20   $0.0064     900
small-a    long_context     9/20   $0.0078    1640
mid-b      extraction      19/20   $0.0022     520
mid-b      classification  19/20   $0.0022     505
mid-b      tool_use        15/20   $0.0035    1120
mid-b      long_context    14/20   $0.0060    1980
large-c    extraction      20/20   $0.0047     780
large-c    classification  20/20   $0.0047     760
large-c    tool_use        18/20   $0.0052    1650
large-c    long_context    17/20   $0.0081    2710

These rows are from the deterministic fake provider. I scripted per-tier pass rates and a latency profile, so the values are reproducible from the seed — but they measure my fake provider, not any vendor. What is real here is the format and the decision rule: $/correct rises for small-a as tiers get harder, because the denominator (passes) collapses faster than the numerator (per-call cost) grows. That inversion is why cost-per-request is the wrong metric.

Near-ties, confidence intervals, and not over-reading small deltas

Looking at the table above with the intervals from the previous section:

• small-a 18/20 vs mid-b 19/20 on extraction is not a difference. The intervals overlap across almost their whole length.
• small-a 9/20 vs large-c 17/20 on long context is a difference, and a large one.
• mid-b 14/20 vs large-c 17/20 on long context is suggestive, not decisive, at this n.

My decision rule, stated as a rule so I cannot rationalize later:

1. Any model below a tier's floor is eliminated. No cost argument reopens it.
2. Among survivors, pick the lowest $/correct only if the pass-rate gap to the next candidate exceeds the interval width at the current n.
3. Otherwise collect more cases. Do not ship a swap on a 3-point delta at n=20.

Illustrative results and the caveats attached to them

Once more, because it matters: the sweep ran against a fake provider with scripted behaviour. What I have verified is that the harness produces the table, that the scoring is strict, that the cost arithmetic rolls up correctly, and that the decision rule eliminates the right rows. What I have not verified is that any real model behaves like small-a on your traffic. That requires running the harness against live APIs on your own eval set, and the numbers will be different.

Per-Request Cost Telemetry

Token accounting edge cases: cached input, reasoning tokens, truncated outputs

Four cases where naive accounting goes wrong:

• Cached input. Cached prompt tokens are typically priced far below fresh input tokens. If you compute cost from prompt_tokens * inputPrice, you will overstate cost on cache-heavy traffic and understate the savings. Track cached_input_tokens separately.
• Reasoning tokens. Billed as output, frequently absent from the completion string. If you estimate output tokens as text.length / 4, you will undercount badly on reasoning models. Read the provider's usage object; never estimate from the string.
• Truncated output (finish_reason: "length"). Cheaper per call and more likely to fail validation. Cost per successful task goes up, which is invisible if you only track spend.
• Retries. Two failed attempts plus one success is one request and three bills. Roll up per attempt, then aggregate to the request, or your per-request cost will be systematically low.

// price table is data, loaded at boot, versioned by effective date
export function makePricing(table) {
return {
  costOf(modelId, usage) {
    const p = table.models[modelId];
    if (!p) throw new Error("no price for " + modelId);

    const freshInput = Math.max(0, (usage.inputTokens ?? 0) - (usage.cachedInputTokens ?? 0));
    const cachedInput = usage.cachedInputTokens ?? 0;
    const output = (usage.outputTokens ?? 0) + (usage.reasoningTokens ?? 0);

    return (
      (freshInput / 1e6) * p.inputPerMTok +
      (cachedInput / 1e6) * p.cachedInputPerMTok +
      (output / 1e6) * p.outputPerMTok
    );
  },
};
}

A versioned price table as data, not hardcoded constants

Prices change. A hardcoded constant means last quarter's invoice gets recomputed at this quarter's rates and your cost trend line is a lie.

## prices/prices-2026-11-01.yaml
effectiveFrom: "2026-11-01"
currency: USD
unit: per_1m_tokens
models:
  small-a:
    inputPerMTok: 0.25
    cachedInputPerMTok: 0.05
    outputPerMTok: 1.00
  mid-b:
    inputPerMTok: 1.50
    cachedInputPerMTok: 0.30
    outputPerMTok: 6.00
  large-c:
    inputPerMTok: 3.00
    cachedInputPerMTok: 0.60
    outputPerMTok: 15.00

These prices are placeholders for the worked example. Copying them into your system instead of reading current vendor price pages is how you discover a 40% accounting error at the end of the quarter.

Store priceTableId on every decision record. Then historical cost is recomputable exactly, and a price change never retroactively rewrites yesterday's numbers.

Attributing cost per request, per tenant, and per route

Three dimensions, all on the decision record:

• per request — requestId, for debugging a specific expensive call
• per tenant — tenantId, for quota enforcement and chargeback
• per route — chainId/taskTier, for answering "is the extraction route getting more expensive or is extraction volume up?"

That last one is the question dashboards usually cannot answer, because they aggregate spend by model. Spend by model tells you large-c cost more this week; spend by route tells you why — either volume grew or escalation rate rose, and those have opposite fixes.

Why cost per successful task survives a model swap

The metric that survives a model change is the one whose denominator also changes.

Worked example, extraction tier, average 800 input + 150 output tokens per request, and a $0.02 human-review cost per failed extraction:

Naive cost per success says small-a is 5.6× cheaper. Price the failures and the gap collapses to 1.25×. The routing decision is still small-a, but the justification changed completely — and if reviews cost $0.20 instead of $0.02, large-c wins.

Now run the same arithmetic on tool_use with a $0.02 review cost:

small-a is still cheapest per successful task — at a 55% raw pass rate. This is the trap in cost-per-success optimization: because the metric is normalized by passes, it can recommend a model that fails 45% of the time, as long as it fails cheaply enough. If those 45% land in front of a user before review, the metric has optimized you into a bad product.

Which is why the floor is not negotiable. tool_use has a 0.80 floor; small-a at 0.55 is eliminated before cost is even considered, and cost optimization happens only inside the feasible set. Cost per successful task is what you minimize and the floor is what you obey — not the other way round.

Latency Telemetry and the Fallback Tax

Separating time-to-first-token from total duration

Two numbers, two meanings:

• TTFT — is the provider queueing? This drives perceived responsiveness for streaming UIs and is the trigger for hedging.
• Total duration — is generation slow? This drives deadline compliance.

A model with 200ms TTFT and 6s total is a very different problem from one with 2s TTFT and 2.5s total, and a single latency_ms field cannot tell them apart. Record both, per attempt.

Measuring the chain as a unit, not the winning model in isolation

If you log latency on the successful attempt only, your p95 is the p95 of your happy path — which is precisely the path that did not hit a fallback. The slow requests, by construction, are the ones you excluded.

Measure from request receipt to final response, and attribute the time to attempts:

requestId=req_8f21  chain=extraction/v7  wall=2610ms  outcome=ok
  attempt0 small-a  ttft=180ms  total=390ms   error=capability:schema_invalid  cost=$0.0004
  attempt1 mid-b    ttft=240ms  total=610ms   error=capability:empty_completion cost=$0.0022
  attempt2 large-c  ttft=310ms  total=1490ms  ok                                cost=$0.0049

Note the bill: $0.0075 for one request, against $0.0004 if attempt 0 had worked. That is the real cost of a capability failure, and it is invisible in any aggregate that counts only successes.

The retry tax: how a 3-attempt fallback chain changes your p95

Fake-provider bench, 500 requests, primary p95 scripted at ~1.2s and a 10% attempt-1 hard failure after a 300ms delay:

$ node latency-bench.mjs --mode primary-only --n 500 --seed 7
p50=608ms  p90=980ms  p95=1184ms  p99=2210ms  mean=$0.00035

$ node latency-bench.mjs --mode chain --n 500 --seed 7
p50=742ms  p90=1502ms  p95=2186ms  p99=3410ms  mean=$0.00091

The chain moved p95 by roughly 1s and mean cost by 2.6×. That is the fallback tax, and it is structural, not a bug:

• 10% of traffic pays the failed attempt's duration plus a full second attempt.
• Those requests were already in the slow part of the distribution, so they push the tail out twice.
• The tax scales with failure rate, so a provider having a bad hour turns a 3-attempt chain into a 3× latency event on the affected share of traffic.

Two consequences. Size your deadline against the chain, not the primary — a deadline that only the primary can meet will be breached on every escalated request. And alert on chain p95 separately from primary p95; when they diverge, your escalation rate is climbing.

Implementing the Router

The model registry: capabilities, price, context limit, eval score

export const REGISTRY = {
"small-a": {
  capabilities: ["text", "json_mode"],
  contextLimit: 128_000,
  maxOutput: 8_192,
  priceTableId: "prices-2026-11-01",
  eval: { setVersion: "eval-2026-10-14",
          scores: { extraction: 0.90, classification: 0.85,
                    tool_use: 0.55, long_context: 0.45 } },
},
"mid-b": {
  capabilities: ["text", "json_mode", "tool_use"],
  contextLimit: 200_000,
  maxOutput: 16_384,
  priceTableId: "prices-2026-11-01",
  eval: { setVersion: "eval-2026-10-14",
          scores: { extraction: 0.95, classification: 0.95,
                    tool_use: 0.75, long_context: 0.70 } },
},
"large-c": {
  capabilities: ["text", "json_mode", "tool_use", "long_context"],
  contextLimit: 1_000_000,
  maxOutput: 32_768,
  priceTableId: "prices-2026-11-01",
  eval: { setVersion: "eval-2026-10-14",
          scores: { extraction: 1.00, classification: 1.00,
                    tool_use: 0.90, long_context: 0.85 } },
},
};

The registry holds measured scores and a pointer to the eval set they came from. The chain builder filters on capabilities and eval.scores[tier] >= floor(tier); it does not sort by price and hope.

The chain executor loop with per-attempt telemetry

import { classify } from "./classify.js";

export async function execute(chain, req, deps) {
const { provider, pricing, telemetry, clock, budget } = deps;
const decision = newDecision(req, chain);
const startedAt = clock.now();

for (let i = 0; i < chain.models.length; i++) {
  const blocked = admit(budget, clock.now());
  if (blocked) { decision.outcome = "capped"; decision.reasonCode = blocked; break; }

  const modelId = chain.models[i];
  const entry = {
    index: i, modelId, startedAt: clock.now(),
    ttftMs: null, durationMs: null, errorClass: null,
    errorCode: null, usage: null, costUsd: 0, flushedToClient: false,
  };
  budget.attempt++;
  telemetry.attemptStarted(decision.requestId, entry);

  try {
    const res = await provider.call(modelId, req, {
      deadlineAt: Math.min(budget.deadlineAt, clock.now() + chain.perAttemptTimeoutMs),
      onFirstToken: () => { entry.ttftMs = clock.now() - entry.startedAt; },
      onFlush: () => { entry.flushedToClient = true; },
    });

    entry.durationMs = clock.now() - entry.startedAt;
    entry.usage = res.usage;
    entry.costUsd = pricing.costOf(modelId, res.usage);
    budget.spentUsd += entry.costUsd;

    const verdict = req.validate(res);
    if (!verdict.ok) {
      const e = new Error(verdict.detail);
      e.code = verdict.detail?.startsWith("empty") ? "empty_completion" : "schema_invalid";
      throw e;
    }

    decision.attempts.push(entry);
    decision.outcome = "ok";
    decision.finalModel = modelId;
    decision.reasonCode = i === 0 ? "primary-ok" : "fallback-ok";
    decision.totalCostUsd = sumCost(decision);
    decision.wallClockMs = clock.now() - startedAt;
    return { decision, response: res };
  } catch (err) {
    entry.errorClass = classify(err);
    entry.errorCode = err.code ?? String(err.status ?? err.name);
    entry.durationMs ??= clock.now() - entry.startedAt;
    if (!entry.costUsd && err.usage) {
      entry.costUsd = pricing.costOf(modelId, err.usage);
      budget.spentUsd += entry.costUsd;
    }
    decision.attempts.push(entry);

    // Hard rule: never fall back once bytes reached the client.
    if (entry.flushedToClient) {
      decision.outcome = "failed";
      decision.reasonCode = "terminal:post-flush";
      decision.totalCostUsd = sumCost(decision);
      decision.wallClockMs = clock.now() - startedAt;
      return { decision, error: err };
    }

    if (entry.errorClass === "terminal") {
      decision.outcome = "failed";
      decision.reasonCode = "terminal:" + entry.errorCode;
      decision.totalCostUsd = sumCost(decision);
      decision.wallClockMs = clock.now() - startedAt;
      return { decision, error: err };
    }
    // retryable | capability => next model
  }
}

decision.outcome ??= "failed";
decision.reasonCode ??= "chain-exhausted";
decision.totalCostUsd = sumCost(decision);
decision.wallClockMs = clock.now() - startedAt;
return { decision, error: new Error(decision.reasonCode) };
}

function sumCost(d) {
return d.attempts.reduce((s, a) => s + (a.costUsd ?? 0), 0);
}

Two details that are easy to get wrong. Costs are recorded on failed attempts too (when the provider reports usage), or your accounting drifts low exactly during incidents. And budget.attempt++ happens before the call, so a provider that hangs still consumes an attempt slot.

Emitting one span per request and one event per attempt

One span, many events. The span's duration is the chain's duration; the events give you the per-attempt story inside it.

{"type":"span.start","name":"router.request","traceId":"4b1e…","spanId":"9c02…",
 "attributes":{"router.chain_id":"extraction","router.chain_version":7,
 "router.task_tier":"extraction","router.eval_set_version":"eval-2026-10-14",
 "tenant.id":"acme","request.idempotency_key":"ik_7d1a"}}

{"type":"event","name":"router.attempt","attributes":{
 "attempt.index":0,"model.id":"small-a","attempt.outcome":"capability",
 "error.code":"schema_invalid","gen_ai.usage.input_tokens":812,
 "gen_ai.usage.output_tokens":143,"attempt.ttft_ms":180,
 "attempt.duration_ms":390,"attempt.cost_usd":0.00035}}

{"type":"event","name":"router.attempt","attributes":{
 "attempt.index":1,"model.id":"mid-b","attempt.outcome":"ok",
 "gen_ai.usage.input_tokens":812,"gen_ai.usage.cached_input_tokens":640,
 "gen_ai.usage.output_tokens":151,"attempt.ttft_ms":240,
 "attempt.duration_ms":610,"attempt.cost_usd":0.00081}}

{"type":"span.end","attributes":{"router.outcome":"ok","router.reason_code":"fallback-ok",
 "router.final_model":"mid-b","router.attempt_count":2,
 "router.total_cost_usd":0.00116,"router.wall_clock_ms":1060}}

The attribute names gen_ai.usage.* follow the OpenTelemetry GenAI semantic conventions, which is worth adopting even loosely — it means an off-the-shelf backend can render your token usage without a custom mapping. The router.* namespace is mine, and the two that earn their keep are router.reason_code and router.attempt_count, because they are the dimensions you will group by during the first incident.

Testing the Multi-Model Router Itself

A deterministic fake provider for chain logic

Router tests fail when they depend on a real model. The fake provider implements the same interface and replays a script, so chain logic is tested without network, cost, or nondeterminism.

// script[modelId] = array of behaviours, consumed in call order per test
export function createFakeProvider(script) {
const cursor = new Map();

return {
  async call(modelId, req, opts) {
    const n = cursor.get(modelId) ?? 0;
    cursor.set(modelId, n + 1);
    const b = (script[modelId] ?? [])[n] ?? { kind: "ok", text: "{}" };

    switch (b.kind) {
      case "ok":
        opts.onFirstToken?.();
        return {
          text: b.text ?? '{"ok":true}',
          finishReason: "stop",
          usage: b.usage ?? { inputTokens: 800, outputTokens: 150 },
        };
      case "http_error": {
        const err = new Error("provider " + b.status);
        err.status = b.status;
        err.usage = b.usage ?? null;
        throw err;
      }
      case "hang":
        return new Promise((_, reject) =>
          setTimeout(() => reject(Object.assign(new Error("stall"),
            { name: b.at === "ttft" ? "FirstTokenTimeout" : "InterTokenStall" })),
            b.ms ?? 50));
      case "malformed":
        opts.onFirstToken?.();
        return { text: "{\"a\":", finishReason: "stop",
                 usage: { inputTokens: 800, outputTokens: 12 } };
      case "empty":
        opts.onFirstToken?.();
        return { text: "", finishReason: "stop",
                 usage: { inputTokens: 800, outputTokens: 0 } };
      case "truncated":
        opts.onFirstToken?.();
        return { text: b.text, finishReason: "length",
                 usage: { inputTokens: 800, outputTokens: 512 } };
      default:
        throw new Error("unknown behaviour " + b.kind);
    }
  },
};
}

Fault injection matrix: 429s, 500s, hangs, malformed JSON, empty completions

One row per fault, with the assertion that must hold. This is the table I would write before the executor and keep in the repo next to it.

Two rows catch real regressions in practice. The inter-token stall row catches the "we added streaming and forgot the flush guard" bug. The cost-cap row catches the "empty completions are free" assumption, which is false.

Replay from recorded responses to catch regressions after config changes

Fake providers test logic. Recorded responses test behaviour against real model output after a config change.

Design: hash (modelId, prompt, decodingParams) to a fixture filename at record time; on replay, look up the hash and return the stored response, failing loudly if a fixture is missing. Then a chain reorder or a max_tokens change produces an obvious diff instead of a mystery cost increase.

## record once, against a pinned model version
RECORD=1 node sweep.mjs mid-b --eval eval-2026-10-14
## replay in CI, no network, deterministic
node sweep.mjs mid-b --eval eval-2026-10-14 --replay fixtures/

The failure mode this catches is subtle and common: you reorder a chain for latency reasons, the new primary returns a slightly different JSON shape, validation fails more often, and escalation rate climbs 15% — with no error rate change and no test failure. Replay turns that into a red build.

Production Guardrails

Shadow mode and canary routes before flipping traffic

Shadow mode: run the candidate chain on real traffic, discard its output, log the decision. Compare pass_rate, $/correct, and wall against the incumbent on the same requests.

The catch, and it is a real one: shadow mode must still respect budget caps, or a bad candidate doubles your bill for the duration of the shadow. Cap shadow traffic at a percentage and a daily dollar amount from day one.

Canary: route a small tenant slice to the new chain, with automatic rollback on pass_rate(tier) < floor over a rolling window — not on error rate. Error rate stays flat while quality drops, and that gap is the reason this post exists.

Budget caps, per-tenant ceilings, and a kill switch for the router itself

Three levels:

• Per request — maxCostUsd from the caller. Stops runaway chains.
• Per tenant per day — enforced before the attempt loop, not after. A tenant that has hit its ceiling gets a 429-style rejection from the router, not a 200 and a surprise invoice.
• Global — a circuit breaker on total hourly spend. If it trips, the router falls back to a static single-model route with no chain.

Almost nobody builds the kill switch. But the router is now a component in your critical path that can fail in ways the underlying provider cannot: bad chains, bad price tables, bad eval scores, a config deploy with an empty model list. A single feature flag that bypasses routing entirely — one model, no fallback, no eval gating — is a 20-line investment that ends an incident in seconds instead of a rollback.

Alerting on quality-cliff drift, not just on error rate

Alerts I would actually page on:

The escalation-rate alert is the one I would keep if I could keep only one. It moves first, it moves smoothly, and it tells you where to look.

Conclusion

The claim that routing across models cuts AI bills is probably right, and it is the least interesting part. Moving work to cheaper capacity always cuts bills. What separates a router from a cost hack is whether it can prove the pass rate held.

The position I would defend: routing is a constrained optimization, and the constraint is a per-tier pass-rate floor that cost is not allowed to override. Cost per successful task is the objective, not the constraint — the tool_use arithmetic above shows it will happily recommend a model that fails 45% of the time if failing is cheap enough. The floor is what stops that. Everything else in this post — the failure taxonomy, the capability class, the strict scoring, the Wilson intervals, the flush guard, the decision record on success — exists to make the floor measurable and enforceable.

There are three pieces I would not ship this without:

1. Strict scoring and n≥200 per tier for any selection decision. At n=20 you cannot distinguish 0.90 from 0.75, so you are selecting on noise.
2. The decision record logged on success, with reason_code and attempt_count. Escalation rate is your earliest drift signal and it is free.
3. The router tested as a component — fake provider, fault matrix, recorded replay — including the kill switch. It sits in the critical path now, and it fails in ways the provider cannot.

What remains unverified: whether Fireworks' router delivers the claimed savings on your traffic. The report I have says it does and does not say how it was measured. That is not something I can settle from a news summary, and it is not something you should settle from a benchmark table either — including the illustrative one in this post, which came from a fake provider I wrote. Run the sweep on your own eval set, against your own tiers, with your own failure costs. The harness is the deliverable; the numbers are local.

Further Reading

• Fireworks AI routing claim — syndicated report — the source seed for this post; summary only, no methodology published.
• OpenTelemetry GenAI semantic conventions — attribute naming for gen_ai.usage.* and model/token spans.
• OpenTelemetry JavaScript documentation — span and event emission from Node.
• Google SRE Book — Addressing Cascading Failures — retry amplification and why per-attempt budgets exist.
• Google SRE Book — Handling Overload — circuit breakers and graceful degradation patterns for the kill switch.
• IETF draft: The Idempotency-Key HTTP Header Field — background on idempotency keys for the duplicate-execution problem.
• Binomial proportion confidence interval (Wilson score interval) — the interval math used for the sample-size table.
• OpenAI API pricing and Anthropic pricing — current price pages; use these instead of the placeholder table in this post.