Testing a Multi-Model Router: Fallback Chains, Quality Cliffs and Cost Telemetry
A short walkthrough of how to test a multi-model inference router before it ships: ordered fallback chains with per-attempt budgets, quality-cliff detection that compares per-tier pass rates against a floor instead of an aggregate, and per-request cost telemetry that rolls token usage up into cost per successful task. What's covered: - Why the three numbers that matter are per-tier pass rate, cost per successful task, and tail latency of the whole chain - Escalation rate as a leading indicator: pass rate and error rate stay flat while the bill climbs - Failure classification before you write a chain: retryable, capability, terminal, and truncated-output degrade - Why connect timeouts, first-token timeouts and inter-token stalls are not the same class, and why a chain must never fall back after the first flushed token - Context-length and content-filter rejections that no fallback fixes - A decision record logged on success, with per-attempt tokens, cost and error class Honesty note from the post: the harness runs against a deterministic fake provider, not live vendor APIs. The result tables exist to show the table shape and the decision rule, and model names are aliases. The router mechanics, failure classification, cost arithmetic and CI behaviour were actually exercised. Full post: https://hackyjs.com/posts/testing-a-multi-model-router-fallback-chains-quality-cliff-detection-and-per-request-cost-telemetry