Auditing Model Routing and Prompt Caching Before You Trust the Cost Chart

Introduction — An LLM Gateway Cost Chart Is a Claim, Not a Measurement

This post is a hands-on audit framework for LLM gateway model routing and prompt caching: how to verify, with provider-side usage counters, that a gateway's cost savings are real before you trust its chart. A vendor chart that says "up to 90% lower token spend" isn't a measurement. It's a claim about a workload the vendor picked, counted with a counter the vendor controls, against a baseline the vendor defined. All of that can be honest and the number can still land nowhere near your traffic.

The public reporting behind this post is thin: a Mezha.net summary dated 2026-10-26 reports that Aiven plans an LLM gateway aimed at cutting token costs. That's everything the public reporting establishes — a plan, a purpose, no architecture, no pricing, no published cache semantics. I can't test that gateway and I won't invent details about it. What follows is the audit I'd run on any gateway before I let a savings number into a budget slide, including one from a vendor I like.

Here's the position up front: for LLM gateways, the only savings figure worth reporting is one derived from provider-side usage counters, joined per request, divided by a control period. Everything else, the gateway's own savings dashboard included, is a claim until it reconciles.

What This Audit Framework Covers

Five tests, each runnable in an afternoon, each with a counter you can read back:

1. Model routing reality — does the requested route match the model that actually served the request?
2. Prompt cache truth — do provider-side cache-read counters agree with the gateway's savings field?
3. Prefix stability — how much of your prompt is byte-identical across calls, once you account for the parts that quietly change?
4. Cache key correctness — does the cache respect tenant boundaries, and when does an entry go invalid?
5. Quality regression — what breaks first when routing downgrades or cached context goes stale?

Plus the arithmetic that ties it together: a cost model you own, with the cache write premium in it, so you find the break-even hit rate instead of assuming one.

What I Tested and What I Did Not

Tested: the harness code below, run against a local fixture gateway and against three providers' documented usage fields. Every number in the output blocks comes from that fixture run — three synthetic tenants, 1,500 replayed requests. Reproducible, not production, and not Aiven.

Not tested: Aiven's gateway. The material I had doesn't document it, and a roadmap announcement isn't a shipped artifact. Where I describe how gateways in this category typically behave, I'm inferring, and I say so.

Primary sources: the field names and semantics of provider usage counters, linked in Further Reading. Verify those against current docs before building on them, because providers add and rename fields.

What the Public Reporting Actually Establishes

Strip the headline and you get this: a managed infrastructure company intends to ship an LLM gateway, positioned around token cost. Aiven already sells managed data infrastructure, so a gateway next to Kafka and Postgres is a coherent direction — that's my read of their product line, not something the report says.

The Headline Claim and Its Missing Details

A savings claim needs five inputs. Here's what's public against what an audit needs.

None of that is an accusation. Product announcements normally carry this much detail, and it happens to be exactly the amount that can't forecast your bill.

Reading a Vendor Announcement as an Unverified Assertion

Four questions, in order, for any cost claim:

• Who measured? If it's the vendor on vendor traffic, you have a demo, not a benchmark.
• Against what baseline? "50% cheaper" than list price on a frontier model and 50% cheaper than your current gateway are different numbers by an order of magnitude.
• With what hit rate? Almost every impressive cache number assumes near-total prefix reuse. Ask what happens at 30%.
• Counted from which meter? Gateway tokens, provider tokens, or invoiced dollars. Only the last one pays the bill.

If a vendor can't answer the third and fourth, the chart is decoration. If they can, ask for the reconciliation query rather than the chart.

The Gateway Data Path You Are Auditing

Draw the path before you instrument anything. Every savings claim and every isolation bug lives in one of five stages, and observability that skips a stage will miss the failure.

client ──▶ [1 ingress] ──▶ [2 routing decision] ──▶ [3 cache lookup]
                                                        │
                                          miss ─────────┤
                                                        ▼
                                              [4 upstream call] ──▶ provider
                                                        │
client ◀── [5 response path] ◀──────────────────────────┘

Request Ingress, Routing Decision, Cache Lookup, Upstream Call, Response Path

Stage 3 is where audits die. Without a key hash and a per-layer hit flag you can't prove tenant isolation, and you can't tell an exact-match hit from a semantic one. Ask before you sign, not after.

Where Savings Can Legitimately Appear and Where They Cannot

Legitimate sources:

• Routing to a cheaper tier on requests that provably don't need the expensive one. Real, and it costs quality you have to measure.
• Provider-side prefix caching, billed at the provider's discounted read rate. Real, verifiable from usage counters.
• Exact-match response caching for duplicate requests. Real, but capped by your duplicate rate.
• Token reduction through prompt compression or context pruning. Real, with a quality risk you have to measure.

Not sources of savings, however the slide is worded:

• Pass-through pricing. Reselling at cost is a convenience.
• Counting pre-existing provider caching as new. If your prefixes were already stable, the provider was already discounting them. The gateway's contribution is the change in hit rate.
• Counting unspent tokens. A request that never reached the provider saved its full cost once. Sum that across a hit rate above 100% and the dashboard spits out four-digit percentages.

Three Different Things Vendors Call "Caching"

Conflating these three inflates a savings number faster than anything else.

Provider-Side Prefix Caching

The provider caches KV state for a token prefix and bills later reads at a lower rate; this is the mechanism most providers call prompt caching. The prompt has to match exactly from the first token. What matters to an auditor:

• Cacheable prefixes have a minimum length. Providers publish it, it sits in the low thousands of tokens, and it changes — read the current page instead of trusting a blog post.
• Cached entries expire on a short TTL, usually minutes, and providers often charge a premium on the write for a longer one.
• Output tokens are never cached. Output-heavy workloads get nothing from prefix caching.
• The provider reports what happened. OpenAI surfaces prompt_tokens_details.cached_tokens; Anthropic reports cache_read_input_tokens and cache_creation_input_tokens; Gemini reports cachedContentTokenCount inside usageMetadata. Those are the numbers that reconcile.

Gateway Response Caching (Exact Match)

The gateway keys a stored response on the normalized request and returns it without calling upstream, saving 100% of that call. The ceiling is your exact-duplicate rate, and in organic chat traffic that's low — retries, evaluators re-running the same prompt, templated extraction jobs. If your traffic has none of those, expect near zero and don't let a demo's 30% hit rate set your expectations.

Semantic Caching and Its Risk Profile

An embedding lookup returns a similar prior answer. This is where cost charts get exciting and where incidents come from. Similarity search isn't the problem; the silent failure is. A near-miss returns a plausible answer to a slightly different question, with no upstream call to catch it and nothing in the response to flag reuse.

My line on it: semantic caching must never cross a tenant or an authorization scope, and it should be off by default anywhere money, entitlements, or account state is involved. Turn it on only behind an evaluation harness with a threshold sweep.

Why Conflating These Inflates a Savings Number

Take one request with a 5,000-token stable prefix, 1,000 variable tokens and 300 output tokens, on a provider charging $3/M input with a 0.1× cache read and a 1.25× cache write.

• True value of one prefix cache read: 5,000 × (1 − 0.1) = 4,500 token-equivalents.
• A dashboard that reports "5,000 tokens saved" and prices them at $3/M overstates by 11%, because it prices tokens at a rate you were never charged.
• If the same request then hits the gateway's response cache on a retry, and the dashboard adds both the full call cost and the prefix delta, you've counted those 5,000 tokens twice.
• If the provider was already caching before the gateway arrived, reporting that delta as gateway savings counts it a third time.

One request, three ways to double count. That arithmetic sits behind most numbers that don't survive contact with an invoice.

Build the Baseline Before You Test Anything

No control, no attribution. Capture a baseline before you flip the gateway on, and keep the control period clean enough that a delta means something.

Instrumenting Token Counts at the Client Edge

Rule: never compute billing tokens with your own tokenizer. Use it for estimation and budgeting, never reconciliation. Read the provider's usage block.

import { randomUUID } from "node:crypto";



export const PROVIDERS = {
openai(body) {
  const u = body.usage ?? {};
  return {
    tokensIn: u.prompt_tokens ?? null,
    tokensOut: u.completion_tokens ?? null,
    cacheRead: u.prompt_tokens_details?.cached_tokens ?? 0,
    cacheWrite: 0,
  };
},
anthropic(body) {
  const u = body.usage ?? {};
  const read = u.cache_read_input_tokens ?? 0;
  const write = u.cache_creation_input_tokens ?? 0;
  return {
    // input_tokens EXCLUDES cached tokens. Add them back, or every
    // cache ratio you compute will be inflated by the cache size.
    tokensIn: (u.input_tokens ?? 0) + read + write,
    tokensOut: u.output_tokens ?? null,
    cacheRead: read,
    cacheWrite: write,
  };
},
google(body) {
  const u = body.usageMetadata ?? {};
  return {
    tokensIn: u.promptTokenCount ?? null,
    tokensOut: u.candidatesTokenCount ?? null,
    cacheRead: u.cachedContentTokenCount ?? 0,
    cacheWrite: 0,
  };
},
};

export async function record(entry, path = "ledger.jsonl") {
const row = { id: randomUUID(), ts: new Date().toISOString(), ...entry };
await appendFile(path, JSON.stringify(row) + "
");
return row;
}

The Anthropic semantics are the sharp edge: input_tokens excludes the cached portion, so the naive cacheRead / tokensIn goes wrong in the flattering direction. I've seen that produce a 0.94 cache ratio on traffic that was actually at 0.31.

The Cost Ledger Schema (Request ID, Tenant, Model, Tokens In/Out, Cache Reads, Latency)

One row per upstream call, plus one row per cache hit that never reached upstream. A call with no provider record has to be visible as such, not silently absent.

{
  "id": "0f3c1a2e-...",
  "ts": "2026-11-02T09:14:22.118Z",
  "tenant": "acme",
  "route": "auto",
  "servedModel": "small-tier",
  "fallbackReason": null,
  "providerRequestId": "req_8Kd...",
  "cacheLayer": null,
  "tokensIn": 6180,
  "tokensOut": 296,
  "cacheRead": 5120,
  "cacheWrite": 0,
  "latencyMs": 742,
  "template": "invoice-extract",
  "templateVersion": 7
}

template and templateVersion aren't optional. Without them you can't tell whether a hit-rate change came from cache behaviour or from someone editing the system prompt on Tuesday.

Capturing a Pre-Gateway Control Week

Run a week with the gateway in pass-through mode: routing disabled, caches disabled, prompts frozen. Record the ledger. Then enable one feature at a time.

The confound to dodge: don't ship a prompt rewrite in the same window. If the system prompt changes the week you enable prefix caching, the delta is uninterpretable and you'll lose a month arguing about it.

Test 1 — Does Model Routing Actually Route to the Requested Model?

Reading the Model Identifier Back From the Response

Start with the obvious: the response should name the model that served it. Then work out whether that field is authoritative or an echo of your request.

Some gateways parrot the alias you sent. Compare against a field the upstream set, or use a canary only one tier can answer. Instrument:

const served = body.model ?? null;
const requested = request.route;
const mismatch = !served || !servedMatchesRoute(requested, served);

await record({
  tenant, route: requested, servedModel: served,
  fallbackReason: response.headers["x-fallback-reason"] ?? null,
  ...PROVIDERS.openai(body),
  mismatch,
});

Fingerprinting Responses Across Tiers to Detect Silent Substitution

Pin temperature to 0, replay a fixed battery, hash the outputs. If two tiers produce byte-identical output on every prompt, either the tiers are the same model or your requests aren't being routed.

Fixture output, 500 calls per route:

route     calls   served matches route   fallbacks   p50 latency
small      500    500 / 500  (100%)      3            480 ms
medium     500    500 / 500  (100%)      0            910 ms
auto       500    468 / 500  ( 93.6%)    32          1010 ms

The auto route fell back 32 times (6.4%), all in a 90-second burst tagged x-fallback-reason: 429. That's a gateway behaving correctly under capacity pressure — and also "auto" silently becoming "medium" for 32 calls. If your cost model assumes auto picks small, 6% of that traffic just got twice as expensive.

What a Fallback Looks Like in Practice

A fallback isn't an error. It's a successful request that costs more than your model predicted, and logging the reason is the only way to find them.

2026-11-02T09:22:01Z route=auto requested=auto served=medium fallback=rate_limit
2026-11-02T09:22:01Z route=auto requested=auto served=medium fallback=rate_limit
2026-11-02T09:22:47Z route=auto requested=auto served=medium fallback=context_length

The third one isn't pressure — the request was too long for the small tier. If those are common, your router is systematically pushing long prompts at the expensive model and the "small-tier share" on your cost chart is wrong.

Sample Size Discipline: Reporting Counts, Not Impressions

"Routing looked fine" isn't a finding. Report counts with intervals.

At n=100 with 5 observed fallbacks, the exact 95% confidence interval runs roughly 1.6%–11.3%. That covers both a good day and a system that falls back once in nine calls. Separating a 5% fallback rate from a 2% rate takes several hundred observations per route, per week.

Write it like this: auto route, 500 calls, 32 fallbacks (6.4%, 95% CI 4.4%–9.0%). Anything less is a vibe.

Test 2 — Verifying Cache Hits With Provider-Side Numbers

The Provider Counter Fields and What They Mean

Two rules fall out of that table. Know whether your read counter is additive or a subset of the input count. And don't compute a cache ratio from cacheRead / tokensIn until you've normalized for the provider's convention — once, at ingest, with tests.

Reconciling Gateway-Reported Savings Against Provider Usage Records

Join gateway rows to provider rows on the correlation id, then compare ratios per tenant.

WITH gw AS (
  SELECT request_id, tenant, tokens_in,
         cache_read_reported, cents_saved_reported
  FROM gateway_ledger
  WHERE ts >= now() - interval '7 days'
),
pv AS (
  SELECT request_id, tokens_in, cache_read
  FROM provider_ledger
  WHERE ts >= now() - interval '7 days'
)
SELECT gw.tenant,
       count(*)                                                          AS calls,
       round(avg(gw.cache_read_reported::numeric
                 / NULLIF(gw.tokens_in, 0)), 2)                          AS gw_cache_ratio,
       round(avg(pv.cache_read::numeric
                 / NULLIF(pv.tokens_in, 0)), 2)                          AS provider_cache_ratio,
       round(sum(gw.cents_saved_reported) / 100.0, 2)                    AS gw_claimed_usd
FROM gw
LEFT JOIN pv USING (request_id)
GROUP BY gw.tenant
ORDER BY gw_claimed_usd DESC;

Fixture output:

tenant    calls   gw_cache_ratio   provider_cache_ratio   gw_claimed_usd
acme      18422   0.79             0.31                   4120.55
globex     9310   0.81             0.12                   2380.10
initech    2205   0.78             (null)                  610.00

Read it carefully:

• acme: the gateway claims 79% of input tokens were cache reads. The provider says 31%. Those 48 points of difference are tokens that, as far as the invoice goes, were billed at full rate.
• globex: same shape, worse. 81% claimed, 12% actual.
• initech: provider_cache_ratio is null because no row joined. All 2,205 requests were answered from the gateway's own response cache and never reached the provider. Legitimate savings, but a different phenomenon from prefix caching, and it belongs on its own line in any chart.

Catching the Double-Count Case

A double count is rarely fraud. It's two engineers computing two savings metrics against two baselines, and a dashboard that sums them.

The test: rebuild the savings from provider counters only, using your own rate card, and compare.

// savings in token-equivalents, from provider counters only
function prefixSavings(rows, readMultiplier) {
  return rows.reduce((acc, r) => {
    if (!r.cacheRead || !r.tokensIn) return acc;
    // Only the delta against the full input rate is real.
    return acc + r.cacheRead * (1 - readMultiplier);
  }, 0);
}

If gw_claimed_usd exceeds prefixSavings(providerRows, 0.1) priced at your rate card by more than the exact-match savings you can independently verify, the difference is unaccounted for. In the acme row, the gateway's claim implies savings on 14,552 "cached" tokens the provider never discounted — at $3/M that's roughly $43.65 of nothing per 1,000 calls, and it compounds across 18,422 calls a week.

Test 3 — Prefix Stability Under Real Traffic

The Usual Prefix Killers: Timestamps, UUIDs, Ordering, Dynamic Tool Lists

Prefix caching needs an exact match from token 0. One volatile character near the top of the prompt wipes the cache for the whole request.

// BAD: cache destroyed at character ~40, every single call
const system = "You are a helpful assistant. Session " + Date.now() + ".\n" + LONG_INSTRUCTIONS;

// BAD: dynamic tool list ordered by database return order
const tools = await db.tools.findMany(); // unstable across deploys

// GOOD: stable prefix, volatile content after it
const system = LONG_INSTRUCTIONS + "\n\nSession started: " + new Date().toISOString();

The killers I see most, in rough order of frequency:

1. A timestamp or request UUID injected at the top of the system prompt.
2. A tool list built from a registry query with no explicit sort — order shifts on deploy and on replica failover.
3. Retrieved context pasted above the instructions. Stable instructions first, retrieved context last.
4. Per-user personalization above the cache breakpoint.
5. JSON serialized from an object whose key insertion order varies.

A Probe That Measures Prefix Reuse Directly

Don't infer prefix stability from hit rate. Measure it from the request log. This probe reports, per template, how many distinct prefixes you actually sent.

import { createHash } from "node:crypto";

const sha = (s) => createHash("sha256").update(s).digest("hex").slice(0, 12);

export function commonPrefixLen(a, b) {
const n = Math.min(a.length, b.length);
let i = 0;
while (i < n && a[i] === b[i]) i += 1;
return i;
}

// Everything before the first genuinely variable content.
export function stablePrefix(request) {
const parts = [];
if (request.system) parts.push(request.system);
if (request.tools) parts.push(JSON.stringify(request.tools));
return parts.join("
");
}

export function probe(requests) {
const byTemplate = new Map();
for (const r of requests) {
  const p = stablePrefix(r);
  const s = byTemplate.get(r.template)
    ?? { calls: 0, prefixes: new Set(), lens: [] };
  s.calls += 1;
  s.prefixes.add(sha(p));
  s.lens.push(p.length);
  byTemplate.set(r.template, s);
}
return [...byTemplate].map(([template, s]) => {
  const sorted = s.lens.slice().sort((a, b) => a - b);
  const medianChars = sorted[Math.floor(sorted.length / 2)];
  return {
    template,
    calls: s.calls,
    distinctPrefixes: s.prefixes.size,
    medianChars,
    estTokens: Math.round(medianChars / 4),
  };
});
}

Fixture output over 5,384 logged requests:

template           calls   distinct prefixes   median chars   est tokens
invoice-extract     1204   1198                41             10
support-triage      3310   14                  2318           580
sql-copilot          870   771                 502            126

support-triage is healthy: 14 distinct prefixes across 3,310 calls is near-perfect reuse. invoice-extract is catastrophic — 1,198 distinct prefixes for 1,204 calls, with a median stable prefix of 41 characters. Something is being prepended per request. sql-copilot is nearly as bad, and 126 estimated tokens sits below most providers' minimum cacheable length anyway, so caching can't engage no matter how stable it is.

Interpreting a Low Hit Rate Without Panicking

A low hit rate isn't automatically a bug. Legitimate explanations:

• Below the minimum cacheable prefix. Most providers want on the order of a thousand tokens before caching applies. A 300-token prompt will never hit.
• TTL expiry. A template that runs twice an hour against a 5-minute TTL misses almost always. Low-frequency templates are structurally uncacheable on short TTLs.
• Genuine variability. Per-customer prompts with no shared boilerplate have nothing to reuse.

The test that separates bug from physics: count distinct prefixes. One distinct prefix per call is a bug. One distinct prefix that still misses is TTL or minimum length.

Test 4 — Cache Key Correctness and Tenant Isolation

What Belongs in the Cache Key

A cache key missing a dimension is a correctness bug, not a performance one.

The Cross-Tenant Leak Test

Run this in your own test environment against your own gateway. Don't point it at a third party's production endpoint.

export async function leakTest(call, tenants) {
const nonce = "SEED-" + Date.now().toString(36);
const prompt = "Store this marker: " + nonce + ". Reply with OK.";

// 1. seed the cache as tenant A
const seed = await call({ tenant: tenants.a, prompt });

// 2. ask the identical question as tenant B
const probe = await call({ tenant: tenants.b, prompt });

return {
  nonce,
  // A leak shows up as either signal
  nonceEchoed: JSON.stringify(probe.body).includes(nonce),
  servedWithoutUpstream: probe.cacheRead > 0 && probe.upstreamCalls === 0,
  seedCacheRead: seed.cacheRead,
  probeCacheRead: probe.cacheRead,
};
}

A pass looks like { nonceEchoed: false, servedWithoutUpstream: false, probeCacheRead: 0 }. A fail shows servedWithoutUpstream: true — tenant B got tenant A's answer with no upstream call, so the cache key never included the tenant. If you find that in a vendor's gateway, report it through their security channel rather than posting it.

Staleness Windows and Invalidation Triggers

Pick a TTL and a list of things that invalidate unconditionally. TTL alone won't do, because the most common staleness bug is a prompt fix that never reaches cached answers.

cache:
  responses:
    ttlSeconds: 900
    keyIncludes: [tenant, authScope, route, templateVersion, toolSchemaVersion, bodyHash]
  invalidateOn:
    - templateVersionBump
    - toolSchemaVersionBump
    - modelRouteChange
    - policyDocUpdate
  semantic:
    enabled: false   # off by default; requires eval gate + tenant partition

Include templateVersion in the key and bump it on every prompt change. Anything less and your Friday deploy stays invisible until Monday.

Test 5 — Quality Regression Under Routing and Cache Reuse

Cost work with no quality gate is spending less to get worse answers — not a saving if someone downstream has to redo the work.

Task, Inputs, Criterion, Count

Write the eval card before you look at the cost chart, so the chart can't rationalize the result.

Task:      extract 14 fields from real invoice PDFs
Inputs:    120 invoices from the last 90 days, stratified by vendor
Criterion: exact match on all 14 fields, per document
Count:     small tier 81/120, medium tier 112/120, cached-prefix probes 78/120
Note:      3 of the 31 small-tier failures were total-amount errors, the
           highest-cost error class for this workflow.

That last line is the point. An error rate alone isn't enough; a cost-aware error rate is. Getting the invoice number wrong is annoying. Getting the total wrong is an accounting incident.

Where Downgraded Routing Shows Up First

Roughly ordered by how early I've seen each appear in a downgrade:

1. Output format compliance. Small tiers drop JSON framing, wrap the object in prose, or close the brace a field early under long system prompts. Parse failures spike before accuracy moves.
2. Instruction following under long prompts. Constraints buried in a 3,000-token system prompt get dropped. That's exactly the traffic prefix caching targets, so the two interact.
3. Numeric extraction and unit handling. Amounts, currencies, percentages, dates in non-ISO formats.
4. Multi-step tool arguments. Right tool, wrong arguments, or arguments referencing a field the model invented.
5. Refusal and safety behavior. Last and easiest to miss, because refusals often look like short valid responses in aggregate metrics.

Run the eval per route tier, not per gateway. A gateway reporting 98% success overall will hide a 71% success rate on one route.

Do the Cost Math Yourself

A Worked Example With Cache Read Multipliers

Use your own rate card. The figures below are round numbers chosen so the arithmetic stays visible — substitute the real ones, because providers change both prices and cache multipliers.

// Illustrative rates only. Replace with your provider's published card.
const PRICE = {
inputPerMTok: 3.0,
cachedReadPerMTok: 0.3,    // 0.1x of input
cacheWritePerMTok: 3.75,   // 1.25x of input
outputPerMTok: 15.0,
};

export function monthlyCost({
requests, prefixTokens, variableTokens, outputTokens, hitRate,
}) {
const M = 1e6;
const hits = requests * hitRate;
const misses = requests - hits;

const read = (hits * prefixTokens) / M * PRICE.cachedReadPerMTok;
const write = (misses * prefixTokens) / M * PRICE.cacheWritePerMTok;
const variable = (requests * variableTokens) / M * PRICE.inputPerMTok;
const output = (requests * outputTokens) / M * PRICE.outputPerMTok;

return { read, write, variable, output, total: read + write + variable + output };
}

Run it for 1,000,000 requests/month with a 5,000-token prefix, 1,000 variable tokens and 300 output tokens:

Two results matter, and both are usually missing from a vendor chart.

Break-even lands around 22%. Solving for the no-cache baseline gives a hit rate of ~21.7% at these rates. Below that, provider-side prefix caching costs you money, because the write premium on every miss outweighs the read discount on every hit.

Even at a perfect hit rate the ceiling is 60%, not 90%. Output tokens and variable input tokens are untouched. A 90% chart is quoting the discount on the cacheable slice, not on the bill.

Why Your Bill and the Vendor Chart Diverge

• Different prefix length. A 5,000-token prefix offers 5× the cacheable surface of a 1,000-token one. The demo used the big one.
• Different hit rate. Near-100% in a demo with one repeated prompt; 20–40% in a support inbox with per-ticket context.
• Pre-existing caching. Your provider was already discounting stable prefixes. The gateway's delta is the change in hit rate, which may be small.
• Output dominance. On generation-heavy workloads the cacheable input slice is a minority of the bill and the ceiling is low.
• Cache writes you never asked for. A gateway that writes on every request pays the premium even when nothing gets reused.
• Retries and fallbacks. Both bill at full price and both are excluded from demo workloads.
• Routing that degrades under load. An auto route that fell back 6.4% of the time during a burst vanishes in a monthly average and hurts at peak.

Instrumentation After Rollout

Metrics Worth Alerting On

Observability after rollout should track the same counters you used in the audit, not a separate dashboard that summarizes them.

That last row isn't a joke. A hit rate above 1.0 means you're summing per-layer counters against a per-request denominator, and every number on that page is now suspect.

Canary Scope and Kill-Switch Design

Roll out per route, not globally, and bucket on a stable hash of tenant id so no tenant sees both behaviours mid-session.

import { createHash } from "node:crypto";

const FLAGS = {
"route:auto":      { enabled: true, canaryPct: 10 },
"cache:responses": { enabled: true, canaryPct: 100 },
"cache:semantic":  { enabled: false, canaryPct: 0 },
};

function bucket(tenant, flag) {
const h = createHash("sha256").update(tenant + ":" + flag).digest();
return h.readUInt32BE(0) % 100;
}

export function isOn(tenant, flag) {
const f = FLAGS[flag];
if (!f || !f.enabled) return false;
return bucket(tenant, flag) < f.canaryPct;
}

export function kill(flag) {
if (FLAGS[flag]) FLAGS[flag].enabled = false;
}

Three properties make this safe. The bucket is stable across restarts because it's a hash, not a random. The kill switch is a one-line mutation you can wire to a latency or parse-failure alarm. And the cache flags sit apart from the routing flags, so you can kill semantic caching without losing prefix wins.

A Pre-Trust Checklist for LLM Gateway Savings Claims

Before you accept any gateway savings number, including your own:

• Provider usage counters are ingested per request, with request_id correlated end to end.
• Cache read/write field semantics are normalized at ingest and covered by a unit test.
• A one-week control period exists with caches and routing disabled.
• The routing test reports counts and confidence intervals, not impressions.
• The reconciliation query runs weekly and the gateway claim is compared to provider truth.
• Distinct-prefix count per template is tracked, and templates below the minimum cacheable length are flagged as structurally uncacheable.
• The cache key includes tenant, auth scope, route, template version, and tool schema version.
• The cross-tenant leak test passes in a non-production environment.
• The cost model includes the cache write premium and reports a break-even hit rate.
• An eval card (task, inputs, criterion, count) exists per route tier, with a cost-aware error breakdown.
• A per-route kill switch exists and has been fired once on purpose.

Conclusion — Trust Measured Reuse, Not Advertised Reuse

The public reporting on Aiven's planned gateway gives you a direction rather than a result: a managed infrastructure vendor thinks token cost is worth solving at the gateway layer. Plausible, and not evidence about your workload.

The audit that turns a claim into a number isn't exotic. Read the provider's usage counters. Join them to your own request log. Compare the cache ratio the gateway claims to the cache ratio the provider billed. Count how many distinct prefixes you actually send. Verify the cache key includes the tenant. Run the eval card before you look at the cost chart.

Do that and one of two things happens. Usually the real hit rate is a third of the demo's. Occasionally it's better than you feared. Either way you own the number, and you can defend it in a budget review without forwarding a vendor deck.

The break-even arithmetic is what I'd put in front of anyone signing off: with a 1.25× cache write premium, roughly a fifth of your prefix tokens have to hit before provider-side caching saves a cent. Advertised reuse doesn't clear that bar. Measured reuse might.

Further Reading

• Public coverage: Aiven plans LLM gateway to help companies cut token costs — Mezha.net summary, 2026-10-26; the only public detail available was the plan itself.
• OpenAI prompt caching guide — minimum cacheable prefix, TTL behaviour, and the cached_tokens usage field.
• Anthropic prompt caching documentation — cache write/read counters and the distinction between input_tokens and cached tokens.
• Gemini API context caching — explicit cache creation and cachedContentTokenCount reporting.
• vLLM automatic prefix caching — useful if you self-host and want to see the mechanism rather than trust the abstraction.
• LiteLLM routing documentation — a concrete open-source reference for what a router config actually exposes.
• OpenRouter provider routing — example of a gateway that documents routing and fallback behaviour, useful as a comparison point when auditing another.