Why the Same Prompt Costs More in Japanese: Tokenizer Variance and Cost Modeling

Two users open the same support widget on the same afternoon and type the same question. One writes English, one writes Japanese. Both get an answer of similar quality and length. On the invoice, the Japanese request costs roughly two to three times more — and for scripts with thinner pretraining coverage, the gap is much wider than that. This isn't a model-quality problem, and swapping models won't fix it. It's tokenizer variance: a pricing property of your stack, your vendor, and your vocabulary. This post covers how to measure that multiplier on your own traffic, how to convert tokens into dollars per locale, and six levers — instrumentation, model selection, routing, prompt hygiene, caching, and output-side constraints — that actually move the number.

What the 17x Token-Usage Claim Actually Says, And What It Doesn't

Sourcing Note on the 17x Figure

The seed for this post is a secondary aggregator item—KuCoin, published 2026-10-30—reporting that "a new study" found AI model token usage can vary by as much as 17x depending on language. I didn't review that study, its corpus, its tokenizer versions, or its methodology. Treat 17x as a headline, not a finding.

What I can say about the number: 17x is a worst-case spread across a wide language set, almost certainly including scripts that fall back to byte-level encoding nearly everywhere. It isn't a per-request multiplier for your traffic. If your product serves English, Spanish, Japanese, and German, the number that governs your bill is probably in the 1.0x–2.5x band—and you need to measure it. Budgeting from a headline is how you end up with an alarm set to the wrong threshold.

The academic work most often cited here is Petrov, La Malfa, Torr, and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages (arXiv:2305.15425). The paper's own framing is that the same content can require over an order of magnitude more tokens in one language than another under production tokenizers. That's directionally the same claim as the aggregator's 17x. Verify the identifier and exact figure against the paper before citing it in anything customer-facing.

The Defensible Part of the Claim

Independent of who reported it, you can measure this today with two commands and a JSON file:

• Every production tokenizer is a BPE variant with a fixed merge table.
• Merge allocation is decided by frequency in a pretraining corpus.
• Any UTF-8 sequence with no useful merge falls back to bytes, and bytes are expensive.

You don't need a study to establish that. You need enc.encode(text).length and a parallel corpus.

The Metric That Matters: Tokens per Unit of Content, Not per Word

The literature metric is tokenizer fertility: tokens per word. It's useful for cross-language comparison but the wrong denominator for cost work, because "word" isn't well-defined across scripts. Japanese and Chinese don't delimit words with spaces, and what a morphological analyzer calls a word in Korean or Turkish may be several BPE tokens.

For billing, use tokens per unit of semantic content—where the content unit is a sentence that says the same thing, taken from a parallel corpus. Characters are a slightly worse but much easier denominator: chars / tokens. It's stable, trivial to compute, and good enough to rank locales against each other. It breaks down when comparing scripts with very different information density per character (a Hangul syllable vs. a Latin letter), which is why the comparison has to be on parallel sentences, not raw string length.

The practical rule: pick one denominator, pick a corpus, and never compare numbers computed with different denominators.

How BPE Tokenizers Turn Language Choice Into a Bill

Byte-Level Fallback Is the Expensive Path

Take one word pair with identical meaning:

The English word is a single token because it appeared often enough in the pretraining corpus for BPE to learn the whole string as a merge. The katakana loanword is the same concept, and it costs six tokens where cl100k has learned katakana merges—or up to 27 if it falls back entirely to bytes, one token per byte, because the base vocabulary is bytes.

That's the whole mechanism. Nothing is "wasted" or misconfigured. The encoder is doing exactly what it was trained to do: represent unseen strings as the smallest pieces it knows, and the smallest piece is a byte.

At the extreme—Burmese, Amharic, Tigrinya, and other scripts with sparse coverage—you pay close to the byte floor systematically. A 3-byte character is 3 tokens. English averages roughly 4 characters per token. The ratio between those two regimes is where 17x comes from, and it's arithmetic, not opinion.

Vocabulary Coverage Is a Training-Data Artifact

cl100k_base ships roughly 100k vocabulary entries. o200k_base ships roughly 200k. Neither number is a language-neutral budget: the merge table is allocated proportionally to observed frequency, so a script that is 0.01% of the corpus gets a rounding error's worth of merges, and the top three web languages get most of them.

This isn't a bug to report. It's the expected outcome of fitting a fixed-size vocabulary to a skewed distribution. The consequence: tokenizer quality for a locale is a property of the vendor's corpus at training time, and it changes between model generations without announcement.

Concretely: Japanese and Korean text improved substantially moving from cl100k_base to o200k_base because the newer vocabulary allocates more merges to those scripts. If you measured Japanese cost in 2023 and still budget from that number, you're probably over-provisioned. If you switched vendors without re-measuring, you may be under-provisioned and not know it.

The Same Sentence Is Not the Same Number of Tokens

You can't compute a per-locale multiplier by taking an English string and running it through Google Translate, because translation isn't length-preserving.

• Japanese and Korean routinely drop explicit subjects and articles that English requires.
• German compounds where English uses phrases, so the target is often shorter in tokens than the character count suggests.
• Arabic and Hindi inflect differently, and the morphological surface form can be longer or shorter than the English equivalent.
• Chinese is denser in characters but each character may cost more tokens than a Latin character does.

A translated corpus also inherits translator style. Two vendors translating the same English sentence into Japanese can produce a 20% difference in token count with identical meaning. If you build a cost model on machine-translated strings, you're partly measuring the translator.

Use a parallel corpus: the same set of sentences, professionally localized, one file per locale. If you already have localized UI strings or help-center articles, you have one.

Tokenizers Differ Between Vendors and Between Generations

The same Japanese string counts differently under cl100k_base, o200k_base, Gemini's SentencePiece vocabulary, and Anthropic's tokenizer. There's no cross-vendor token standard, and no reason to expect one.

This is why vendors publish token-counting endpoints rather than shipping the vocabulary: OpenAI exposes a tokenizer page and the tiktoken library, Anthropic has a /v1/messages/count_tokens endpoint, and Google exposes countTokens for Gemini. The endpoint exists because client-side counts drift, and a drifted count means your rate limiter, context-budget logic, and cost dashboard are all slightly wrong in ways that don't throw errors.

A Reproducible Token-Count Harness

Counting Tokens per Locale With tiktoken in Node

import { readFileSync } from "node:fs";


// corpus.json: [{ "locale": "en", "text": "..." }, ...]
// Same sentences, professionally localized. One file, one row per locale per sentence.
const corpus = JSON.parse(readFileSync(new URL("./corpus.json", import.meta.url), "utf8"));

const encodingName = process.env.ENCODING ?? "cl100k_base";
const enc = get_encoding(encodingName);

// [...text].length counts code points, not UTF-16 units.
// text.length would overcount astral characters (emoji, rare CJK ext) by 2x.
const charCount = (s) => [...s].length;

const byLocale = new Map();

for (const { locale, text } of corpus) {
const tokens = enc.encode(text);
const acc = byLocale.get(locale) ?? { locale, chars: 0, tokens: 0 };
acc.chars += charCount(text);
acc.tokens += tokens.length;
byLocale.set(locale, acc);
}

const baseline = byLocale.get("en");
if (!baseline) throw new Error("corpus must contain an 'en' baseline row");

const rows = [...byLocale.values()].map((r) => ({
locale: r.locale,
chars: r.chars,
tokens: r.tokens,
"chars/token": +(r.chars / r.tokens).toFixed(2),
"cost index vs en": +(r.tokens / baseline.tokens).toFixed(2),
}));

rows.sort((a, b) => b["cost index vs en"] - a["cost index vs en"]);
console.table(rows);

enc.free();

Run it against a second encoding to see generation drift:

ENCODING=cl100k_base node token-cost-by-locale.mjs
ENCODING=o200k_base  node token-cost-by-locale.mjs

Reading the Output: A Six-Locale Worked Example

Below is output from that harness over a six-sentence billing-support corpus, one sentence per locale, encoded with cl100k_base. This is an anecdote-scale measurement—six sentences, one domain, one tokenizer version. It's a worked example, not a benchmark. Your numbers will differ.

Two things move these numbers more than anything else:

1. Tokenizer version. The same Japanese sentence under o200k_base costs meaningfully fewer tokens than under cl100k_base. A generation change from your vendor is a silent cost change in both directions.
2. Text domain. Support tickets, product names, SKUs, and legal boilerplate behave differently. An English-heavy corpus of internal jargon will look worse per token than plain prose, because jargon is also poorly covered. Measure your actual strings.

Note the second-order oddity: the Japanese row has fewer characters than English and more tokens. Any cost model built on character count would have told you Japanese was cheaper.

What I Verified, What I Inferred, and What I Did Not Verify

I want the boundary explicit, because it's the difference between an engineering argument and a confident-sounding guess.

• Confirmed, rerunnable: the tokenizer mechanics above; every number in the table, which the harness reproduces given the same corpus and encoding; the byte floor arithmetic (9 katakana characters = 27 UTF-8 bytes = up to 27 tokens); the existence of vendor token-counting endpoints.
• Inferred, not measured on your stack: which locales dominate a given product's traffic mix; whether your generation-side output inflation matches your input-side inflation; whether your vendor's newer model actually improved coverage for the specific scripts you serve.
• Unverified: the KuCoin report's 17x figure, the corpus behind it, and the exact measurement methodology of the paper it describes. I didn't review either primary artifact. Don't put 17x in a planning doc.

From Tokens to Dollars: Cost per Resolved Request

The Cost Formula for a Multilingual Feature

Per request, the arithmetic is boring:

cost = (input_tokens  × input_price_per_token)
     + (output_tokens × output_price_per_token)

The useful number is one level up:

cost_per_resolved_request[locale] = sum(request costs in locale) / resolutions in locale

Worked example, using a published rate card of $2.50 per million input tokens and $10.00 per million output tokens. Check the current card; these move.

A 2.5x per-request delta. Nobody approved a 2.5x price increase; it arrived with the localization launch.

Output Tokens Inflate the Gap

Note the multiplier above is larger on the output side (2.2x applied to a token that costs 4x more per unit than input). Output tokens are the expensive side of the invoice, so output inflation dominates.

Two interactions worth knowing:

• max_tokens is a hard ceiling, not a budget. It bounds a single response, not monthly spend. If you set it once and never look at it, you're capping the wrong quantity.
• Stop sequences truncate, but not predictably. A model answering in Japanese may produce a polite closing ("ご不明な点がございましたら…") your English-tuned stop sequence never matches, so the response runs to max_tokens. Measure truncation rate per locale, not globally.

Second-Order Costs: Context Windows, RAG Chunking, Embeddings, Latency

Token count isn't just the price column:

• Context window. A 200k-token window holds roughly 45k tokens of Japanese where it holds 200k tokens of English. Retrieved-context budgets that work in English silently truncate elsewhere.
• RAG chunking. Fixed character chunk sizes produce wildly different token counts per locale. Chunk by tokens or you'll trip the embedding model's input limit on some locales and never on others.
• Embeddings. Embedding spend scales with tokens per document, so your Japanese index costs more to build and query than your English one.
• Latency. Time-to-last-token tracks output token count more than semantic length. Japanese responses are slower for the same answer.

The Hidden Attack Surface: A Language Picker Is a Cost Multiplier

Per-Request Rate Limits Under Unauthenticated Users

If your free tier limits requests—20 per hour per IP—and a switchable locale changes per-request cost by 2–6x, then the limit is denominated in the wrong unit. An unauthenticated caller who can flip a locale selector gets more compute for the same quota. That's a cost-amplification bug, not a security bug, but it shows up on the same invoice.

Mitigation: rate-limit and quota on tokens, not requests. A token budget is locale-neutral by construction. If your edge doesn't know token counts, gate on a per-locale weight derived from your measured cost index—it doesn't have to be exact, it has to be monotonic.

Full request bodies and locale-switching details are deliberately out of scope here; the defensive point is that the quota unit must match the billed unit.

Why Budget Alarms Miss a Locale Shift

An alert on "daily spend" fires when total spend moves. It doesn't fire when a single locale quietly becomes 40% of spend while total volume holds steady—because at the dashboard level, that looks like growth.

Locale has to be a first-class dimension in your metrics, not a tag you add when something breaks.

Mitigation 1 — Make Cost-Per-Locale a Measured Metric

Instrumenting Cost as a Span Attribute

Cost belongs next to latency, on the same span, computed at request time so it survives vendor price changes and dashboard rewrites.

const PRICES = {
// USD per token. Keep this in config, not in code, and date-stamp it.
"model-a": { input: 2.5 / 1e6, output: 10.0 / 1e6 },
"model-b": { input: 0.5 / 1e6, output: 1.5 / 1e6 },
};

export function recordLlmSpan(span, { locale, model, usage }) {
const price = PRICES[model];
if (!price) {
  span.setAttribute("llm.cost.unknown", true);
  return;
}

const cost = usage.input_tokens * price.input + usage.output_tokens * price.output;

span.setAttributes({
  "llm.locale": locale,
  "llm.model": model,
  "llm.usage.input_tokens": usage.input_tokens,
  "llm.usage.output_tokens": usage.output_tokens,
  "llm.cost.usd": cost,
});
}

Set llm.cost.unknown loudly. A model missing from the price table is a silent zero in every aggregate downstream, and silent zeros are how cost dashboards lie.

The Three Reports That Change Decisions

Three queries, no more:

1. Top locales by total spend. Not by request count. These diverge, and the divergence is the finding.
2. Cost per resolved request by locale. The denominator matters: resolutions, not requests. A locale with high resolution rate can be cheap per resolution and expensive per request.
3. Cost per resolved request ÷ revenue per user, same locale. If Japanese costs 2.5x and monetizes at 1.3x, you have a margin problem specific to one locale—which is actionable. An average across locales is not.

Mitigation 2 — Tokenizer-Aware Model Selection

Compare Tokenizers on Your Own Corpus, Not a Leaderboard

A model with a larger or better-covered vocabulary can win on total cost even at a higher per-token price. This is the one place where tokenizer variance works in your favor: if vendor B's vocabulary covers Japanese well enough to halve the token count, vendor B can be cheaper at double the rate.

The comparison table you want:

Read that carefully: model-b is roughly 2x the price per token and still cheaper per Japanese request, because its vocabulary doesn't punish katakana. Per-token price isn't the comparison; cost per resolved request in the target locale is.

Measure ja tokens with the vendor's own counting endpoint, not with a local approximation. And run the quality gate—a cheaper tokenization that degrades answer quality isn't cheaper per resolution.

Where This Lever Is Weakest: Single-Vocabulary Vendors

If your vendor offers a single vocabulary across its whole lineup, you have no tokenizer choice, only a model choice within one merge table. The ratio is then effectively fixed at the vendor level, and the remaining levers are the ones below. Say so in your design doc, because it changes which mitigations are worth engineering.

Mitigation 3 — Language-Aware Routing

Route by Locale to the Cheapest Model That Clears a Quality Bar

Routing is straightforward once you have per-locale cost and per-locale quality:

const ROUTES = {
en: { primary: "model-a", fallback: "model-b" },
ja: { primary: "model-b", fallback: "model-a" }, // model-b wins on ja tokenization
default: { primary: "model-a", fallback: "model-a" },
};

export function pickModel(locale, qualityOk) {
const route = ROUTES[locale] ?? ROUTES.default;
return qualityOk[locale] ? route.primary : route.fallback;
}

The qualityOk gate is not decoration. Routing without a per-locale eval silently degrades the non-English experience—the model you routed to because it was cheaper may be cheaper because its tokenizer handles the script poorly and its training data for that script was thinner. Cheap tokenization and weak capability correlate more often than vendors like to admit.

Hold a regression eval per locale in CI. If a route change drops Japanese answer accuracy by two points, the routing saves money you can't afford.

The Translation-First Pipeline Trap

The tempting architecture: translate input to English, run the task, translate back. It looks like it converts every locale into your cheapest locale.

Cost it honestly:

• Two extra model calls per request (in and out), each with its own token count.
• The task call now runs on translated text, which means entity fidelity risk—names, product codes, addresses, and dates are exactly what translators normalize or mangle.
• You've added two failure modes and two latency hops.

In most cases the round trip isn't cheaper than paying the 2–3x input premium, especially with short prompts. It can win for very long retrieved contexts where the task model is expensive and the translation model is small—but that's a specific configuration you must measure, not a general rule.

Mitigation 4 — Prompt Compression and Template Hygiene

Static Instructions Are the Cheapest Thing to Fix

Every system prompt token is paid for on every request in every locale. At a 3x Japanese multiplier, that token costs three times as much there.

• Trim verbose instructions. Most system prompts carry 30–50% redundancy.
• Deduplicate few-shot examples that no longer match production inputs.
• Move invariant policy (tone, refusal policy, formatting rules) into vendor-supported configuration if available, rather than into the prompt every call.
• Version the template and put the version in your metrics, so a prompt edit's cost impact is visible.

Compress Retrieved Context, Not the User's Question

The user's question is short and semantically dense—compressing it destroys the signal. The retrieved context is long and redundant, and that's where the savings are.

Summarize or select chunks before they enter the window. Then measure the quality delta per locale. A summarizer that preserves 98% of English answer quality may preserve less for Japanese, because the summarizer shares the same vocabulary bias as everything else in your stack.

Where Compression Breaks: Low-Resource Languages First

Aggressive compression degrades low-resource languages first. Your compression model has the same tokenizer fertility problem, and its summarization ability tracks the same pretraining coverage. Languages with fewer merges are also languages with thinner supervision for the "keep the important parts" objective.

Detection is boring and effective: hold a per-locale eval set and run it on every compression change. If English holds at 98% and Japanese drops to 89%, the compressor is now a locale-specific quality bug that also happens to cost less money.

Mitigation 5 — Caching, and Why Absolute Savings Depend on Locale

Exact-Match and Prefix Caching for LLM Responses

import { createHash } from "node:crypto";

export function cacheKey({ locale, model, templateVersion, prompt, context }) {
const body = JSON.stringify({ locale, model, templateVersion, prompt, context });
return "llm:" + createHash("sha256").update(body).digest("hex");
}

locale is not optional in that key. A cache key that omits it will serve a Japanese user an English cached answer—the savings look great and the product is broken. Same for model (a different model means a different answer) and templateVersion (a prompt edit invalidates every entry, and forgetting that poisons your eval results).

Prefix caching, where the vendor supports it, is the highest-leverage version of this: a long static system prompt gets cached once and the per-request input cost drops on the cached portion. The benefit scales with prompt length, which means it scales with how much you were over-paying.

The Asymmetry: Cache Savings Scale With Locale Cost

A Japanese request that costs 3x produces 3x the absolute saving on a cache hit. Cache hit-rate work pays back fastest in your most expensive locale—the opposite of where teams usually look, because they optimize the locale with the highest request volume.

Rank locales by absolute spend, not volume, when deciding where to spend engineering time on caching.

Mitigation 6 — Control the Output Side

Constrain Generation Structurally

Output tokens carry the higher per-token price, so constrain them structurally:

• Structured outputs / JSON schemas. A schema-conformant response cannot ramble.
• Hard max_tokens per task, set from your p95 measured length plus headroom, not from a round number.
• Explicit length instructions per locale, calibrated against your eval rather than translated from the English instruction.

Measure Answer-Length Inflation per Locale

This is the diagnostic that keeps you honest about which problem you have:

Task: 200 identical questions answered in en and ja.
Metric: mean output tokens per answer.
Observed: en 180 tokens, ja 396 tokens.

Is that a tokenizer problem or a verbosity problem? Split it: compare characters per answer first. If Japanese answers are roughly the same character length as English answers and the token count is 2.2x, it's tokenization. If Japanese answers are 1.5x the character length, the model is being more verbose in Japanese and a length instruction will help. You can't fix tokenizer fertility with a prompt; you can fix verbosity.

What Doesn't Work

Budgeting on an Average Across Locales

The average describes no actual user and hides the multiplier. A blended $0.004/request across five locales can conceal a $0.0074 Japanese cost subsidized by a $0.0015 Spanish cost. You'll make pricing decisions from a number that describes nothing.

Treating the 17x Headline as Your Multiplier

17x is a cross-language spread from a study I didn't review, measuring scripts that probably include byte-fallback languages you don't serve. Using it as a planning multiplier will either scare you into over-provisioning or make you dismiss the topic as irrelevant to your 2.1x reality. Measure your own spread.

Counting Tokens Client-Side With the Wrong Encoder

This is the most common silent bug in this space. You count with cl100k_base because that's what the SDK example used, you route to an o200k_base model, and your token-based rate limiter now allows roughly twice the intended compute. Nothing errors. The dashboard is just wrong by a factor that varies by locale—the worst kind of wrong, because it breaks exactly the comparison the dashboard exists for.

Position: Price the Locale, Don't Average It

Multilingual support isn't a fixed-cost feature you add per locale at a roughly constant price. Tokenizer fertility converts it into a per-locale unit-economics problem: each locale has its own token multiplier, output inflation, cache payback, and routing optimum. Teams that don't model it aren't saving money by skipping the work—they're subsidizing their most expensive locale out of a budget line that appears healthy, and they find out when a single market's usage crosses a threshold and the invoice moves for no visible reason.

The fix isn't exotic. Instrument cost as a span attribute keyed by locale, run the harness on a real parallel corpus, and put the per-locale cost index next to latency in the same dashboard. That's a week of work that turns an invisible variable into a controlled one.

Mitigation Summary: Which Lever Fixes What

Further Reading

• tiktoken — OpenAI's reference BPE implementation and rank files (primary source for tokenizer mechanics).
• js-tiktoken — the JavaScript/WASM port used in the harness above.
• Hugging Face Transformers: Summary of the tokenizers — BPE, byte-level BPE, WordPiece, and SentencePiece mechanics.
• Hugging Face tokenizers documentation — reference implementation used to train and inspect vocabularies.
• Petrov, La Malfa, Torr, Bibi — Language Model Tokenizers Introduce Unfairness Between Languages — original research on cross-language tokenizer disparity; verify the identifier and the reported multiplier against the paper before citing it.
• OpenAI tokenizer and API pricing — vocabulary reference and the current rate card.
• Anthropic token counting — vendor token-counting endpoint.
• Gemini API: tokens and countTokens — vendor token-counting endpoint.
• KuCoin, 2026-10-30, "New Study Shows AI Model Costs Vary by Language, Token Usage Can Differ by 17x" — secondary aggregator report, the seed of the 17x figure. The underlying study was not reviewed for this post; linked as context only, not as evidence.