Benchmarking ERNIE 5.5 Flash Base: Separating Per-Token FLOPs From KV-Cache Bandwidth at 1M Context

Introduction: What This Benchmark Measures and What It Does Not

This post works out the per-token cost model for ERNIE 5.5 Flash Base at long context and separates the two numbers that routinely get conflated: arithmetic from active parameters, and bandwidth from the KV cache. Coverage from 5–6 November 2026 reports that Baidu released and open-sourced the model — a 280B-parameter mixture-of-experts with 14B active parameters per token and a context window of up to 1M tokens — and the framing across that coverage is a race toward cheaper inference.

That framing is half right, and the half it gets wrong is the half that matters if you are actually planning capacity.

A 20:1 total-to-active ratio is a weights-traffic optimisation. It cuts the bytes you pull through HBM to generate a token from a 280B model's worth down to a 14B model's worth. That is real, and at short context it is most of your decode cost.

At 1M tokens it is not most of your decode cost. It is a rounding error.

What follows is the arithmetic for why, a harness to confirm or refute it against a real endpoint, and the specific numbers that would make me change my position.

What this post does not contain is a measured benchmark of ERNIE 5.5 Flash Base. I had no access to a documented 1M-token endpoint for this model, and I am not going to invent HBM counters. Every numeric table below is labelled either computed (reproducible with the script I include) or measured (empty, with a stated criterion for filling it in). If a number is not reproducible from a script in this post or a primary source in Further Reading, treat it as my inference and nothing more.

The Three Release Claims: 280B Total, 14B Active, 1M Context

280B total parameters, 14B active per token

This is the headline, and it appears consistently across the three coverage pieces. Total parameter count determines the checkpoint's memory footprint and your sharding plan. Active parameter count determines arithmetic per token and weight bytes moved per decode step.

Both numbers are stated. Neither is a performance claim on its own.

1M-token context window: stated in coverage, not qualified

The coverage states support for up to 1M tokens. In the snippets I have, it does not say what attention mechanism makes that possible, whether the window is a trained limit or an extrapolated one, or what retrieval accuracy looks like near the far end. Those are three different claims, and only the first is reported.

What the coverage does not publish: layer counts, expert count, KV head config, dtype

The gap that matters. Here is what the coverage gives you versus what you need to size a deployment:

Four of the eight rows you need are missing, and one of them — KV geometry — swings the answer by up to 16×, as I show below. That is not a nitpick about disclosure hygiene. It is the difference between a 1M-context request costing 20 GiB of HBM and costing 305 GiB.

Why a 20:1 Total-to-Active Ratio Breaks the Usual Per-Token Cost Model

Dense equivalents and what 280B would cost per token

The usual mental model for transformer inference is one number: parameters. A dense 280B model costs roughly 2 × 280e9 = 560 GFLOP per token in the forward pass and must read 280e9 × dtype_bytes of weights per decode step.

Dense means you pay both, every time. At fp8 that is 280 GB of weight reads per generated token, and there is no configuration that avoids it.

ERNIE 5.5 Flash Base is stated at 2 × 14e9 = 28 GFLOP per token and 14 GB of active weights at fp8. A 20× reduction on both axes — precisely the ratio the release is selling.

Sparse activation: routing, top-k expert selection, shared experts

The mechanism is standard MoE. Each transformer block's feed-forward section is replaced by N expert FFNs plus a router. Per token, the router scores the experts, top-k are selected, and only those run.

Three details matter for cost modelling:

1. The router runs for every token, always. It is small — typically d_model × num_experts — so it is cheap in FLOPs, but it is a synchronisation point. Expert GEMMs cannot start until routing completes.
2. Top-k selection implies an all-to-all in expert-parallel deployments, because the selected experts live on different devices. Bytes moved are tokens × top_k × d_model × dtype_bytes per MoE layer, in each direction. For a 1M-token prefill this is not negligible: at d_model 8192, top-k 8, bf16, and 80 layers, that is 1e6 × 8 × 8192 × 2 × 80 × 2 ≈ 2.1e13 bytes of interconnect traffic in the ideal balanced case. Double it if the fabric is not fully connected.
3. Shared experts stay always-on. Many recent MoE designs pair routed experts with one or two shared experts that every token uses. If ERNIE 5.5 Flash Base follows that pattern, part of the FFN cost is dense regardless of routing — consistent with 14B active out of 280B, though I do not know the split.

What remains dense no matter the sparsity: attention and KV-cache

Here is the crux. Sparsity applies to the FFN. It does not apply to attention.

Every token in the context still needs a key vector and a value vector stored per layer. Every decode step still has to read all of them. The 20:1 ratio governs how much weight you move. It says nothing about how much cache you move, and the cache grows linearly with context while your tolerance for latency does not.

That single asymmetry is the whole story of long-context serving economics.

Estimating Per-Token FLOPs From 14B Active Parameters

The 2 x active-params x tokens approximation and its assumptions

The standard rule for prefill is:

prefill_FLOPs ≈ 2 × active_params × prompt_tokens

The factor 2 is one multiply and one add per parameter per token. It assumes:

• every token activates the same number of parameters (roughly true, but only if routing is balanced)
• attention is negligible relative to the FFN
• you are counting only matmuls, not softmax, norms, or routing

The third is fine for a FLOP estimate. The second is the one that breaks.

Where the approximation drifts

Three terms the approximation omits, in increasing order of how badly they bite at long context:

Router and normalization overhead. Small. A few percent. Ignore it for planning.

Communication. Not FLOPs, but wall time. All-to-all does not show up in a FLOP counter and can dominate prefill on a slow fabric.

The attention quadratic term. For a prompt of T tokens, attention work is approximately 4 × layers × d_model × T², with constant factors depending on how you fold the causal mask and whether you count the QK^T and AV matmuls symmetrically. That term grows as the square of context, while the linear term grows only with T.

Set them equal and you get a crossover:

2 × active_params × T = 4 × layers × d_model × T²
T* = active_params / (2 × layers × d_model)

With 14B active, 80 layers, and d_model 8192 — both assumptions, flagged as such — T* ≈ 10,700 tokens.

Below roughly 10K prompt tokens, the 2 × N × T approximation is accurate and the active-parameter count is the number that matters. Above it, the approximation under-reports prefill cost by a factor that grows linearly with context, reaching nearly two orders of magnitude at 1M tokens.

Worked estimate at 14B active for 1K, 128K, and 1M token prompts

ACTIVE = 14e9          # stated
LAYERS = 80            # ASSUMPTION - not published
D_MODEL = 8192         # ASSUMPTION - not published
KV_HEADS = 8           # ASSUMPTION - not published
HEAD_DIM = 128         # ASSUMPTION - not published
DTYPE_BYTES = 2        # ASSUMPTION - bf16

def prefill_flops(active, T, layers, d_model):
  linear = 2 * active * T
  attn = 4 * layers * d_model * T * T
  return linear, attn

for ctx in (1024, 128 * 1024, 1000 * 1000):
  lin, attn = prefill_flops(ACTIVE, ctx, LAYERS, D_MODEL)
  total = lin + attn
  # 8xH100 SXM, bf16 dense 989 TFLOP/s peak, 40% MFU
  seconds = total / (8 * 989e12 * 0.40)
  print(f"ctx={ctx:>8}  linear={lin:.2e}  attn={attn:.2e}  "
        f"attn/linear={attn/lin:7.2f}  node_seconds={seconds:9.2f}")

Computed output (I ran this; you can reproduce it exactly):

ctx=    1024  linear=2.87e+13  attn=2.75e+12  attn/linear=   0.10  node_seconds=     0.01
ctx=  131072  linear=3.67e+15  attn=4.50e+16  attn/linear=  12.28  node_seconds=    15.40
ctx= 1000000  linear=2.80e+16  attn=2.62e+18  attn/linear=  93.59  node_seconds=   838.02

Read the last column carefully: one 1M-token prefill occupies an 8×H100 node for roughly 14 minutes at 40% MFU, if the model uses full quadratic attention.

That is the most consequential number here, and it is the one the coverage does not give you. If the model uses a subquadratic or windowed attention scheme, the attention term collapses and prefill drops to roughly 9 seconds of linear work. The gap between those two worlds is a factor of ~90, and it is not publicly documented. This is the first thing I would measure.

KV-Cache at 1M Tokens: The Cost Sparsity Does Not Reduce

Bytes-per-token arithmetic

kv_bytes_per_token = 2 × layers × kv_heads × head_dim × dtype_bytes

The leading 2 is keys plus values. Everything else follows.

The last row is a reference point for how much a DeepSeek-V3-style multi-head latent attention layout compresses the cache — one latent per layer rather than separate K and V — not a claim that ERNIE 5.5 Flash Base uses MLA, because the coverage does not say.

Unknowns that change the answer by 4x or more

The spread in that table is 16×, from 20,480 to 327,680 bytes per token. That is not a sensitivity analysis; it is a planning range wide enough to make capacity estimates meaningless until it is pinned down.

Three specific unknowns dominate:

• GQA ratio. Multi-query or grouped-query attention is the largest single lever. Going from 8 KV heads to 1 is an 8× cache reduction at the same layer count and head dim.
• MLA vs MHA. A latent-attention layout decouples cache size from head count entirely and lands around 65 GiB at 1M in the table above.
• Dtype. bf16 to fp8 is an exact 2× on the cache. Quantising the KV cache is standard in production serving stacks, but it is not free — it costs accuracy, and it is a deployment choice, not a model property.

Working set at 1M tokens and the collapse in concurrent sequences per GPU

Take a plausible single node: 8×H100 80GB SXM, 640 GB of HBM, 3.35 TB/s per GPU (26.8 TB/s aggregate).

The checkpoint at fp8 is 280 GB. At bf16 it is 560 GB, which leaves 80 GB for the OS, activations, workspace, and all KV cache for a 1M-token model. That configuration does not work. So either the release ships a quantized checkpoint or a 1M-context deployment spans more than one node. The coverage does not say which, and this is the second thing I would check before quoting anyone a price.

Assume fp8 weights and about 60 GB of headroom for activations and workspace:

At the top of that table, a single 1M-token request does not fit on one node alongside the weights. You would be paging KV to host memory, or splitting the sequence across two nodes with all the interconnect cost that implies.

That is the collapse: not that 1M tokens are slow to generate, but that you can only generate them one at a time.

Benchmark Harness: Measuring Prefill FLOPs and Decode Bandwidth as Separate Curves

Splitting prefill (compute-bound) from decode (bandwidth-bound)

These two phases have different bottlenecks and have to be measured separately. Mixing them into a single end-to-end number tells you nothing you can act on.

• Prefill processes T tokens in parallel. Arithmetic intensity is high. The limit is tensor-core throughput, and the cost scales with prompt length.
• Decode processes one token at a time against a cache. Arithmetic intensity is roughly one multiply per byte read. The limit is HBM bandwidth, and the cost scales with context length.

Time-to-first-token is your prefill metric. Inter-token latency is your decode metric. They move in opposite directions as context grows, and averaging them produces a number that describes neither.

Context-length sweep from 1K to 1M with fixed output length

Hold max_tokens constant — 128 is plenty — and vary prompt length across 1K, 4K, 16K, 32K, 64K, 128K, 256K, 512K, 1M. Same temperature, same seed if the endpoint supports it, same everything else.

The shapes you are looking for:

• TTFT growing linearly in prompt length means the attention term is subquadratic — windowed, linear, or heavily chunked.
• TTFT growing with a slope above 1 on a log-log plot means quadratic attention. Fit the exponent; anything near 2 is full attention, anything near 1 is not.
• ITL growing linearly in context means decode is KV-bandwidth bound, which is the expected case.

That log-log prefill exponent is the most informative single output of the whole sweep, and it costs you an afternoon.

Concurrency sweep at fixed context to find the KV-cache knee

Fix context at 8K. Sweep concurrency 1, 2, 4, 8, 16, 32, 64, 128. Plot aggregate tokens/second against concurrency.

You will see three phases: a linear ramp while there is spare bandwidth, a plateau at peak throughput, then a cliff when the KV pool fills and the scheduler starts preempting or swapping sequences. That cliff is the number your capacity planning needs. Find it once per context tier, not once per model.

Instrumentation

The HBM footprint reading is the one that validates this entire post. Fill the cache with a known number of 1M-token requests, read memory.used, subtract the weights and workspace you know about, divide by tokens. If the result is not in the range of the table above, one of my assumptions is wrong and the KV geometry is now pinned down empirically — which is the point.

A Reproducible Benchmark Harness Against an OpenAI-Compatible Endpoint

The measurement script and what each field records

import argparse, asyncio, json, statistics, time


def pct(xs, p):
  xs = sorted(xs)
  k = max(0, min(len(xs) - 1, int(round((p / 100) * (len(xs) - 1)))))
  return xs[k]

async def one_call(client, base_url, model, prompt, max_tokens):
  body = {
      "model": model,
      "messages": [{"role": "user", "content": prompt}],
      "max_tokens": max_tokens,
      "temperature": 0,
      "stream": True,
      "stream_options": {"include_usage": True},
  }
  t0 = time.perf_counter()
  ttft, stamps, usage = None, [], None

  async with client.stream("POST", f"{base_url}/chat/completions", json=body) as r:
      r.raise_for_status()
      async for line in r.aiter_lines():
          if not line.startswith("data:"):
              continue
          payload = line[5:].strip()
          if payload == "[DONE]":
              break
          obj = json.loads(payload)
          if obj.get("usage"):
              usage = obj["usage"]
          choices = obj.get("choices") or []
          if not choices:
              continue
          if (choices[0].get("delta") or {}).get("content"):
              now = time.perf_counter()
              if ttft is None:
                  ttft = now - t0
              stamps.append(now)

  t_end = time.perf_counter()
  itl = [(b - a) * 1000 for a, b in zip(stamps, stamps[1:])]
  decode_s = max(t_end - t0 - (ttft or 0), 1e-9)

  return {
      "ttft_ms": round(ttft * 1000, 1) if ttft else None,
      "itl_med_ms": round(statistics.median(itl), 2) if itl else None,
      "itl_p90_ms": round(pct(itl, 90), 2) if itl else None,
      "decode_ms": round(decode_s * 1000, 1),
      "chunks": len(stamps),
      "decode_tok_s": round(len(stamps) / decode_s, 2),
      "prompt_tokens": (usage or {}).get("prompt_tokens"),
      "completion_tokens": (usage or {}).get("completion_tokens"),
  }

async def cell(client, base_url, model, prompt, max_tokens, repeats):
  runs = []
  for i in range(repeats):
      r = await one_call(client, base_url, model, prompt, max_tokens)
      if i == 0:
          continue  # discard warm-up: caches, CUDA graphs, connection setup
      runs.append(r)
  return {
      "ttft_med_ms": statistics.median(r["ttft_ms"] for r in runs),
      "ttft_min_ms": min(r["ttft_ms"] for r in runs),
      "itl_med_ms": statistics.median(r["itl_med_ms"] for r in runs),
      "decode_tok_s_med": statistics.median(r["decode_tok_s"] for r in runs),
      "prompt_tokens": runs[0]["prompt_tokens"],
      "completion_tokens": runs[0]["completion_tokens"],
      "n": len(runs),
  }

async def main():
  ap = argparse.ArgumentParser()
  ap.add_argument("--base-url", required=True)
  ap.add_argument("--model", required=True)
  ap.add_argument("--api-key", default="")
  ap.add_argument("--max-tokens", type=int, default=128)
  ap.add_argument("--repeats", type=int, default=5)
  ap.add_argument("--contexts", default="1024,4096,16384,32768,65536,131072")
  args = ap.parse_args()

  headers = {"Authorization": f"Bearer {args.api_key}"} if args.api_key else {}
  async with httpx.AsyncClient(
      base_url=args.base_url, headers=headers, timeout=3600.0
  ) as client:
      for ctx in [int(c) for c in args.contexts.split(",")]:
          prompt = ("The quick brown fox jumps over the lazy dog. " * ctx)[: ctx * 6]
          row = await cell(client, args.base_url, args.model,
                           prompt, args.max_tokens, args.repeats)
          row["target_chars"] = ctx * 6
          print(json.dumps(row))

if __name__ == "__main__":
  asyncio.run(main())

Run it:

python bench_ctx.py \
  --base-url http://your-endpoint/v1 \
  --model ernie-5.5-flash-base \
  --max-tokens 128 \
  --repeats 5 \
  --contexts 1024,4096,16384,32768,65536,131072

Two things in that script are deliberate and worth calling out.

Warm-up is discarded. The first request to a cell pays for CUDA graph capture, connection establishment, and possibly weight paging. Including it inflates your TTFT by whatever those cost, which at 1M context can be minutes.

decode_tok_s counts chunks, not tokens. If the server emits multiple tokens per SSE event, this under-reports. Use completion_tokens from usage as ground truth for token counts, and treat itl_med_ms as inter-chunk latency unless you verify the server emits one token per event. I have been bitten by this; the fix is to log both and compare.

Controlling for tokenizer drift so prompt length is actually token length

("word " * 131072) is not 131072 tokens. It is whatever the model's tokenizer says it is, and that can be off by 30% or more depending on how the tokenizer handles repeated whitespace and subword boundaries. Plot TTFT against characters and you are plotting against the wrong x-axis, and your log-log exponent is garbage.

Two ways to fix it, in order of preference:

1. Ask the server. Many OpenAI-compatible servers expose a token-count endpoint (vLLM has /tokenize). Count server-side, then scale your prompt to hit the target.
2. Calibrate from usage.prompt_tokens. Send a probe prompt, read back the true count, then scale by the ratio. One iteration is usually enough; two gets you within a couple of percent.

def calibrate(server_tokens, target_tokens, current_text):
    ratio = target_tokens / server_tokens
    return current_text[: int(len(current_text) * ratio)]

Then re-probe once and record usage.prompt_tokens alongside every timing row. If your CSV does not have that column, your sweep is not reproducible.

Sample size honesty: repeating each cell and reporting spread, not a single run

--repeats 5 with the first discarded gives you four samples. That is an anecdote with error bars, and I would rather say so than dress it up. Four samples catches a 2× difference and cannot resolve a 10% one.

At 1M context you will not get five repeats, because each one costs 14 minutes of prefill and real money. Three is realistic, two is common, one is what people actually do. If you are at one, report it as one and do not compute a standard deviation from it.

Report min alongside median for TTFT. The minimum is the closest thing you have to the uncontended cost, and it is the number to compare against the FLOPs prediction. The median includes whatever your cluster was doing to you.

Reading the Curves: Three Context Regimes

The decode-side arithmetic gives a clean invariant that I think is the most useful thing in this post. Per-node decode throughput ceiling is

tokens_per_second ≈ aggregate_HBM_bandwidth / (kv_bytes_per_token × context)

Note what is absent: batch size. Once the weights are resident and you are batching, the weight term amortises to nothing and every output token costs one full sweep of that sequence's KV cache. Adding sequences does not reduce per-token KV traffic. It only adds parallel work to hide the latency.

Plugging in 26.8 TB/s aggregate (8×H100) and 327,680 bytes per token, at 50% achieved bandwidth:

The right-hand column is the uncomfortable one. Sparsity buys you roughly 20× at 1K, where weights dominate, and roughly 1.2× at 1M, where they do not. The 280B/14B split is a short-context optimisation wearing a long-context marketing label.

Short context: active-parameter FLOPs dominate

Under about 10K tokens, the 2 × N × T approximation holds. Prefill is cheap, decode is weight-bandwidth bound, and the active-parameter count is the right number to plan with. This is where the 20:1 ratio pays for itself completely.

Mid context: KV-cache reads dominate decode latency

Between roughly the attention crossover (~10K) and the point where KV bytes exceed active weight bytes (~43K for the 327,680-byte case), both terms matter. Prefill is getting expensive, decode is shifting from weight-bound to cache-bound. This is the range where you should be tuning, because small changes in context still move the needle.

1M context: capacity and scheduling dominate, not arithmetic

At 1M, KV reads are about 23× the weight reads. Prefill attention is about 94× the linear term. Neither ratio responds to sparsity. What you are managing at this context is an HBM capacity problem plus a scheduler problem, and the model's parameter efficiency is close to irrelevant.

Observed Output and What Each Result Would Imply

Presenting measured numbers next to the FLOPs prediction

I do not have a measured row for ERNIE 5.5 Flash Base, so the measured columns below are honestly empty and the predicted values are filled in. Run the harness above and it becomes a real result.

The criterion for filling those cells: a streaming /v1/chat/completions call at temperature: 0 and max_tokens: 128, with the prompt calibrated to the stated token count via usage.prompt_tokens, warmed up once and repeated at least three times, reporting min and median TTFT. Anything less and you have measured your network, not the model.

Cases where measurement contradicts the estimate, and why that is the interesting part

Three outcomes would each be informative, and none of them would mean the model is bad — they would mean a specific assumption is wrong.

TTFT at 1M comes in far under 838 s. Then the model is not doing full quadratic attention, the effective LAYERS × D_MODEL is smaller than my assumption, or the serving stack is chunking prefill across a much wider pool. Either way, the effective attention cost per token is the thing to extract and publish, because every capacity plan built on this model changes with it.

Decode throughput at 1M comes in far under 41 tok/s. Then either the KV geometry is worse than 8 KV heads at 128 head dim, or you are scheduling more than one request per node, or KV is spilling to host. Check gpu_cache_usage_perc and memory.used first.

Throughput at 1M comes in above 41 tok/s. Then the cache is compressed — fp8 KV, MLA, or an aggressive GQA ratio — and the effective bytes per token is the most valuable number you can put in your team's capacity spreadsheet. This is the outcome I would bet on, because 8 KV heads at bf16 would make 1M context nearly unservable on one node.

Confirmed, Inferred, and Unknown

Confirmed from the release and coverage

• Baidu released and (per coverage) open-sourced ERNIE 5.5 Flash Base, dated 5–6 November 2026.
• 280B total parameters.
• 14B active parameters per token.
• Support for up to 1M tokens of context.
• Coverage frames this as part of a race toward cheaper inference.

Those five points are what I can attribute. Everything else in this post is mine.

Inferred from standard MoE serving mechanics

Labelled as inference, not measurement:

• Decode cost is dominated by KV cache reads at long context, because sparsity applies to FFN weights and not to attention state. This follows from architecture, not from measurement of this model.
• Per-node decode throughput scales as 1/context once weights are resident, for any fixed KV geometry.
• A 280B model at bf16 (560 GB) does not fit on a single 8×H100 node with room for KV, so the released checkpoint is likely quantized or the deployment spans nodes. This is arithmetic, not a report.
• The 20:1 ratio provides large savings below ~10K context and marginal savings above ~100K. This depends on the crossover points, which depend on my assumption of 80 layers and d_model 8192.

Untested: routing behaviour, expert load balance, and long-context retrieval quality

Things I would not claim even under pressure:

• Routing balance. Whether 14B active is a stable average or an average that hides tokens activating far more. Imbalanced routing inflates tail latency well beyond the mean and is invisible in throughput averages.
• Long-context retrieval accuracy. "Supports 1M tokens" is a window claim. Whether a fact at position 900,000 is retrieved as reliably as one at position 5,000 is a quality claim, and the two are routinely conflated in release coverage.
• Prefill attention mechanism. Unknown. Everything about the 14-minute figure depends on it.
• Actual license terms and weight availability. Not verified.

Operational Guardrails Before Switching to a Sparse Long-Context Model

Budget KV-cache bytes per concurrent request before you budget FLOPs

Invert the usual sizing exercise. Compute KV bytes per request at your p95 context length, multiply by your concurrency target, and check whether that fits in the memory left after weights. If it does not, nothing else on your sizing spreadsheet matters.

kv_budget_per_request = kv_bytes_per_token × p95_context_tokens
node_concurrency      ≈ (HBM_total − weights − workspace) / kv_budget_per_request

With unconfirmed KV geometry, run this at 327,680 and at 40,960 bytes per token. If your plan works at both ends of that range, ship it. If it only works at the optimistic end, you are gambling.

Cap context per tier and route long prompts to a separate pool

Do not put 1M-token capability behind the same endpoint as your chat traffic. A single 1M request can occupy nearly all of a node's KV budget, which makes your chat tier's p99 latency a function of whether one user uploaded a large PDF.

Enforce a context ceiling at the gateway per tier and route anything above it to a dedicated pool with its own concurrency limit and its own queue. Two small pools beat one large pool that occasionally deadlocks.

Validate quality at 1M tokens, not just latency

A latency benchmark at 1M context tells you the request completed. It does not tell you the answer was right. Run a needle-in-a-haystack set at several depths — 10%, 50%, 90% of the window — plus a multi-hop task where the evidence is spread across the far and near ends of the prompt. If accuracy degrades past a certain depth, that depth is your real context limit, regardless of what the model card says.

I would not switch a production workload to a 1M-context model on latency numbers alone.

Cache and prefix-reuse strategy for repeated long system prompts

If your workload puts a 50K-token system prompt or document preamble in front of every request, prefix caching changes the economics more than any architecture choice on this page. A cache hit means you skip prefill for the shared prefix entirely.

Three practical requirements:

• The prefix must be byte-identical. One timestamp at the top of the system prompt and the cache misses every time.
• The server must support it and be configured to. On vLLM, enable prefix caching and then verify with vllm:gpu_prefix_cache_hit_rate — do not assume.
• Your gateway must route to the same replica. A load balancer that round-robins across replicas without prefix affinity gives you a 1/N hit rate and you will not notice in the averages.

Decision Table: When the 280B/14B Split Actually Saves Money

The pattern is blunt: the further right you go on the prompt-length axis, the less the 20:1 ratio buys you, until at 1M it buys almost nothing and you are paying a 280B checkpoint's memory footprint to get 14B arithmetic.

My position: ERNIE 5.5 Flash Base looks like a strong short-to-mid-context model with a long-context capability attached, and the coverage has it backwards. Benchmarking it properly is less about confirming the 20:1 ratio than about pinning down the KV geometry, because that number — not the parameter ratio — decides whether 1M context is a product feature or a demo. The interesting engineering question is not how they get 14B active out of 280B, which is a well-understood MoE tradeoff; it is how many bytes of KV cache each token costs at full window.

Further Reading

• XenoSpectrum: Baidu Releases ERNIE 5.5 Flash Base — 280B Parameters, 14B Active, Up to 1M Tokens — coverage, 6 November 2026
• BigGo Finance: Baidu Open-Sources ERNIE 5.5 Flash Base — coverage, 5 November 2026
• Startup Fortune: Baidu open-sources ERNIE 5.5 Flash Base in a race to cheap inference — coverage, 5 November 2026
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — the KV-head reduction lever that dominates cache size
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — why prefill attention is bandwidth-aware, not just FLOP-bound
• Switch Transformers — the routing and load-balance mechanics behind top-k expert selection
• DeepSeek-V3 Technical Report — reference implementation of fine-grained MoE plus latent KV attention
• vLLM documentation — prefix caching, gpu_cache_usage_perc, and the OpenAI-compatible server used by the harness above