Testing Tencent Hy4 Flash on Long-Context RAG: Cost, Latency, and Recall Tradeoffs

Testing Tencent Hy4 Flash on Long-Context RAG: Cost, Latency, and Recall Tradeoffs

pr0h0••20 min read•ai
AI Usage (99%)
On this page

What Tencent Announced for Hy4 Flash, and What the Spec Sheet Leaves Out

Tencent announced Hy4 Flash on 17 November 2026, pitching it as a fast, low-cost model for everyday workflows. The two numbers that made it travel are the ones that matter if you are building a long-context RAG pipeline: a 1M-token context window and $0.07 per million tokens. This post turns those two numbers into a decision — a cost model you can run, a latency curve showing where prefill becomes the binding constraint, a recall harness you can point at your own corpus, and the crossover point where retrieval stops paying for itself.

I have not run Hy4 Flash. Anything here that looks like a measurement is either arithmetic from the published price or an assumption I have labelled as one. The harness below is real and runnable; the Hy4 Flash numbers are yours to fill in once you have endpoint access. I would rather give you a checklist that survives contact with the endpoint than a table of invented p95s.

That distinction matters. A price-per-million-token figure is the part of an inference announcement you can compute against immediately, and it is also the number most likely to be read as a quote when it is really a floor. Here is the split I am working with.

ClaimStatus
1M-token context windowStated in the announcement
$0.07 per million tokensStated in the announcement
Announcement date, 17 Nov 2026Stated by Tencent
Whether $0.07 is input, output, or blendedNot stated in the material I saw
Prefix/prompt caching support or discountNot stated
Tokenizer behaviour, especially for CJKNot stated
Rate limits, concurrency, regional availabilityNot stated
Measured TTFT, throughput, or recall on Hy4 FlashNot measured by me

Confirmed Details — 1M Context, $0.07 per Million Tokens, November 2026 Launch

The headline is a real step change in list price. At $0.07/M, a full 1M-token prompt costs $70. A 4,000-token RAG prompt costs $0.00028. Those are the two ends of the design space, and every decision in this post lives somewhere between them.

Unstated Variables That Move Every Number — Output Pricing, Prefix Caching, Tokenizer, Rate Limits, Regional Availability

Five gaps change the cost model more than the headline price does.

Output pricing. Every pipeline I have built is input-heavy and output-light: 4K in, 300 out. If output is priced on a different tier, which is common, the blended cost is dominated by whichever side is larger. That flips for agent loops that emit long tool-call payloads.

Prefix caching. This is the biggest unknown for agent workloads. An agent loop re-sends a growing transcript on every turn, and without caching you rebill the entire prefix each time.

Tokenizer. The chars/4 heuristic is fine for English prose and wrong for CJK. Tencent's model is likely to be strong on Chinese text, and Chinese typically lands near one token per character. A budget tuned on English will overflow by roughly 4× on Chinese documents.

Rate limits and concurrency. A $0.07 price means little if you can only hold a handful of concurrent streams. Batch extraction runs on throughput, not unit price.

Regional availability. Latency depends on where the endpoint sits relative to your workers. A 100ms network floor and a 5s prefill are different problems with different fixes.

Benchmark Design — Task, Corpus, Criteria, and Counts

This section exists so you can run it. Two years of long-context claims have taught me that the only number worth quoting is one produced by a harness you control, on documents you own.

Corpus and Query Construction for a Real Long-Context RAG Workload

The corpus I use for this kind of evaluation is deliberately boring: 400 financial filings and vendor contracts, averaging 18,000 tokens each, plus a 120-question set where every question has a ground-truth answer that appears verbatim in exactly one document. No paraphrased needles, no synthetic filler. The questions fall into three families:

  • Single-fact lookup — "what is the termination notice period" — the easy case.
  • Aggregation — "sum the late fees across all schedules" — needs several spans.
  • Conditional — "does the liability cap apply if the breach is wilful" — needs the clause plus its exception.

The conditional family is where stuffed context usually loses to retrieval, and it is the family most needle tests skip entirely.

Metrics — Recall@k, Field-Level Exactness, TTFT, p95 Latency, Billed Tokens

Five metrics, each with a fixed definition, so a result cannot be quietly reinterpreted later:

MetricDefinition
Recall@kFraction of questions where every gold span is in the context sent to the model
Field exactnessExact string match on all extracted fields, per document
TTFTTime from request send to first streamed content token
p95 latency95th percentile of total wall time across the run
Billed tokensSum of input tokens reported by the API's usage object, not my estimate

That last one matters more than it looks. Estimating tokens and multiplying by a price produces fiction. Reading the usage field is the only way to keep the cost model honest, and it also tells you whether your tokenizer estimate is drifting.

What the Harness Controls and What It Does Not

It controls prompt construction, chunk boundaries, retrieval parameters, concurrency, and retry policy. It does not control serving-side batching, queue depth, or how the provider routes requests. That is why you record p95 rather than mean, and why you record the time of day you ran.

⚠️

A single afternoon of latency samples on a shared endpoint is a snapshot, not a spec. Re-run after any provider-side change and treat the old numbers as history.

The Hy4 Flash Cost Model, Worked Out at $0.07 per Million Tokens

This part is pure arithmetic from the stated price, so it is the part I can state with confidence.

Cost per 1K Requests Across Three Pipeline Shapes

cost-model.mjs
const PRICE_PER_MTOK = 0.07; // USD — the only published number
const usd = (tokens) => (tokens / 1e6) * PRICE_PER_MTOK;

const shapes = {
retrieve_4k:      { tokensPerCall: 4_000,   callsPerRequest: 1 },
stuff_200k:       { tokensPerCall: 200_000, callsPerRequest: 1 },
agent_flat_12x6k: { tokensPerCall: 6_000,   callsPerRequest: 12 },
};

for (const [name, s] of Object.entries(shapes)) {
const per1k = s.tokensPerCall * s.callsPerRequest * 1_000;
console.log(
  name.padEnd(18),
  String(per1k).padStart(12),
  "tok/1k-req",
  "$" + usd(per1k).toFixed(2)
);
}

Observed output:

retrieve_4k              4000000 tok/1k-req $0.28
stuff_200k             200000000 tok/1k-req $14.00
agent_flat_12x6k        72000000 tok/1k-req $5.04

At this price, a retrieval pipeline costs about a quarter per thousand requests. That is not a budget line; it is rounding error. Stuffing 200K tokens per request costs $14 per thousand — fifty times more, and still cheap in absolute terms for a back-office job.

The agent-loop figure is the one I distrust most, and the script below shows why.

agent-loop-growth.mjs
// An agent loop re-sends the transcript each turn. The transcript grows.
const PRICE_PER_MTOK = 0.07;

function agentLoopTokens({ start, growth, turns }) {
let total = 0;
for (let i = 0; i < turns; i++) total += start + i * growth;
return total;
}

const flat = 6_000 * 12;
const growing = agentLoopTokens({ start: 6_000, growth: 1_000, turns: 12 });
const usd = (t) => (t / 1e6) * PRICE_PER_MTOK;

console.log("flat estimate    ", flat, "$" + usd(flat * 1000).toFixed(2));
console.log("growing transcript", growing, "$" + usd(growing * 1000).toFixed(2));
flat estimate     72000 $5.04
growing transcript 138000 $9.66

The growing transcript is about 90% higher than the flat "12 calls × 6K" estimate, because turn 11 is not 6K tokens, it is 16K. Without prefix caching — the feature that collapses this back down — an agent loop's cost is quadratic in turn count. That is the first thing I would verify on the Hy4 Flash pricing page.

Finding the Retrieval-versus-Stuffing Crossover Point

The useful question is not "is stuffing expensive" but "at what corpus size does the retrieval stack stop paying for itself." Retrieval costs tokens plus infrastructure; stuffing costs only tokens. Set them equal:

crossover.mjs
const PRICE_PER_TOKEN = 0.07 / 1e6;
const per1k = (tokens) => tokens * 1_000 * PRICE_PER_TOKEN;
const RETRIEVAL_TOKENS = 4_000; // query + k chunks + system prompt

for (const infra of [0.10, 0.40, 1.00, 5.00]) {
const breakEven =
  (per1k(RETRIEVAL_TOKENS) + infra) / (1_000 * PRICE_PER_TOKEN);
console.log(
  'infra $' + infra.toFixed(2) + '/1k-req -> stuff up to ' +
  Math.round(breakEven).toLocaleString('en-US') + ' tokens'
);
}
infra $0.10/1k-req -> stuff up to 5,429 tokens
infra $0.40/1k-req -> stuff up to 9,714 tokens
infra $1.00/1k-req -> stuff up to 18,286 tokens
infra $5.00/1k-req -> stuff up to 75,429 tokens

This is the most actionable number in the post. If your corpus per request is under roughly 10K tokens and your retrieval stack costs about $0.40 per thousand queries, the vector database is not earning its keep. Skip it, stuff the documents, and spend the engineering time on a regression suite instead.

Above that, retrieval wins on cost, and it wins decisively — at 200K stuffed tokens the retrieval stack would need to cost more than $13.72 per thousand requests before stuffing became the cheaper option. Almost nothing costs that much. The price cut does not make stuff-everything rational for RAG; it makes it rational for batch work, where the numbers look completely different.

Why the Sticker Price Is a Floor, Not a Quote

Three things push actual spend above the model: output tokens at an unknown rate, retries on truncated or malformed responses, and non-cached prefix repricing in multi-turn work. A 2% retry rate on a $14/1K-request pipeline is $0.29/1K of pure waste — small here, but it scales with whatever volume you actually run.

Latency — Prefill Dominates Past 100K Tokens

I cannot give you Hy4 Flash's TTFT. I can give you the shape of the curve and the reason prefill, not decoding, is the binding constraint for long-context RAG.

TTFT Scaling by Input Size

Prefill is compute-bound and scales roughly linearly with input tokens at fixed hardware. Decode is memory-bandwidth-bound and scales with output tokens. For a 200K-token input and a 300-token answer, prefill is not merely the larger term — it is nearly the only term.

The model below takes two inputs I do not have from the announcement: sustained prefill throughput and a fixed queue-plus-network overhead. Both are assumptions.

ttft-model.mjs
// ASSUMPTIONS, not measurements: fix these with your own probe.
const OVERHEAD_MS = 250;     // queue + network, assumed
const PREFILL_TOK_S = 20_000; // sustained prefill throughput, assumed

const ttft = (tokens) => OVERHEAD_MS + (tokens / PREFILL_TOK_S) * 1000;

for (const n of [4_000, 32_000, 100_000, 200_000, 1_000_000]) {
console.log(String(n).padStart(9), 'in ->', Math.round(ttft(n)) + 'ms TTFT');
}
     4000 in -> 450ms TTFT
    32000 in -> 1850ms TTFT
   100000 in -> 5250ms TTFT
   200000 in -> 10250ms TTFT
  1000000 in -> 50250ms TTFT

Under those assumptions, 100K tokens of context buys you a five-second wait before the first character appears. At 200K it is ten seconds. A one-million-token prompt is a fifty-second prefill, and that is the optimistic case where nothing queues behind you.

The conclusion does not depend on the exact throughput constant. It holds for any prefill rate in the low tens of thousands of tokens per second, which is the region every long-context endpoint I am aware of operates in. If Hy4 Flash's "high-speed" claim means materially better than that, it is the most interesting thing about the model and the first number I would measure.

Tail Latency and Realistic Timeout Budgets

Your timeout must cover the p95, not the mean. On shared endpoints the tail is where retries come from, and retries on a 200K-token request are expensive in both money and wall time. A budget that works:

Shapep95 targetTimeoutBehaviour on breach
Interactive retrieval, 4K1.5s4sFall back to cached answer or a "narrow your question" state
Batch extraction, 100K30s90sRetry once, then park the document for review
Agent step, 6–20K3s12sAbort the turn, return the partial tool result

The interactive row is the one that rules out big context. Human patience for a search box is roughly two seconds; no amount of cheap tokens buys that back.

Recall — Stuffed Context versus Retrieved Context

Here I have to be explicit about what I know and do not. I have not scored Hy4 Flash on the 120-question set. What follows is the harness that would produce the score, the published evidence on how long-context models generally behave, and the failure modes I would look for first.

Why Needle-in-a-Haystack Results Do Not Predict RAG Quality

The single-needle test — plant one sentence in a long document, ask for it — is a test of lexical retrieval inside the attention window. It is a useful smoke test and a terrible quality metric, for three reasons.

First, a needle is verbatim in the context; real questions require the model to recognise that a paraphrased clause answers them. Second, one needle is one hop; aggregation questions need several spans combined. Third, needles are usually placed in otherwise unrelated filler, so the model faces no distractor that looks like the answer. Real corpora are full of near-miss clauses — the 2024 indemnity clause in one contract and the 2025 one in another — and picking the wrong one is a silent failure that exact-match scoring catches and needle tests never will.

The academic record on this is consistent. Liu et al.'s "Lost in the Middle" (arXiv:2307.03172) showed accuracy depends on where in the context the relevant information sits, with a U-shaped curve favouring the start and end. Hsieh et al.'s RULER (arXiv:2404.06654) found that most models' effective context is well short of their advertised window once tasks require multi-hop reasoning over the whole input. A 1M-token window is a capability claim; an effective 1M-token window is a much stronger one, and the announcement does not distinguish between them.

Position Effects and Middle-Of-Context Degradation

The practical consequence of the U-curve is a prompt-assembly rule: put instructions first, the question last, and the highest-value evidence adjacent to one of those two anchors. In a stuffed pipeline you control this ordering. In a retrieval pipeline you also control it, which is one of the underrated arguments for retrieval — you are choosing what goes where rather than hoping the ranking falls out well.

If you want to test position effects on Hy4 Flash, take twenty of the 120 questions and run each one five times with the gold span rotated through the front, the 25% mark, the midpoint, the 75% mark, and the end. Same prompt length every time. Twenty questions × five positions is a hundred calls — trivial at $0.07/M — and it tells you whether you need to engineer around the middle.

Failure Modes Observed, and the Ones That Remain Suspected

From long-context pipelines generally, the failures I would expect to reproduce here:

  • Distractor capture. The model returns a clause that matches the question's vocabulary but belongs to a different party or year.
  • Silent truncation. Context exceeds the ceiling, the provider truncates from one end, and the answer comes back confidently wrong.
  • Aggregation drift. Summing or comparing across many spans in one pass loses items in the middle third.
  • Instruction decay. System-prompt constraints applied on page one and forgotten by page four hundred.

Suspected but unverified for this model: how strongly the effective window trails the 1M claim, and whether CJK documents behave differently from English at the same token count. Both are one afternoon of calls away from being facts.

💡

Report recall as a fraction with the count beside it — "41/50 fully correct, all 9 failures in the tax-ID field" — not as a percentage in isolation. The failure clustering is the actionable part.

Building the Long-Context RAG Pipeline in Node

The pipeline below is deliberately small. It assumes one endpoint shape and no framework, so the budgeting logic is visible rather than buried.

Prompt Assembly, Token Budgeting, and Truncation Policy

prompt-budget.mjs
// Swap this for the real tokenizer once you know which one the endpoint uses.
// English prose: ~4 chars/token. CJK: closer to 1 char/token — do not reuse.
export function estimateTokens(text) {
const cjk = (text.match(/[\u3000-\u9fff]/g) ?? []).length;
const rest = text.length - cjk;
return Math.ceil(cjk + rest / 4);
}

export function buildPrompt({
system,
question,
chunks,              // pre-ranked, best first
maxInputTokens,
reservedForOutput,
}) {
const ceiling = maxInputTokens - reservedForOutput;
const fixed = estimateTokens(system) + estimateTokens(question);
let budget = ceiling - fixed;

if (budget <= 0) {
  throw new RangeError('system + question already exceed the input ceiling');
}

const kept = [];
for (const chunk of chunks) {
  const t = estimateTokens(chunk.text);
  if (t > budget) break;      // stop at the first chunk that does not fit
  kept.push(chunk);
  budget -= t;
}

return {
  prompt: [system, ...kept.map((c) => c.text), question].join('\n\n'),
  kept: kept.length,
  dropped: chunks.length - kept.length,
  estimatedInputTokens: ceiling - budget,
};
}

Two design choices worth defending. The loop breaks at the first chunk that does not fit rather than skipping it and continuing — skipping produces a context with holes in the ranking, and a hole is worse than a shorter list. And the CJK split is not decorative: if you point an English-tuned chars / 4 estimate at Chinese contracts, the budget check passes and the request fails at the API with a length error, or worse, gets silently truncated.

Fallback Behaviour When Context Overflows or Latency Spikes

call-with-fallback.mjs
export async function callWithFallback(body, { timeoutMs = 12_000, retries = 1 } = {}) {
for (let attempt = 0; attempt <= retries; attempt++) {
  const ac = new AbortController();
  const timer = setTimeout(() => ac.abort(), timeoutMs);
  try {
    const res = await fetch(ENDPOINT, {
      method: 'POST',
      headers: { 'content-type': 'application/json', authorization: 'Bearer ' + process.env.HY4_KEY },
      body: JSON.stringify(body),
      signal: ac.signal,
    });

    if (res.status === 413 || res.status === 400) {
      // Length problem: retrying is pointless. Halve the context and surface it.
      return { ok: false, reason: 'context_overflow', status: res.status };
    }
    if (res.status === 429 || res.status >= 500) {
      if (attempt === retries) return { ok: false, reason: 'upstream', status: res.status };
      await new Promise((r) => setTimeout(r, 400 * 2 ** attempt));
      continue;
    }
    return { ok: true, data: await res.json() };
  } catch (err) {
    if (err.name === 'AbortError') return { ok: false, reason: 'timeout' };
    throw err;
  } finally {
    clearTimeout(timer);
  }
}
}

The important part is the asymmetry: a 400 or 413 is a deterministic failure — the same request will fail identically — so retrying it wastes money and time. A 429 or 5xx is transient and deserves backoff. Conflating the two is the most common bug I see in inference clients, and with cheap tokens it gets expensive fast because people retry generously.

Where the Tradeoffs Bite by Workload

The same price per token behaves very differently depending on who is waiting on the other end of the call.

Agent Loops — Many Small Calls, Cost Is Not the Constraint

A twelve-turn agent that makes twelve 6K-token calls bills somewhere between $5 and $10 per thousand tasks depending on prefix caching. Neither number should influence the design. What should: a 5-second prefill inserted into a loop that runs twelve times adds a minute of dead time per task. Agent loops want small contexts and fast TTFT. Long context is the wrong tool here, and low price does not change that.

Batch Extraction — Few Huge Calls, This Is the Sweet Spot

Overnight extraction over 20,000 contracts, one call per document, no human waiting. A 150K-token stuffed prompt costs $0.0105 per document. Ten thousand documents cost $105. Recall is measured against a 200-document human-labelled sample, and anything below the bar goes to a review queue. This is where the price change is genuinely transformative: a job that was previously uneconomical to run daily is now cheaper than the storage it reads from.

Interactive Retrieval — Latency Budget Kills the Big-Context Option

A search box with a two-second budget cannot send 200K tokens, no matter what they cost. The decision is made by the latency table, not the price table. Retrieval stays, and the $0.07 price just makes the reranking pass free.

A Decision Table for Choosing a Long-Context RAG Pipeline Shape

WorkloadShapeWhy
Interactive Q&A, corpus < 10K tokens/requestStuff everythingRetrieval stack costs more than the tokens
Interactive Q&A, corpus > 10K tokensRetrieve top-k, then answerPrefill latency, not price, is binding
Batch extraction, documents 50–200KStuff one document per call$0.01/doc, no retrieval loss, no recall risk
Batch extraction, documents > 500KChunk and map-reducePrefill time and truncation risk
Agent loopSmall context, aggressive tool schemasTTFT × turn count dominates
Anything multi-turnVerify prefix caching firstOtherwise cost is quadratic in turns

Guardrails for Cheap Long Context

Cost Circuit Breakers and Per-Request Token Ceilings

Cheap tokens encourage sloppy prompts. Put a hard ceiling in the client — reject any assembled prompt over your own limit before it hits the network — and a daily budget that trips a feature flag rather than a pager. A bug that loops a 200K-token request is $0.014 per iteration; at a thousand iterations an hour you notice, but only if something is counting.

Untrusted Document Content and Prompt Injection Surface

Stuffing 200K tokens of retrieved documents means 200K tokens of untrusted input sitting directly in the instruction path. A contract PDF with a hidden line of text can attempt to redirect the extraction. The defensive rules are unchanged from any LLM pipeline, and they apply more sharply here because stuffed context has no chunk boundary to reason about:

  • Delimit document content explicitly and instruct the model to treat it as data, not instructions.
  • Never let a document's content select a tool or a URL. Validate tool arguments against an allowlist outside the model.
  • Prefer structured output with a schema validator; a response that fails validation is a failure, not something to parse leniently.
  • Treat high-stakes extractions as requiring a second, independent check — a different span check, not a re-prompt of the same model.

The OWASP LLM Top 10 still lists prompt injection as LLM01, and nothing about cheaper context changes that ranking.

Regression Evals Before Any Price-Driven Migration

Before moving a workload from retrieval to stuffing because the arithmetic says so, run the harness on the same 120 questions for both shapes and compare field exactness. If stuffing loses 4 points of exactness to save $0.28 per thousand requests, you have made a bad trade and the arithmetic was never the whole question.

Reproducing the Harness

  1. Assemble the 400-document corpus and the 120-question set with gold spans marked.
  2. Run the cost scripts above with your real token counts from the usage field, not estimates.
  3. Probe TTFT at 4K, 32K, 100K, 200K, and 1M input tokens, twenty samples each, and record p50 and p95 separately.
  4. Run the position-rotation test on twenty questions.
  5. Run both pipeline shapes on the full set and compare field exactness with the failure clustering named.
  6. Re-run after any provider-side change. Prices and serving behaviour both move.

Where I Land — Retrieve for Interactive, Stuff for Batch

The $0.07 price is real and it is the cheapest long-context input pricing I have seen announced. It changes the answer for batch extraction, where a 150K-token stuffed document now costs about a cent — a workload that was previously a retrieval problem mostly because retrieval was cheaper, not because it was better.

It does not change the answer for interactive RAG. Prefill dominates the latency budget past 100K tokens, and no price gets a human to wait five seconds for a search result. It does not change the answer for agent loops either, where the constraint is turns × TTFT and, without prefix caching, a quadratic bill.

The claim I would most like to be wrong about: that Hy4 Flash's effective context trails its 1M window the way most 2026 long-context models do. That is a guess, marked as one, and a hundred calls would settle it. Until then, the responsible default for long-context RAG is retrieve for interactive, stuff for batch, and measure before you migrate either.

Further Reading

Share this post

More posts

Loading comments…