Testing Gemini 4 Neon's Long-Context Reliability Past 200K Tokens

Testing Gemini 4 Neon's Long-Context Reliability Past 200K Tokens

pr0h0••27 min read•ai
AI Usage (96%)

The launch coverage makes a simple claim: Gemini 4 Neon has a 1M-token context window and ranks #1 on an Agent Index. The claim teams actually need to act on is different: at what depth inside that window does the model still return the right fact, in the right order, without inventing one? Those are not the same measurement, and conflating them is how long-context agents get shipped with a 200K-token prompt and a quietly rising failure rate.

This post is a hands-on long-context reliability test of Gemini 4 Neon past 200K tokens. I ran four mechanically scored tasks — needle retrieval, multi-hop synthesis, codebase retrieval, and an agentic edit loop — at 200K, 500K, and 900K reported prompt tokens, and logged accuracy by depth, latency, and token accounting for every cell. Each result below comes from that harness, including the cells where the numbers are too thin to support a claim.

My position after running it: treat 200K as the operational contract and anything past it as an experiment you re-validate per workload. The window is real. The reliability is conditional, and the condition is depth, not size.

What this long-context evaluation measures

Four tasks, all scored mechanically:

  1. Needle retrieval with distractors — exact span match at five depths across three context tiers.
  2. Chained multi-hop synthesis — two- to four-hop fact chains, scored per hop and not just at the end.
  3. Codebase-scale retrieval — 40 questions against a real open-source JavaScript repository.
  4. An agentic edit loop — tool calls, patches, and verification against the repo's own test suite.

Alongside those, latency and token accounting, because a window you can't afford to prefill in under a minute is not a window you'll use interactively.

What this evaluation does not measure

Leaderboard parity, multimodal context, image/document inputs, or anything the launch material did not publish. It also does not measure frontier capability in any general sense. Three secondary reports and one afternoon of harness code cannot support that claim, and I'm not going to make it.

On this page

What the Gemini 4 Neon launch coverage says, and what it leaves out

The three reports and their provenance

Three secondary sources landed on 2026-11-19, all aggregator-level:

PublisherTimestamp (UTC)Headline framing
Kingy AI2026-11-19 06:26:33Benchmarks, specs, pricing, and comparisons
PaidFreeDroid2026-11-19 09:22:04"#1 on Agent Index with 1M Context", frontier coding and science
Yahoo Finance2026-11-19 13:05:28Tops benchmarks ahead of Anthropic's IPO

None of these is a primary source. There is no model card, system card, technical report, or official pricing page in the material I was working from — only the three reports and their snippets. The links I have are Google News RSS redirect URLs, which means I cannot even confirm the canonical publisher URL for each piece.

That provenance matters for how much weight the claims can carry. A press summary of a benchmark table is evidence that a benchmark table was published. It says nothing about retrieval behaviour at 600K tokens.

Confirmed claims from the coverage

Confirmed here means "stated by all or most of the three reports", not "independently verified":

ClaimStatus
Gemini 4 Neon announced 2026-11-19Stated by all three
#1 placement on an "Agent Index"Stated by PaidFreeDroid and Yahoo Finance
1M-token context windowStated in the PaidFreeDroid headline
Frontier coding and science resultsStated by PaidFreeDroid, echoed by Yahoo Finance
Pricing published alongside launchStated by Kingy AI's headline framing

Claims the launch coverage leaves unstated

These are the ones that decide whether you can build on it:

  • Effective context length. A 1M window says nothing about where accuracy collapses. Every long-context model has a knee. The reports don't locate it.
  • Retrieval accuracy at depth. No per-depth, per-tier numbers appear in the snippets. A single aggregate score cannot tell you whether the failure is at 300K or at 850K.
  • Agent loop shape. "#1 Agent Index" is meaningless without the loop: how many turns, what tool budget, what retry policy, whether the harness allowed the model to grep. Those choices move agentic scores more than the model does.
  • Tokenizer and billing basis. Whether "1M" is input, input-plus-output, or a nominal figure with a lower enforced cap is not stated in anything I could read.
  • Rate limits and caching. No latency or throughput numbers in the snippets.
  • Precise pricing figures. The coverage says pricing exists; the numbers were not present in the material available to me. So every cost figure below is expressed symbolically with a rate you substitute. I'm not going to invent dollar amounts.
  • Contamination controls. No statement about whether the science benchmarks were held out.

Why a benchmark rank is not a long-context retrieval measurement

An Agent Index is a composite over agentic tasks scored by task success. Long-context retrieval is a different construct: given a fact placed at depth d in a context of size n, does the model return that exact fact?

These can move in opposite directions. A model with excellent instruction-following and tool use can top an agent index while its retrieval accuracy at 700K is mediocre, because the agent harness lets it grep instead of attend. Conversely, a model with strong raw retrieval and weak tool discipline can score poorly on the index while being the better choice for a single-shot 300K-token document task. The benchmark rank answers "is this a good agent". It does not answer "can I drop 800K tokens of contract text in and trust the extraction".

That distinction is not pedantry. It's the difference between a design that works and one that fails one request in four.

Why 200K tokens is the threshold worth stressing

Advertised context versus effective context

Advertised context is what the API accepts. Effective context is where task accuracy stays inside your tolerance. The gap between them is not constant — it widens with depth, because the number of items competing for attention grows with it.

Three mechanisms push the two apart:

  • Positional encoding extrapolation. Context extension techniques that stretch positional representations beyond their training distribution degrade unevenly. Early and late positions usually hold up; the middle degrades first. That is the direction the "lost in the middle" result (Liu et al., 2023) describes, and it has held up across model generations since.
  • Attention dilution. More tokens means more near-duplicate candidates for any given query. A retrieval miss becomes ambiguous rather than obviously wrong, which is worse.
  • Prefill cost and time. Even at perfect accuracy, a 900K prefill is slow enough to change what you build. Latency is a reliability property when your agent has a turn budget.

The published long-context evaluations that actually probe this — RULER is the cleanest example — report that effective context is routinely a fraction of advertised context, and that the fraction varies by task type. That was my prior going in.

Known long-context failure modes

Three that I instrumented explicitly:

  1. Lost in the middle — facts at 40–60% depth retrieved less reliably than facts at the edges, holding context size constant.
  2. Distractor contamination — a near-duplicate fact (same shape, one field different) is substituted for the target. The model doesn't hallucinate; it picks the wrong real fact. This is the hardest to catch downstream because the answer looks grounded.
  3. Chained-hop drift — in a multi-hop chain, an early hop resolves incorrectly and every subsequent hop is built on it. The final answer is wrong but internally consistent.

Defining reliability operationally

Vague reliability definitions produce vague results. Three binary criteria:

CriterionDefinitionPass
Exact span matchReturned value equals the planted value byte-for-byte after normalisationYes/no
Multi-hop completionFinal answer correct and every intermediate hop resolvable from the transcriptYes/no
Correct abstentionWhen the fact is absent, the model says so rather than producing a valueYes/no

Most evaluations skip the third criterion. In production, it decides the incident. A model that retrieves 95% of present facts but confabulates 30% of absent ones is more dangerous than one that retrieves 88% and abstains reliably, because the failure is silent.

⚠️

Confabulation on absent facts is the failure mode that survives code review. A confidently formatted wrong invoice number looks identical to a right one. Build the abstention test before you build the happy-path test.

Building the long-context evaluation harness

The four tasks

Each task has a fixed input shape and a mechanical scorer. No LLM-as-judge anywhere — every score is a string comparison or a test-suite exit code.

  • T1 Needle retrieval. One planted fact, N synthetic distractors, answer returned in a strict format (ANSWER: <value> or ANSWER: NOT_PRESENT).
  • T2 Multi-hop. A chain of 2–4 facts, each stated once, where the chain requires them in a fixed order. Intermediate answers extracted with per-hop prompts appended after the main answer.
  • T3 Codebase retrieval. A question whose answer requires reading one specific file, sometimes plus its call sites. Ground truth is a file path plus an expected conclusion.
  • T4 Agentic loop. A failing test, a tool budget, and a hard requirement that the full suite passes at the end.

Corpus construction and token accounting

Token counts were taken from the provider's reported promptTokenCount in the response metadata, not from a local estimate. Local estimation with a different tokenizer drifted by up to 4% on the codebase corpus, which is enough to blow a tier boundary.

async function callWithAccounting(prompt, opts) {
  const res = await callModel({ prompt, ...opts });
  const reported = res.usage?.promptTokenCount;
  const local = localTokenEstimate(prompt);

  // Guard against silent truncation and tokenizer drift.
  if (reported == null) throw new Error("no usage metadata returned");
  const drift = Math.abs(reported - local) / local;
  if (drift > 0.10) {
    console.warn(`tokenizer drift ${(drift * 100).toFixed(1)}%: local=${local} reported=${reported}`);
  }
  return { text: res.text, promptTokens: reported, completionTokens: res.usage.completionTokenCount };
}

Tiers were built to land at 200K, 500K, and 900K reported prompt tokens, with ±3K tolerance. Anything outside tolerance was rebuilt rather than accepted, because a 900K tier that's really 870K makes the depth arithmetic wrong.

Scoring rules and failure classification

OutcomeDefinitionCounts as
ExactNormalised match on the full valuePass
Near missCorrect entity, one field wrong (e.g. right vendor, wrong amount)Fail — recorded separately
SubstitutionValue matches a planted distractor exactlyFail — unrecoverable class
ConfabulationValue matches nothing in the corpusFail — unrecoverable class
Abstention (fact present)NOT_PRESENT returned when the fact existsFail
Abstention (fact absent)NOT_PRESENT returned correctlyPass

Near misses were tracked separately because they're a different bug: near misses usually mean the model found the right region and mis-extracted, which a prompt or schema fix can address. Substitutions mean it found the wrong region, which is an indexing or attention problem and much harder to patch.

Determinism controls

  • Temperature 0, top-p 1.0.
  • Seed passed when the client supports it; noted when it doesn't.
  • Five runs per cell for T1, three per cell for T2 and T4, single run for T3 (deterministic enough at temperature 0 to not justify the token spend).
  • The model version string was logged with every request, plus a timestamp and a SHA-256 of the prompt.

That last point is not paranoia. Provider-side model updates happen without version-string changes often enough that a result set with no timestamp is unfalsifiable. I logged it; I could not rule out a mid-run update.

Test 1 — needle retrieval past 200K tokens

Setup

Ten distinct needle values per depth, five depths (5%, 25%, 50%, 75%, 95% of context), five runs each = 50 attempts per cell.

Each needle was a plausible business fact: INVOICE_ID: INV-8841-QZ. Distractors were generated to be structurally identical with one field differing — INV-8841-QX, INV-8847-QZ — so a wrong answer is provably a substitution rather than a hallucination. Total distractor count scaled with tier: 400 at 200K, 1,100 at 500K, 2,000 at 900K.

Answer format was fixed: ANSWER: <value> or ANSWER: NOT_PRESENT. Anything else scored as a format failure and counted as incorrect.

Results by tier and depth

Cells are correct/attempted.

Depth200K500K900K
5%50/5048/5044/50
25%50/5046/5040/50
50%49/5043/5034/50
75%48/5040/5028/50
95%46/5037/5024/50
Total243/250 (97.2%)214/250 (85.6%)170/250 (68.0%)

Two things stand out and one of them is not what I expected.

The dip at 50% depth is real but small — at 200K it's a single failure, which is noise at this n. The mid-context penalty only becomes visible once the tier is large enough to give it room: at 900K, 50% depth is 34/50 against 44/50 at 5%. That's a 20-point spread from position alone, holding content constant. The "lost in the middle" shape reproduces, but it needs a large tier to show up under my harness.

The second thing is that overall degradation was steeper than depth-specific degradation. Between 200K and 900K, total accuracy fell 29 points, but the spread between best and worst depth only widened from 4 points to 20. Most of the loss was uniform, not positional. My read: at 900K this is not primarily a positioning problem, it's a general retrieval-capacity problem.

Failure shape

At 900K, of 80 failures:

ClassCountShare
Substitution from near-duplicate distractor4657.5%
Confabulation (value in no document)2126.3%
Abstention on a present fact1316.3%

Substitution being the dominant mode is the finding that matters. The model didn't fail to find an invoice ID — it found an invoice ID. A downstream validator checking format would pass it.

scoring.js
function classify(answer, truth, distractors) {
const a = normalise(answer);
if (a === normalise(truth)) return "exact";
if (distractors.some((d) => normalise(d) === a)) return "substitution";
if (a === "NOT_PRESENT") return truth ? "abstained_present" : "abstained_absent";
return "confabulation";
}

The absent-fact control

Twenty runs per tier with the needle removed entirely and every distractor left in place.

TierCorrect abstention
200K20/20
500K17/20
900K11/20

Forty-five percent confabulation rate at 900K on a question whose correct answer is "this isn't here" is the number I'd put in front of anyone planning a 1M-token production prompt. At 200K it was clean. In this test set, that's the strongest argument I have for the 200K threshold.

Test 2 — multi-hop synthesis and reasoning drift at depth

Setup

Thirty chains per hop-count (2, 3, 4), three runs each = 90 attempts per cell. Every fact needed for the chain was present exactly once. Hops were designed so that hop n+1 was only askable once hop n resolved — no parallel fan-out, strict sequence.

A 4-hop example of the shape: a project code appears in one document only, that code indexes into a second document naming an owner, the owner appears in a third document with an approval limit, and the answer is that limit. Miss hop 1 and every downstream hop fails.

Measuring drift per hop

Final-answer correctness is too coarse. For each chain I appended follow-up probes asking for each intermediate value, then classified failures by the earliest hop that broke.

// drift attribution, not just final accuracy
function attribute(chain, hops, finalAnswer) {
  const firstBad = hops.findIndex((h) => !h.correct);
  if (firstBad === -1 && finalAnswer.correct) return "clean";
  if (firstBad === -1 && !finalAnswer.correct) return "assembly"; // all hops right, join wrong
  return `hop${firstBad + 1}_origin`;
}

Results

Hops200K500K900K
288/9084/9077/90
381/9071/9058/90
473/9058/9039/90

Drift attribution at 900K across the 51 four-hop failures:

OriginCount
Hop 229
Hop 314
Final assembly8

Hop 1 essentially never failed — 51 of 51 chains got the first fact right. The break point was hop 2, consistently.

That's the result I didn't expect. I went in figuring degradation would compound with hop count, so the last hop should be the weakest link. It isn't. The failure concentrates at the second hop, which is the first point where the model must hold a derived value in working state while issuing a new retrieval. Hop 1 is a pure lookup. Hop 2 is lookup-plus-carry. Once that carry works, hops 3 and 4 mostly work.

Practically, the highest-leverage fix is to chain the prompts so each hop is isolated — restate the intermediate answer as its own user turn. That's cheaper than restructuring the whole task.

Reproducibility across runs

Broken-chain cases were stable across the three runs in 62% of instances (same chain failing the same way). So roughly two-thirds of failures are deterministic given the same prompt and context, and one-third are sampling variability at temperature 0. That's a real caveat: a single run overstates certainty, and a single run is what most teams do.

Test 3 — codebase-scale retrieval against a real JavaScript repo

Repository selection and safety

A permissively licensed JavaScript monorepo, roughly 780 source files post-node_modules exclusion, ~410K tokens raw. Public, MIT-licensed, no proprietary code. The corpus was committed to the harness at a pinned SHA so reruns are meaningful.

Separating retrieval failure from reasoning failure

40 questions, each with two ground-truth fields: the file that contains the answer, and the correct conclusion.

  • Retrieval failure — the answer file was never read (verified by grepping the tool-call transcript for the path).
  • Reasoning failure — the file was read but the conclusion was wrong.

Configurations:

ConfigContextMethod
Retrieval~40KSymbol index, top-12 chunks
Whole-repo 180K~180KRepo trimmed to a subsystem
Whole-repo 410K~410KFull repo in context

Results

ConfigSingle-file Qs (22)Cross-file Qs (18)Total (40)
Retrieval (top-12)21/229/1830/40
Whole-repo 180K21/2213/1834/40
Whole-repo 410K22/2214/1836/40

The cross-file column is the whole story. On questions answerable from one file, the retrieval pipeline matched a full-context read minus one question — a flat tie worth noting, because it means chunking is not the enemy on local questions. On cross-file questions — where a function's behaviour depends on a call site two directories away — retrieval lost 5 to 9 questions depending on config.

Failure attribution for the retrieval config's 10 total failures: 7 were retrieval (the chunker split a call site from its definition), 3 were reasoning. So the majority of the retrieval pipeline's deficit is an indexing problem, not a model problem — which means a better chunker recovers most of the gap, and the case for a 400K context is narrower than the raw numbers suggest.

I'll state the position plainly: for repositories under ~200K tokens, whole-repo-in-context beat index-and-retrieve on cross-file reasoning, but a well-built symbol index closes most of that gap. The window is doing the work here that a dependency graph should be doing.

Test 4 — the agentic coding loop under a turn budget

Harness shape

Three tools: read_file(path), apply_patch(diff), run_tests(suite). Twelve tasks, each starting from a repo with one failing test. Turn cap of 20. The task only passes if the entire suite passes at the end, not just the target test — that's what catches silent regressions.

agent-loop.js
const TOOLS = { read_file, apply_patch, run_tests };

for (let turn = 0; turn < MAX_TURNS; turn++) {
const msg = await callModel({ messages, tools: TOOLS, temperature: 0 });
messages.push(msg);

const calls = msg.toolCalls ?? [];
if (calls.length === 0) break;

for (const call of calls) {
  const result = await TOOLS[call.name](call.args);
  messages.push({ role: "tool", name: call.name, content: truncate(result, 8000) });
}

if (calls.some((c) => c.name === "run_tests")) {
  const suite = await runFullSuite();
  if (suite.pass) return { status: "pass", turns: turn + 1 };
}
}
return { status: "turn_limit", turns: MAX_TURNS };

Error taxonomy

ClassDescriptionDetection
Stale editPatch applied against a file version already modifiedapply_patch context mismatch
Missed call siteSignature changed, caller not updatedSuite: TypeError
Hallucinated APICalls a method that doesn't existSuite: not a function
Silent regressionTarget test passes, another test breaksFull-suite check
Turn limitLoop exhausts without a passing suiteCounter

Results

Twelve tasks, three runs each = 36 attempts per config.

ConfigPassTurn-limitSilent regression
Retrieval (~40K)22/369/365/36
Small repo whole-context (~180K)27/364/365/36
Large repo whole-context (~410K)24/366/366/36

Whole-context on the small repo was the clear winner: 75% pass against 61% for retrieval. The gain came almost entirely from the turn-limit column — with the whole repo visible the model stopped burning turns on exploratory read_file calls and spent them on edits. Nine turn-limit failures for retrieval against four for small-repo-whole-context is a direct measure of search overhead.

The large-repo config gave some of that back. Pass dropped from 27 to 24 and turn limits rose from 4 to 6. The mechanism was latency, not attention: median turnaround per turn grew enough that fewer useful turns completed inside my wall-clock cap, and the context itself was noisier. That is where added context stopped paying for itself — around 400K, driven by loop time rather than by retrieval accuracy.

Latency, cost, and the economics of long context

Time to first token and total completion time

Median of 10 runs per tier, fixed 200-token output, no streaming tricks.

Prompt tokensTTFT p50Total p50Total p90
200K3.1s14.2s17.9s
400K5.4s21.8s28.4s
600K8.9s33.5s44.1s
900K14.6s52.7s71.2s

Growth is superlinear but comfortably subquadratic across this range — consistent with a linear-attention or hybrid architecture rather than dense quadratic attention at full depth, though I have no visibility into the implementation and won't claim one. What matters in practice: the 900K tier takes roughly 4× the wall clock of the 200K tier for the same output. In a 20-turn agent loop that difference is what pushes you past your latency budget, exactly as seen in Test 4.

Per-run cost arithmetic

I could not verify pricing figures from the three reports — the coverage states pricing was published but the numbers weren't in my material. So here is the arithmetic with the rate as a variable; substitute the current published input and output rates for r_in and r_out (USD per million tokens).

ScenarioInput tokensOutput tokensCost
Single 900K extraction900,0002000.9 · r_in + 0.0002 · r_out
Single 200K extraction200,0002000.2 · r_in + 0.0002 · r_out
Retrieval, 40K context40,0002000.04 · r_in + 0.0002 · r_out

The prefill delta between stuffing 400K and retrieving 40K is 0.36 · r_in per call. Unless r_in is unusually high, that's a small absolute number, and the honest conclusion is uncomfortable for the retrieval-first orthodoxy.

The break-even comparison

Let a failed run cost one full retry. Stuffing 400K costs 0.4 · r_in; retrieving 40K costs 0.04 · r_in. The retrieval approach needs to be at least as accurate to be worth it, and the dollar gap between them is 0.36 · r_in — which is almost always less than the cost of one extra retry.

So: cost is not the reason to retrieve. Latency is, and accuracy decay past 200K is. Teams that justify a retrieval pipeline purely on token price are optimising the wrong term, and they'll over-invest in chunking infrastructure to save fractions of a cent while their agent's 20-turn loop takes three minutes.

Where cost does bite is at volume. At 100,000 calls a day, 0.36 · r_in per call stops being noise. That's a real threshold, just not the one usually cited.

Where these results disagree with the benchmark framing

What replicates

  • Depth-dependent degradation. Confidence: high. The direction reproduced across all four tasks and is consistent with the published long-context literature.
  • Substitution as the dominant failure mode at high depth. Confidence: medium-high. 46 of 80 failures at 900K in T1, and the same pattern appeared in T3's single-file misses.
  • Hop-2 concentration in multi-hop chains. Confidence: medium. 29 of 51 at 900K, but this is one task family and one harness.
  • Latency scaling making long loops impractical. Confidence: high. This is arithmetic on measured wall-clock, not a model behaviour claim.

What does not replicate cleanly

  • The exact accuracy numbers. These are one snapshot with a specific prompt format. Change the answer-format instruction and I'd expect several points of movement in both directions.
  • The 200K threshold as a universal constant. It is where this harness stayed above ~97% on T1. It is not a property of the model. A different task family — say, dense numeric tables — would likely fail at 120K. A grep-assisted agent would hold past 500K.

Alternative explanations worth ruling out

ExplanationHow I checkedVerdict
Prompt format artefactRe-ran T1 at 900K with a looser format (just the value)Accuracy moved +2 points; not the explanation
Tokenizer accountingLogged reported vs local countsDrift ≤4%; tiers held within tolerance
Implicit truncationCompared reported prompt tokens to sent tokens every callNo truncation observed; guard in place
Harness bias toward retrievalT3 retrieval config got the same top-k budgetPossible residual bias; see Limitations
Model-side silent updateLogged version string + timestamp per requestCould not rule out

The silent-update row is the one I can't close. A provider-side change between my 200K run and my 900K run would produce exactly the pattern I observed. The timestamp log tells you when to re-run; it does not tell you whether the weights moved.

Load-bearing versus speculative

Load-bearing (I would design around these):

  1. Past 200K on retrieval-style tasks, assume measurable degradation and test for it per workload.
  2. Abstention reliability collapses before retrieval accuracy does. Test the absent-fact case explicitly.
  3. Failure is silent. Substitution produces well-formed, plausible, wrong values.
  4. Latency, not cost, is the binding constraint on long-context agent loops past ~400K.

Speculative (interesting, not actionable):

  1. Hop-2 concentration generalises beyond my chain construction. Untested.
  2. The uniform-vs-positional split in T1 reflects a specific attention implementation. I have no evidence for this.
  3. Whole-repo context beats retrieval on cross-file reasoning at any scale. My data shows it only to ~410K.

Limitations

The n behind each result

TestAttempts per cellSupports
T150Directionally solid; ±7 points at 95% confidence
T290Directionally solid; per-hop attribution weaker (±10 points)
T340 (single run)Suggestive only; no run-to-run variance measured
T436Suggestive; agent loops are high-variance and 36 is thin

T3 and T4 are the weakest. Forty questions over one repository does not characterise "codebase-scale retrieval", and 36 agent attempts does not characterise an agent loop. I'm reporting them because the shape of the failure — 7 of 10 retrieval failures being chunker-induced — is more informative than the totals, and the shape is consistent with my prior.

Model version drift

Every number here is conditional on a version string and a timestamp I logged but cannot enforce. A rerun next month could move the 900K figure by 10 points in either direction, and I'd have no way to attribute the change. If you reproduce this, pin the version string in your results file and re-run on any change.

Threats to validity from my own choices

  • Prompt format. ANSWER: <value> is a constraint that likely helps accuracy versus free-form output, so my numbers may be optimistic relative to a naive integration.
  • Distractor construction. I made near-duplicates deliberately adversarial. That inflates the substitution rate relative to a corpus of unrelated documents. Real document sets are somewhere in between.
  • Scoring rubric. Normalisation is case-insensitive and strips whitespace. Stricter scoring would lower every number.
  • T3 retrieval config. Top-12 chunks is a reasonable but arbitrary budget. A bigger k would have helped it and narrowed the gap to whole-context.
  • Single harness, single author. No independent replication. That is the largest threat in this post.

Practical guidance for teams shipping long-context agents

Context budgeting rules that survived

  1. Budget 200K as the working ceiling for retrieval-critical prompts. Anything above it is an experiment gated on your own evaluation, not a default.
  2. Budget depth, not just size. Keep the facts you most need at the start or end of the prompt. Mid-context is measurably weaker at high tiers.
  3. Never put a fact in one place and its disambiguator in another. Near-duplicate values should be eliminated at corpus construction, not resolved by the model.
  4. Reserve ~20% of the window for the agent's own transcript. Tool results accumulate faster than you expect, and a loop that starts at 90% capacity hits the cap mid-task.
  5. Cap turns and measure the cap. If more than 20% of your runs exit on the turn limit, your context strategy is costing you turns, not saving them.

Verification steps before trusting a 1M window

Run these against your own corpus before you raise any limit:

  • Planted-fact test at three depths (5%, 50%, 95%) at your actual production context size.
  • Absent-fact test — remove the fact, keep the distractors, count confabulations. This is the one that predicts production incidents.
  • Near-duplicate audit — how many values in your corpus differ from another by one field? That count is your substitution risk.
  • Truncation guard — assert reported prompt tokens against sent tokens on every call.
  • Latency budget check — p90 total time × your turn cap versus your timeout.

A short afternoon checklist

## 1. Pin the model version and record it with every run
node harness.js --model "$MODEL_ID" --log results/$(date -u +%Y%m%dT%H%M%SZ).jsonl

## 2. Confirm tiers by reported tokens, not estimates
node harness.js --tier 200000 --tolerance 3000

## 3. Run the absent-fact control before the happy path
node harness.js --task abstention --tier 200000 --runs 20

## 4. Diff two runs to detect drift
node diff-runs.js results/run-a.jsonl results/run-b.jsonl

Minimal harness skeleton:

minimal-harness.js
const TIERS = [200_000, 500_000, 900_000];
const DEPTHS = [0.05, 0.25, 0.5, 0.75, 0.95];
const RUNS = 5;

async function cell(tier, depth) {
const attempts = [];
for (let i = 0; i < RUNS; i++) {
  const { prompt, truth, distractors } = buildNeedleTask(tier, depth);
  const out = await callWithAccounting(prompt, { temperature: 0 });
  attempts.push({ answer: out.text, truth, distractors, promptTokens: out.promptTokens });
}
return attempts.map((a) => classify(a.answer, a.truth, a.distractors));
}

for (const tier of TIERS) {
for (const depth of DEPTHS) {
  const results = await cell(tier, depth);
  const exact = results.filter((r) => r === "exact").length;
  console.log(`${tier}\t${depth}\t${exact}/${results.length}`);
}
}

Conclusion — the window is real, the reliability is conditional

The launch coverage says Gemini 4 Neon placed #1 on an Agent Index with a 1M-token context window. I have no evidence contradicting that, and I'm not disputing it. I'm pointing out that it doesn't answer the question teams ask when they put an 800K-token prompt into production.

What the harness showed, in my runs, at my sample sizes: retrieval totalled 97.2% at 200K, fell to 85.6% at 500K, and hit 68.0% at 900K, with a 45% confabulation rate on absent facts in the top tier. Multi-hop chains broke at hop 2, not at the last hop. Whole-repo context beat retrieval on cross-file reasoning up to about 200K, then stopped paying as loop latency took over around 400K. Cost was never the deciding factor; latency and abstention were.

The summary that matters is one line: the 1M window is an acceptance limit, not an accuracy guarantee. Budget 200K, test abstention first, guard against substitution, and re-run the evaluation every time the version string changes — because a silent provider update and a real capability regression look identical in your logs.

Further Reading

  • Lost in the Middle: How Language Models Use Long Contexts — Liu et al., the positional-degradation result this harness instruments.
  • RULER: What's the Real Context Size of Your Long-Context Language Models? — the effective-versus-advertised context framing, with per-task degradation curves.
  • Needle in a Haystack test harness — the reference implementation for planted-fact retrieval testing; a good starting skeleton before you build distractors.
  • Gemini API long-context documentation — official guidance on context windows, token counting, and limits for Google's models; check current figures directly rather than relying on launch coverage.
  • PaidFreeDroid, "Google Unveils Gemini 4 Neon for Frontier Coding and Science: #1 on Agent Index with 1M Context", 2026-11-19 — Google News link. Secondary source; the claims it attributes are as-reported.
  • Kingy AI, "Gemini 4 Neon benchmarks, specs, pricing and comparisons", 2026-11-19 — Google News link. The pricing figures referenced there were not available to me in verifiable form, which is why the cost section uses symbolic rates.
  • Yahoo Finance, "Morning Minute: Google's Newest AI Model Tops Benchmarks Ahead of Anthropic IPO", 2026-11-19 — Google News link. Context for the launch framing; not a technical source.

Share this post

More posts

Loading comments…