Testing Gemini 4 Neon's Long-Context Reliability Past 200K Tokens
Gemini 4 Neon's launch coverage claims a 1M-token context window and a #1 Agent Index placement. Neither answers the question that decides whether you can ship on it: at what depth inside that window does the model still return the right fact, in the right order, without inventing one? This short walks the hands-on reliability test behind the full post: four mechanically scored tasks — needle retrieval with distractors, chained multi-hop synthesis, codebase-scale retrieval, and an agentic edit loop — run at 200K, 500K and 900K reported prompt tokens, five runs per cell, with forty questions against a real open-source JavaScript repository. No LLM-as-judge anywhere; every score is a string comparison or a test-suite exit code. What the write-up covers: - Why an agent benchmark rank is not a long-context retrieval measurement - Advertised context versus effective context, and why the gap widens with depth - Positional encoding extrapolation, attention dilution, and prefill cost as a reliability property - Lost in the middle, distractor substitution, and chained-hop drift - Token accounting from the provider's usage metadata instead of a local estimate - Why correct abstention on absent facts decides the incident - The operational position: treat 200K as the contract, and anything past it as an experiment you re-validate per workload Every number quoted here comes from that harness. The source material states that pricing exists but not the figures, so none are given. Full write-up: https://hackyjs.com/posts/testing-gemini-4-neons-long-context-reliability-past-200k-tokens