What Private Datasets Hide: Auditing Intelligence Index v5 Model Rankings

Introduction: Reading Intelligence Index v5 as a Hypothesis, Not Evidence

On 2026-10-28, OfficeChai reported that Artificial Analysis published Intelligence Index v5, that the new version runs on majority private datasets, and that the ordering moved: OpenAI models gained ground while Muse Wonder and Kimi dropped. That is the extent of what the public report in front of me establishes. No per-model score deltas, no item counts, no provenance for the private sets. I am not going to fill those in with guesses.

This post covers what a majority-private index can and cannot tell you, what the shift costs you in reproducibility, and how to audit a v5 model ranking against your own workload in a single afternoon.

My position: a majority-private index is a defensible product decision and a worse instrument for you. Public AI benchmarks had one measurable, embarrassing failure mode — contamination. Private sets remove your ability to observe that failure mode without removing the failure mode itself. In exchange you get a number whose error bars you cannot see, produced from a set you cannot inspect, scored against a rubric you may only partly know.

The useful response is neither to discard the ranking nor to trust it. Demote it from evidence to hypothesis, then spend an afternoon converting that hypothesis into a measurement you control.

What Intelligence Index v5 Actually Changed

From Public Benchmark Aggregation to Majority-Private Sets

The v4-era pattern is familiar. Pull MMLU, GPQA, HumanEval, AIME, some coding-agent benchmark; normalise each to 0–100; weight; sum. Every component is public, so every component is also in somebody's pretraining corpus and every component can be re-run by anyone with a GPU budget.

The report describes v5 as shifting the balance so most contributing evaluations are private. "Majority private" is a claim about the composition of the item pool — not about the scoring code or the aggregation weights. Those may still be public. The summary does not say, and I have not verified it.

The Reported Ranking Moves: OpenAI Gains, Muse Wonder and Kimi Drop

What the report states is directional:

• OpenAI models gained position under v5.
• Muse Wonder dropped.
• Kimi dropped.

What it does not state, in the summary I have: which specific models, by how many points, which sub-evaluations absorbed the movement, or whether both index versions scored the same model snapshots. All four are load-bearing if you are deciding whether to migrate a production workload. A model that drops two points may have dropped because one sub-benchmark was re-weighted, or because the private set leans into a domain it is weak in. Those are different facts wearing the same headline.

What the OfficeChai Report Does and Does Not State

Separating these now saves confusion later:

What "Private Dataset" Means Mechanically

Held-Out, Licensed, and Vendor-Supplied Are Three Different Things

"Private" describes access control, not methodology. Three common cases, three very different risk profiles.

Held-out. The evaluator wrote or curated the items and never published them. Contamination risk is low, since the items were not public before the run. The real risks are curation bias — whoever wrote the items picked problems they find interesting — and drift, where a set frozen in 2025 is used to rank models from 2027.

Licensed. Items bought or licensed from a third party: exam banks, ticketing corpora, code review comments. Access is controlled; provenance is opaque. A licensed pool can easily be text that appeared on the public web at some point, so the fact that you cannot download it says nothing about whether a model's pretraining data contained it.

Vendor-supplied. The lab whose model is being scored contributed items, funding, or review. Highest risk, hardest to detect from outside, because "we partnered with providers on domain-specific evaluations" reads like a strength in a press release.

The report says "majority private datasets" and stops. The distinction above is what I would want before treating any delta as meaningful, and right now it is unknown to me.

Private Does Not Automatically Mean Contamination-Free

Contamination is not a property of access control. It is a property of the relationship between the item pool and the training corpus. A private set is contaminated if:

• it was scraped from public web text, then access-restricted afterwards;
• it was generated by a model whose outputs were conditioned on a public benchmark;
• it is a canary split of a public benchmark — same distribution, different instances — in which case style-level overfitting transfers even though memorisation does not;
• a vendor saw the items before the model shipped.

None of those are detectable from outside, and only the last is detectable by the evaluator if they bother to ask. Honest framing: private sets reduce the probability of the crudest contamination attack (train on the literal test items) and hide the evidence for the subtler ones.

Why Majority-Private Is a Spectrum, Not a Switch

"We are majority private" is compatible with a wide range of realities:

• 51% of items private, 49% public — the aggregate is still substantially driven by benchmarks anyone can contaminate.
• 90% private, but the visible public slice is what people anchor on when they quote the number.
• 90% private with confidential item counts per category, so you cannot tell whether a category delta rests on 400 items or 12.

Weighting matters as much as composition. If a private sub-evaluation carries 25% of the aggregate and holds 50 items, sampling error on that component swamps most of the reported movement. Without published counts and weights you cannot compute it. That is the actual loss.

The Reproducibility Budget: What a Private Index Costs You

What You Can Still Verify From Outside the Index

Plenty survives, and it is worth being precise, because teams tend to throw out the whole report once they learn the items are private.

• Directional movement. Three sources of movement — dataset composition, model change, prompting/harness change — all produce directional signal. As a prior for what to test, direction is still useful.
• Model identity and version strings. If you can identify the exact snapshots scored, you can check whether their behaviour changed between your runs, independent of any index.
• Cost and latency. Measurable on your own traffic. If the index says a model moved up while its price per million output tokens tripled, that is a fact you can confirm directly.
• Cross-index agreement. Compare v5 against a different methodology — arena-style pairwise preference, an open-source harness you run yourself. Agreement across independent methodologies is weak but real evidence.
• Sub-evaluation ordering. If category breakdowns are published, "model A beats model B on coding but loses on long context" is testable even when the items are not.

What You Can No Longer Verify At All

• Whether the items are correct. Label noise on a private set is invisible, and a 3% label error rate can move an aggregate further than most model deltas.
• Whether the rubric rewards what you care about. Exact-match on a normalised string is a different measurement from an LLM judge on a 1–5 scale.
• Whether run-to-run variance was reported. Single-sample evaluation of a stochastic model is a coin flip dressed as a measurement.
• Whether a category delta rests on enough items to be non-noise.
• Whether any ranked model saw the items during training.

Mapping Index Claims to Independent Verification Options

Benchmark Gaming Under Private Evaluations

The Old Attack Surface: Public-Set Contamination

Public benchmarks had a known exploit: put the test set in pretraining, or tune on it. Cheap, deniable, effective. The defence the community settled on was item retirement — rotate items as they leak — which is expensive and eventually exhausts the pool.

The New Attack Surface: Distribution Mismatch and Vendor Proximity

Hiding the items kills only the most blatant version of the attack. Three others survive, and two get easier.

Distribution mismatch. A private set encodes someone's choice of domain. If the evaluator's items skew toward, say, multi-step arithmetic with clean formatting, models tuned for that style win the aggregate while being no better on your messy real inputs. This is not the lab cheating; it is a systematic mismatch between their distribution and yours. It existed with public sets too — but public sets could at least be inspected, so you could measure the mismatch.

Vendor proximity. When labs contribute items or feedback, the loop between "what gets measured" and "what gets optimised" shortens. No bad intent required: an evaluation shaped with a lab's input tends to reward that lab's house style.

Rubric gaming. If the rubric is public and the items are not — the likely split, since rubrics are easier to publish without leaking items — you can still optimise for it: verbosity, format conformance, hedging patterns, refusal calibration. Those transfer to unseen items in the same style.

Why Hidden Sets Shift Gaming Rather Than Remove It

The invariant: any optimisation target that is visible gets optimised. Publish a rubric and hide the items, and the rubric becomes the target. Hide both and the evaluator's judgement becomes the target, which is worse — unobservable and unaccountable.

The strongest version of a private index is not "you cannot see the items." It is: items private, collection methodology and rubric public, item counts and run counts public, and a committed rotation schedule. That combination is auditable at the level that matters without disclosing a single item.

Auditing a Leaderboard Delta With Your Own Harness

The whole point of doing this yourself is that the index answers "how did these models rank on someone's items" while you need "which model should I ship." Here is the process I use.

Step 1 — Pin the Index Version and Record the Date

Write down what you are responding to before anything else: index name, version, the date you read it, the exact model identifiers named. Leaderboards are living documents. A URL is not a citation unless it is versioned or archived.

Step 2 — Define a Task-Specific Eval Set That Matches Your Workload

Pull 100–300 real inputs from your own logs, redacted, stratified so rare classes appear at a rate you care about rather than the rate they occur. Label once, carefully, then freeze. If you cannot get 100, take 50 and accept that you will only detect large effects — see the interval arithmetic below.

The failure mode here is building a set that is easy rather than representative. Sample from production traffic, not from your test suite.

Step 3 — Run the Same Prompts Across Models and Persist Raw Outputs

Same prompt template, same decode parameters, same case order, every model. Persist the raw output string, the model string the API returned, the timestamp, and a hash of the prompt. Do not score inline. Store first, score as a separate step, so you can rescore without re-spending tokens.

Step 4 — Score Deterministically and Show the Confusion, Not the Average

An aggregate accuracy number hides exactly the information you need. Report per-class recall and a confusion matrix, and bucket the failures: wrong label, refused, format violation, truncated. A model at 90% accuracy that refuses 8% of inputs is a different product decision from one at 90% with a flat error spread.

For paired comparisons — same cases, two models — do not compare two independent accuracies. Run McNemar's test on the discordant pairs. At n=200 and a two-point gap, the paired test is often the difference between "signal" and "noise," and the unpaired one almost never is.

A Dependency-Free Node.js Harness for Repeatable Runs

Deliberately dependency-free: Node 20 built-ins only, with a stub adapter so you can verify the scoring path before spending a cent on tokens.

import { createHash } from "node:crypto";


const INDEX = {
name: "artificial-analysis-intelligence-index",
version: "v5",
observedOn: "2026-10-28",
};

// Decode params must be identical across models or the comparison is void.
const DECODE = { temperature: 0, maxTokens: 64 };

const CASES = [
{ id: "c1", input: "I was charged twice this month", expected: "billing" },
{ id: "c2", input: "The export button throws a 500", expected: "bug" },
{ id: "c3", input: "How do I reset my API key?", expected: "howto" },
{ id: "c4", input: "Where is my invoice for March?", expected: "billing" },
];

// Stub adapter: hard-coded answers so the scoring path can be tested
// without network calls. c4 is deliberately wrong so the confusion
// matrix is non-empty.
const STUB = { c1: "billing", c2: "bug", c3: "howto", c4: "howto" };

const ADAPTERS = {
stub: async (c) => STUB[c.id] ?? "unknown",
// openai: async (c) => (await client.chat.completions.create({...})).choices[0].message.content,
// kimi:   async (c) => (await otherClient.chat.completions.create({...})).choices[0].message.content,
};

const norm = (s) => s.trim().toLowerCase().replace(/[^a-z]/g, "");

function summarize(model, rows) {
const n = rows.length;
const correct = rows.filter((r) => norm(r.output) === r.expected).length;
const confusions = new Map();
for (const r of rows) {
  const got = norm(r.output);
  if (got === r.expected) continue;
  const key = `${r.expected} -> ${got}`;
  confusions.set(key, (confusions.get(key) ?? 0) + 1);
}
const worst = [...confusions.entries()].sort((a, b) => b[1] - a[1])[0];
return {
  model,
  n,
  correct,
  accuracy: Number((correct / n).toFixed(3)),
  worst: worst ? `${worst[0]} (${worst[1]})` : "-",
};
}

mkdirSync("runs", { recursive: true });
const caseHash = createHash("sha256")
.update(JSON.stringify(CASES))
.digest("hex")
.slice(0, 8);
const runId = `${INDEX.version}-${new Date().toISOString().slice(0, 10)}-${caseHash}`;

console.log(`run ${runId}`);
console.log(`index ${INDEX.name} ${INDEX.version} recorded ${INDEX.observedOn}`);
console.log(`cases ${CASES.length} decode ${JSON.stringify(DECODE)}\n`);

const results = [];
for (const [model, call] of Object.entries(ADAPTERS)) {
const rows = [];
for (const c of CASES) {
  const output = await call(c);
  rows.push({ runId, model, caseId: c.id, expected: c.expected, output });
  appendFileSync(
    "runs/results.jsonl",
    JSON.stringify({ runId, model, caseId: c.id, expected: c.expected, output, ...DECODE }) + "\n"
  );
}
results.push(summarize(model, rows));
}

const cols = ["model", "n", "correct", "accuracy", "worst"];
console.log(cols.map((c) => c.padEnd(10)).join(""));
for (const r of results) {
console.log([r.model, r.n, r.correct, r.accuracy, r.worst].map((v) => String(v).padEnd(10)).join(""));
}

Self-test output from the stub adapter (one deliberate mismatch at c4):

run v5-2026-10-29-<caseHash>
index artificial-analysis-intelligence-index v5 recorded 2026-10-28
cases 4 decode {"temperature":0,"maxTokens":64}

model     n         correct   accuracy  worst
stub      4         3         0.75      billing -> howto (1)

The run id and the eight-character case hash depend on the date you run it and on the fixture contents, so yours will differ. Everything from the n column rightwards is deterministic given the fixtures — exactly the property you want before pointing the harness at a paid API. Confirm the scoring path with the stub, then swap in real adapters and re-run.

A Worked Example: Does the v5 Delta Apply to My Task?

Precision matters here, because it is easy to dress a stub test up as a measurement. I ran the harness above with the stub adapter. I have not run a 200-case comparison of the v5 movers on real traffic, and I will not print numbers I did not measure. What follows is the design and the arithmetic, which is the part that transfers.

Task, Inputs, Criterion, and Count

Task: route inbound support messages into 5 queues
       (billing, bug, howto, refund, account).
Inputs: 200 redacted production messages, stratified so refund and
        account each appear at 15% instead of their 3% live rate.
Criterion: exact label match after lowercasing and stripping punctuation.
           "refused" and "format violation" counted as errors, tracked separately.
Count: 3 models x 200 cases x 1 decode at temperature 0.
       Estimated cost: negligible at 64 output tokens per call.

Observed Results and What They Do Not Prove

The only output I have observed is the stub self-test: 3 of 4 correct, one confusion, billing -> howto. That proves the harness scores, persists, and surfaces confusions. It proves nothing about any model.

When you do run it, here is what the output does and does not support. Suppose model A lands at 184/200 and model B at 180/200. That does not establish A is better. It establishes that on 200 stratified cases with an exact-match criterion at temperature 0, A was correct on four more items. To claim A is better you need the paired breakdown: how many cases did each model get right that the other got wrong? If the discordant pairs split 9 and 5, McNemar at n=14 discordant will not hand you a small p-value. The honest conclusion is "indistinguishable at this sample size."

Where My Sample Size Is Too Small to Conclude Anything

Two pieces of arithmetic worth carrying around:

• n=200 at an observed 92% accuracy gives a standard error of ≈1.9 points, so a 95% interval of roughly ±3.8 points. A two-point leaderboard delta is inside that interval.
• n=50 at an observed 82% gives a standard error of ≈5.4 points, so roughly ±10.6 points. At that size you can detect a disaster, not a ranking change.

Practical rule: if the reported delta between two models is smaller than about four points, a 200-case task-specific set will not settle it. You need paired analysis on a few hundred cases and a delta larger than the interval — or you need to accept that you are choosing on cost and latency, not quality.

Practical Decision Rules for Model Selection

When a Leaderboard Move Should Change Your Choice

• The delta is large relative to your own set's interval, and your set agrees in direction.
• The movement is concentrated in the category you actually use, not the aggregate.
• The model string you are being asked to adopt is unchanged but its behaviour measurably differs (provider-side update), and your regression set catches it.
• The move survives a second, independent methodology — an open harness you run yourself, or arena-style preference data.

When a Leaderboard Move Should Not Change Your Choice

• The delta is a couple of points on an aggregate you cannot decompose.
• The index changed dataset composition and model versions in the same release, so the delta is confounded.
• The category driving the move has an unpublished item count.
• Your workload's inputs look nothing like the evaluator's domain. A benchmark that rewards clean formatted inputs says little about your 400-token support tickets full of typos.

Cheap Guardrails: Version Pinning, Canary Prompts, and Regression Runs

Three things that cost almost nothing and beat any leaderboard:

1. Pin model versions explicitly. Never depend on a floating alias. If the provider silently rotates the snapshot behind an alias, your evaluation history is worthless.
2. Keep 20 canary prompts with known-good outputs and run them on every deploy and every provider config change. Twenty cases will not rank models, but they catch "the provider changed the default temperature" and "output format broke" immediately.
3. Run the full regression set on a schedule, not on demand. Weekly is enough. Store results as JSONL so you can diff last week's failures against this week's.

What a Private-Set Index Should Publish to Be Auditable

If I were writing the disclosure contract for a majority-private index, this is the minimum that would let me use the number without re-deriving everything.

Dataset Provenance and Collection Dates

For each contributing evaluation: held-out, licensed, or vendor-supplied; when the items were collected; whether the pool has been rotated since the last version. Vendor-supplied items should be labelled per model, and any model scored substantially on items its own vendor contributed should carry a marker. That single disclosure would defuse most of the proximity concern.

Scoring Rubric, Sampling Method, and Run Counts

• Exact-match, regex, or LLM-judge — and for judges, which judge model and what prompt.
• Decode parameters per model, including whether provider defaults were overridden.
• Number of independent samples per item, and whether variance across samples is reported.
• Item counts per category and aggregation weights. Without these, no reader can compute whether a category delta is larger than sampling noise.
• Confidence intervals or standard errors on the aggregate.

Known Coverage Gaps and Failure Reporting

A section listing what the index does not measure beats another decimal place on the aggregate. Same for the failure taxonomy: "89.4" is less informative than "89.4, with 4.1% refusals and 2.7% truncations at the token cap," because those two failure modes tell a deployer completely different things.

Confirmed Facts vs. Inference

What the Public Reporting Establishes

• Artificial Analysis launched Intelligence Index v5 (OfficeChai, 2026-10-28).
• The v5 index is described as using majority private datasets.
• The reported ranking movement is OpenAI models up, Muse Wonder and Kimi down.

That is it. Everything else here is either general mechanism or design advice, labelled as such.

What Remains Untested or Unknown

• The magnitude of any delta. Unknown to me.
• Whether v4 scores were recomputed on the v5 pool, or whether the two versions are being compared across different datasets and different model snapshots. Unknown — and the single most consequential open question.
• The mix of held-out, licensed, and vendor-supplied items. Unknown.
• Item counts per category, aggregation weights, sampling variance. Unknown.
• The rubric type, and whether an LLM judge was involved. Unknown.
• Whether the private sets are contaminated via public-web provenance or model-generated items. Untestable from outside, possibly untested inside.
• Whether the movement reproduces on my own workload. Untested — that is the entire point of the harness above.

Conclusion

A majority-private index is the predictable end state of public benchmark contamination, and it is not dishonest. It is a different kind of artefact: a well-informed opinion from an evaluator you have to trust, rather than a measurement you can check.

Treat the v5 movement as a prior. OpenAI gaining, Muse Wonder and Kimi dropping is a reason to re-run your own comparison, not a reason to migrate. The cost of finding out is one afternoon, 200 stratified cases from your own logs, a pinned decode configuration, and a scoring step that shows you a confusion matrix instead of a single number.

What I want most from v5 is not more items or a bigger leaderboard. It is item counts, weights, provenance labels, and run counts — the four numbers that would let an outside engineer compute whether a three-point delta means anything at all.

Further Reading

• Artificial Analysis — Intelligence Index — the primary source for the index itself; check the current methodology page for item counts and rubric details as they are published.
• OfficeChai report on Intelligence Index v5 — the source summary supplied for this post, dated 2026-10-28; linked as syndicated.
• EleutherAI lm-evaluation-harness — run the public benchmarks yourself rather than quoting them.
• Stanford CRFM HELM — a contrasting aggregation methodology with published per-scenario breakdowns.
• OpenAI simple-evals — small, readable reference implementations of common eval scoring paths.
• LMArena — pairwise human preference data, methodologically independent of static item pools.
• McNemar's test — the paired test you want for same-cases model comparison; the unpaired comparison overstates significance.