What Private Datasets Hide: Auditing Intelligence Index v5 Rankings
Intelligence Index v5 moved to majority private evaluation sets, and the reported ordering shifted: OpenAI models gained ground while Muse Wonder and Kimi dropped. That is where the public report stops — no per-model score deltas, no item counts, no provenance for the private sets. This video walks through what a majority-private index can and cannot tell you. Private is an access label, not a methodology: held-out items, licensed items and vendor-supplied items carry very different risks, and only the first avoids contamination before the run rather than hiding it afterwards. The pipeline still collects items, holds them back or licenses them, scores each model and aggregates — but only the ranking is published, so label noise, run-to-run variance, item counts per category and whether a ranked model saw the items all stay invisible. The useful response is to demote the ranking from evidence to hypothesis. Record the index version and the date you read it, pin the exact model snapshots, and run those snapshots on tasks drawn from your own workload. Directional movement, cost, latency and a second independent methodology still give you something checkable. Read the full audit checklist, the verification table and the section on what hidden sets do to benchmark gaming at https://hackyjs.com/posts/what-private-datasets-hide-auditing-intelligence-index-v5-model-rankings