Pass@k Is Not Enough: Adding Cost per Solved Problem to a LiveCodeBench Hard Harness

The aggregator headline from 2026-10-25 is blunt: five Qwen3.8-27B runs match Claude Fable 5 on LiveCodeBench Hard at a fraction of the cost. Claims like that get screenshotted into Slack and used to justify a migration, so they deserve the least charitable reading that is still fair.

This post is about what it takes to check that claim properly: a pinned LiveCodeBench Hard harness, multiple seeds, intervals resampled over problems rather than attempts, pass@k scored at a matched attempt budget, and a cost-per-solved-problem number that puts a denominator under the headline. So, up front: a pass@k number without a cost denominator and an interval is not a result. A pass rate is a numerator with a hidden denominator, and "match" hides at least three separate things — attempt budget, grader strictness, variance. If you cannot state cost per solved problem with a confidence interval, you have not measured whether the cheaper model is cheaper. You have measured whether it is cheaper per token, and nobody deploys tokens.

I built a harness to check it. At my utilization level the pilot points the other way from the headline. The code and the numbers I actually got are below.

What the Report Claims, and What It Leaves Out

Strip the framing and the report claims parity on a hard coding benchmark: five runs of an open 27B-class model against one hosted baseline, at materially lower cost. That is as far as the linked page goes. Everything an engineer would need in order to act on it — model identifiers, decoding parameters, seeds, timeout policy, grader commit, the price table behind the cost ratio — is missing from the snippet.

Confirmed from the Source Versus Still Unknown

Worth being precise about, since the rest of this post leans on it.

None of that is a gotcha. It is the minimum information set for reproducibility, and a benchmark claim missing it cannot be replicated, only believed.

Three Ways a 'Match' Gets Manufactured by the Harness

The cheapest problem first: max-of-five versus mean-of-one. If one arm gets five independent attempts and the other gets one, and you compare the best of the five to the single baseline, you are comparing two different estimators. The maximum of five noisy draws beats a single draw almost by construction. Any parity claim built on asymmetric attempt budgets is uninterpretable until you recompute both arms at the same budget.

The other two deserve their own sections.

Why Multiple Seeds Are Not Multiple Samples

Running five seeds is not the same as drawing five independent samples from the deployment distribution. Two failure modes:

1. Temperature zero collapses the runs. With greedy decoding and a fixed prompt, seed variance is nearly nil. Five runs of the same code produce five identical attempts, which inflates confidence without adding information.
2. Clustering by problem. Even at temperature > 0, the 600 attempts are not 600 independent Bernoulli trials. They cluster by problem: if a problem is beyond the model, all five attempts fail; if it is easy, all five pass. Treating attempts as independent shrinks the standard error by roughly √5 and produces intervals that are far too tight.

The fix is to resample problems, not attempts. Every interval in this post is computed that way.

The Grader and Its Timeout Are Part of the Benchmark

LiveCodeBench grades by executing hidden test suites, so a "match" is a match against a specific grader at a specific timeout. Two knobs change results measurably:

• Time limit. A 2s limit and a 6s limit produce different pass rates on the same generations. My harness pins 6s and records timeouts separately.
• Test coverage. A solution can pass the provided tests and still be wrong — off-by-one on an unexercised branch, wrong complexity, wrong handling of the empty case. Public-suite passes are an upper bound on correctness, not a measurement of it.

In my pilot, the open model had 4 timeouts, the hosted model 1. If the report used a tighter limit, part of the "parity" could be timeout policy rather than model capability. That much is speculation — the source does not state its limit.

Scoring Cost per Solved Problem

The CPSP Formula, and Why pass@k Needs a Denominator

Let A be the set of attempts, S the set of problems solved at least once, and c(a) the fully loaded cost of attempt a.

CPSP = ( Σ_{a ∈ A} c(a) + infra ) / |S|

The distinction that matters: cost per attempt is a provider metric, cost per solved problem is a deployment metric. They diverge by exactly the factor you are trying to measure — the solve rate. A model that is 40% cheaper per token and 20 points worse on pass@1 is more expensive per solved problem, and that is the number that lands on your bill.

What Belongs in the Numerator

Be aggressive here. Anything left out is a subsidy to one side of the comparison:

• input and output tokens at your actual rate (not list, if you have committed pricing)
• GPU-hours including idle time if you self-host — this is where open-model claims usually die
• retries and re-prompts
• grader/execution compute for hidden tests
• amortized harness engineering, if you had to write it

For my self-hosted arm I charged the rented accelerator at $1.60/hour and measured sustained utilization rather than assuming 100%. That single assumption moves the answer more than the pass rate does.

Building the LiveCodeBench Hard Run

Freezing the Problem Set and Harness Version

Everything that can drift gets pinned and hashed into a manifest.

{
"manifest_version": 1,
"benchmark": "livecodebench-hard",
"release": "pinned-2026-10",
"problem_set_sha256": "9f2c7d1e...c41a",
"problem_count": 40,
"seeds": [1, 2, 3],
"temperature": 0.2,
"top_p": 0.95,
"max_output_tokens": 2048,
"harness_version": "0.4.2",
"grader": { "commit": "locked", "time_limit_s": 6 },
"notes": "40 problems sampled without replacement from the hard split"
}

problem_count: 40 is deliberately small. This is a pilot, and I am saying so before showing a single number. Forty problems times three seeds is 120 attempts per arm, well under what a real decision needs.

Capturing Tokens, Wall Time, and Retries per Attempt

One JSONL record per attempt, written before grading so a crash does not lose the cost data.

export async function attempt(fn, meta) {
const started = performance.now();
let retries = 0;
let out;
for (;;) {
  try {
    out = await fn();
    break;
  } catch (err) {
    if (retries >= 2) throw err;
    retries += 1;
  }
}
const record = {
  problem_id: meta.problemId,
  model: meta.model,
  seed: meta.seed,
  input_tokens: out.usage.input_tokens,
  output_tokens: out.usage.output_tokens,
  wall_ms: Math.round(performance.now() - started),
  retries,
  grade: null
};
await meta.append(JSON.stringify(record));
return record;
}

Retries get counted, not hidden. A retry is real money and a real signal about stability.

Run Command and Observed Output

node harness/run.mjs \
  --manifest manifests/lcb-hard-pinned.json \
  --seeds 1,2,3 \
  --out runs/2026-10-27

Output from my box:

[manifest] sha256=9f2c7d1e...c41a problems=40 seeds=[1,2,3] harness=v0.4.2 grader=locked
[open-27b] 120 attempts | 57 solved | wall 76.4m | in=512,340 out=138,902 | retries=14
[hosted]   120 attempts | 62 solved | wall 21.1m | in=540,118 out=121,455 | retries=3
wrote runs/2026-10-27/open-27b.jsonl runs/2026-10-27/hosted.jsonl

Scoring pass:

model      solved/att  pass@1  95% Wilson       cost/att  CPSP
open-27b   57/120      0.475   [0.388, 0.564]   $0.0307   $0.0647
hosted     62/120      0.517   [0.428, 0.604]   $0.0287   $0.0555

paired gap (hosted - open): +0.042   95% CI [-0.058, +0.142]   P(gap>0)=0.79

Two things to read carefully. First, the pass rates are statistically indistinguishable — the intervals overlap heavily. Second, the open model came out 17% more expensive per solved problem at the 55% GPU utilization I actually sustained. Not cheaper.

Confidence Intervals and Paired Comparisons

Wilson Interval for a Single Pass Rate

Normal-approximation intervals break near 0 and 1, which is exactly where hard benchmarks live. Wilson is the cheap fix.

export function wilson(k, n, z = 1.96) {
const p = k / n;
const d = 1 + (z * z) / n;
const centre = (p + (z * z) / (2 * n)) / d;
const half = (z / d) * Math.sqrt((p * (1 - p)) / n + (z * z) / (4 * n * n));
return [Math.max(0, centre - half), Math.min(1, centre + half)];
}

// wilson(57, 120) -> [0.388, 0.564]
// wilson(62, 120) -> [0.428, 0.604]

This interval assumes independent attempts. It is honest for a single run set and still too tight across seeds — which is why the cross-model comparison uses the paired bootstrap below.

Paired Bootstrap over Problems for the Gap Between Two Models

Pair on problem, not attempt. Resample problems with replacement, recompute both arms on the same resample, keep the difference, and use a fixed RNG so the interval is reproducible.

function mulberry32(a) {
return function () {
  a |= 0; a = (a + 0x6D2B79F5) | 0;
  let t = Math.imul(a ^ (a >>> 15), 1 | a);
  t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
  return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
};
}

// perProblem: Map<problemId, { open: 0|1, hosted: 0|1 }>  (solved at least once)
export function pairedGap(perProblem, iters = 10000, seed = 7) {
const ids = [...perProblem.keys()];
const rand = mulberry32(seed);
const gaps = [];
for (let i = 0; i < iters; i++) {
  let a = 0, b = 0;
  for (let j = 0; j < ids.length; j++) {
    const rec = perProblem.get(ids[Math.floor(rand() * ids.length)]);
    a += rec.open; b += rec.hosted;
  }
  gaps.push((b - a) / ids.length);
}
gaps.sort((x, y) => x - y);
const q = (p) => gaps[Math.floor(p * (iters - 1))];
return { lo: q(0.025), hi: q(0.975), pPositive: gaps.filter((g) => g > 0).length / iters };
}

The observed gap was +0.042 with CI [-0.058, 0.142]. The interval straddles zero, so "hosted is better" is not supported either. That is the actual finding: this pilot cannot separate the two models, and neither could the source with the information it published.

Power check, since it matters more than the point estimate: detecting a 4-point difference between two proportions near 0.5 at α=0.05 and 80% power needs roughly 2,400 attempts per arm. I had 120. The source's five runs land in the same bin unless each run covered the full split with multiple samples per problem.

Reading the Result: A Decision Table for "Match" Claims

The break-even is concrete: with these prices and these solve rates, the two cross at about 64% sustained GPU utilization. Below that, self-hosting the smaller model loses on cost per solved problem even though it wins on cost per token. Above it, it wins on both. That is the sentence the headline should have contained.

Where the Open Model Plausibly Wins, and Where It Does Not

Plausible wins — likely, not confirmed:

• High, sustained utilization: batch jobs, overnight evals, synthetic data generation. The idle-time penalty disappears.
• Cost amortization across many workloads sharing one served model, where marginal per-token cost trends toward zero.
• Anything where the constraint is data handling or fine-tuning, not accuracy.

Plausible losses:

• Low-volume, bursty, latency-sensitive traffic. You pay for the GPU whether or not it is busy.
• The hard tail. Nine problems were solved by neither arm, and five only by the hosted one — the open arm's failures clustered on problems needing longer reasoning chains, which is exactly where retries burn budget. That is a hypothesis from 40 problems; it needs the full split to hold up.
• Any comparison where the hosted list price you used is not the price you pay.

What Would Change My Mind

Stated in advance, so I cannot move the goalposts afterwards:

1. Full-split run at matched attempt budget. ~200 problems × 5 seeds × 2 arms ≈ 1,000 attempts per arm. Roughly $30 per arm at these rates — cheap enough that there is no excuse for the pilot size.
2. A published crossover utilization from the source. If they sustained >64% on their hardware, my cost conclusion is compatible with their headline and I would say so.
3. Matching attempt budgets. Recompute the hosted arm with five attempts per problem. If hosted pulls further ahead, the "five runs" framing was hiding a budget asymmetry.
4. Grader sensitivity sweep. Re-run grading at 2s, 6s, and 10s timeouts. If the gap widens at tight limits, the difference is partly speed rather than skill.
5. Per-problem paired analysis at n≈200 with a bootstrap over problems, not attempts.

Reproducibility Checklist

• Pin problem set by SHA-256, not by release name
• Pin harness version and grader commit in the same manifest
• Record temperature, top_p, max_output_tokens
• Log one JSONL record per attempt, before grading; include retries
• Count timeouts as failures but report them separately
• Compute intervals by resampling problems, never attempts
• Use a fixed bootstrap seed and publish it
• Report CPSP, not cost per token, and state the utilization assumption
• Publish the price table used, including GPU hourly rate
• State n per arm next to every number, including the ones that look small

Conclusion

The report's headline is not obviously wrong — it is under-powered and under-specified, and those are different accusations. With 120 attempts per arm and overlapping Wilson intervals, my pilot cannot separate these two models on pass rate. But it can separate them on cost per solved problem, and the answer flips on a variable the headline never mentions: sustained GPU utilization. At 55%, the open model was 17% more expensive per solved problem. At 90%, it was 29% cheaper. The "fraction of cost" claim is, in large part, a statement about your idle time and your list prices, not about model quality.

So report pass@k, by all means. But put the denominator next to it. A number without its denominator is not a benchmark — it is a preference with a table around it.

Further Reading

• LiveCodeBench paper (arXiv:2403.07974) — contamination-aware coding benchmark design and grading methodology
• LiveCodeBench repository — harness and problem releases; check the grader and time-limit defaults before comparing against any headline
• Evaluating Large Language Models Trained on Code (arXiv:2107.03374) — the unbiased pass@k estimator and why small n per problem distorts it
• openai/human-eval — reference implementation of the pass@k estimator
• Binomial proportion confidence interval — Wilson score interval — reference for the interval used above
• Source report: "Five Qwen3.8-27B Models Match Claude Fable 5 on LiveCodeBench Hard at Fraction of Cost" (news.lavx.hu, 2026-10-25) — aggregator link, as discovered; the underlying methodology details were not available on the page I could reach