Scoring Partial Progress in Agent Evals: A GDPx-Inspired Harness in JavaScript

This post builds a JavaScript harness for scoring partial progress on long-horizon agent evals: checkpoint decomposition, dependency-weighted partial credit, trace-based graders you can re-run against a saved attempt, and calibration fixtures that catch a harness lying to you. It uses SemiAnalysis's GDPx benchmark as the motivating case, because the reported headline — a 12% top score for the best frontier model — is exactly the kind of number a partial-credit scoring model is meant to interrogate.

On 2 November 2026, FourWeekMBA and Crypto Briefing both reported that SemiAnalysis had released an agent benchmark called GDPx, and that the best model on it scored 12%. Both reports identify the leading model as "Opus 5.5." The coverage frames that as agentic capability lagging far behind headline model quality.

Maybe. But 12% is a suspiciously round number for a suite meant to measure long-horizon work, and my first reaction as someone who has built eval harnesses is not "agents are bad." It is: what does one point of that 12% mean?

I could not reach the GDPx task set or its scoring rule from the sources available to me. So this post does two things. First, it states plainly what the coverage does and does not establish. Second — and this is most of the post — it builds, in JavaScript, the harness I would want before quoting a number like 12% in a decision document: checkpoint decomposition, dependency-weighted partial credit, trace-based graders that can be re-run against a saved run, and calibration fixtures that catch the harness lying to you.

My position, stated up front: binary all-or-nothing scoring of long-horizon agent tasks is the wrong instrument, and a low ceiling on such a suite is at least as much a measurement-format artefact as a capability ceiling. That is an inference from how these suites are usually built, not a finding about GDPx specifically. I flag it as such throughout.

The code here runs on Node with no dependencies. The scores I show come from scripted policies in a stub sandbox, not from any frontier model — I did not run Opus 5.5, and nothing here should be read as a per-checkpoint profile of it.

What the GDPx Coverage Actually Establishes

The Headline Claim and Who Is Actually Making It

The claim chain matters. SemiAnalysis is the benchmark author. FourWeekMBA and Crypto Briefing are reporting on SemiAnalysis's release. I am reading the reporting. That is three hops from the task set, and each hop tends to drop the denominator: what counts as a "task," what counts as "solved," and whether a partially completed long-horizon task earns anything at all.

The reported facts are narrow: a benchmark named GDPx exists, it was released by SemiAnalysis, the top score is 12%, and the top model is identified as Opus 5.5. Everything else in the coverage — the "agents are behind" framing — is interpretation layered on top.

Confirmed, Reported, and Unverified: A GDPx Fact Table

Claim	Status for me	Basis
SemiAnalysis released a benchmark named GDPx	Reported	FourWeekMBA and Crypto Briefing, both published 2026-11-02
The top score on GDPx is 12%	Reported	Same two reports
The top model is named "Opus 5.5"	Reported, unverified against a primary source	Same two reports; I could not confirm the model identifier independently
GDPx uses binary pass/fail rather than partial credit	Unknown	Not stated in the coverage I could reach
GDPx task count, domains, step budgets, tool surface	Unverified	Not stated in the coverage I could reach
12% implies agents lack long-horizon capability	Inference	Mine, and conditional on the scoring rule

The gap between row two and row four is the whole argument. A 12% top score is a fact about reporting. What it means depends on whether the suite awards credit for getting 80% of the way through a twelve-step task and then failing the last verifiable step. I do not know that, and neither does anyone quoting the headline.

What I Could Not Check in the Available GDPx Coverage

I could not check: the number of tasks, whether tasks are generated or hand-authored, whether scoring is per-task binary or per-checkpoint, whether there is a step budget, whether tool errors are injected, whether the harness reports pass@k, and whether "12%" is a mean over tasks or a resolved-task rate.

⚠️

If you are citing the GDPx number in a planning doc, cite it as "reported top score of 12% on a benchmark whose scoring rule I have not read." Anything stronger is you supplying details the sources did not.

The rest of this post is about building the instrument that would let you answer those questions about any long-horizon suite, including your own.

Why Binary Pass/Fail Undercounts Long-Horizon Agents

The All-or-Nothing Scoring Failure Mode

Consider a task with five verifiable sub-goals in a strict dependency chain: fetch the source, parse it, detect anomalies, write a report, verify the report's totals reconcile. A binary grader returns 1 only if the report exists, is well-formed, and its totals match. Everything else is 0.

Now compare two agents. Agent A fetches, parses, detects all seven anomalies, then writes a report with a missing totals section. Agent B does nothing and returns "I cannot complete this." Both score 0. The suite cannot tell them apart, which means a suite of such tasks cannot distinguish "agents are hopeless" from "agents are 80% of the way there and lose it on the final formatting step."

That compression is not a minor effect on long-horizon suites. It is the dominant effect. If the modal outcome is "fails the last dependent step," the binary mean is pinned near zero no matter how much of the pipeline the model actually built.

What a 12% Binary Score Does and Does Not Tell You

A binary resolved-task rate of 12% tells you: 12% of tasks were fully and verifiably completed. It does not tell you the distribution of how far the other 88% got, whether failures clustered at a particular dependency depth, or whether the failures were recoverable (a retry would have fixed it) or structural (the model never understood the task).

Those three things are what you need in order to act. "12%" tells you to be cautious about deploying agents unattended. It does not tell you whether to invest in better tool-error handling, better state tracking, or a shorter task decomposition — and those are three very different engineering bets.

Partial Progress Is a Signal, Not a Consolation Prize

I am not arguing for participation trophies. A checkpoint-level score is a higher-resolution measurement of the same thing, and it is more falsifiable. If your harness says an agent reaches dependency depth 3 of 5 on 70% of tasks and depth 5 on 8%, you can make a prediction: give it a shallower task and the success rate should jump. If it does not, your harness is measuring something other than progress. That is a testable claim, which is more than a single scalar gives you.

The cost is that checkpoint scoring is gameable in ways binary scoring is not. Much of this post is about paying that cost down.

Designing a Partial-Credit Scoring Model

Decomposing a Long-Horizon Task Into Verifiable Checkpoints

The rule I use: a checkpoint must be checkable without reading the agent's prose. If verifying it requires interpreting what the agent said it did, it is not a checkpoint, it is a vibe.

For the ledger-reconciliation task I use throughout:

export const task = {
  id: "ledger-reconcile-v1",
  maxSteps: 24,
  completionCheckpoint: "report_verified",
  checkpoints: [
    { id: "source_fetched",   dependsOn: [],                   kind: "trace" },
    { id: "rows_parsed",      dependsOn: ["source_fetched"],   kind: "state" },
    { id: "mismatches_found", dependsOn: ["rows_parsed"],      kind: "state" },
    { id: "report_written",   dependsOn: ["mismatches_found"], kind: "artifact" },
    { id: "report_verified",  dependsOn: ["report_written"],   kind: "artifact" },
  ],
  penalties: [
    { id: "false_completion", kind: "declared_done_unverified", value: 0.15 },
  ],
};

kind is documentation, not dispatch — it tells the next engineer where the evidence for that checkpoint lives. dependsOn is the load-bearing field.

Weighting Checkpoints by Dependency Depth, Not by Count

Uniform weights are wrong here, and they fail in a specific direction: they over-reward early scaffolding. Fetching a file is easy; verifying that a written report reconciles is hard. If you give both 0.2, a model that fetches and parses gets 40% for the part a shell script does.

Depth-based weighting approximates the probability that the whole chain held:

function depthOf(checkpoints, id, seen = new Set()) {
  if (seen.has(id)) throw new Error(`dependency cycle at ${id}`);
  seen.add(id);
  const cp = checkpoints.find((c) => c.id === id);
  const parents = cp.dependsOn ?? [];
  if (parents.length === 0) return 0;
  return 1 + Math.max(...parents.map((p) => depthOf(checkpoints, p, new Set(seen))));
}

export function weightsFor(checkpoints) {
  const raw = checkpoints.map((c) => 1 + depthOf(checkpoints, c.id));
  const total = raw.reduce((a, b) => a + b, 0);
  return new Map(checkpoints.map((c, i) => [c.id, raw[i] / total]));
}

For the task above: depths are 0, 1, 2, 3, 4, so raw weights are 1, 2, 3, 4, 5 (total 15), giving 0.067, 0.133, 0.200, 0.267, 0.333. A model that clears the first three earns 6/15 = 0.400 instead of 0.600 under uniform weights.

This is a modelling choice, not a law. The property I care about is that credit grows with the length of the confirmed chain, so the score is roughly monotone in "how far did the dependency graph actually hold."

Ordering Rules: Credit for Sequence, Not Just Presence

The most common bug in hand-rolled partial credit is awarding a checkpoint because its precondition is present in the final state, even if it was produced before the thing it depends on. Ordering must be enforced by the scorer, not inferred.

Two mechanisms:

Dependency blocking. If any parent is not pass, the checkpoint is blocked and earns zero. It is not skipped, and it does not leave the denominator.
Trace ordering assertions for checkpoints where sequence is the point (a write must come after the read it derives from).

That first rule is where most implementations quietly cheat: dropping blocked checkpoints from the denominator makes an agent that fails immediately score 0/0, which some codebases render as 1.0. Never remove blocked checkpoints from the total.

Penalties for Confident-but-Wrong Paths

Partial credit creates an exploit: stop after the cheap checkpoints, declare success, and bank 40%. A declaredComplete flag that is true while the completion checkpoint has not passed should cost more than the remaining credit is worth in expectation.

function applyPenalties(task, attempt, status) {
  const applied = [];
  for (const p of task.penalties ?? []) {
    if (p.kind === "declared_done_unverified") {
      const declared = attempt.finalState?.declaredComplete === true;
      const verified = status.get(task.completionCheckpoint) === "pass";
      if (declared && !verified) applied.push({ id: p.id, value: p.value });
    }
  }
  return { total: applied.reduce((a, p) => a + p.value, 0), applied };
}

A second useful penalty is thrash: the same failing (tool, args) pair repeated three or more times. Cap the total penalty so a single bad habit cannot drive a genuinely productive run to zero.

The Harness Skeleton in JavaScript

The Task, Attempt, and Checkpoint Contracts

Three shapes, and keeping them separate is what makes graders re-runnable:

// Attempt: what the harness observed. Frozen after the run.
{
  taskId: "ledger-reconcile-v1",
  trace: [ { n: 1, tool: "fetchSource", args: { path: "in/ledger.csv" }, ok: true, output: "40 rows" } ],
  finalState: { declaredComplete: false, rows: [ /* ... */ ], mismatches: [ /* ... */ ] },
  artifacts: { "out/report.md": "# Reconciliation\n..." },
  meta: { steps: 9, aborted: false, seed: 1337 }
}

The attempt is the only input graders get. No live sandbox handle, no network, no clock. That constraint is what makes "re-run the scorer against a saved trace from last month" possible.

A Scorer Registry Keyed by Checkpoint ID

const graders = new Map();
export function register(id, fn) {
  if (graders.has(id)) throw new Error(`duplicate grader for ${id}`);
  graders.set(id, fn);
}
export function getGrader(id) {
  const fn = graders.get(id);
  if (!fn) throw new Error(`no grader registered for checkpoint ${id}`);
  return fn;
}

Throwing on a missing grader is deliberate. A silently-skipped checkpoint inflates every score in the run, and you will not notice for weeks.

Running an Attempt With a Step Budget and Hard Abort

export async function runAttempt({ task, agent, sandbox }) {
  const trace = [];
  const controller = new AbortController();
  let n = 0;

  for (n = 1; n <= task.maxSteps; n++) {
    if (controller.signal.aborted) break;

    let action;
    try {
      action = await agent.next({ n, trace: publicView(trace), signal: controller.signal });
    } catch (err) {
      trace.push({ n, tool: "<agent>", ok: false, error: String(err) });
      break;
    }

    if (!action || action.type === "done") {
      trace.push({ n, tool: "<done>", ok: true });
      sandbox.markDeclaredComplete();
      break;
    }

    const result = await sandbox.invoke(action, { signal: controller.signal });
    trace.push({
      n,
      tool: action.tool,
      args: redact(action.args),
      ok: result.ok,
      output: truncate(result.output, 500),
    });
    if (result.fatal) controller.abort();
  }

  return {
    taskId: task.id,
    trace,
    finalState: sandbox.snapshotState(),
    artifacts: sandbox.snapshotArtifacts(),
    meta: { steps: Math.min(n, task.maxSteps), aborted: controller.signal.aborted },
  };
}

publicView(trace) is doing real work: it strips grader-relevant detail before showing the trace back to the agent. More on why in the leakage section.

Capturing the Trace: Tool Calls, Outputs, and Final State

Record args after redaction, not before, and record the ok flag separately from output. A grader that has to string-match an error message to determine failure is a grader that breaks when someone edits a message.

Everything in the trace should be JSON-serialisable and small. Truncate outputs at capture time; a 200 KB tool response in every attempt is how eval suites turn into storage bills.

Writing Graders That Do Not Lie

State-Based Graders Versus Transcript-Based Graders

A transcript grader asks "did the agent call writeFile with the report?" A state grader asks "is there a report at out/report.md with a totals section?" The transcript version passes when the write targeted a read-only path, or when the sandbox rejected it and the agent moved on.

// Transcript-based: lies when the write failed.
register("report_written_bad", ({ trace }) => ({
  pass: trace.some((s) => s.tool === "writeFile" && s.args.path === "out/report.md"),
  detail: "write was attempted",
}));

// State-based: checks the artifact that actually exists.
register("report_written", ({ artifacts }) => {
  const md = artifacts["out/report.md"];
  if (typeof md !== "string") return { pass: false, detail: "out/report.md absent" };
  const hasTotals = /^## Totals$/m.test(md);
  return { pass: hasTotals, detail: hasTotals ? "totals section present" : "totals section missing" };
});

Default to state. Use transcript graders only where the call itself is the deliverable — retry behaviour, ordering, or "did it verify before writing."

Artifact Assertions on Files and Emitted Data

Assert invariants, not golden strings. A golden string tells you the agent reproduced your reference; an invariant tells you the artifact is correct.

register("report_verified", ({ artifacts, finalState }) => {
  const md = artifacts["out/report.md"] ?? "";
  const claimed = Number(md.match(/total mismatch count:\s*(\d+)/i)?.[1]);
  const actual = finalState.mismatches?.length;
  if (!Number.isInteger(claimed) || !Number.isInteger(actual)) {
    return { pass: false, detail: "totals not machine-readable" };
  }
  return {
    pass: claimed === actual,
    detail: `report claims ${claimed}, state holds ${actual}`,
  };
});

This recomputes from the agent's own state rather than from a stored answer, so it survives fixture edits.

Tool-Call Trace Assertions for Ordering and Retries

Fault recovery is only measurable if you know the fault was injected. Make the grader fail loudly when it was not:

register("recovered_from_fault", ({ trace }) => {
  const firstFail = trace.find((s) => s.tool === "fetchSource" && !s.ok);
  if (!firstFail) {
    return { pass: false, detail: "fault was not injected — harness bug, not agent failure" };
  }
  const retry = trace.find((s) => s.n > firstFail.n && s.tool === "fetchSource" && s.ok);
  return {
    pass: Boolean(retry),
    detail: retry ? `retried successfully at step ${retry.n}` : "never retried successfully",
  };
});

That early return is the difference between "the model cannot recover from tool errors" and "my injector was off." I have seen the second one reported as the first one.

Grader Idempotency and Re-Running Against a Saved Trace

Graders must be pure. Freeze the attempt on entry so mutation throws in strict mode:

function deepFreeze(o) {
  for (const v of Object.values(o)) if (v && typeof v === "object") deepFreeze(v);
  return Object.freeze(o);
}

export function scoreOnce(task, attempt) {
  return scoreAttempt(task, deepFreeze(structuredClone(attempt)));
}

Then a cheap self-check that catches most accidental impurity:

const a = scoreOnce(task, attempt);
const b = scoreOnce(task, attempt);
assert.deepEqual(a, b); // same input, same output, no state carried

If that assertion fails, one of your graders is reading a counter, a timestamp, or a module-level cache.

Handling Irreversible Steps Safely in a Scored Run

Sandboxing Destructive Checkpoints

Any checkpoint that deletes, overwrites, or provisions something needs a sandbox boundary that lives in the harness, not in the prompt. Concretely:

No real network. Tool implementations resolve from a map; unknown tools throw.
One temp directory per attempt, deleted on completion.
Path arguments are resolved and prefix-checked against the attempt root before any write.
No shell tool at all in the graded run. If a task genuinely needs one, run it in a container and score only on files that appear in the output mount.

function resolveIn(root, p) {
  const abs = path.resolve(root, p);
  if (!abs.startsWith(path.resolve(root) + path.sep)) {
    throw new Error(`path escape: ${p}`);
  }
  return abs;
}

Failing Closed When a Checkpoint Cannot Be Verified

Three states, and blocked is not a synonym for fail:

Status	Meaning	Credit	Denominator
`pass`	grader returned true	weighted share	included
`fail`	grader ran, returned false	0	included
`blocked`	a dependency did not pass	0	included
`harness_error`	grader threw or was missing	run is void	excluded from reporting entirely

A harness_error should never be silently converted to fail. The run gets thrown out and reported as an infrastructure failure, because a broken grader and a bad model look identical in an aggregate.

Keeping Abuse-Enabling Detail Out of the Task Fixtures

Fixtures for destructive checkpoints should describe effects, not procedures. "The provisioning step must be idempotent and must not be retried after a 4xx" is a testable checkpoint. A fixture containing working credentials or a copy-pasteable destructive command is a liability that ships with your repo and gets indexed. Use synthetic data with the same shape as production and no real identifiers.

Probing the Failure Modes the Ceiling Exposes

Long-horizon failure is not one thing. These four probes separate them, and each one is a checkpoint or a policy variant rather than a new benchmark.

State Tracking: Does the Agent Still Know Where It Is?

Instrument the task with an "interrogation" checkpoint at depth 2 or 3: the sandbox asks the agent to state, in a structured field, which sub-goals it has completed and which remain. Grade against the harness's own record.

register("state_self_report", ({ finalState }) => {
  const claimed = new Set(finalState.selfReport?.completed ?? []);
  const actual = new Set(finalState.verifiedCompleted ?? []);
  const missing = [...actual].filter((x) => !claimed.has(x));
  const invented = [...claimed].filter((x) => !actual.has(x));
  return {
    pass: missing.length === 0 && invented.length === 0,
    detail: `missing=${missing.join(",") || "-"} invented=${invented.join(",") || "-"}`,
  };
});

invented is the interesting column. Agents that claim completed work that never happened are not just wrong, they are wrong in a way that poisons every downstream step.

Tool-Error Recovery Under Injected Faults

Use a seeded fault injector so the run is reproducible:

function mulberry32(seed) {
  return function () {
    seed |= 0; seed = (seed + 0x6D2B79F5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

Then declare faults per tool as a probability plus a cap, record the seed in attempt.meta, and grade recovery from the trace as shown earlier. Report the recovery rate separately from the overall score — mixing them hides whether a model is bad at the task or bad at being interrupted.

Planning Drift and Goal Regression Across Steps

Drift shows up as a step that is valid in isolation but inconsistent with the plan established earlier. Cheap proxy: a tool_coverage grader that checks whether the tool the agent used at step n was ever referenced in the agent's own stated plan.

register("plan_consistency", ({ trace }) => {
  const planned = new Set(trace[0]?.plan?.map((s) => s.tool) ?? []);
  if (planned.size === 0) return { pass: false, detail: "no plan emitted" };
  const used = trace.filter((s) => s.tool !== "<done>").map((s) => s.tool);
  const offPlan = used.filter((t) => !planned.has(t));
  return {
    pass: offPlan.length <= 1,
    detail: `${offPlan.length} off-plan calls: ${offPlan.join(",") || "-"}`,
  };
});

This is a heuristic, and it penalises legitimate replanning. Set the tolerance above zero and treat a large jump as the signal.

Context Loss at Long Step Counts

Run the same task family at step budgets of 8, 16, 32, and 64 by padding with verifiable no-op steps. If the score declines with budget while the checkpoint set is unchanged, you are measuring context handling, not task difficulty. This is the cheapest diagnostic in the list and the one most likely to produce an actionable finding.

Aggregating Checkpoints Into One Number

The Partial-Credit Curve and Monotonicity Checks

export function assertMonotonic(ordered) {
  for (let i = 1; i < ordered.length; i++) {
    if (ordered[i].score < ordered[i - 1].score) {
      throw new Error(
        `non-monotonic: ${ordered[i - 1].id}=${ordered[i - 1].score} > ${ordered[i].id}=${ordered[i].score}`
      );
    }
  }
}

Feed it policies ordered by how much of the chain they complete. If the ordering breaks, your weights or your blocked-checkpoint handling are wrong.

Reporting the Progress Distribution, Not Just the Mean

export function summarize(scores) {
  const s = [...scores].sort((a, b) => a - b);
  const q = (p) => s[Math.floor(p * (s.length - 1))];
  return {
    n: s.length,
    mean: +(s.reduce((a, b) => a + b, 0) / s.length).toFixed(3),
    p10: q(0.1),
    median: q(0.5),
    p90: q(0.9),
    max: s.at(-1),
  };
}

A mean of 0.33 with a p90 of 0.9 and a median of 0.05 tells a completely different story from a mean of 0.33 with a flat distribution. Report the shape.

pass@k and Best-of-n Under Partial Credit

Binary pass@k does not transfer cleanly, so define a pass threshold τ and compute the same unbiased estimator over "scored at least τ":

export function passAtK(scores, k, tau = 0.8) {
  const n = scores.length;
  const c = scores.filter((s) => s >= tau).length;
  if (n - c < k) return 1;
  let prod = 1;
  for (let i = 0; i < k; i++) prod *= (n - c - i) / (n - i);
  return 1 - prod;
}

Report (τ, k) next to every pass@k number. A pass@5 at τ=0.8 and a pass@5 at τ=0.5 are not comparable and will be compared anyway if you do not label them.

Worked Example: A Small GDPx-Style Task With a Known Ceiling

The Worked-Example Task and Its Checkpoint Set

The ledger-reconcile-v1 task above: 40 synthetic rows in in/ledger.csv, 7 planted mismatches, a report that must include a ## Totals section whose machine-readable count matches the state. Weights from depth: 0.067, 0.133, 0.200, 0.267, 0.333.

Running One Attempt and Reading the Trace

The agent below is a scripted policy, not a model. It completes three checkpoints, writes a report with a missing totals section, and declares itself done.

$ node run-example.mjs --policy drift --seed 1337
task: ledger-reconcile-v1   steps=9   aborted=no
checkpoint            w      status   credit  detail
source_fetched        0.067  pass     0.067   fetched in step 1
rows_parsed           0.133  pass     0.133   40 rows, 0 bad amounts
mismatches_found      0.200  pass     0.200   7 mismatches
report_written        0.267  fail     0.000   totals section missing
report_verified       0.333  blocked  0.000   blocked by report_written
penalty false_completion                        -0.150
------------------------------------------------------------
raw credit  0.400   penalty 0.150   final score  0.250
binary pass@1                                 0.000

Observed Result Next to the Claim It Supports

Policy (scripted)	Checkpoints confirmed	Raw credit	Penalty	Score
`noop`	none	0.000	0.000	0.000
`fetch-only`	1	0.067	0.000	0.067
`parse-only`	1–2	0.200	0.000	0.200
`mismatch-only`	1–3	0.400	0.000	0.400
`drift`	1–3, false done	0.400	0.150	0.250
`reference`	1–5	1.000	0.000	1.000

What this supports: the harness separates two behaviours that binary scoring merges. noop and drift both score 0.000 under pass@1; under checkpoint scoring they are 0.000 and 0.250, and the difference is exactly the information you would need to decide whether to invest in formatting robustness or in basic task comprehension.

What this does not support: anything about front-running models. The scripted policies were written by me to make drift land where it does. The monotonicity check over [noop, fetch-only, parse-only, mismatch-only, reference] returns 0.000, 0.067, 0.200, 0.400, 1.000 — non-decreasing, which validates the weighting, not the model.

summarize() over the five ordered policy scores returns { n: 5, mean: 0.333, p10: 0, median: 0.2, p90: 0.4, max: 1 }. That is arithmetic on the table above, reproducible with the code in this post.

Where Partial-Credit Scoring Goes Wrong

Score Gaming and Reward Hacking Against Your Own Graders

The moment partial credit exists, "write something that makes the grader return true" is a valid strategy. Three defences that cost little:

Recompute from state, never compare to a stored answer. The report_verified grader derives the expected count from finalState.mismatches, not from a constant.
Require two independent signals for expensive checkpoints. Artifact exists and its contents reconcile against state.
Randomise fixture values per seed. If the expected mismatch count is 7 every run, a model can learn 7.

Grader Leakage Into the Agent Prompt

If the agent sees checkpoint ids or grader details, it will target them. That is not a model failure, it is a harness failure. Keep two views:

export function publicView(trace) {
  return trace.map(({ n, tool, args, ok }) => ({ n, tool, args, ok })); // no output, no grader feed
}

And keep checkpoints, penalties, and any expected values entirely out of the system prompt. If your agent could read task.checkpoints at runtime, your score is measuring prompt-reading.

False Progress From Partially-Written State

Existence is not correctness. out/report.md existing with zero bytes is a file and a lie. Every artifact grader needs at least one content invariant, and the invariant should be cheap to state: a section header, a numeric field that parses, a checksum over rows the agent wrote itself.

The related failure is a partially-written state that looks complete: three of seven mismatch records present, report generated from those three, totals internally consistent. Pure internal consistency passes. That is why at least one checkpoint must compare against data the agent did not produce — the source rows.

Calibrating the Harness Before You Trust It

Seeding Runs With Known-Good and Known-Bad Solutions

Before you run a single model, the harness must score a hand-written reference solution at exactly 1.000 and a no-op at exactly 0.000. Both are cheap to write and both catch the majority of weighting bugs. Add a "plausible but wrong" solution — completes everything except the verification step — and assert it lands strictly between them.

assert.equal(scoreOnce(task, reference).score, 1);
assert.equal(scoreOnce(task, noop).score, 0);
const p = scoreOnce(task, plausibleButWrong).score;
assert.ok(p > 0 && p < 1, `plausible-but-wrong landed at ${p}`);

Reproducing a Slice of a Public Benchmark Locally

I cannot reproduce a slice of GDPx, because I do not have its task set or graders. Say that out loud before you claim a local reproduction: what you can do without the task set is reproduce the shape — dependency-chained checkpoints, injected faults, a step budget — and check that your harness produces a monotone curve across capability-ordered policies.

If the task set is published, the reproduction is: pin the task set by commit SHA, port the graders one-to-one, seed your injector identically, and diff your per-checkpoint results against the published ones. Any variance at that point is your port, not the benchmark.

Comparing Your Scores to a Published Number Without Overclaiming

If your local slice scores higher than the reported number, the honest statement is "my slice of the task family scored X." Not "model Y beats GDPx." A slice is a subset, subsets are easier, and the headline number came through two hops of reporting that may not even use your scoring rule.

Running This in CI Without Lying to Yourself

Determinism, Seeds, and Flake Budgets

Record the seed in attempt.meta and pin it in CI.
Pin the model snapshot identifier, not the marketing name. Providers update aliases.
Set temperature to 0 and still expect variance — it reduces, it does not eliminate.
Declare a flake budget and fail the build when it is exceeded: flakeRate = runsWhereScoreChangedOnRerun / totalRuns, threshold 0.02.

Threshold Policy and Regression Gates

Two gates, not one:

## eval-gates.yaml
min_score: 0.55          # absolute floor for the task family
max_regression: 0.04     # allowed drop vs the rolling 5-run baseline
min_pass_at_k: { k: 5, tau: 0.8, value: 0.30 }

The absolute floor stops silent rot. The regression delta stops a good model version from masking a harness change. Keep them as separate failures so you know which one fired.

Tracking Score Drift Across Model Versions

Store one JSON per run:

{ "model": "provider/model", "modelVersion": "2026-10-14", "harnessSha": "a1b2c3d",
  "taskSetSha": "e4f5g6h", "seed": 1337, "scores": { "ledger-reconcile-v1": 0.25 } }

Then plot per-task scores across modelVersion. A model that improves the mean while regressing three specific checkpoints is a model with a narrower improvement than the headline suggests, and the per-checkpoint view is the only place that shows up.

Conclusion: Score the Trajectory, Not Just the Destination

The reported 12% top score on GDPx is one number from one hop of reporting, and the framing around it — agents are far behind — is an interpretation that may or may not survive contact with the scoring rule. I could not check the rule, and I said so. What I can say with confidence is that a long-horizon agent suite scored all-or-nothing throws away most of the signal it paid to collect. Two agents that both "fail" can differ by 80% of the dependency chain, and that difference is the whole engineering roadmap.

Partial credit is the right instrument, but it is an instrument that lies easily. Weight by dependency depth, never drop blocked checkpoints from the denominator, grade state before transcript, make graders pure so you can re-score a saved trace, fail closed on unverifiable checkpoints, and validate the whole thing against a hand-written reference and a no-op before a model ever touches it.

The most useful sentence in any eval report is not the mean. It is "the failures clustered at dependency depth 3, in the step that verifies the artifact it just wrote." A binary ceiling cannot say that. A checkpoint harness can.

Introduction: A 12% Ceiling Is a Scoring Problem Before It Is a Capability Problem