Using Eval Gates to Decide When a Cheap Open-Weight Model Is Good Enough for an Agent Task

Introduction: Deciding When a Cheap Open-Weight Model Is Good Enough

On 2026-10-15, XenoSpectrum reported that Intel's CEO described DeepSeek, Kimi, and GLM as "good enough" while the price gap between Western and open-weight models kept widening. Stocktwits called the same day the token economy's first deflationary shock. The Financial Times ran a tokenomics piece — managing AI and infrastructure spend instead of discovering it on the invoice.

This post shows how to turn that kind of headline into an eval gate: a golden set, a scorer, and thresholds kept in version control, so a model swap becomes a reviewable diff instead of an argument. You'll see what "good enough" has to mean for one agent task, how to score tool-call reliability alongside final answers, and how cost-per-success decides which tasks a cheap open-weight candidate can actually take over.

I won't argue with the direction of that price curve. I'll argue with the phrase itself: "good enough" isn't a property of a model, it's a property of a task. The only honest way to act on a price gap is an eval gate that decides, per task, whether a cheap open-weight candidate clears a bar you wrote down and can review in a pull request. A CEO quote and a publisher's framing are signals to test, not results to adopt.

What "Good Enough" Means for One Agent Task

A gate needs three things before it needs any tooling: a fixed input distribution, a pass criterion, and a cost unit. That means writing the task down in the shape the result will be reported in:

Task: extract 20 invoice fields from real PDFs.
Inputs: 40 invoices sampled from production traffic, hardened and PII-scrubbed.
Criterion: exact match on all 20 fields, plus valid tool arguments.
Count: 40 cases.

Which model, which temperature, which provider — those stay variables you're allowed to change once the task definition is frozen. If the criterion moves while you're testing, you aren't running a gate, you're negotiating with yourself.

Task-Level Accuracy, Not Benchmark Scores

Leaderboard transfer doesn't hold. A model that tops a general reasoning benchmark can still fumble a narrow JSON-extraction task, because the benchmark rewards broad recall while your task rewards exactness on a fixed schema with a fixed vocabulary of field names. The failures are usually boring: a field name pluralised differently, a date normalised into a format your downstream parser rejects, a null emitted as the string "null".

Sample from the real workload — 30 to 100 actual inputs, stratified if your inputs are stratified. Not three prompts. Three prompts is an anecdote, and saying so isn't weakness; it's the line between a measurement and a vibe.

Tool-Call Reliability Is a Separate Axis

For agent tasks, split the score in two: final-answer correctness and tool-call correctness. The second one covers schema conformance, argument hallucination (an invoice id that appears nowhere in the input), missing required fields, and retry count.

Cheap models tend to lose their advantage here. If a model only reaches the right answer after three malformed tool calls, it isn't cheap in cost-per-success terms — you paid for four inference passes and three wasted tool executions to get one usable result. Final-answer-only scoring hides exactly that failure mode, which is why the next section exists.

The Eval Gate: Golden Set, Scorer, Threshold

Three artifacts, all checked into the repo: a golden set of inputs with expected outputs or expected tool sequences, one scorer function per task, and a threshold config. Keeping thresholds in version control instead of a slide is the whole point — a model swap turns into a reviewable diff with a number attached.

A Minimal Eval Gate Runner in Node

// eval-gate.js
// The adapter below is a stub: no provider SDK, no credentials, no network.


const THRESHOLDS = {
minPassRate: 0.9,
maxCostPerSuccessUsd: 0.005,
minToolCallConformance: 0.95,
};

// adapter(input) resolves to:
// { finalAnswer, toolCalls: [{ name, args, ok }], usage, costUsd, retries }
export async function runGate({ adapter, scorer, cases = goldenSet, thresholds = THRESHOLDS }) {
let passed = 0;
let checks = 0;
let conformant = 0;
let spendUsd = 0;
let retries = 0;

for (const testCase of cases) {
  const run = await adapter(testCase.input);
  spendUsd += run.costUsd;
  retries += run.retries;

  for (const call of run.toolCalls) {
    checks += 1;
    if (call.ok) conformant += 1;
  }

  if (scorer({ expected: testCase.expected, actual: run.finalAnswer }).ok) {
    passed += 1;
  }
}

const passRate = passed / cases.length;
const toolCallConformance = checks === 0 ? 1 : conformant / checks;
const costPerSuccessUsd = passed === 0 ? Infinity : spendUsd / passed;

const reasons = [];
if (passRate < thresholds.minPassRate) {
  reasons.push("pass rate " + passRate.toFixed(2) + " < " + thresholds.minPassRate);
}
if (toolCallConformance < thresholds.minToolCallConformance) {
  reasons.push("tool conformance " + toolCallConformance.toFixed(2) + " < " + thresholds.minToolCallConformance);
}
if (costPerSuccessUsd > thresholds.maxCostPerSuccessUsd) {
  reasons.push("cost/success " + costPerSuccessUsd.toFixed(4) + " > " + thresholds.maxCostPerSuccessUsd);
}

return {
  verdict: reasons.length === 0 ? "PASS" : "FAIL",
  passRate,
  toolCallConformance,
  costPerSuccessUsd,
  spendUsd,
  retries,
  cases: cases.length,
  reasons,
};
}

Run it against a stub adapter first. If the gate passes a deliberately broken adapter, the scorer is wrong — and you found that out for free.

Gating on the Trajectory, Not Just the Last Message

Agent tasks fail mid-loop. A step-level assertion list is a few lines of code and catches the expensive failures:

Step 4 is where the cost math turns: those two retries are billable even though the run scored as a failure, and a final-answer-only scorer never sees them.

Cost-Per-Success Accounting

Define the metric as total spend divided by passing cases, not spend divided by tokens. The numerator swallows failed retries, wasted tool calls, prompt tokens spent on cases that never worked, cache misses, and — if you score with a model-as-judge — the judge's own cost. The denominator is passing cases only.

Tokens per dollar is a supplier metric. Cost per passing case is the number that decides whether the pricing gap from the source material is actionable or irrelevant for your workload.

Wiring the Gate Into Model Routing

Register every candidate behind one interface — generate(taskId, input) — so the routing layer never imports a provider SDK directly. Route on gate result, with a fallback ladder to the expensive model whenever the cheap one fails a case or blows its retry budget.

Then shadow it: run the candidate alongside production traffic, score offline, and promote only after the gate holds across more than one sample window. One green run on a Tuesday isn't a promotion signal.

Sample Gate Results From a Small Run

The table below uses synthesized numbers to make the output shape concrete. Treat it as a template, not as evidence about any named provider — the only trustworthy version of this table is the one your own gate prints.

For the failing task, the gate output names the reason outright:

{
  "verdict": "FAIL",
  "passRate": 0.686,
  "toolCallConformance": 0.81,
  "costPerSuccessUsd": 0.0049,
  "reasons": ["pass rate 0.69 < 0.9", "tool conformance 0.81 < 0.95"]
}

The split is the useful part. One task clears, one doesn't, and the failing task's cost per success is nearly 2.3x the passing task's despite the nominally cheaper model — because 11 of 35 cases burned tokens without producing a usable answer.

What the Pricing-Deflation Reports Do and Don't Establish

Confirmed from the source material: XenoSpectrum reported on 2026-10-15 that Intel's CEO called DeepSeek, Kimi, and GLM "good enough" as pricing gaps emerged. Stocktwits published a piece the same day describing an AI deflationary shock hitting the token economy. The Financial Times published a tokenomics piece on managing AI and infrastructure costs. Global Times reported an expert arguing that the OpenAI CFO's cybersecurity-risk framing around Chinese AI models is driven by business interests amid open-weight competition.

Inference, clearly marked: that these reports describe a durable trend rather than a moment. Note the competing-interest angle without endorsing either side — a security argument from a company selling closed models and a deflation argument from outlets covering open-weight competition both carry incentives. Test the claim on your workload instead of adjudicating the politics.

What the source material does not contain: per-token price tables. Any specific price delta I wrote here would be invented, so I'm not writing one. Pull current prices from the providers you're actually evaluating.

Failure Modes and Defenses

Gate gaming. A scorer that checks format and not content will pass a model emitting perfect JSON with the wrong values. Defense: field-level exact match, held-out cases the candidate's prompt has never seen, and occasional blind human review of a passing sample.

Eval leakage. Golden-set inputs drift into system prompts or few-shot examples, and the gate starts measuring memorisation. Defense: keep golden sets in files the application never imports, and rotate a held-out slice on every model swap.

Distribution drift. The gate holds in March; the workload changes by June. Defense: sample production inputs on a schedule, add cases, and re-score unchanged thresholds so the drift shows up explicitly.

Locale and language gaps. Open-weight candidates vary widely outside English, and an aggregate pass rate hides it. Defense: stratify the golden set by locale and report per-slice pass rates instead of one global number.

Latency regression. Cheaper and slower is a different product. Defense: track p95 latency and retry counts alongside cost-per-success in the same gate output, so a swap can't pass on cost alone.

Conclusion: Gate the Task, Not the Model

The deflationary pressure those reports describe is real enough to test against — but "good enough" is only decidable with an eval gate you own, on tasks you own, with a threshold you can review in a diff. A market take and a CEO quote are inputs to your decision, not a replacement for it.

Start with one agent task, 30 to 50 real inputs, and a single cost-per-success threshold. Add the second task only where the gate holds across more than one sample window. The cheap open-weight model doesn't have to be good enough in general. It has to be good enough at the one thing you're paying it to do.

Further Reading

• XenoSpectrum — Intel CEO calls DeepSeek, Kimi and GLM "good enough"
• Stocktwits — AI is experiencing its first deflationary shock
• Financial Times — Tokenomics: smart strategies to deal with the costs of AI and infrastructure
• Global Times — expert response on open-weight competition and cybersecurity claims