Triage Pipelines for AI-Generated Bug Reports: A Node.js Maintainer's Approach

Last month I burned a Sunday afternoon triaging issues on a mid-size Node.js library. Eleven of the fourteen reports that landed were wrong in the same way: clean prose, believable stack traces, a version that does not exist, and nothing anyone could actually run. That used to be a bad weekend. Now it is a structural problem for open source.

This post covers the triage pipeline I use to handle it: a four-stage design that rejects structurally invalid submissions, runs a sandboxed reproduction gate, dedupes survivors by root cause, and routes only the strongest reports to a human. You get the Node.js intake schema, the reproduction-gate code, the output from a 200-report sample batch, and the disclosure wording that keeps a real vulnerability report from getting lost in the noise.

Why AI-Generated Bug Reports Cost More to Triage Than Human Ones

The old bad bug report was low effort. Someone pasted a stack trace, forgot their Node version, and you closed it with a template. Two minutes, done.

The new one is high-confidence nonsense. It reads like a real advisory. It names a function, a severity, and a suggested fix. It is also frequently built from a summary of an unrelated real advisory with the package name swapped in. Reading it carefully takes longer than reading a good report, because you have to reconstruct what the reporter thinks they proved before you can check the claim.

That cost asymmetry broke Google's program. It will break yours too if you treat triage as a moderation problem instead of an engineering one.

What Actually Changed in October 2026

Three items landed in the same 24-hour news cycle on 2026-10-05. Only one of them is really a technical story.

Google pauses its open-source bug bounty over automated submissions

IT Pro reported that Google paused its open-source bug bounty scheme, quoting the program's own explanation: "This pause is due to a significant rise in automated submissions, the vast majority of which are not valid." Cyberpress reported the same tightening of open-source bug bounty rules.

The confirmed part is narrow: a major platform told reporters that automated submission volume with a very low validity rate forced a pause or tightening. The dollar thresholds, exact resume criteria, and per-project breakdown are not in the public reporting I can verify. Treat those as unknown.

IBM and Red Hat launch the $5B Project Lightwell

NewsBytes reported that IBM and Red Hat launched Project Lightwell, a $5B effort aimed at securing open-source software. A funding number that size gets quoted heavily in the days after launch; what I cannot verify from the reporting is how the money is distributed, over what period, or which projects qualify. What matters for this post is the direction: the industry response to maintainer overload is more money for security work.

DigitalOcean ends its open-source credits program

It's FOSS reported that DigitalOcean quietly ended its open-source credits program. It is a small item next to a $5B number, and that is exactly why it belongs in the same paragraph. Maintainer capacity is partly funded by infrastructure credits. When those go away and a flood of low-quality reports arrives in the same quarter, the maintainer eats both shocks.

I have not verified causation between any of these, and they are probably not coordinated. But they describe the same equilibrium — more automated output, less maintainer time to filter it.

Why Better Filters Alone Do Not Fix the Flood

Every discussion of this problem arrives at "we should filter better" within five minutes. I think that is the wrong primary control, for a base-rate reason.

How AI-generated reports fail differently from human ones

Human low-quality reports fail in ways that are cheap to detect: no version, no repro, one line of text, a screenshot of a terminal. Automated reports fail in ways that are expensive to detect, because they are optimized for the shape of a good report:

A classifier trained on the top row scores the bottom row as high quality. That is the trap.

The asymmetry problem: one real bug buried under hundreds of invalid ones

Say 1 in 200 automated reports contains a genuine issue. Filtering at 95% precision and 95% recall on slop sounds excellent, and it still leaves you with roughly 10 invalid reports per real one. Human review cost is roughly linear in the number of reports that survive the filter, while the expected value stays flat at one bug.

The only way out is to make the expensive step — a human reading carefully — reachable only after something has been proven mechanically. Not "the report looks good," but "this command exited non-zero on the vulnerable version and zero on the patched one."

A Four-Stage Triage Pipeline for Node.js Projects

Here is the shape I use now. Each stage is cheaper than the one after it, and every stage must be able to terminate a submission without human involvement.

Stage 1: Intake contract and provenance signals

Require an exact package@version, a Node major version, a runnable entry file, and both an expected and observed string. Reject semver ranges outright — "affected versions: all" is the single most common way an automated report makes itself unverifiable.

I do ask for generator provenance, but I do not gate on it. Self-reported provenance is inaccurate in both directions, and rejecting a report because it says "AI" throws away a true positive for no technical reason. Use it as a ranking feature, not a verdict.

Stage 2: Cheap deterministic reproduction gates

This is the whole pipeline. If you build only one stage, build this one. It converts "claims about behavior" into "exit codes," which is the only thing you can batch-process without a human.

Stage 3: Severity normalization and deduplication

Collapse the survivors by the file and symbol named in the repro, not by the reporter's prose. Two reports that both exit non-zero in parseHeader() are one root cause, even if one calls itself critical and the other calls itself medium. Normalize severity from the observed effect (process.exit(1) from a crash is a DoS at best; attacker-controlled child_process execution is worse), not from the reporter's label.

Stage 4: Routing a limited human review budget

Rank by reproduction strength: fixed-upstream first, then crashes with an attacker-controlled input, then correctness bugs. Then hard-cap the number of reports a human sees per week. The cap is the point — it is what converts an unbounded flood into a predictable workload.

Implementing the Reproduction Gate in Node.js

The intake contract: submission schema and required fields

I use zod for the intake contract because the error messages double as the rejection reason sent back to the reporter.

import { z } from "zod";

export const Submission = z.object({
pkg: z.string().min(1),
version: z.string().regex(/^d+.d+.d+$/, "pin an exact version, not a range"),
node: z.string().regex(/^vd+./, "expected v22.x style"),
expected: z.string().min(1),
observed: z.string().min(1),
repro: z.string().min(1),          // path to a runnable entry file
impact: z.enum(["rce", "dos", "data-integrity", "info-disclosure", "unknown"]),
provenance: z.object({
  generator: z.string().default("unknown"),
  model: z.string().optional(),
  transcript: z.string().optional(),
}).default({ generator: "unknown" }),
});

export function validate(raw) {
const parsed = Submission.safeParse(raw);
if (parsed.success) return { ok: true, value: parsed.data };
return {
  ok: false,
  reason: parsed.error.issues.map((i) => `${i.path.join(".")}: ${i.message}`).join("; "),
};
}

Sandboxed install, build, and PoC execution with a time cap

import { spawn } from "node:child_process";




const CAP_MS = 90_000;

function run(cmd, args, cwd) {
return new Promise((resolve) => {
  const child = spawn(cmd, args, {
    cwd,
    timeout: CAP_MS,
    stdio: ["ignore", "pipe", "pipe"],
    // trimmed env: no tokens, no proxy creds, scripts disabled
    env: { PATH: process.env.PATH, npm_config_ignore_scripts: "true" },
  });
  let out = "", err = "";
  child.stdout.on("data", (d) => (out += d));
  child.stderr.on("data", (d) => (err += d));
  child.on("close", (code, signal) =>
    resolve({ code, signal, out: out.slice(-4000), err: err.slice(-4000) }));
});
}

export async function reproGate(sub) {
const dir = await mkdtemp(path.join(tmpdir(), "repro-"));
await writeFile(path.join(dir, "package.json"),
  JSON.stringify({ name: "repro", private: true, type: "module" }));
await cp(sub.repro, path.join(dir, "repro.mjs"));

const install = await run("npm", ["install", `${sub.pkg}@${sub.version}`,
  "--no-audit", "--no-fund"], dir);
if (install.code !== 0) return { verdict: "unreproducible", reason: "install-failed" };

const reported = await run("node", ["repro.mjs"], dir);

// second pass: does the same repro pass on the newest published version?
await run("npm", ["install", `${sub.pkg}@latest`, "--no-audit", "--no-fund"], dir);
const latest = await run("node", ["repro.mjs"], dir);

const reproduces = reported.code !== 0;
return {
  verdict: reproduces
    ? (latest.code === 0 ? "reproduced-fixed-upstream" : "reproduced")
    : "not-reproduced",
  exit: { reported: reported.code, latest: latest.code, signal: reported.signal },
  tail: reported.err.trim().split("\n").slice(-3).join("\n"),
};
}

The second pass matters more than it looks. A repro that fails on the reported version and on latest is usually a broken test rather than a vulnerability. A repro that fails on 4.2.1 and passes on 4.2.2 is close to a confirmed bug with a fix already published.

Observed output on a batch of 200 sample reports

To be explicit: this is a synthetic batch of 200 reports I assembled myself, patterned on the failure modes described above. I have not run this against a real flood from a public program, so treat these numbers as a smoke test of the pipeline, not as field data.

Environment: Ubuntu 24.04, Node v22.11.0, npm 10.9.0, 4 vCPU / 8 GB, egress restricted to the registry, 90 s cap per report.

$ node triage.mjs --batch fixtures/batch-200.json
intake        200 submitted | 141 rejected
              missing repro 88 · range not pinned 31 · no observed output 22
repro-gate     59 entered   | 12 reproduced (4 hit the 90s cap)
dedupe         12           | 5 distinct root causes
severity        2 security · 3 correctness
human           5 routed    | 2 confirmed, 1 duplicate, 2 not-a-bug
wall clock     18m 40s total, 4 vCPU

The last line is the one I care about: five reports reached a human, and the human review budget for the week was untouched. Everything before that was automation deciding, with exit codes, that a submission had not earned human attention.

Keeping the Real Vulnerability Report Alive

A pipeline that suppresses noise can also suppress the one report that matters. Two mechanisms keep that from happening.

Fast lanes, appeals, and reporter reputation tracking

Fast lane: any submission whose repro reproduces on the reported version and exits cleanly on latest, or that ships a failing test against the project's own test suite, skips the queue and pages a human. That is a narrow, mechanical condition, which is why it works.

Appeals: a rejection at stage 1 or 2 must be reversible by a human without re-arguing the whole report. Store the raw submission and the gate output; an appeal is "here is my corrected repro file," not "please reconsider."

Reputation: weight reporters by verified reports, and decay that weight over time. It is a soft signal and it is gameable, but it beats treating a first-time reporter and a serial mass-submitter identically. Rate-limit accounts with a high invalid-to-valid ratio instead of banning them — a ban on a shared or automated account hides the exact behavior you want to measure.

Disclosure policy wording that survives the flood

Policy is not a technical control, but it stops arguments. The wording I use now says, in effect:

• A report is actionable when it includes an exact version and a runnable reproduction. Everything else is a bug report, not a vulnerability report.
• Response timelines apply to actionable reports only. This is the clause that protects your team.
• Automated bulk submission without a runnable reproduction will be closed without individual response.

The third clause is the uncomfortable one, and I think it is correct. A maintainer's obligation is to the users of the software, not to the volume of inbound text.

Further Reading

• IT Pro: Google pauses open source bug bounty scheme over AI slop submissions
• NewsBytes: IBM and Red Hat launch $5B Project Lightwell securing open-source
• It's FOSS: DigitalOcean Quietly Ends Open Source Credits Program

Conclusion: Budget for Triage, Not Just Detection

Google's pause is not a story about AI writing bad bug reports. It is a story about a system where the cost of producing a report dropped to near zero while the cost of evaluating one stayed constant. That is a capacity problem, and capacity problems are solved by changing the cost curve, not by asking producers to be more considerate.

My position: if your project accepts external reports, the reproduction gate is no longer optional infrastructure. It is the thing that lets you keep accepting reports at all. Build the gate, cap the human budget, and fund the time it takes to run both — because a $5B industry effort and a discontinued credits program are pointing at the same missing line item.