Why Lockfile Pinning Alone Won't Catch Hallucinated Imports from Coding Agents

Why Lockfile Pinning Alone Won't Catch Hallucinated Imports from Coding Agents

pr0h0••23 min read•ai
AI Usage (85%)
On this page

Introduction — the build that broke even though the lockfile was clean

A coding agent adds a handler and one line to the diff. The import points at a package that was never published, but CI installs exactly what package-lock.json pins, every integrity hash matches, and the failure only surfaces in the file that uses it. This post explains why lockfile pinning alone cannot catch hallucinated imports from coding agents, sorts those bad imports into four shapes that each need a different control, and shows a registry probe plus a CI gate you can rebuild in about forty lines of JavaScript.

Here is the shape of that diff:



That third import looks fine. The name follows npm conventions, it's unscoped the way most utility packages are, and the agent used it with an API that reads plausibly. CI runs:

npm ci --ignore-scripts

and exits 0. 412 packages installed, every integrity hash matched. Nothing in package-lock.json changed, because nothing needed to change — the import lives in src/, and npm ci never opens src/.

The build dies one step later, in the compiler or the bundler:

src/report.ts:3:30 - error TS2307: Cannot find module 'node-json-canonicalize-stream'
or its corresponding type declarations.

The framing that usually follows — "the lockfile was clean, so the dependency was fine" — is a category error. A lockfile is an integrity record for packages that were already declared. It has never been a statement about whether the names your source imports are real.

The trigger for this post is a HackerNoon piece published 2026-11-19, Your AI Coding Assistant Is Writing Code for Libraries That No Longer Exist, which frames agent-generated imports of dead or nonexistent libraries as a dependency-validation problem. That framing is right. The fix it implies — pin harder — is not. Pinning is the control you already have, and it is orthogonal to this failure.

My position: treat "does this name exist, and should it be here?" as a separate gate from "are these bytes the bytes we agreed on?" The second question is solved and boring. The first is unanswered in most repos, cheap to implement, and it still only catches about half of the ways an agent can get an import wrong. I'll also argue that the highest-value single control isn't a registry probe at all — it's a type check against a strict resolver. A probe can never catch an API that doesn't exist inside a package that does.

What lockfile pinning actually guarantees in npm

How npm resolves, writes, and replays a lockfile

Three commands, three different jobs, and people conflate them constantly.

npm install resolves the graph: it walks package.json, asks the registry for version metadata, picks versions satisfying the semver ranges, checks the cache, and then writes the result into package-lock.json. Resolution and record-writing are one step.

npm ci replays it. Per the npm docs it requires an existing lockfile, removes node_modules before installing, and installs exactly the versions in the lockfile. No range resolution, no upgrades. If package.json and the lockfile disagree it refuses outright:

npm error `npm ci` can only install packages when your package.json and
npm error package-lock.json or npm-shrinkwrap.json are in sync. Please update
npm error your lock file with `npm install` before continuing.

The lockfile itself (v3 format, the packages map) is keyed by install path. A trimmed entry, with illustrative values:

{
  "lockfileVersion": 3,
  "packages": {
    "node_modules/example-utils": {
      "version": "2.4.1",
      "resolved": "https://registry.npmjs.org/example-utils/-/example-utils-2.4.1.tgz",
      "integrity": "sha512-…",
      "license": "MIT",
      "dependencies": { "example-core": "^1.2.0" }
    }
  }
}

Here's what's in that entry: a version, a tarball URL, a hash, and the package's own declared dependencies. Here's what isn't: anything derived from your source code. The lockfile has no concept of an import. It records what npm decided to fetch, not what your files reference.

What the integrity hash does and does not protect

The integrity field is an SRI hash over the tarball bytes. It answers exactly one question: are these the same bytes that were fetched when the lockfile was written?

That's worth having. It stops silent tarball mutation, makes installs reproducible, and alongside a committed lockfile it pins the whole transitive tree. It is also completely silent on:

  • whether the package belongs in your dependency tree at all;
  • whether the published code matches the source repository;
  • whether the pinned version is deprecated, abandoned, or hijacked;
  • whether the API you're importing exists.

The hash is computed over whatever got uploaded. Publish something malicious and pin its hash, and the hash is correct. Integrity isn't authenticity, and neither is legitimacy.

The one question npm ci can never answer

Does every bare specifier in my source tree resolve to a dependency I declared on purpose?

It can't answer that, because it doesn't open source files. It reads two manifests and a lockfile, then fetches. A hallucinated import in src/ is invisible to it — and invisible to a perfect lockfile too, which is the part that bites. No amount of pinning converts a source-level name into a manifest-level fact.

npm's hoisted node_modules layout makes the question harder still. A bare specifier resolves if the package exists anywhere in the flattened tree, including as a transitive dependency of something unrelated. So a hallucinated-looking import sometimes does resolve, and the failure moves from "build broken" to "works locally, breaks after someone bumps a parent dependency." More on that in the lockfile-diff section.

The four shapes of a hallucinated import from a coding agent

These aren't one bug, and they don't share a fix. Conflating them is how "we pin our deps" ends up offered as a response to all four.

Shape 1 — the package name that never existed anywhere

Simplest case: the model emits a name that was never registered. node-json-canonicalize-stream in the intro is one I made up for this post — deliberately plausible the way model output is plausible: a real builtin prefix, a real concept (JSON canonicalisation), a real packaging convention (-stream).

Detection is trivial and deterministic. The registry returns 404. This is the shape a registry probe was built for, and the only one it catches with zero false positives.

Shape 2 — the deprecated, renamed, or archived package

The name exists, resolves, installs, hash matches. It's also dead. The model learned the ecosystem's older idioms and reproduces them with confidence.

Examples of the class: request carries a long-standing deprecation notice pointing at its own repository issue; tslint was deprecated in favour of typescript-eslint; node-sass in favour of sass. Read the exact deprecation text and the date each flag was set live from the registry rather than trusting a blog post — including this one.

The mechanic that matters: deprecation is a per-version field, not a package-level property. A package can carry deprecated on v0.x while dist-tags.latest points at a healthy v3. Check the field on the version you're actually installing, not on the name.

Shape 3 — the real package with the wrong API surface

This is the expensive one, and no registry check will ever catch it.

The package is real, current, maintained. The model gets the entry point or export shape wrong. Concrete pattern: csv-parse moved its synchronous API from csv-parse/lib/sync to csv-parse/sync in the v5 line. A model trained across both eras will hand you the old subpath — a resolution error on a v5 install, and something worse on a mixed one.

Which is why I think the dependency gate is necessary and insufficient. The only control that reliably catches this shape compiles against the real type surface:

npx tsc --noEmit

Not a clever insight — that's the point. The cheapest effective hallucination detector in a TypeScript repo is one you already have, and teams that skip it in CI for speed are skipping the check that catches the largest class by frequency.

Shape 4 — the plausible name that belongs to someone else

The name exists. It isn't deprecated. It has a repository field and a version. It also isn't what the model thought it was: a thin wrapper published by an unrelated account, or a name that was free until recently and got registered, or a near-miss of a popular package.

There's a nasty dynamic that pushes agents into this shape. When an agent proposes a name and the install 404s, the loop's next move is often to "fix" it by substituting a similar name that does resolve. Existence checking by trial-and-error at the keyboard manufactures shape 4 out of shape 1.

Detection here is heuristic, never deterministic. Signals: age, maintainer count, repository liveness, download volume, provenance. Every one of them has a false-negative story. More below.

ShapeExists in registryCaught by lockfile diffCaught by tsc --noEmitCaught by registry probe
1 · never existednonoyesyes
2 · deprecatedyesnonoyes (per-version flag)
3 · wrong API surfaceyesnousually (if typed)no
4 · someone else's nameyesnonoheuristic only

Why a lockfile diff is not a dependency gate

Regenerating the lockfile is a one-command bypass

npm install <name> writes package.json and package-lock.json in a single operation. An agent with shell access will run it, and the diff it produces is internally consistent: manifest and lockfile agree, npm ci passes, and the only human-visible evidence is a few hundred lines of JSON.

There's no mechanism by which a lockfile objects to a package being added. It has no policy surface. A "gate" that consists of look at the lockfile during review is a social convention, not a control — and social conventions don't survive an agent that runs npm install twelve times a session.

Review fatigue inside a 4,000-line lockfile diff

Adding one package with a normal transitive tree usually lands anywhere from a few hundred to a couple thousand lines of lockfile diff, because npm rewrites the whole resolved tree. Nobody reads it. Reviewers scroll to the bottom, decide nothing looks weird, and move on — exactly the depth of review a squat with a plausible name survives.

The defensible review surface is the two-line package.json diff. That's the file where a human can apply judgement, and it's the file most CI policies ignore because "the lockfile is the source of truth." The lockfile is the artifact. It isn't the decision.

The cases where the lockfile never changes at all

Three situations where the lockfile diff is empty and the build is still wrong:

  1. Phantom imports of transitive deps. The agent imports a package already in the tree as a dependency of something else. npm's hoisted layout resolves it, the lockfile is unchanged, the code works today. When the parent drops the dependency or hoisting order shifts, it breaks — with no diff to blame.

  2. Shape 3. The package is already declared; only the import specifier is wrong. Zero lockfile movement.

  3. Workspace and file:/link: dependencies. In a monorepo an import may resolve through a workspace path the lockfile records as a link, so renaming a package inside the workspace produces no lockfile delta worth reviewing.

A registry existence check that fits in CI

The goal is a gate that runs in seconds, needs no auth, and blocks only on facts. Four steps.

Step 1 — extract bare specifiers from the source tree

A regex over import/require/import() is crude but adequate, provided you're explicit about what it misses — dynamic specifiers, computed requires. About 40 lines, no dependencies:

scripts/import-audit.mjs
import { builtinModules } from "node:module";



const SOURCE_EXT = new Set([".js", ".jsx", ".mjs", ".cjs", ".ts", ".tsx"]);
const SPECIFIER = /(?:froms*|imports*(s*|requires*(s*)["']([^"']+)["']/g;

async function* walk(dir) {
for (const entry of await readdir(dir, { withFileTypes: true })) {
  if (entry.name === "node_modules" || entry.name.startsWith(".")) continue;
  const full = path.join(dir, entry.name);
  if (entry.isDirectory()) yield* walk(full);
  else if (SOURCE_EXT.has(path.extname(entry.name))) yield full;
}
}

const builtins = new Set(builtinModules.flatMap((m) => [m, "node:" + m]));

function packageName(spec) {
const parts = spec.split("/");
return spec.startsWith("@") ? parts.slice(0, 2).join("/") : parts[0];
}

const found = new Map();
for await (const file of walk(process.argv[2] ?? "src")) {
const text = await readFile(file, "utf8");
for (const match of text.matchAll(SPECIFIER)) {
  const spec = match[1];
  if (spec.startsWith(".") || builtins.has(spec)) continue;
  const name = packageName(spec);
  if (!found.has(name)) found.set(name, new Set());
  found.get(name).add(path.relative(process.cwd(), file));
}
}

const pkg = JSON.parse(await readFile("package.json", "utf8"));
const declared = new Set();
for (const field of ["dependencies", "devDependencies", "peerDependencies", "optionalDependencies"]) {
for (const name of Object.keys(pkg[field] ?? {})) declared.add(name);
}

const undeclared = [...found.keys()].filter((n) => !declared.has(n)).sort();
console.log(JSON.stringify({ imported: found.size, undeclared }, null, 2));

The scoped-package handling is the part that matters: @scope/name/sub/path has to reduce to @scope/name, not @scope. Get it wrong and you get a wall of false positives on any repo using scoped tooling packages.

Step 2 — diff declared dependencies against observed imports

On a fixture with two files, one declared dependency (csv-parse), and imports of node:fs, path, ./util.js, csv-parse/sync, and node-json-canonicalize-stream:

node scripts/import-audit.mjs src
{
  "imported": 2,
  "undeclared": ["node-json-canonicalize-stream"]
}

That's the whole first half of the gate: a name used in source that appears in no manifest. It doesn't tell you whether the name exists, and it doesn't tell you whether the other imports are correct. Step 2 produces candidates, not verdicts.

Two edge cases worth handling up front rather than discovering in production: a monorepo package importing itself by name shows up as undeclared (allowlist workspace names), and a package legitimately provided by the platform (@types/node, edge-runtime globals) may need the same treatment.

Step 3 — query the registry for existence, age, and deprecation

For each candidate, fetch the packument — the full document, not the abbreviated installer variant, because the trust fields we need (time, maintainers, per-version deprecated, dist.attestations) only live there.

scripts/registry-probe.mjs
export async function probe(name) {
const url = "https://registry.npmjs.org/" + name.replace("/", "%2f");
const res = await fetch(url, { headers: { accept: "application/json" } });

if (res.status === 404) return { name, exists: false };
if (!res.ok) return { name, error: "registry " + res.status };

const doc = await res.json();
const latest = doc["dist-tags"]?.latest;
const version = latest ? doc.versions?.[latest] : undefined;

return {
  name,
  exists: true,
  latest: latest ?? null,
  firstPublish: doc.time?.created ?? null,
  lastPublish: doc.time?.modified ?? null,
  maintainers: doc.maintainers?.length ?? 0,
  deprecated: version?.deprecated ?? null,
  repository: version?.repository?.url ?? null,
  provenance: Boolean(version?.dist?.attestations),
};
}

The 404 branch is the only one that yields a hard block. Everything else is evidence for a human or a scoring function.

Step 4 — score new packages before they land

Score only packages that are new relative to the base branch, never the whole tree. A repo with 400 legacy dependencies has 400 sets of reasons to complain, and a gate that complains about all of them gets muted in week one.

scripts/score.mjs
const DAY = 86_400_000;

export function score(pkg) {
if (!pkg.exists) {
  return { verdict: "block", points: null, reasons: ["name not in registry"] };
}

const reasons = [];
let points = 0;

if (pkg.firstPublish) {
  const ageDays = (Date.now() - Date.parse(pkg.firstPublish)) / DAY;
  if (ageDays < 30) { points += 3; reasons.push("first published " + Math.round(ageDays) + "d ago"); }
}
if (pkg.deprecated) { points += 2; reasons.push("latest version deprecated: " + pkg.deprecated); }
if (!pkg.repository) { points += 2; reasons.push("no repository field"); }
if (pkg.maintainers <= 1) { points += 1; reasons.push("single maintainer"); }
if (!pkg.provenance) { points += 1; reasons.push("no provenance attestation"); }

return { verdict: points >= 4 ? "review" : "allow", points, reasons };
}

Run against hand-written fixture objects (fabricated inputs, so you can reproduce them exactly):

inputverdictpointsreasons
{ exists: false }block—name not in registry
12-day-old package, 1 maintainer, repo present, no provenancereview4first published 12d ago; single maintainer; no provenance attestation
6-year-old package, 3 maintainers, repo present, no provenanceallow1no provenance attestation
package whose latest version carries a deprecated stringreview3latest version deprecated: …; no repository field

The thresholds are arbitrary; tune them to your repo. The structure isn't. block is reserved for the one deterministic fact, and everything heuristic lands in review. That separation is what keeps people trusting the output.

Reading the registry signals correctly

First-publish date versus last-publish date

time.created is when the name was first registered. time.modified moves on every publish and every metadata change. The registry sets both, so a publisher can't backdate them.

A young time.created is a meaningful signal, since most squatting and typo-registration happens on fresh names. But it's asymmetric — a package first published in 2014 isn't thereby trustworthy. Account takeover of long-dormant packages is a known attack pattern, and in that case time.created looks excellent while the tarball is brand new. If you read only one date, read time.created for the name and compare it against the version's publish time in time[version]. A decade-old package with a two-day-old latest version deserves the same scrutiny as a new name.

maintainers is an array of { name, email }. Zero or one is a signal, not a verdict — plenty of genuinely good small utilities have a single maintainer, and most of your tree would fail a strict threshold.

Repository liveness is worth checking because dead links correlate with abandoned packages, but don't over-read a live one. The repository field is self-declared metadata, not a build provenance claim. A live GitHub repo doesn't prove the published tarball came from it. Only provenance attestations make that connection.

Deprecation flags and dist-tag drift

deprecated is free text set by maintainers, and it lives on the version object. Check it on the version you're about to install. A probe that only reads dist-tags.latest will miss a deprecated pinned version — precisely the case an agent trained on old code produces.

Dist-tag drift is the related trap. latest is a pointer maintainers can move, and it can point at an older line (a maintained v1 while v2 sits under next). A gate that reports dist-tags.latest as "the version" can report the wrong one. Compare against the version actually being added.

Provenance attestations and signature verification

When a package is published from a public repository through a supported CI provider with --provenance, the version object carries dist.attestations, and consumers can verify the link between tarball, source commit, and build (npm provenance docs). The client can check it:

npm audit signatures

Strongest signal on the list, and the least available — a large share of the ecosystem still publishes without it. Absence of provenance is therefore close to zero information, which is why it contributes one point and never a block.

⚠️

Do not let the probe become a dependency of your install step. Registry calls from CI are rate-limited and unauthenticated; cache results keyed by name plus version, and treat a registry error as review, not as pass.

Where these signals lie to you

  • Age says nothing about a hijacked account.
  • Maintainer count punishes healthy single-maintainer utilities and rewards orgs that never review their own publishes.
  • Repository link is self-declared and can point at an unrelated, legitimate-looking repo.
  • Provenance proves build origin, not intent. A malicious maintainer can build malicious code with provenance from their own repository.
  • Download counts are gameable, and mostly measure transitive usage rather than quality.

Every signal here is a prior, not a proof. The gate's job is to make a human look at the two or three new names a month that deserve it. Not to make the decision.

Wiring the gate into the pipeline

Gate the dependency delta, not the pull request

Compute the set difference between package.json on the base branch and on the head branch, and probe that set only. In a typical repo that's zero to three names per PR. It runs in about a second and — the useful part — it produces the same result whether the agent added the dependency via npm install, via a manual manifest edit, or via a lockfile-only change.

Cover the lockfile in the delta too, to catch packages added without ever appearing in the manifest. Rare under npm ci, but it's one line to check.

Blocking mode versus advisory mode

Start advisory. Run the probe in report-only mode for a sprint, collect the verdicts, count how many review results a human would have had to clear. Then promote exactly one rule to blocking — name not in registry — and leave the score advisory indefinitely.

Blocking on a heuristic is how a security gate gets disabled by the first person who needs to ship on a Friday. Blocking on "this name returns 404" is a rule nobody argues with, because there's no legitimate argument against it.

Setting an honest false-positive budget

Pick a number before you ship: at most two advisory review results per week per team, say. If the gate exceeds it, the thresholds are wrong, not the team. Track two numbers — block rate and review clearance rate — and be honest that a low block rate can mean "agents rarely hallucinate" or "the extraction step misses the imports." Same metric, different causes.

💪

Run the extraction step over the whole tree once, on a branch where you know the audit output is clean. That baseline tells you how much noise your monorepo conventions generate before you turn the gate on for anyone else.

Running the check on your own repository

The extraction script and what its output looks like

The fixture transcript above is exactly what scripts/import-audit.mjs prints on the two-file tree, and you can rebuild it in about a minute: a package.json declaring one dependency, and a source file with five import statements, one of them fabricated. Two non-relative specifiers survive filtering, one of them undeclared.

A real, reproducible result in the weak sense — deterministic given the fixture, and reproducible by you without registry access.

The registry probe and what fields to read

The probe's output depends on live registry state, so I won't print values here that I can't verify. Read these fields in this order, and record them:

FieldPathUse
existenceHTTP status of /namethe only hard block
latest versiondist-tags.latestanchor for per-version fields
first publishtime.createdsquat prior
latest publishtime[latestVersion]takeover prior
maintainersmaintainers.lengthabandonment prior
deprecationversions[latest].deprecateddeprecation block candidate
repositoryversions[latest].repository.urlliveness check only
provenanceversions[latest].dist.attestationsstrongest, scarcest signal

What the report claims versus what I could verify here

Keeping these apart is the point of the exercise.

Confirmed by primary sources. npm ci installs from the lockfile and refuses to run when the manifest and lockfile are out of sync — stated in the npm CLI docs. The lockfile records versions, resolved URLs, integrity hashes, and per-package dependency lists; it contains nothing derived from source files — visible in any lockfile. Integrity hashes are computed over tarball contents — that is what SRI is. Provenance attestations exist as a dist.attestations field and can be verified with npm audit signatures — documented by npm.

Confirmed by running the fixture. The extraction script identifies a bare specifier that appears in no manifest, and reduces csv-parse/sync to csv-parse correctly. That much is mechanical.

Reported, not independently verified by me. The HackerNoon article describes agents generating imports for libraries that no longer exist or are deprecated. As far as I can see, the excerpt doesn't report a sampled failure rate across a defined corpus — so any percentage figure you see quoted from it should be treated as unmeasured. I have no base-rate data of my own for how often hallucinated imports occur per thousand generated imports. That number matters for deciding how much effort the gate deserves, and it should be measured on your own agent logs rather than borrowed.

Inference, explicitly marked as such. My ranking of the four shapes by frequency — I believe shape 3 is the most common and shape 1 the loudest, because shape 3 is the only one that doesn't announce itself. That ordering comes from debugging agent-generated code, not from a measurement. It's also why I weight tsc --noEmit above the registry probe in the checklist below, and it's falsifiable: instrument your own repo, and if shape 1 dominates, the probe deserves the higher weight.

Defense Checklist

ControlCatchesCostNotes
npm ci in CI, never npm installmanifest/lockfile driftfreealready standard
tsc --noEmit on every PRshapes 1 and much of 3lowhighest value per second
Strict linker (pnpm / Yarn PnP)phantom imports of transitive depslowremoves the works-locally case
Registry delta probe, blocking on 404shape 1lowdeterministic only
Deprecation check on the pinned versionshape 2lowper-version field
Score new names, advisoryshape 4 priorslownever block on heuristics
Review the package.json diff, not the lockfileall shapes, weaklyfreehuman judgement where it applies
npm audit signaturestampered/unattested publisheslowcoverage is partial
--ignore-scripts in CIpostinstall abusefreebaseline hygiene
Agent proposes deps, does not install themthe whole npm install bypassprocess changethe real fix

That last row changes the shape of the problem. An agent that can run npm install has write access to your dependency tree. An agent that has to emit a proposal does not. Everything above it is detection. That row is prevention.

What This Approach Still Misses

Shape 3, mostly. A registry probe answers "is this name real and healthy?" It cannot answer "does client.messages.createStream exist in version 4.2.1?" Only compiling or type-checking against the installed package does that, and only if the package ships types. For untyped packages the check degrades to "it resolved."

Dynamic and computed specifiers. require(someVar) and template-built import paths are invisible to the regex extractor. They're also where an agent can smuggle a name past the gate entirely.

Transitive drift. A real, well-maintained package can add a malicious or broken dependency in a patch release. Gating the direct dependency delta doesn't see it. That belongs to lockfile review and provenance, and it isn't solved here.

Correctness as distinct from existence. An agent can import a real, healthy, correctly-versioned package that is simply the wrong tool — a duplicate of an internal utility, or a heavyweight library where a standard-library call would do. Existence checks say nothing about intent.

Provenance coverage. Because most of the ecosystem has no attestation, the absence signal is weak and will stay weak for a while. Don't build a block rule on it.

The unmeasured base rate. Until someone publishes a count — imports checked, imports hallucinated, by shape, per model and version — the ROI of this gate is an argument rather than a number. That's the most useful thing a team running coding agents at scale could contribute here, and it's cheap to collect: log the extraction output, log the verdicts, log what happened next.

Further Reading

Share this post

More posts

Loading comments…