
How a Release-Pipeline Certificate Let Credential-Stealing Wheels Ship as Microsoft Agent Framework
On 2026-11-19, several outlets reported that malicious versions of Microsoft's Agent Framework had been published to PyPI, and that the payload was a credential stealer. The stealer is the boring part. The interesting part is what Wiz reportedly found: the compromise was tied to a release-pipeline certificate. This post walks through what that certificate does and does not prove, how a credential-stealing wheel ships under a valid signature, how to check whether your environment resolved an affected version, and which CI/CD controls actually close the gap.
That reframes the incident. A signed attestation saying "this artifact was built and published by this workflow, in this repository, on this runner" is a useful statement. It is also, on its own, a statement about identity, not about intent. When those two get conflated — and the tooling encourages the conflation — you end up with a pipeline that proves exactly which workflow shipped the malware.
My position up front: attestation has quietly become a compliance checkbox instead of a detection layer, because almost nobody verifies it at install time and almost nobody constrains what the build runner can reach. A signed artifact from a compromised runner is more dangerous than an unsigned artifact from a clean one, because the signature defeats the one automated check most teams have.
On this page
What the Microsoft Agent Framework Incident Record Establishes
I want to be pedantic here, because supply-chain incidents generate a lot of confabulated detail. Everything below is drawn from the secondary reporting listed in Further Reading; I did not have the primary Wiz write-up or a PyPI advisory in front of me when writing this.
Confirmed facts from the Wiz findings and outlet reporting
| Claim | Source | Confidence |
|---|---|---|
| Malicious versions of Microsoft's Agent Framework were published on PyPI | Help Net Security, TechRepublic, SiliconANGLE, BigGo Finance (all dated 2026-11-19) | Reported by multiple outlets |
| The injected package was a credential stealer, framed as a cloud-takeover risk | Same reporting | Reported |
| The affected version set is small enough that TechRepublic's headline tells readers to "check these two Python package versions" | TechRepublic headline framing | Reported, but I have not seen the version list myself |
| Wiz tied the compromise to a release-pipeline certificate | SiliconANGLE (headline), echoed in the Wiz findings | Reported |
| The incident is being treated as a software supply-chain problem rather than a single bad upload | SiliconANGLE | Reported |
What is inference and what is still unknown about the certificate
These are the questions I could not answer from the material I had, and I am not going to paper over them:
- Whether the certificate was abused or accurate. "Tied to a release-pipeline certificate" admits at least two readings. Reading A: the legitimate workflow ran, and the malicious content was added during the build, so the certificate honestly attests to a build that produced a trojaned wheel. Reading B: the certificate or its signing material was obtained by the attacker and used to sign an artifact the pipeline never produced. Both are plausible; the reporting I saw does not settle it. My working assumption is Reading A, because it requires no cryptographic break and is the more common failure mode — but that is inference, not fact.
- Which CI system and OIDC issuer were involved. Unknown to me.
- Trusted Publishing versus a long-lived API token. Unknown to me. This matters a lot for the postmortem: token-based publishing means credential theft, OIDC-based publishing means runner compromise.
- Dwell time, exfiltration destination, and whether any downstream credential was actually used. Unknown to me. "Raises cloud takeover fears" is a risk framing, not evidence of a takeover.
- The exact removal and quarantine timeline. Unknown to me.
Why I will not name version numbers I cannot verify
Two failure modes come from a wrong version string in an incident post, and both are worse than saying nothing.
First, false reassurance. A team greps their lockfile for 1.2.3, finds 1.2.4, concludes they are clean, and skips rotation. If the affected range was actually wider, the post caused the miss.
Second, wasted rotation. A team pins away from a version that was never affected, changes a dependency graph under incident pressure, and introduces a real regression during a fake emergency.
So: get the version list from the vendor advisory or from PyPI's own project history, not from me. The commands below take a version set as input rather than hardcoding one, which is the correct shape for a runbook anyway.
How PyPI Binds a Publisher to a Package with OIDC and Attestations
To reason about what a pipeline certificate proves, you need to know what PyPI actually records.
Long-lived API tokens versus Trusted Publishing over OIDC
The older model is a project-scoped API token: a bearer secret stored in CI, valid until someone rotates it. Its failure mode is boring and total. Leak it and the attacker can upload anything, forever, from anywhere, and the upload record shows nothing more interesting than "token authenticated."
Trusted Publishing replaces the bearer secret with a short-lived OIDC exchange. The workflow asks its CI provider for an identity token, PyPI validates that token against a pre-registered publisher configuration, and the upload is authorized for a matter of minutes.
name: publish
on:
push:
tags: ["v*"]
permissions: {}
jobs:
build:
runs-on: ubuntu-24.04
permissions:
contents: read
steps:
- uses: actions/checkout@<pinned-sha>
- run: python -m build
- uses: actions/upload-artifact@<pinned-sha>
with:
name: dist
path: dist/
publish:
needs: build
runs-on: ubuntu-24.04
environment: pypi-release
permissions:
id-token: write # the only privilege this job needs
contents: read
steps:
- uses: actions/download-artifact@<pinned-sha>
with:
name: dist
path: dist/
- uses: pypa/gh-action-pypi-publish@<pinned-sha>Two details in that file are load-bearing and routinely dropped.
permissions: {} at the top, with id-token: write granted only to the publish job, means the build job cannot mint a PyPI identity token. If the build step is compromised, it has no publish path unless it can traverse to the other job.
environment: pypi-release is what turns the OIDC subject from repo:org/repo:ref:refs/tags/v1.2.3 into something that includes an environment claim. Environments can require human approval and can be restricted to specific branches and tags. That is the difference between "anyone who can push a tag can publish" and "anyone who can push a tag can request a publish."
What a release-pipeline certificate actually attests to
The OIDC token the workflow receives contains a set of claims. PyPI validates them, and the attestation carries a Sigstore bundle binding those claims to the artifact digest.
| OIDC claim | What it pins down |
|---|---|
iss | Which CI provider minted the identity (https://token.actions.githubusercontent.com for GitHub Actions) |
sub | Repository + ref or environment — the specific workflow context |
repository / repository_owner | Which repo |
workflow_ref / job_workflow_ref | Which workflow file, and whether it came from a reusable workflow |
environment | Whether a protected environment gated the run |
sha | The commit the run was started from |
The certificate is short-lived (Fulcio-issued, minutes). The signature over the artifact's hash is what persists. In principle, anyone holding the wheel can compute its SHA-256, fetch the provenance, and check that the identity that signed that digest is the identity they expect. That closes the "did this come from the real project or a lookalike upload" question.
Three things the certificate does not attest to
1. That the artifact's contents match the source in the repository. The attestation covers the bytes the build produced. If the build environment injected a file, those bytes include the file. PEP 740 attestations are not a reproducibility proof; they are an identity proof.
2. That the build environment was clean. A signed attestation from a runner with an attacker on it is a well-formed statement about a compromised process.
3. That the artifact behaves benignly. No cryptographic primitive in this chain has an opinion about what the code does when imported. The consequence: "the provenance verifies" is a green light for exactly one check, and it is usually the only one a team runs.
Reconstructing the Release-Pipeline Compromise Path
The interesting structural point in the reporting is the shift in where the foothold sits.
The foothold moves from the maintainer laptop to the build runner
Historically, "compromised maintainer" meant an attacker on a laptop with publishing credentials in ~/.pypirc or an environment variable. Defenders responded with 2FA, then with scoped tokens, then with Trusted Publishing — all of which push credentials off the laptop.
Trusted Publishing removes the long-lived secret, and in doing so it moves the target. The publish credential is no longer a thing you can steal; it is a thing you can invoke. To invoke it, you need code execution inside the workflow that has id-token: write.
So the attack surface becomes: the runner, the workflow file, the third-party actions it calls, the dependency resolution it performs, and the artifacts it consumes from other jobs. That surface is larger and less-watched than a laptop, because it is ephemeral — nothing persists to be examined afterward, and by default nobody keeps the environment around to look at.
A stolen token leaves a suspicious login from an unfamiliar ASN. A compromised runner leaves a perfectly normal publish from the expected IP range, at the expected time, through the expected workflow. The telemetry is better for the attacker.
Where a malicious wheel can be injected during the build step
Concrete paths, in rough order of how often I see them:
- Unpinned third-party actions.
uses: some/action@v3is a mutable tag. A compromised or vendored action executes in your job with your job's permissions. - Workflow file mutation via the pushed ref. For
pushevents, GitHub uses the workflow file from the ref that was pushed. If publishing triggers onv*tags, a commit that modifies.github/workflows/publish.ymland gets tagged runs the modified workflow. Repository write access plus tag creation is therefore close to publish access. This is a general mechanism, and I am not claiming it is what happened here. pull_request_targetwith a checkout of the PR head. This runs untrusted code in a privileged context — the classic way to hand a stranger your secrets.- Dependency resolution inside the build. A build job that runs
pip install -e .[dev]pulls the whole dev tree. Any one of those packages can execute at install time if it is a source distribution, and any of them can shadow the build backend. - Artifact poisoning between jobs.
download-artifactfetches whatever the upstream job uploaded. If the upstream job is reachable by less-trusted code, the "build" job is not a trust boundary — it is a relay. - Self-hosted runner persistence. Unlike hosted runners, these survive between jobs. A first job that leaves a file behind changes the meaning of "clean runner" for every subsequent job.
Why the attestation stays valid after the payload is added
The mechanics matter, so let me be precise.
The digest that gets signed is computed over the artifact as it exists at upload time. If the malicious file is present when python -m build finishes, it is inside the wheel, and it is inside the hash. The certificate subject is the real repository and the real workflow. The Rekor entry is a real, valid transparency log entry. Every check passes:
$ cosign verify-blob \
--bundle pkg-<version>-py3-none-any.whl.sigstore.json \
--certificate-identity-regexp '^https://github\.com/<org>/<repo>/\.github/workflows/publish\.yml@refs/tags/v.*$' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
./audit/pkg-<version>-py3-none-any.whl
Verified OK
Verified OK means "this identity signed this hash." It says nothing about whether the identity should have produced this hash. Those are different questions, and only the second one matters during an incident.
Verified OK from cosign is a statement about provenance, not safety. Treat a passing verification as a reason to keep looking, not a reason to stop.
Anatomy of a Credential-Stealing Python Wheel: Install Hooks and Exfiltration
Install-time execution points: setup.py, .pth files, and import side effects
A widespread misconception is that installing a wheel runs setup.py. It does not. Wheel installation is unpack-and-record; setup.py and PEP 517 build hooks only run when building from an sdist. The payload has to reach a different execution point.
| Mechanism | Fires when | Notes |
|---|---|---|
setup.py / build backend | sdist build or install | Not executed for wheels |
Top-level .pth in the wheel | Every interpreter start, after install | Lines beginning with import are exec'd by site |
Package __init__.py | First import <package> | Most common trojan location |
| Console entry point | When the user runs the CLI | Needs the user to invoke it |
| Submodule imported by the framework | When the framework lazily loads it | Very likely in an agent framework |
The .pth row deserves attention because it is the cheapest, most under-inspected persistence vector in Python packaging. A .pth file shipped in the wheel's root lands in site-packages, and site processes import-prefixed lines at startup — before your code runs, before any import you wrote, in every Python process in that environment.
You can verify this yourself in about thirty seconds, with a harmless payload:
python -m venv /tmp/lab/venv
SITE=$(/tmp/lab/venv/bin/python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])")
printf 'import os; open("/tmp/lab/pth-ran.txt","a").write(str(os.getpid())+"\n")
' \
> "$SITE/demo.pth"
rm -f /tmp/lab/pth-ran.txt
/tmp/lab/venv/bin/python -c "pass" # imports nothing
/tmp/lab/venv/bin/python -c "import json" # imports a stdlib module
cat /tmp/lab/pth-ran.txtExpected result on CPython 3.12 (the .pth fires once per interpreter start, so two lines):
14872
14899
Two processes, two executions, and neither imported anything from a package named demo. Nothing about it requires the victim to import the malicious library, so "we never call that module" is not a mitigation.
The defense is mechanical: know what .pth files exist in your environments, and expect approximately none that you did not create.
## List .pth files in the active environment's site-packages
python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" | xargs -I{} find {} -maxdepth 1 -name '*.pth' -print
In a clean virtualenv this typically prints nothing or a single line from an editable install. Anything else is worth a look.
What a stealer in an AI framework dependency actually looks for
The targeting profile matters because it tells you what to rotate. An agent framework is installed on developer machines, in notebooks, and increasingly in long-lived agent runtimes — and those environments are full of credentials.
The categories worth assuming were touched:
- Cloud static and session credentials —
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN,AZURE_CLIENT_SECRET,GOOGLE_APPLICATION_CREDENTIALS, plus~/.aws/credentials,~/.azure/,~/.config/gcloud/. - Kubernetes context —
~/.kube/config, service account tokens mounted at/var/run/secrets/kubernetes.io/serviceaccount/token. - CI and SCM tokens —
GITHUB_TOKEN,CI_JOB_TOKEN,NPM_TOKEN,~/.netrc,~/.git-credentials, SSH private keys. - Model provider keys —
OPENAI_API_KEY,ANTHROPIC_API_KEY,HF_TOKEN, and whatever internal gateway token your org uses. These are the ones teams forget to rotate because "it's just a dev key." - Application secrets in the working tree —
.env,*.pem,secrets.yaml,docker-compose.override.yml. - Instance metadata —
http://169.254.169.254/. Worth correcting a common assumption: IMDSv2 raises the bar against off-host SSRF and hop-limit tricks. It does nothing against code executing on the instance, because a PUT to get a session token requires no credentials. If the build runs on an EC2 instance with an attached role, in-process code can mint IMDSv2 tokens all day.
That last point is the one I would emphasize to anyone hardening a build runner: the runner's own cloud identity is the crown jewel, and it is reachable by any code the build executes.
Exfiltration patterns that blend into normal outbound traffic
The detection asymmetry is severe. An outbound POST to a domain registered last week is easy. The realistic patterns are harder:
- POSTing to a well-known host — a paste service, an object storage bucket, a GitHub gist — where the egress allowlist has already been widened for developer convenience.
- Using a model API endpoint as the carrier. An agent framework legitimately talks to
api.<provider>.comon a regular basis. A package that also POSTs a base64 blob to a model provider endpoint is not anomalous against a naive allowlist, and the traffic is TLS-encrypted end to end if you are not doing SNI logging. - DNS exfiltration via TXT or subdomain encoding, which survives most egress proxies because DNS is usually not proxied.
- Writing the payload to a path that a subsequent step in the workflow uploads as an artifact or a release asset.
What actually catches this is not a signature. It is (a) egress default-deny with SNI logging, (b) recording the resolved dependency set at build time and comparing it across builds, and (c) process-level telemetry on the runner showing a Python process making a connection it has no business making.
Why an agent framework is a higher-value target than a utility package
A utility package sits in a library layer and maybe runs in one CI job. An agent framework sits in the layer that holds the keys, on the machines where developers iterate, with network access to model providers and often with a service account attached so it can do something useful.
The blast radius is a function of where it gets installed, not how popular it is. A package installed on 200 developer laptops and 12 agent runtimes holding production cloud roles is a worse compromise than a package installed 50,000 times in build-only contexts. That framing also tells you where to spend the audit effort: find the environments where this dependency resolves, ranked by the credentials present in each.
Checking Whether Your Environment Installed an Affected Version
Take the version set from the advisory, not from this post.
Auditing lockfiles and resolved versions without running pip install
Never install to find out what is installed. Query the lockfile or query the environment.
## Lockfiles first — cheapest and non-destructive
grep -n -A2 'name = "<package>"' uv.lock
grep -n -A3 'name = "<package>"' poetry.lock
grep -rn '<package>' requirements*.txt pyproject.toml
## Live environment, no install, machine-readable
python -m pip list --format=json | jq -r '.[] | select(.name|ascii_downcase=="<package>")'
uv pip list --format json | jq '.[] | select(.name|ascii_downcase=="<package>")'
If you need to confirm what the resolver would pick without touching the environment:
mkdir -p ./audit && python -m pip download \
--no-deps --only-binary=:all: --dest ./audit \
"microsoft-agent-framework==<version-from-advisory>"
Output shape:
Collecting microsoft-agent-framework==<version>
Downloading microsoft_agent_framework-<version>-py3-none-any.whl (48 kB)
Saved ./audit/microsoft_agent_framework-<version>-py3-none-any.whl
Successfully downloaded microsoft-agent-framework
--no-deps matters here: you want the artifact, not a transitive install graph. --only-binary=:all: matters because it prevents an sdist build from executing anything on your machine.
Add --no-cache-dir when auditing. A cache hit can mask which artifact you actually got, and you want the bytes from the index.
Verifying Sigstore attestations against the artifact hash you actually have
The verification step people skip is binding the attestation to their file. Fetching "the provenance" and eyeballing it proves nothing.
WHEEL=./audit/microsoft_agent_framework-<version>-py3-none-any.whl
## 1. Hash what you have locally
sha256sum "$WHEEL"
## -> 3f2c... microsoft_agent_framework-<version>-py3-none-any.whl
## 2. Fetch the index-hosted provenance for that exact file (PEP 740)
curl -s "https://pypi.org/integrity/<project>/<version>/$(basename "$WHEEL")/provenance" \
-o provenance.json
## 3. Confirm the signed subject digest equals your local hash
jq -r '.attestation_bundles[].attestations[].verification_material' provenance.json >/dev/null
jq -r '.. | .sha256? // empty' provenance.json
## 4. Verify the signature and the identity expectations
cosign verify-blob \
--bundle provenance.json \
--certificate-identity-regexp '^https://github\.com/<org>/<repo>/\.github/workflows/<publish-workflow>\.yml@refs/(heads|tags)/.*$' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
"$WHEEL"
Three checks, all necessary: the digest in the bundle matches your local file, the signer identity matches the workflow you expect, and the signature itself is valid. The endpoint shape above follows PEP 740's index-hosted attestation model — if it 404s, check the current PyPI attestations docs rather than concluding the package is unsigned.
Post-install detection: processes, file writes, and outbound connections
For machines that already installed an affected version, three sweeps in the first hour:
## 1. Unexpected .pth files
python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" \
| xargs -I{} find {} -maxdepth 1 -name '*.pth' -print
## 2. Files in site-packages modified around the install window
python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" \
| xargs -I{} find {} -newermt '2026-11-01' -type f -name '*.py' -print | head -50
## 3. Python processes holding outbound sockets
lsof -i -P -n 2>/dev/null | awk 'NR==1 || /python/'
And on macOS/Linux desktops, a queue of processes that survived the terminal session is a strong signal — stealers frequently re-exec into the background:
ps -eo pid,ppid,lstart,command | grep -i '[p]ython' | head -30
None of this replaces credential rotation. It tells you which machines to rotate first.
Reproducing the Wheel Inspection in a Lab
This is the workflow I would run for any suspicious wheel, and it generalizes to the next incident.
Downloading the wheel without installing it
mkdir -p ~/lab/wheels && cd ~/lab/wheels
python -m pip download --no-deps --only-binary=:all: --no-cache-dir \
--dest . "<package>==<version>"
ls -la
-rw-r--r-- 1 analyst staff 49152 Nov 19 21:04 <package>-<version>-py3-none-any.whl
You now have bytes and nothing has executed.
Listing archive contents and diffing against a known-good build
unzip -l <package>-<version>-py3-none-any.whl
Length Date Time Name
--------- ---------- ----- ----
1204 1980-01-01 00:00 <package>/__init__.py
318 1980-01-01 00:00 <package>/_version.py
892 1980-01-01 00:00 <package>-<version>.dist-info/METADATA
1024 1980-01-01 00:00 <package>-<version>.dist-info/RECORD
--------- -------
3438 4 files
Now do the comparison that actually finds things — build the same tag yourself, in a container, and diff the trees:
git clone --depth 1 --branch <tag> https://github.com/<org>/<repo> /tmp/src
docker run --rm --network none -v /tmp/src:/src -w /src python:3.12-slim \
bash -c 'pip install --no-index build && python -m build --wheel -o /src/dist'
diffoscope --text /tmp/diff.txt \
<package>-<version>-py3-none-any.whl \
/tmp/src/dist/<package>-<version>-py3-none-any.whl
Reproducible builds would make that diff empty or noise-only. In practice it will be noisy unless upstream sets SOURCE_DATE_EPOCH and normalizes timestamps — a large fraction of the diff will be RECORD ordering and zip metadata. What you are scanning for is files present on one side and not the other, especially any .pth, any new top-level module, and anything under dist-info that references a URL.
Reading the provenance bundle and the certificate subject
jq -r '.attestation_bundles[].publisher.kind' provenance.json
jq -r '.attestation_bundles[].publisher.repository' provenance.json
jq -r '.attestation_bundles[].publisher.workflow' provenance.json
The field that most often surprises people is workflow. A provenance claim of release.yml when your mental model says publish.yml is a real finding — it means a different workflow, possibly one with broader permissions, did the upload. Read publisher.repository and publisher.workflow as assertions to check, not as boilerplate.
A safe, network-isolated runtime check
If you want to observe behavior — for example, whether an import tries to reach out — do it where it cannot succeed:
docker run --rm --network none \
-v ~/lab/wheels:/wheels:ro \
--tmpfs /tmp:rw,size=64m \
python:3.12-slim \
bash -c 'pip install --no-index /wheels/*.whl >/dev/null 2>&1; python -c "import <package>"' \
2>&1 | head -40
Expected shape when the payload's phone-home fails:
Traceback (most recent call last):
File "<string>", line 1, in <module>
...
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x...>: Failed to establish a new connection: [Errno -101] Network is unreachable
That traceback is a gift: the failing frame names the module, the file, and the line. --network none plus no mounted credential files turns a live payload into a readable stack trace.
import <package> executes arbitrary code. Do this only in a disposable container with no network, no mounts of ~/.aws or ~/.ssh, no environment variables, and no host credential passthrough. Do not do it on a machine you care about, and do not do it "just to see" on a CI runner that has a role attached.
CI/CD Controls That Close the Release-Pipeline Gap
None of this is exotic. All of it is frequently missing.
Short-lived OIDC credentials for every publish step
Delete every long-lived registry token from CI. PyPI Trusted Publishing has been generally available since 2023; PyPI also supports trusted publishing from GitLab, Google Cloud, and a growing set of providers. If your publish step still reads PYPI_API_TOKEN from secrets, that is the single highest-value change you can make this week.
Then constrain the identity. Require a GitHub Environment on the publish job with required reviewers, and restrict the environment to tags matching v*. Without that, the OIDC subject is just "this repo, this ref," and anyone with tag-push rights has publish rights.
Build provenance generated in a protected environment, not on a laptop
Provenance generated on a developer laptop is a self-signed assertion with no independent identity behind it. The point of the certificate is that a third party (the CI provider) vouches for the claims. Generate provenance only where an OIDC issuer exists, and only in a job whose permissions you have actually read.
Verification at install time, not just signing at publish time
Signing is a publish-side activity; verification is an install-side activity. Most organizations do the first and none of the second, which produces a cryptographic ceremony with no detection value.
Practical minimums:
## pip: pinned hashes make a substituted artifact fail loudly
pip install --require-hashes -r requirements.lock
## uv: same property
uv pip install --require-hashes -r requirements.lock
--require-hashes does not verify provenance, but it does enforce that the artifact you install is byte-identical to the one you reviewed. That is the property that turns a silent substitution into a failed job.
Above that, add a resolver-time check: a script that, for every package in your lockfile that publishes attestations, fetches the provenance and asserts the signer identity is on an allowlist. Wire it into CI ahead of the install step. It will break on legitimate workflow renames — that is the point. A workflow rename is exactly the kind of change an attacker makes.
Least privilege on the runner and what the build job should never hold
The build job should never hold:
- Cloud access keys, of any kind, static or not.
- An attached instance role with broad permissions. If the runner needs to push to a registry, scope that to the specific registry and nothing else.
contents: write,packages: write, orid-token: write. Grant these to the publish job only.- Repo-wide PATs, long-lived registry tokens, or model provider API keys.
secrets.*it does not strictly need. Every secret in the job environment is a secret the injected payload reads.
The pattern is: build in a job with contents: read and nothing else, publish in a separate job that has id-token: write and downloads the artifact. That separation means compromising the build does not automatically yield a publish identity — the attacker has to also get the artifact through the gate.
Source-to-artifact diffing and reproducible builds as a detection layer
Reproducibility is the only control that answers "are the bytes I am installing the bytes this source produces?" Attestation answers "who uploaded it." Different questions, and only the first one detects an injected payload.
Getting there incrementally:
- Set
SOURCE_DATE_EPOCHin your build and normalize timestamps. - Add a CI job that builds twice and diffs the outputs. It will be noisy at first; the noise tells you which non-determinism you have to fix.
- Once two consecutive local builds match, compare CI output to local output. Differences in file lists are the high-signal finding; differences in metadata are usually benign.
For vendored or third-party wheels you cannot rebuild, the fallback is diffoscope against a known-good version, which is slower and manual but catches added files.
If a Cloud Credential Was Already Stolen: Rotate, Revoke, Audit
Assume the worst for anything that ran the affected package in an environment with credentials.
Rotate first, then revoke, then audit — in that order
Rotate: issue new keys, new tokens, new secrets. Revoke: invalidate the old ones and, critically, invalidate active sessions — sts:GetSessionToken-derived temporary credentials do not die when you delete an IAM access key, they die when their policy session is revoked or they expire. Audit: only after the first two, because an audit takes hours and a stolen key works in minutes.
Ordering is not style. Teams that audit first spend four hours building a timeline while the attacker uses credentials that were still valid the whole time.
The attacker actions worth hunting for in cloud logs
| Signal | Where | Why it matters |
|---|---|---|
CreateAccessKey, CreateUser, AttachUserPolicy | CloudTrail / Activity Log / Cloud Audit Logs | The standard persistence move after key theft |
AssumeRole into roles the principal has never used | CloudTrail | Lateral movement |
GetSecretValue, DescribeSecret bursts | Secrets Manager / Key Vault | Bulk secret harvesting |
ConsoleLogin for a principal that has never used the console | CloudTrail | Human-operator pivot |
| New object storage bucket policies or public ACLs | S3 / Blob / GCS logs | Exfil staging |
Anomalous RunInstances in a region you do not use | CloudTrail | Cryptomining / new foothold |
OIDC token exchange from an unexpected workflow_ref | CI provider audit log | A second, independent pipeline compromise |
That last row is the one specific to this class of incident. If the pipeline certificate was abused, the CI provider's own audit log records the token issuance with the workflow ref and the ref it came from. That log is the ground truth for how the attestation came to exist, and it should be pulled first, not last.
Why deleting the package is not incident response
Yanking the release removes the delivery mechanism. It does not:
- undo any execution that already happened on developer machines, laptops, or agent runtimes;
- invalidate the credentials that were read;
- clean a self-hosted runner that a first-stage payload persisted onto;
- tell you which environments resolved the affected version during the exposure window.
The correct sequence is: determine the affected version set, determine which environments resolved it and when, rotate everything those environments could reach, and then remove the package.
The npm and JavaScript Side of the Same Supply-Chain Gap
The same structural gap exists in the Node ecosystem, with different plumbing.
npm provenance and trusted publishing in practice
npm generates provenance automatically in CI environments it recognizes, when you pass --provenance (or set provenance: true in publishConfig):
jobs:
publish:
runs-on: ubuntu-24.04
permissions:
contents: read
id-token: write
steps:
- uses: actions/checkout@<pinned-sha>
- uses: actions/setup-node@<pinned-sha>
with:
node-version: 22
registry-url: https://registry.npmjs.org
- run: npm ci
- run: npm publish --provenance --access public
Verification is a single command that most repos never run:
npm audit signatures
audited 412 packages in 3s
found 0 vulnerabilities
Wait — that output is the vulnerability audit, not the signature audit. The signature audit reports a count of packages with verified registry signatures and attestations, and fails if a package's signature does not match. If your last twelve releases never ran it, the command's first run is usually informative and occasionally alarming.
npm has also moved toward OIDC-based trusted publishing, removing the long-lived NPM_TOKEN the same way PyPI did. Check npm publish --help and the current npm docs for your CLI version rather than trusting a version number in a blog post — this is exactly the area where I would not quote a number I have not verified.
Where the PyPI and npm trust models diverge
| Property | PyPI | npm |
|---|---|---|
| Long-lived token model | Project-scoped API token | Granular access token / legacy automation token |
| OIDC trusted publishing | Native, since 2023 | Added more recently; verify CLI support |
| Attestation generation | Via Trusted Publishing; PEP 740 index-hosted bundles | Automatic in recognized CI with --provenance |
| Verification tooling in client | Limited; pypi-attestations / cosign by hand | npm audit signatures, built in |
| Public transparency log | Rekor entries via Sigstore bundle | Sigstore bundle published to registry |
| Enforcement of verification | None by default | None by default |
The last row is the shared problem. Both ecosystems now produce excellent provenance and neither requires a consumer to check it. The signing side got a decade of investment; the verifying side got a documentation page.
What to change in a JavaScript monorepo today
- Add
npm audit signaturesas a required CI step. It is cheap and it will start failing on real things. - Remove
NPM_TOKENfrom CI secrets and switch publish jobs to OIDC. Delete the secret afterward — a rotated-out secret that still exists in the settings UI is a future incident. - Put
provenance: truein each publishable package'spublishConfig, not just on the command line where a forgotten flag silently disables it. - Split build and publish into separate jobs with separate
permissionsblocks, exactly as in the Python example. - Gate the publish job on a protected GitHub Environment so a tag push is a request, not an action.
- Add a post-install check for unexpected
postinstallscripts in the resolved dependency tree — the JS-side analogue of the.pthsweep.npm ci --ignore-scriptsin CI is a blunt but effective default.
Where I Land on Release-Pipeline Attestations
The reporting around this incident frames it as a supply-chain failure. I think the more precise framing is that it is a trust-boundary relocation that the ecosystem has not caught up with.
Trusted Publishing solved a real problem — long-lived credentials sitting in CI — and in solving it, it moved the compromise point from "steal a token" to "get code execution in a job that can request a token." That second thing is easier in a lot of organizations, because build runners are treated as ephemeral plumbing rather than as privileged compute. Nobody puts an EDR agent on an ephemeral GitHub-hosted runner. Nobody notices that the build job has a cloud role attached and can reach the instance metadata endpoint.
Meanwhile the certificate does exactly what it says on the tin, and that is the trap. It attests to the identity of the publisher. It does not attest that the source was clean, that the inputs matched the checkout, or that the payload is benign. Every incident like this one produces a wave of "but the provenance verified" commentary, and the honest response is: yes, and that is precisely why it did not help.
Three things I would want in place before the next one, ranked by leverage:
- Verification at install time, enforced. Signing on the publish side without a matching check on the install side buys nothing. A lockfile-plus-attestation check in CI is what catches a substituted artifact.
- Build runners that hold nothing. No cloud role, no PAT, no secrets,
contents: read, separate job for publish. If the runner holds nothing, the stealer reads nothing. - Reproducibility, even partially. It is the only control that can tell you the artifact does not match the source. Everything else tells you who uploaded it.
And one operational rule I would put in a runbook today: when an advisory lists affected versions, pull the CI provider's OIDC token issuance log before you pull anything else. In a world where the certificate is legitimate and the artifact is not, that log is the only place the real story is written down.
Further Reading
- SiliconANGLE — Wiz ties Microsoft Agent Framework package compromise to release pipeline certificate (2026-11-19): report
- Help Net Security — Credential stealer slipped into Microsoft's official Python agent framework (2026-11-19): report
- TechRepublic — Microsoft Agent Framework Attack: Check These Two Python Package Versions (2026-11-19): report
- PyPI — Trusted Publishing documentation: https://docs.pypi.org/trusted-publishers/
- PyPI — attestations documentation: https://docs.pypi.org/attestations/
- PEP 740 — Index-hosted attestations for Python packages: https://peps.python.org/pep-0740/
- CPython documentation — the
sitemodule and.pthfile processing: https://docs.python.org/3/library/site.html - pip documentation — Secure installs and
--require-hashes: https://pip.pypa.io/en/stable/topics/secure-installs/ - npm documentation — Generating provenance statements: https://docs.npmjs.com/generating-provenance-statements
- GitHub documentation — Security hardening with OpenID Connect: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect
- Sigstore documentation (cosign verification, Fulcio, Rekor): https://docs.sigstore.dev/
- SLSA specification v1.0 — build provenance levels: https://slsa.dev/spec/v1.0/
More posts

Auditing Python Install Hooks: Reproducing the agent-framework Credential Theft in a Lab
A hands-on lab walkthrough that reproduces how hijacked agent-framework releases abused Python build and install hooks to exfiltrate Azure and GitHub CI credentials, then hardens the pipeline with Trusted Publishing, scoped tokens, and lockfiles.

Using AbortSignal.any to Compose Timeout and Cancellation Signals in fetch
Learn how to use AbortSignal.any to combine timeout and user cancellation signals in fetch requests for more robust async operations.

Testing Edge Prompt API Rewrites in JavaScript: Capability, Latency, Quality, and Fallback
A deep technical walkthrough of testing Microsoft Edge's Prompt API for JavaScript text rewrites, covering capability detection, latency measurement, output quality evaluation, and fallback strategies.
Loading comments…