Certificate Revocation Is a Release-Engineering Problem, Not a PKI Team Problem

Certificate Revocation Is a Release-Engineering Problem, Not a PKI Team Problem

pr0h0•
pkicertificate-revocationrelease-engineeringfedorasoftware-supply-chain
AI Usage (86%)

Introduction

A Linux distribution pushed its release date because of a root certificate. If you own a build pipeline, that sentence should make you flinch: a third-party trust anchor was sitting on the critical path between feature freeze and a downloadable ISO, and nobody noticed until it fired. This post traces how a CA revocation reaches Fedora-style build pipelines, separates what the public report establishes from what remains inference, and gives you checks you can run before a trust-store change becomes a schedule slip.

My position: certificate revocation is a release-engineering problem, not a PKI-team problem. The PKI team produces a decision ("this root is no longer trusted") and a date. Everything after that — rebuilding buildroots, re-signing artifacts, re-running validation, moving Go/No-Go — belongs to whoever owns the release. Teams that treat ca-certificates as an OS file that quietly updates itself at runtime will keep discovering it's a build input, usually at the worst possible moment.

What the public record actually establishes

Confirmed from the source

The report published by news.lavx.hu on 23 October 2026 states two things:

  • Fedora 45's release moved to November 2026.
  • The stated cause was an eMudhra certificate authority incident that forced mass revocation.

That's the entire confirmed set. No CVE, no advisory ID, no list of affected roots, no enumeration of rebuilt artifacts, and nothing about which packages or images were touched.

What the report does not confirm about the revocation mechanism

The report doesn't say whether remediation was root removal from the trust store or per-certificate revocation via CRL/OCSP. Those have different downstream shapes: root removal changes the bundle your build chroots download, while per-certificate revocation changes what a verifying client accepts at runtime. If someone tells you which happened, ask for the primary document.

Also unconfirmed, and worth flagging as unlikely: that Fedora's package signing keys were affected. Fedora signs RPMs with GPG keys managed through its own signing infrastructure, not with X.509 certificates chained to a public CA. A CA removal doesn't invalidate an RPM signature — exactly the distinction that gets lost in the incident chatter. Treat that as inference, not as a sourced claim.

The one thing the timeline does support: a slip measured in weeks, not months, fits re-running composes and re-validating artifacts rather than re-architecting anything. That's my reading of the schedule, not a statement from the report.

How a CA revocation reaches a distro build pipeline

The CA bundle is a build dependency, not just a runtime file

On Fedora-family systems, ca-certificates ships the source anchors, and update-ca-trust extracts them into the bundles that TLS libraries actually read:

$ ls /etc/pki/ca-trust/extracted/pem/
tls-ca-bundle.pem  email-ca-bundle.pem  objsign-ca-bundle.pem

Anything that consumes tls-ca-bundle.pem at build time inherits the revocation — a wider set than most inventories capture:

  • package managers fetching over HTTPS inside a buildroot (dnf, pip, npm, go mod download, cargo)
  • artifacts that bake in a trust store: container base images, Java cacerts, Node images, installer and live media
  • verification steps that chain to a public CA — RFC 3161 timestamp countersignatures, Authenticode, keyless signing via Fulcio

CA certificates and package signing keys are different anchors that get conflated

These are separate systems with separate failure modes, and incident response that merges them burns time on unnecessary work.

VerificationAnchor typeEffect of a CA root removal
rpm --checksig pkg.rpmDistro GPG keyNone — signature still valid
gpg --verify on a release tarballMaintainer GPG keyNone
openssl verify -CAfile bundle.pem leaf.pemPublic CA hierarchyFails if the chain touches the removed root
RFC 3161 timestamp validationPublic CA hierarchyCan fail even though the artifact is untouched
Keyless/cosign verificationFulcio root + TUF rootFails only if the Fulcio chain is affected

The dangerous row is the timestamp one: revocation can invalidate the evidence around an artifact without changing the artifact itself. That's a verification problem, not a rebuild problem, and it deserves its own runbook.

How a revocation cascades from trust store to release date

From revocation to rebuilt buildroot

The chain that actually blocks a release, in order:

  1. Upstream publishes the removal; the distro's ca-certificates maintainer bumps the package.
  2. A new buildroot is composed with the updated bundle.
  3. Every package build that fetched anything over TLS during %build or %prep now runs against a different trust set. If bootstrap isn't hermetic, the inputs to the build changed, so reproducibility checks flag it.
  4. Container base images rebuild, then every layered image on top.
  5. Installer, live, and cloud images rebuild — large, slow, and validated as a set.
  6. Artifacts re-sign and re-checksum.
  7. The QA matrix re-runs against the new compose.

Steps 3 and 4 are where the time goes, because each is a fan-out you can't shortcut without shipping an artifact whose trust inputs you can't describe.

From rebuilt buildroot to a moved release date

Once a compose is invalidated after a freeze, the schedule stops belonging to engineering. Go/No-Go has to happen again, release notes have to be regenerated, mirror staging slots may need re-booking, and any downstream consumer that validated an RC has to re-validate. A two-to-three-week slip is the honest cost of that, and it matches what the report describes.

A reproducible revocation-readiness check

Run this against a representative build host, not your laptop. The goal is a number you can put in front of a release manager.

Enumerate what actually trusts public CAs

## How many anchors does the system trust today?
trust list --filter=ca-anchors --format=pem-bundle > /tmp/anchors.pem
grep -c "BEGIN CERTIFICATE" /tmp/anchors.pem

## What declares a dependency on the bundle package or its extracted files?
rpm -q --whatrequires ca-certificates
rpm -q --whatrequires /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

## Inside a buildroot, what pulls TLS in transitively?
dnf repoquery --whatrequires ca-certificates --recursive --qf '%{name}\n' | sort -u | wc -l

Your counts will differ from any number quoted in a blog post; what matters is having a baseline you can diff. Save the sorted output to git and re-run it whenever the bundle version changes.

Time a trust-store change in a scratch channel

The useful measurement is the ratio between a warm rebuild and a cold one, because the cold path is what a revocation forces.

trust-store-drill.sh
set -euo pipefail

## One-off: remove a throwaway root from a scratch image and see what breaks.
podman build -t localhost/trust-probe -f Containerfile.trust . 

for i in 1 2 3; do
podman rmi -f localhost/trust-probe >/dev/null 2>&1 || true
/usr/bin/time -f "cold run %e s"   podman build --no-cache -t localhost/trust-probe -f Containerfile.trust .
done

## What did the change actually touch inside the image?
podman run --rm localhost/trust-probe sh -c 'trust list --filter=ca-anchors | grep -c "^pkcs11:id"'

Record three numbers: cold build wall-clock, count of affected packages, and count of steps that needed a human. That third number is the one that predicts your slip.

Separate repeatable prerequisites from one-off dependencies

BucketExamplesAutomation status
Repeatablebundle bump, buildroot recreate, image rebuild, checksum refreshMust be one command each
Repeatable but slowISO/live/cloud image buildsRun on a nightly schedule, not at freeze
One-offre-signing ceremony, mirror re-booking, release-note rewritesNamed owner required
Manual and undocumented"ask the person who knows the compose flags"Release blocker

Anything in the last row is why a CA incident becomes a schedule event.

What to automate before the incident

Inventory and blast-radius mapping

Generate the trust inventory on a schedule and commit it. The audit question you must be able to answer is: which artifacts did we build while this root was trusted? That requires build timestamps plus a record of the bundle version baked into each artifact. If you can't answer it, you can't scope a revocation — you can only rebuild everything, which is the expensive option taken under time pressure.

A rehearsed bundle rollback path

Removing a root breaks things. Ship the rollback before the removal:

sudo dnf downgrade ca-certificates
sudo update-ca-trust extract
openssl verify -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem cert.pem

update-ca-trust extract is idempotent, so a rollback is a package downgrade plus one command. Rehearse it in a scratch channel. Don't discover during an incident that your golden image hardcodes an extracted bundle.

⚠️

Don't "fix" a revocation-induced build failure with --nogpgcheck, sslverify=false, or NODE_TLS_REJECT_UNAUTHORIZED=0. That converts a visible trust change into silent, permanent trust in everything.

Offline and pinned-trust build modes

Hermetic builds are the real mitigation: if a build can't reach the network, a trust-store change can't alter its inputs.

  • Use mock with a pre-populated --installroot and --disablerepo for anything on the release path.
  • Vendor dependencies and pin checksums rather than fetching during %build.
  • In Node tooling, make CA trust explicit instead of ambient: NODE_EXTRA_CA_CERTS for additional anchors, --use-openssl-ca where the system store is the intended source, and tls.createSecureContext({ ca }) when a service must pin a specific hierarchy. For internal registries, npm config set cafile <bundle> beats disabling verification.
  • For sigstore/cosign verification, pin the Fulcio root and TUF root you trust, and verify with --offline where transparency log proofs are cached — so a public-CA event can't silently swap your verification root.

Signing key custody kept independent of CA trust

GPG signing keys and X.509 CA trust must not share a rotation event. Keep release signing keys in an HSM or dedicated signing service, with a documented rotation procedure triggered by key compromise only — never by a CA decision. If you do use X.509-based signing (Authenticode, keyless), document explicitly that a CA revocation can force re-signing, and give that its own alert path.

What is still unconfirmed and what would resolve it

Three things would turn this post's inferences into facts, and each changes the response:

  • The primary advisory. A Fedora devel-announce or security advisory naming the affected certificates and the remediation mechanism would tell you whether this was root removal or per-certificate revocation. If per-certificate, runtime clients were the exposure; if root removal, build pipelines were.
  • The CA's own incident statement. eMudhra's disclosure would establish scope and whether misissuance was involved.
  • A CCADB record or root-store policy decision. A Mozilla root store removal record would fix the date the change landed upstream, which is the input to every downstream schedule.

Until those exist, treat the mechanism as unknown. The readiness work above doesn't depend on the answer.

Conclusion

The uncomfortable lesson isn't that a CA can be distrusted — that part is the system working. It's that most organizations can't say which of their artifacts were produced under a trust anchor that has since been withdrawn, so their only safe response is a full rebuild, and a full rebuild after freeze is a schedule slip.

Make the bundle a versioned, inventoried build input. Keep signing keys off the CA trust graph. Rehearse the rollback. Then a revocation becomes a two-day release-engineering job with a known owner instead of a release date moving to November.

Further Reading

Share this post

More posts

Comments