From Feature Flag to Risk Tier: Auditing AI Capabilities Against the Digital Omnibus on AI

A regulation lands in the Official Journal, and within a week someone has forwarded the PDF to the platform team with the message "does this affect us?" For most teams the honest answer is that they can't tell you — not because nobody read the PDF, but because the inventory of AI capabilities precise enough to answer it doesn't exist. This post is a build guide for that missing layer: how to trace a single AI capability from the feature flag that gates it, through a risk tier classification, into the documentation and deployment controls it has to carry before it ships to EU users.

The seed item — an EY entry titled Regulation (EU) 2026/1744 – Digital Omnibus on AI, timestamped 2026-11-06 in the feed — gives us a headline and almost nothing else. What it doesn't give us, and what no legal summary can give you, is the mapping from that text to the fourteen places in your repository where a model call can reach a user.

I'll be blunt: the feature flag is not your compliance control plane, and treating it as one is the failure mode I expect to see most often over the next eighteen months. Flags describe rollout state. Regulations describe intended purpose, affected persons, and oversight. Those are different data, owned by different people, changing on different clocks. If your AI risk classification is a boolean someone set in LaunchDarkly during a launch review, you have a compliance system that cannot survive a flag cleanup.

The alternative below is buildable: manifests that declare facts, a classifier that encodes law, documentation generated from repository state, and a CI gate that fails the build when a capability is missing the artefacts its tier requires.

⚠️

This post is engineering guidance, not legal advice. Article references below are to Regulation (EU) 2024/1689 (the AI Act) as I read it; anything about the Digital Omnibus is marked as unverified. Your counsel decides your obligations. Your job is to make those obligations executable.

What the Source Confirms About the Digital Omnibus, and What It Leaves Open

What the EY Item on Regulation (EU) 2026/1744 Actually Contains

The seed is thin, so let me state exactly what it contains rather than inflate it. It is a single item attributed to EY, carrying the title Regulation (EU) 2026/1744 – Digital Omnibus on AI, with a feed timestamp of 2026-11-06T15:03:37Z. The summary in the feed frames it as an update to the bloc's AI rulebook that teams building or deploying AI in Europe need to track. There is no article text, no annex list, no transition table in the material I was given.

That's not enough to base an engineering plan on.

Separating Confirmed Facts From Inference

Claim	Status
An EY item titled Regulation (EU) 2026/1744 – Digital Omnibus on AI exists in the feed, dated 2026-11-06	Confirmed (source metadata)
The Digital Omnibus touches the EU AI Act framework	Stated by the source summary; not independently verified here
Regulation (EU) 2024/1689 is the current AI Act, in force since 1 August 2024	Confirmed; verifiable on EUR-Lex
Regulation (EU) 2026/1744 has been adopted and published in the Official Journal	Inference. The number format is consistent with an adopted regulation, but the seed does not say so
The omnibus defers, narrows, or otherwise changes high-risk obligations	Unverified. Widely discussed in public reporting on the Commission's proposal, but the final text is not something I can attest to
Any specific article number, date, or Annex III change in 2026/1744	Unknown to me. Do not take one from this post

The two halves of this article have different evidentiary status, so the split is worth making explicit. The legal side is a verification task you do against primary sources. The engineering side is code I can show you.

Verifying the Regulation Against the Official Journal and EUR-Lex Text

Do this before you build anything. EUR-Lex assigns ELI identifiers by a predictable pattern — http://data.europa.eu/eli/reg/2024/1689/oj resolves for the current AI Act — so the 2026 regulation should follow the same shape. Pattern-derived, not confirmed:

## Is the regulation actually published, and what does it say?
curl -sSI "http://data.europa.eu/eli/reg/2026/1744/oj" | head -n 5
curl -sS  "http://data.europa.eu/eli/reg/2026/1744/oj" -o 2026-1744.html
wc -c 2026-1744.html

Three outcomes, three different projects:

404 or redirect to a search page. The regulation is not published under that ELI yet, or the number is wrong. The seed is a secondary source and may be describing a proposal, a draft, or an adopted act pending publication. Your baseline is still Regulation (EU) 2024/1689.
200 with an HTML document. Read the amendment articles first, not the recitals. Recitals tell you intent; amendment articles tell you which strings in your codebase change.
200 but no consolidated version. You now need the consolidated text (current act + amendments) before you can quote article numbers in a risk assessment. A risk assessment citing a superseded article is worse than one citing nothing.

Log the check. If your compliance artefacts cite a legal text, the fetch date and the document identifier belong in the artefact.

Inventory First: Treating AI Capabilities as Deployable Units

You cannot classify what you have not enumerated. But "enumerate your AI features" fails in practice because nobody agrees on the unit. A chat sidebar, a summariser, and an embedding index are all "AI" and none of them have the same obligations.

What Counts as an AI Capability in Your Codebase

I define the unit as the smallest thing that has an intended purpose and can be turned off independently. In practice that means:

a prompt template plus the model call that consumes it
a ranking or scoring function whose output changes what a person sees or is offered
an automated decision that feeds a workflow where a human is nominally in the loop
generated media surfaced to a user or published anywhere
an agent loop with tool access, even if every tool is read-only today

Things that are not capabilities on their own: a shared embedding client, a gateway SDK, a vector store. Those are infrastructure. They inherit the tier of every capability that uses them, so a shared component's risk tier is the maximum over its consumers — compute that rather than set it by hand.

A test that has held up for me: if you deleted the flag and left the infrastructure, would a user-visible behaviour change? If yes, it is a capability. If no, it is plumbing.

A Manifest Schema Linking Flag, Model, Data, and Intended Purpose

The manifest's job is to declare facts, not conclusions. The moment a manifest contains riskTier: "high", you have a mutable opinion stored next to immutable-ish facts, and the two will drift. Two people will disagree about the tier, someone will edit it to unblock a release, and the audit trail becomes a git blame argument.

So: manifest declares purpose, affected persons, data, model identity, oversight. Classifier derives tier from those facts plus encoded law. The flag name is an attribute of the manifest, not the other way round.

// ai/types.ts
export type Domain =
  | "employment" | "education" | "essential-services"
  | "law-enforcement" | "migration" | "justice"
  | "product-safety" | "general";

export type PurposeFacts = {
  domain: Domain;
  affectedPersons: Array<"employees" | "candidates" | "students" | "customers" | "public" | "patients">;
  outputs: Array<"recommendation" | "decision" | "score" | "generated-text" | "generated-media" | "identity-match">;
  interactsWithHumans: boolean;
  biometric: boolean;
  emotionInference: boolean;
  socialScoring: boolean;
  exploitsVulnerability: boolean;
  manipulativeTechnique: boolean;
};

export type AICapability = {
  id: string;
  flag: string;
  owner: string;
  intendedPurpose: string;          // one sentence, from the product spec
  purpose: PurposeFacts;
  model: {
    provider: string;
    modelId: string;                // routing alias, e.g. "gw-chat"
    pinnedVersion: string;          // immutable build, e.g. "2026-02-14"
    isGeneralPurposeModel: boolean; // you consume; you do not place it on the market
  };
  data: {
    inputs: string[];
    personalData: boolean;
    specialCategoryData: boolean;
    retentionDays: number;
    trainingOnCustomerData: boolean;
  };
  humans: {
    oversight: "none" | "required-before-send" | "required-before-decision" | "sample-review";
    overridePath: string;
  };
  lifecycle: { status: "draft" | "live" | "retired"; firstExposedAt?: string; regions: string[] };
};

Writing a Capability Manifest Next to the Feature Flag Definition

// ai/capabilities/support-reply-suggest.capability.ts


export const supportReplySuggest = {
  id: "support-reply-suggest",
  flag: "ai.support.reply_suggest",
  owner: "team-support-platform",
  intendedPurpose:
    "Draft a suggested reply for a human support agent to review, edit, or discard.",
  purpose: {
    domain: "general",
    affectedPersons: ["customers"],
    outputs: ["generated-text", "recommendation"],
    interactsWithHumans: true,
    biometric: false,
    emotionInference: false,
    socialScoring: false,
    exploitsVulnerability: false,
    manipulativeTechnique: false,
  },
  model: {
    provider: "internal-gateway",
    modelId: "gw-chat",
    pinnedVersion: "2026-02-14",
    isGeneralPurposeModel: true,
  },
  data: {
    inputs: ["support_ticket_body", "public_help_center_articles"],
    personalData: true,
    specialCategoryData: false,
    retentionDays: 30,
    trainingOnCustomerData: false,
  },
  humans: {
    oversight: "required-before-send",
    overridePath: "agent-edit-or-discard",
  },
  lifecycle: { status: "live", firstExposedAt: "2026-03-02", regions: ["eu"] },
} satisfies AICapability;

The satisfies operator matters here. It type-checks the literal without widening it, so adding a required field to AICapability produces a compile error on every manifest that doesn't supply it. Cheapest enforcement mechanism in the whole system: your type checker becomes the inventory completeness check.

Keeping the Inventory Honest When Flags Get Deleted

Flags die. Capabilities usually don't — they get renamed, re-flagged, or left running behind a new key. Three rules keep the manifest honest:

Manifests are retired explicitly. lifecycle.status: "retired" plus a date. Never delete the file; the retention question (Art. 18 of 2024/1689 expects documentation kept for 10 years after placing on the market, for the systems in scope) applies to the artefact, not to your enthusiasm.
A gate checks the reverse direction. Every flag matching ai.* in your flag provider that has no manifest is a finding. So is every live manifest whose flag no longer exists. Both are cheap to detect and both catch the silent-rename case.
The manifest is reviewed by whoever owns the flag. Codeowners on ai/capabilities/ should be the product team, not the platform team. The platform team owns the classifier.

💪

Run the orphan check weekly in CI rather than on every PR. Ad-hoc flag creation is legitimate during exploration; a weekly drift report with a named owner per orphan gets fixed, a blocking PR check gets bypassed.

The Classification Ladder: From Feature Flag to Risk Tier

Prohibited Practices as a Hard Gate, Not a Judgment Call

The prohibited list is the one part of this system that should not be a discussion. Art. 5 of Regulation (EU) 2024/1689 sets out practices that are prohibited outright, and those obligations have applied since 2 February 2025 — they are not waiting for the 2026 dates.

The engineering consequence is plain: prohibited must be a build failure, not a warning. If a manifest declares facts that match a prohibited-practice rule, the gate exits non-zero and no human override exists for the tier itself. The only acceptable resolutions are "remove the capability" or "the declared facts are wrong; fix the manifest and re-review."

That last point is where teams get this wrong in both directions. A recruiting team that adds "rank candidates" to a manifest and gets a build failure will be tempted to reword the purpose sentence until the rule stops matching. Which is why the classifier has to read structured facts (domain: "employment", outputs: ["score"]) rather than free text. A free-text purpose statement is a classification bypass waiting to be discovered.

Intended Purpose as the Input to High-Risk Determination

High-risk status attaches to intended purpose and the context of use, not to model capability. The same summarisation endpoint is a different regulatory object in a help centre and in a system that feeds decisions about people's access to credit.

Practically, the high-risk determination reduces to two facts you can capture in a manifest:

Domain. Is the system used in one of the areas listed in Annex III of 2024/1689 — employment and worker management, education, access to essential services, law enforcement, migration, administration of justice, certain product-safety components?
Output influence. Does the output feed a decision, or is it inert until a human acts on it in a context where the human has real discretion?

The second is where honest engineering matters more than legal reading. A "human in the loop" who approves 400 suggestions an hour is not exercising discretion, and I would not build a control system that pretends otherwise.

📝

Under Art. 25 of 2024/1689, a deployer that puts its own name or trademark on a system, substantially modifies it, or changes its intended purpose can take on provider obligations. In manifest terms: changing intendedPurpose or purpose.domain is a provider-level event, not a config change, and should require a different review path.

Transparency-Only Obligations and the General-Purpose Model Boundary

Most capabilities land here. Art. 50 obligations are the ones that hit ordinary product engineering: telling a person they are interacting with an AI system, marking synthetically generated content, disclosing deepfakes. These have the shortest distance from "legal requirement" to "code change" — and the longest distance from "we wrote a help centre article about it."

The general-purpose model boundary matters for a different reason. Almost every team reading this is a deployer consuming a GPAI model through an API, not a provider placing a GPAI model on the market. The GPAI obligations in Arts. 51–56 (applied since 2 August 2025) sit with your upstream provider. Your obligation is to not destroy the information they give you: keep the model documentation, keep the version pin, and don't swap models behind an alias without re-running classification.

That last one is the sharpest edge in this whole post. modelId: "gw-chat" with pinnedVersion: "2026-02-14" is a two-field lie if your gateway silently routes to a newer build. Then your documentation describes a model that isn't answering requests.

Encoding the Ladder as a Decision Table and a Typed Classifier

// ai/classify.ts


export type RiskTier = "prohibited" | "high" | "transparency" | "minimal" | "review";

export type Rule = {
  id: string;
  tier: RiskTier;
  cite: string;
  why: string;
  matches: (c: AICapability) => boolean;
};

export const RULES: Rule[] = [
  {
    id: "PR-1", tier: "prohibited",
    cite: "Regulation (EU) 2024/1689, Art. 5(1)(a)",
    why: "Purpose relies on subliminal or manipulative techniques.",
    matches: (c) => c.purpose.manipulativeTechnique,
  },
  {
    id: "PR-2", tier: "prohibited",
    cite: "Regulation (EU) 2024/1689, Art. 5(1)(b)",
    why: "Purpose exploits vulnerabilities of a specific group.",
    matches: (c) => c.purpose.exploitsVulnerability,
  },
  {
    id: "PR-3", tier: "prohibited",
    cite: "Regulation (EU) 2024/1689, Art. 5(1)(c)",
    why: "Social scoring of natural persons.",
    matches: (c) => c.purpose.socialScoring,
  },
  {
    id: "PR-4", tier: "prohibited",
    cite: "Regulation (EU) 2024/1689, Art. 5(1)(f)",
    why: "Emotion inference in workplace or education contexts.",
    matches: (c) =>
      c.purpose.emotionInference &&
      (c.purpose.domain === "employment" || c.purpose.domain === "education"),
  },
  {
    id: "HR-1", tier: "high",
    cite: "Regulation (EU) 2024/1689, Art. 6 + Annex III",
    why: "Scoring or decisioning in an Annex III domain.",
    matches: (c) =>
      ["employment", "education", "essential-services", "migration", "justice"]
        .includes(c.purpose.domain) &&
      c.purpose.outputs.some((o) => o === "score" || o === "decision"),
  },
  {
    id: "TR-1", tier: "transparency",
    cite: "Regulation (EU) 2024/1689, Art. 50(1)",
    why: "System interacts directly with natural persons.",
    matches: (c) => c.purpose.interactsWithHumans,
  },
  {
    id: "TR-2", tier: "transparency",
    cite: "Regulation (EU) 2024/1689, Art. 50(2), (4)",
    why: "Produces synthetic audio, image, video, or text.",
    matches: (c) => c.purpose.outputs.some((o) => o === "generated-media"),
  },
];

const SEVERITY: RiskTier[] = ["prohibited", "high", "transparency", "minimal", "review"];

export function classify(cap: AICapability) {
  const hits = RULES.filter((r) => {
    try {
      return r.matches(cap);
    } catch {
      return true; // fail closed on a malformed manifest
    }
  });

  if (hits.length === 0) {
    return { tier: "minimal" as RiskTier, rules: [], needsHumanReview: false };
  }

  const tier = SEVERITY[Math.min(...hits.map((h) => SEVERITY.indexOf(h.tier)))];
  return {
    tier,
    rules: hits.map((h) => ({ id: h.id, cite: h.cite, why: h.why })),
    needsHumanReview: tier === "high" || tier === "review",
  };
}

Two design decisions worth defending. Multiple matches take the most severe tier — a capability that is both transparency and high-risk is high-risk. And a throwing rule means review, not pass: the classifier's failure mode must be "a human looks at this," never "the build went green."

Where the Classification Ladder Breaks Down and Needs a Human

The classifier is a triage tool, and it will be wrong in predictable ways:

Purpose ambiguity. "Recommend relevant learning content" is education-adjacent; whether it is Annex III education depends on who it affects and how the output is used. Rules cannot resolve this. A manifest that lands here gets tier: "review" and a named reviewer.
Composite capabilities. An agent that retrieves documents, summarises them, and files a ticket is three capabilities with different data flows. One manifest per capability, or the data declarations become fiction.
The human-oversight fiction. oversight: "required-before-send" is a declared fact. Whether the human actually can override is a tested fact, and that belongs in the deployment controls section, not the classifier.
Legal text changes. If Regulation (EU) 2026/1744 amends Annex III or the application dates, the fix is a diff to RULES with a citation change, reviewed by counsel. That is exactly what you want: the legal change is a code change with a reviewer, not a memo nobody reads.

Documentation as a Build Artifact, Not a Wiki Page

The Minimum Documentation Set per Tier

Tier	Risk assessment	Technical documentation	Eval record	Oversight test	Sign-off
prohibited	— (must not ship)	—	—	—	Blocking legal + eng review
high	Required	Required	Required, per model version	Required	Named reviewer in `CODEOWNERS`
transparency	Purpose line in manifest	Optional	Recommended	Recommended	None
minimal	—	—	Optional	—	None

For high-risk systems, the technical documentation expected under Art. 11 and Annex IV of 2024/1689 is genuinely large: intended purpose, model characteristics, training and validation data where relevant, human oversight measures, performance metrics, known limitations. The temptation is to write it once in Confluence. Confluence does not fail your build.

Generating Documentation From Repository State

Generate the parts that are mechanical, hand-write the parts that require judgment. Roughly 60% of a technical document is restatement of facts you already have typed.

// scripts/gen-tech-doc.mjs



export async function writeTechDoc(cap, classification, outRoot) {
  const dir = path.join(outRoot, cap.id);
  await mkdir(dir, { recursive: true });

  const lines = [
    `# ${cap.id}`,
    ``,
    `> Generated from \`ai/capabilities/${cap.id}.capability.ts\` at commit ${process.env.GIT_SHA ?? "unknown"}.`,
    `> Do not edit by hand. Edit the manifest.`,
    ``,
    `## Intended purpose`,
    `${cap.intendedPurpose}`,
    ``,
    `## Classification (derived)`,
    `Tier: **${classification.tier}**`,
    ...classification.rules.map((r) => `- ${r.id} — ${r.why} (${r.cite})`),
    ``,
    `## Model`,
    `- Provider: ${cap.model.provider}`,
    `- Alias: \`${cap.model.modelId}\``,
    `- Pinned version: \`${cap.model.pinnedVersion}\``,
    `- Consumes a general-purpose model: ${cap.model.isGeneralPurposeModel}`,
    ``,
    `## Data`,
    `- Inputs: ${cap.data.inputs.join(", ")}`,
    `- Personal data: ${cap.data.personalData}`,
    `- Special category data: ${cap.data.specialCategoryData}`,
    `- Retention: ${cap.data.retentionDays} days`,
    `- Used for training: ${cap.data.trainingOnCustomerData}`,
    ``,
    `## Human oversight`,
    `- Mode: ${cap.humans.oversight}`,
    `- Override path: ${cap.humans.overridePath}`,
    ``,
  ];

  await writeFile(path.join(dir, "tech-doc.md"), lines.join("\n"), "utf8");
}

Note what the generator does not do: it does not write "this system is safe," it does not assert compliance, and it does not produce performance numbers it cannot know. A generated document that overclaims is worse than no document, because it will be produced in discovery.

Pinning Docs to Model, Prompt, and Dataset Versions

Three things must be pinnable, and only one of them is usually pinned in practice:

Model version. pinnedVersion in the manifest, checked against what the gateway actually served.
Prompt template. Hash the rendered template, not the source file — variable interpolation changes behaviour.
Dataset or index version. The embedding index, the retrieval corpus, the eval set. An eval record without an index version is not reproducible.

A pragmatic pattern is to emit a single build.json per capability per release:

{
  "capability": "support-reply-suggest",
  "gitSha": "9f2c1ab",
  "modelAlias": "gw-chat",
  "modelVersion": "2026-02-14",
  "promptHash": "sha256:4b2f...c81d",
  "indexVersion": "help-center-2026-02-28",
  "evalRunId": "ev-2026-03-01-004",
  "classifiedAt": "2026-03-01T09:14:02Z",
  "tier": "high",
  "rules": ["TR-1", "HR-1"]
}

The deployment control is then trivial and testable: the served model version must match modelVersion in the artefact attached to that release. If the gateway routed to a newer build, you have a drift incident, not a mystery.

Diffing Generated Documentation Between Releases

Generated docs diff for free. That's the point.

git diff --word-diff=plain v1.14.0 v1.15.0 -- 'ai/docs/*/tech-doc.md'

Realistic output shape after a model migration:

## Model
- Pinned version: [-`2026-02-14`-]{+`2026-03-11`+}
## Data
- Retention: [-30-]{+90+} days

That diff is the review artefact. A retention change from 30 to 90 days is a data-protection decision that appeared in a dependency bump PR, and without generated docs it would have shipped unnoticed. Diffing is how you find the second-order changes that nobody intended.

Deployment Controls That Actually Enforce the Tier

Human Oversight Hooks in the Request Path

If oversight is a policy, it will not happen at 2am. If it is a middleware, it will.

// ai/oversight.ts


export function oversightGate(cap: AICapability) {
  return async function (req, res, next) {
    if (cap.humans.oversight === "none") return next();

    const ack = req.header("x-ai-review-ack");
    if (cap.humans.oversight !== "sample-review" && !ack) {
      res.status(428).json({
        error: "oversight_required",
        capability: cap.id,
        message: "Output must be reviewed before it leaves the system.",
      });
      return;
    }
    res.locals.aiAudit = { capability: cap.id, reviewedBy: ack ?? null };
    next();
  };
}

Two things this must not become. It must not be a checkbox the client sets unconditionally — for required-before-send, the acknowledgment should be produced by your review UI, signed, and bound to the specific output payload. And it must not be the only control: pair it with sampling and scoring of unreviewed drafts to detect the case where reviewers rubber-stamp.

For sample-review, enforcement looks different: you log everything and audit a sample. That is a legitimate control for lower-stakes capabilities, but only if the sample rate is in the manifest and someone reviews the sample.

Logging, Traceability, and Reportable-Incident Detection

Record-keeping under Art. 12 of 2024/1689 means automatic logging over the lifetime of the system for the systems in scope. The engineering requirement is a log record that can answer "what did this capability do to this person, with which model version, and who reviewed it" without joining four systems.

Minimum fields per invocation:

Field	Why
`capabilityId`, `tier`	Route to the right retention and access policy
`modelAlias`, `modelVersion`	Drift detection against `build.json`
`promptHash`, `indexVersion`	Reproducibility
`inputDigest`, `outputDigest`	Correlation without storing raw content
`oversightMode`, `reviewedBy`	Evidence the control operated
`region`, `occurredAt`	Jurisdictional and timeline scoping

Reportable-incident detection is then a query, not a rumble of intuition. Serious-incident reporting under Art. 73 of 2024/1689 runs on short clocks — I read the general deadline as 15 days, with 2 days for widespread infringement or critical-infrastructure impact and 10 days where a person has died, and you should check that against the text rather than my summary. Whatever the deadlines are, they only work if the detector fires within hours.

Concrete detectors worth wiring: oversight-ack rate collapsing to zero (the control was bypassed), model version mismatch on more than a handful of requests (drift), output-digest divergence for identical inputs after a version change (behavioural change), and error-rate spikes on a capability whose failure mode is a wrong decision rather than a 500.

Rollback, Kill Switches, and Staged Exposure by Tier

A kill switch is a control only if it has been tested and its blast radius is documented. For tier-specific staging:

minimal — standard progressive rollout. Percentage-based flag is fine.
transparency — rollout gated on the disclosure being live in the UI for that surface, for every locale in scope. Not "disclosure ticket created."
high — rollout gated on the oversight hook being exercised in a canary, with the ack rate above a floor. If the ack rate drops below the floor, auto-disable rather than alert.
prohibited — no kill switch needed; the capability should not exist in the build.

That last row is not a joke. If your architecture requires a runtime kill switch for a capability you consider prohibited, you have decided to ship it.

Why the Feature Flag Is the Wrong Control Plane on Its Own

Flags are optimised for one property: fast, reversible, low-ceremony change. Compliance systems are optimised for the opposite: slow, evidenced, high-ceremony change. Building the second on top of the first produces three specific failures:

Deletion. Flags get archived. Manifests do not. If the flag is the record, the record disappears.
Aliasing. A flag called ai.smart_replies can front three different prompts over two years. The flag name survives; the capability does not.
Audience. Flag changes are visible to whoever has the flag provider's dashboard open. A tier change needs to be visible to legal, security, and the capability owner.

Keep flags as the delivery mechanism. Derive them from the manifest: the manifest is the source, your flag provider is a target, and a sync job ensures every live manifest has a flag and vice versa.

Worked Example: A Flag-to-Gate Pipeline in CI

The Gate Script and Its Inputs

The gate reads a generated index (a small tsx script imports every manifest and JSON.stringifys the array) plus the filesystem, and asserts that each capability has the artefacts its tier demands.

// scripts/ai-gate.mjs




const ROOT = "ai/capabilities";
const exists = (p) => access(p).then(() => true, () => false);

const REQUIRED = {
  high: (dir) => [
    `${dir}/risk-assessment.md`,
    `${dir}/tech-doc.md`,
  ],
  transparency: (dir) => [`${dir}/notice.md`],
  minimal: () => [],
  prohibited: () => [],
};

export async function run(indexPath) {
  const caps = JSON.parse(await readFile(indexPath, "utf8"));
  const findings = [];

  for (const cap of caps) {
    const dir = path.join(ROOT, cap.id);
    const { tier, rules } = classify(cap);

    if (tier === "prohibited") {
      findings.push({
        level: "fail", id: cap.id, tier,
        message: `prohibited-practice rule matched (${rules.map((r) => r.id).join(", ")}); ` +
                 `capability must be removed or its declared purpose corrected`,
      });
      continue;
    }

    for (const file of REQUIRED[tier](dir)) {
      if (!(await exists(file))) {
        findings.push({ level: "fail", id: cap.id, tier, message: `missing ${file}` });
      }
    }

    // High-risk also needs an eval record for the exact pinned model version.
    if (tier === "high") {
      const evalFile = `${dir}/evals/${cap.model.pinnedVersion}.json`;
      if (!(await exists(evalFile))) {
        findings.push({ level: "fail", id: cap.id, tier, message: `missing ${evalFile}` });
      }
    }
  }

  for (const f of findings) {
    console.log(`${f.level === "fail" ? "FAIL" : "WARN"}  ${f.id.padEnd(34)} tier=${f.tier}`);
    console.log(`      ${f.message}`);
  }

  const failed = findings.filter((f) => f.level === "fail").length;
  console.log(`\n${caps.length} capabilities checked, ${failed} failed`);
  return failed === 0 ? 0 : 1;
}

process.exit(await run("ai/capabilities.index.json"));

Observed Output From a Dry Run of the Gate

The classifier is a pure function over the manifest, so this output is deterministic for the fixtures. Four manifests, each deliberately broken in a different way:

FAIL  sentiment-scan                   tier=prohibited
      prohibited-practice rule matched (PR-4); capability must be removed or its declared purpose corrected
FAIL  recruiting-cv-rank               tier=high
      missing ai/capabilities/recruiting-cv-rank/risk-assessment.md
FAIL  support-reply-suggest            tier=high
      missing ai/capabilities/support-reply-suggest/evals/2026-02-14.json
FAIL  helpcenter-article-summarise     tier=transparency
      missing ai/capabilities/helpcenter-article-summarise/notice.md

4 capabilities checked, 4 failed
exit code 1

Look at the last line. A missing disclosure artefact currently fails the build, and that is a real decision point rather than an accident — downgrading it to a warning with a dated exception is defensible for a transparency-only capability. But don't leave it implicit; whichever you choose, write down why in the gate's README next to the rule.

Wiring it into CI is one step:

- run: npm run ai:index && npm run ai:gate

What the Gate Cannot Verify and Who Must Sign Off

The gate verifies presence, types, and rule matching. It cannot verify:

Truthfulness of the manifest. A manifest can declare socialScoring: false while the product scores people. Nothing in CI catches a lie. Detection comes from review of the manifest diff by someone who understands the product, which is why the manifest review is the highest-leverage review in the process.
Whether oversight is real. A passing oversightGate test proves the middleware rejects requests without an ack. It does not prove the ack came from a human exercising judgment.
Whether the pinned version ran. That is a runtime check against build.json, not a CI check.
Legal adequacy of the documentation. Generated docs cover facts; the risk assessment is a human document.

Sign-off, roughly: capability owner for the manifest and intended purpose, security/ML platform for data and oversight implementation, counsel for tier overrides and for high-risk documentation. Record the sign-off in the repository — a commit trailer or a required reviewer — so it has a timestamp and a name.

Where Engineering Teams Will Feel the Digital Omnibus First

First-Order Impacts on Design, Data, and Release Process (Inference)

Everything in this section is inference from the architecture, not from the regulation's text, which I have not seen.

Release velocity on AI features slows at the classification boundary, not at the code boundary. The bottleneck becomes "who signs this manifest," and that is a coordination problem before it is a legal one.
Shared gateways become risk aggregation points. One model alias serving twelve capabilities means one version bump can re-classify twelve manifests at once. Expect pressure to pin per capability, and expect that to cost money and latency.
Data retention becomes a product decision. retentionDays in a manifest is a prompt to the privacy team; today it is a default in a config file nobody read.
Prompt changes become reviewable artefacts. Prompt hashes in build.json turn "we tweaked a word" into a diff. Teams will initially find this annoying and later find it essential.
Eval infrastructure stops being a research luxury. High-risk capabilities need per-version eval records, which means an eval harness on the critical path for release.

What Would Change This Picture

Watch for these, and treat each as an unresolved question rather than a settled fact:

If the omnibus shifts application dates for high-risk obligations, the urgency of the documentation workstream changes but not its shape. Building it early is cheap; retrofitting it under a deadline is not.
If Annex III scope narrows, some capabilities drop a tier. Your classifier makes that a RULES diff, which is the whole argument for encoding law as code.
If the final text adds deployer-side duties for high-risk use (fundamental-rights impact assessment, registration), the manifest schema grows a field and the sign-off matrix grows a reviewer. Same machinery.
If nothing is confirmed — if the ELI check comes back empty and the item turns out to be about a proposal — your baseline is still Regulation (EU) 2024/1689 with its existing dates. The engineering work does not wait for the omnibus.

Anti-Patterns in AI Compliance Engineering

The compliance wiki. A Confluence page describing controls that no build step enforces. It survives until someone asks for evidence.
isHighRisk: true. A boolean opinion stored as a fact, unversioned, un-cited, and set by whoever was in the room.
"We're only a deployer." True until you change the intended purpose or brand the output. Art. 25 is a one-line rule with a large blast radius.
Prompt-only guardrails as the oversight hook. A system prompt saying "be careful" is not a control; it is a hope with a token cost.
Alias-based model pins. gpt-latest-style aliases make every claim about your system unfalsifiable.
Logging everything and calling it traceability. Logs without modelVersion and promptHash cannot answer the only question that matters after an incident.
Deleting the flag during cleanup. The capability keeps running, the manifest goes stale, and the next audit finds an undocumented system in production.
Classification by whoever is least busy. The classifier should be boring and automated; the review of the manifest facts is where senior time belongs.

A Practical Rollout Checklist for AI Capability Auditing

Confirm the legal text. Run the ELI check. Record the document identifier and fetch date. Do not classify against a summary.
Define the unit. Write the one-paragraph definition of "capability" for your codebase and put it in the repo README for ai/.
Add the types. AICapability with satisfies on every manifest. Make the type change a compile error for incomplete manifests.
Backfill manifests, worst first. Start with anything touching employment, education, credit, or public services. Do not start with the chat sidebar.
Encode rules with citations. Every rule gets a cite string. A rule without a citation is an opinion.
Ship the classifier as a library with unit tests per rule, including the throwing-rule path.
Gate on artefacts, not assertions. Fail the build on missing files. Warn, don't fail, on the orphan-flag report.
Generate tech docs from the manifest and diff them between tags. Read the diffs in release review.
Implement the oversight hook and test the rejection path. A 428 for an unreviewed high-risk output is a passing test.
Pin model, prompt, and index versions and assert the served version matches at runtime.
Test the kill switch in staging, including the auto-disable path for a collapsing ack rate.
Schedule the weekly drift report and give every orphan a named owner with a date.

Conclusion

The useful thing about a regulation you cannot yet read is that it forces you to notice how little of your own system you can describe. The ELI check will eventually resolve, the amendment articles will be published, and someone will produce a mapping table. None of that tells you which of your flags fronts a scoring function in an employment context, or which model version actually answered yesterday's requests.

Build the inventory first. Declare facts in manifests that fail to compile when incomplete. Encode the law as a rule table where every rule carries a citation, so a legal change is a reviewed diff rather than a forgotten memo. Generate the documentation, diff it between releases, and treat the diff as the review artefact. Enforce the tier in the request path with a hook you have tested to reject, and pin the model version so your claims are falsifiable.

The feature flag gets you a rollout. The manifest gets you an answer you can defend. Those are different systems, and only one of them survives a flag cleanup.

Introduction