Auditing VS Code 1.147's Scheduled AI Agents: Model Routers, Permissions, and Reproducible Runs

Auditing VS Code 1.147's Scheduled AI Agents: Model Routers, Permissions, and Reproducible Runs

pr0h0••34 min read•tech
AI Usage (88%)
On this page

Introduction — VS Code 1.147 stopped suggesting and started scheduling

VS Code 1.147 ships three capabilities that only make sense read together — scheduled AI tasks, browser control, and model routers — and this post is an audit plan for the combination: how to scope permissions for an unattended agent, how to log what actually ran, and how to keep scheduled runs reproducible before you trust any of it without a human watching.

The factual basis is thin, so start with what is actually confirmed. Visual Studio Magazine reported on 2026-11-18 that VS Code 1.147 adds scheduled AI tasks, browser control, and model routers. That is the entire factual payload. Everything past this paragraph is engineering judgement about what those three features mean together, and I will keep the line between the two visible throughout — a secondary trade-press summary is a weak basis for a security posture, and pretending otherwise would be the worst thing this post could do.

Read the three as one story and the shape is obvious: the editor is being moved from an interactive assistant that answers when spoken to, into a scheduler that wakes up on its own, drives a browser, and picks a model through a policy layer. Each feature alone is unremarkable. Together they describe an unattended process that holds your credentials.

What changed in 1.147: scheduled AI tasks, browser control, and model routers as one orchestration story

The three announced capabilities, taken at face value:

  • Scheduled AI tasks — the editor runs AI work on a trigger instead of a keystroke.
  • Browser control — the agent drives a browser session: it reads rendered pages and interacts with them.
  • Model routers — model selection becomes a policy decision instead of a per-request choice.

Chat features fail closed. If you do not type, nothing happens. A scheduler fails open in the sense that it runs whether or not you are watching. Add browser control and the agent now processes content it did not author, from a network it does not control. Add a router and where your source code goes becomes a policy resolution rather than something you decided in the moment. The composition is what matters, not any single item.

The technical position this post takes: an editor agent is a credential-bearing process, and should be audited like one

Stated up front so you can disagree with something specific: any agent running inside your editor should be treated as a process holding your user's credentials, your shell's reach, your browser's cookies, and your network's egress — not as a feature you toggle. That framing has consequences. The interesting questions stop being "is the model good" and become "what can it read, what can it write, where can it send bytes, and can I reconstruct what happened afterwards". If a capability cannot be logged, it should not be scheduled.

Concretely, I would not enable an unattended schedule on a developer workstation until two things hold: the run produces a structured record you can diff, and the agent's output lands as a reviewable artifact — a diff, a patch, a report — rather than a side effect on your working tree. Everything below is either support for that claim or a mechanism for meeting it.

Scope and method — a single trade-press report dated 2026-11-18, so confirmed facts are thin and inference is labelled throughout

I have not run VS Code 1.147. I have not read its release notes, its settings schema, or its telemetry documentation. The public detail available is one report dated 2026-11-18 naming three features. Therefore:

  • Statements about what the features are are attributed to that report.
  • Statements about how they behave are inference, phrased as "likely", "appears to", or "untested".
  • Code here is portable scaffolding, not a reverse-engineered integration. Where I show output, I label it illustrative unless I say otherwise.

That is an uncomfortable place to write from. A vendor changelog gives you the mechanism and lets you reason about bounds; a trade-press summary gives you the headline and leaves the bounds to you. The right response is not to fill the gap with confident prose — it is to build the audit path first and let the documentation fill itself in.

What the 1.147 report claims — and what it leaves unstated

The three named features, restated precisely without embellishment

Restating minimally, because embellishment is how summaries become folklore:

  1. Scheduled AI tasks exist. The report does not describe the trigger vocabulary.
  2. Browser control exists. The report does not describe which browser it drives or which profile it uses.
  3. Model routers exist. The report does not describe the routing criteria or the fallback behaviour.

That is all of it. Note what is missing: no permission model, no logging surface, no statement about which credentials the browser session carries, no statement about whether task definitions live in user settings or in the workspace.

Confirmed vs. unconfirmed table: trigger model, permission model, logging surface, browser session scope

QuestionStatusNotes
Are scheduled AI tasks a 1.147 feature?Confirmed by the reportNamed in the 2026-11-18 Visual Studio Magazine article
Is browser control a 1.147 feature?Confirmed by the reportSame source
Are model routers a 1.147 feature?Confirmed by the reportSame source
Trigger vocabulary (cron? interval? on-save? on-open?)UnstatedDo not assume cron semantics; interval drift matters for cost
Permission model (per-task approval? workspace trust integration?)UnstatedThe single most important unknown
Run logging (does the editor emit a per-run record?)UnstatedAssume no structured log until you see one
Browser profile used by browser controlUnstatedWhether it reuses your logged-in profile is decisive for blast radius
Where task definitions are storedUnstatedDetermines whether a schedule is reviewable in a pull request
Whether routers fall back silentlyUnstatedDetermines whether reproducibility is even achievable

Why a secondary source forces a different audit posture than a vendor changelog

A changelog enumerates settings, defaults, and known limitations; you read the mechanism and derive the bounds. A secondary report gives you the nouns. The practical difference is that you cannot pre-write a policy document from a secondary source — you can only pre-write the instrumentation that will tell you what the policy should be.

So the posture flips: not "read the docs, then enable" but "enable behind instrumentation, observe, then decide". The wrapper script, the audit record, and the diff review come first; the feature flag comes last. If that ordering sounds backwards for a developer tool, consider that you would not deploy an unattended CI job that logs nothing — this is the same object with a different host process.

What would confirm the open questions (release notes, settings schema, telemetry docs) and what changes if they come back differently

Concrete verification steps that do not require guessing at identifiers:

## Inspect the shipped settings and their defaults for anything schedule/router related.
code --version
ls ~/.vscode/extensions 2>/dev/null | head
## Dump the resolved default settings for the running build.
code --list-extensions --show-versions > /tmp/ext.txt 2>&1; echo "exit=$?"
## Find the settings schema bundled with the install (path varies by OS/build).
find /usr/share/code /Applications/Visual\ Studio\ Code.app -name "*.json" -path "*schema*" 2>/dev/null | head

The output is not the point. The point is that the settings schema is the authoritative source for defaults and it ships locally — read the defaults instead of trusting a blog post, including this one.

What changes depending on what you find:

  • If task definitions live in workspace files, a schedule becomes a reviewable code change and the audit story gets easier — treat it like CI config.
  • If they live in user settings, schedules are invisible to code review and you need a separate inventory process.
  • If browser control reuses the default profile, treat every scheduled browser task as having access to every account you are logged into, and do not enable it on a primary workstation.
  • If the router exposes its resolution, the reproducibility problem is mostly solved; if it does not, you are back to black-box testing.

The trust boundary moved — your editor now holds the credentials

The security-relevant change is not that AI writes code. It is that the process making decisions now runs with your user identity, in your session, with your environment.

Reach of a scheduled task: workspace files, terminal, network egress, browser session, git remote

Enumerate the reach explicitly, because vague fear is useless and specific fear is actionable:

SurfaceWhy it matters for an unattended run
Workspace filesRead access to .env, credential files, private config, and any vendored secrets
TerminalCommand execution implies everything the shell can do, including curl to arbitrary hosts
Network egressExfiltration channel; also the channel for injected instructions
Browser sessionCookies, local storage, logged-in SaaS accounts, extension state
Git remotePush access means the ability to publish changes, not just make them

An interactive assistant touching all five is a tool. An unattended one touching all five is a service account with a UI. That reframing is the whole argument.

Threat model: indirect prompt injection from repo content, dependency READMEs, fetched pages, and issue text

The realistic attacker here is not a person at your keyboard. It is content the agent reads:

  • a dependency README containing instructions aimed at an agent
  • a fetched documentation page
  • a GitHub issue body pulled in as context
  • a commit message, or a code comment in a vendored file
  • a rendered page during a browser task

Injection through content is the default failure mode for browser control, not an edge case — the agent's job is to read untrusted text and then act. If the same process can execute a command, the distance between "read hostile text" and "run hostile command" is one tool call.

⚠️

Do not test prompt-injection resistance against production accounts. Use a throwaway browser profile, a disposable repo, and a test account with no privileges beyond the fixture.

Blast radius of an editor-context agent versus a CI runner — shared keychain, logged-in browser, uncommitted work

A CI runner is built around the assumption that its code is hostile: credentials are injected per job, the filesystem is ephemeral, the network is usually constrained. A developer laptop assumes the opposite.

PropertyCI runnerEditor agent on a laptop
Credential scopePer-job, narrowly scopedWhatever is in your keychain and env
FilesystemEphemeral checkoutHome directory, SSH keys, other repos
Browser stateNoneWhatever you are logged into
NetworkOften restrictedUnrestricted, including your VPN
Uncommitted workN/APresent, and overwritable
Blast radiusOne jobEverything the user can reach

The last two rows are why I would not put a write-capable schedule on a workstation. Uncommitted work is the least protected state a developer has, and it is exactly what a helpful agent is most likely to "fix".

Why "it only runs locally" is not a containment argument

Local execution constrains where code runs, not what it can reach. A local process with your cookies can still make authenticated requests to your SaaS accounts. A local process with your SSH key can still push. A local process with unrestricted egress can still exfiltrate. Locality is a deployment property; containment is a permission property. Confusing them is how laptops become the softest target in an otherwise well-run environment.

Anatomy of a scheduled run: trigger, context, tools, router, output sink

To audit something you need a model of it. Here is the one I would use, and it holds whether or not 1.147 implements any of it explicitly.

The five moving parts: trigger, assembled context, tool surface, model or router decision, output sink

  1. Trigger — what started this run, and who owns the schedule.
  2. Assembled context — the exact bytes placed in front of the model: files, diffs, issue text, fetched pages.
  3. Tool surface — which tools were available, not merely which were called. Availability is the attack surface; invocation is just the sample.
  4. Model or router decision — which provider and model actually processed the input, and on what grounds.
  5. Output sink — where the result went: a diff, a comment, a commit, a file write, a browser action.

If you cannot name all five for a given task, you cannot audit it. Note that (2) and (4) are the two almost never recorded by default, and exactly the two that matter for data handling and reproducibility.

State that persists between runs — caches, workspace storage, working tree, browser profile

Run N+1 is not run N. Between them, any of this can change and change the outcome:

  • the model provider's cache and any prompt caching
  • the editor's workspace storage (conversation history, embeddings, indexes)
  • your working tree, including half-finished edits
  • the browser profile: cookies, service-worker state, extension storage
  • the git index, which may now hold a staged change from run N

This is the quiet reproducibility killer. Two runs of the same task on the same commit can diverge solely because the browser still holds a session or the workspace index was rebuilt. If your comparison method only looks at final output text, you will blame "model nondeterminism" and stop investigating.

Questions to answer before enabling any schedule (who owns it, what happens on failure, who reads the output)

Write the answers down. If any answer is "nobody" or "I guess", do not enable the schedule.

  • Who owns this schedule, and what happens when that person is on leave?
  • What is the failure behaviour — retry, back off, or stop? A retry loop against a paid router is a billing incident.
  • Who reads the output, and in what form?
  • What is the maximum cost per run and per day, and what enforces it?
  • What is the maximum time per run, and what kills a stuck browser task?
  • Does the task write to the workspace, or only propose changes?

Model routers — the policy layer that decides who sees your code

This feature has the longest tail, because it converts a per-request choice into a standing policy — and standing policies are invisible until something goes wrong.

What a router plausibly routes on: task class, cost ceiling, latency, provider availability, configured preference

Inference, clearly labelled: routers in this class generally route on some combination of task classification, cost ceiling, latency target, provider availability, and an explicitly configured preference order. I do not know which of these 1.147 exposes. What matters is that every criterion carries a data-handling consequence, and none of them are things a security reviewer would naturally think to ask about.

Data-handling consequences: retention windows, training opt-outs, and residency now depend on a routing decision you did not make explicitly

If the router picks the provider, the router picks the retention window. It picks whether your code lands on a provider where you configured a training opt-out and one where you did not. It picks the region. A compliance conversation that starts with "which model do you use" now answers "it depends, per request, on a policy I configured once in an editor setting and have not read since".

That is not automatically bad — cost-aware routing is genuinely useful, and keeping cheap classification on a small model is good engineering. It is bad when it is unobservable. The useful part and the dangerous part are the same mechanism.

Audit question that matters: can you prove after the fact which model processed which input

This is the only router question I care about. Not "does it route well" — "can I reconstruct the mapping from input to model, after the fact, without asking the vendor".

If you cannot, then you cannot answer a data-subject request, cannot scope an incident, and cannot reproduce a run. One log line per request fixes it:

{
  "ts": "2026-11-19T09:14:02.118Z",
  "taskId": "nightly-lint-triage",
  "router": { "policyId": "default-cost-aware", "requested": "auto" },
  "resolved": { "provider": "provider-a", "model": "small-classifier", "reason": "task-class:classification" },
  "promptSha256": "9f2c…",
  "tokensIn": 3184,
  "tokensOut": 412
}

Illustrative shape, not captured from a real 1.147 run. The essential field is reason: without it, a substitution looks identical to a first-choice selection.

Silent substitution and fallback chains as a reproducibility hazard

Fallbacks are the reproducibility problem wearing a reliability costume. A provider goes down, the router falls through to the next model, the task completes, the run is green — and the output came from a model you did not pin, under a retention policy you did not evaluate. Nothing in the UI necessarily says so.

For scheduled tasks this is worse than for interactive ones, because nobody is watching for the quality shift. Two defensive rules follow:

  • If the router exposes a "no fallback" mode, use it for anything that produces a committed artifact.
  • If it does not, log the resolution and treat an unexpected resolved value as a failed run, not a successful one.

Lock-in mechanics — router policy expressed as editor settings versus policy expressed in-repo

A router policy stored in editor settings is a configuration you cannot review, cannot diff across the team, and cannot carry to another editor or to CI. The same policy stored in the repo as a plain file is a code change with an author and a reviewer.

The distinction is not aesthetic. It decides whether "who changed the routing to send this repo to that provider" is an answerable question. Prefer in-repo. If the editor only supports settings-based policy, mirror the intended policy in a repo file and add a check that the two agree — imperfect, but it turns silent drift into a failing test.

Browser control — the largest new attack surface

Command execution at least has a vocabulary you can deny. A browser is a general-purpose interpreter of untrusted content with a credential store attached.

Session scope: which profile, which cookies, which extension state, which logged-in accounts

The first question to answer, and the one the report does not answer: which profile does browser control use? Three possibilities with very different risk:

  • A dedicated automation profile — best case; scope is whatever you logged into there.
  • A fresh ephemeral context per run — good for tests, useless for anything requiring auth.
  • Your everyday profile — worst case; the agent inherits every session you have.

Until you have verified which, assume the worst case and test on a machine where the worst case is harmless.

Indirect prompt injection through page content is the default failure mode, not an edge case

An agent driving a browser is, by construction, executing instructions derived from page content. That is the feature. So the security question is not "can it be injected" — it can, trivially — but "what can an injected instruction reach".

If the answer includes terminal execution plus network egress, injection is remote code execution with extra steps. If the answer is a read-only DOM query returning text into a log, injection is noise. The permission surface is the mitigation; no prompt fixes this.

Containment baseline: dedicated browser profile, no production credentials, egress allowlist, loopback-only targets for tests

The baseline I would require before enabling a scheduled browser task anywhere real:

  • Dedicated profile, created for automation, containing no production sessions.
  • No credentials in that profile that can mutate production state.
  • Egress allowlist at the host or proxy layer, not in the prompt.
  • For test fixtures, serve locally: http://127.0.0.1:8080 only.
  • Explicit run timeout, and a hard kill for the browser process.
## Minimal egress sanity check: what does the automation host actually resolve and reach?
getent hosts example.com >/dev/null && echo "dns-ok"
curl -sS -o /dev/null -w '%{http_code}\n' --max-time 5 http://127.0.0.1:8080/ || echo "fixture-down"

Observed-result hygiene matters here: a fixture that is down returns a connection error, and an agent that "handles" that error by fetching a fallback URL has just made an unplanned network request. Check the fixture is up before the run, not during triage.

What to check in a captured browser session trace before trusting an automated UI test

A green UI test proves the assertion passed, not that the agent did the intended thing. Review the trace for:

CheckFailing signal
Navigation targetsAny host outside the allowlist
Form submissionsPOSTs to endpoints the test never mentioned
Auth stateRequests carrying cookies from a non-automation profile
File downloadsAny download at all, in a task that should not download
Console errors swallowedRetries that silently changed the assertion path
TimingSteps that passed only because of a race

The recurring theme: the assertion is a single bit, the trace is the evidence. Keep the trace.

Permission design for an agent that runs unattended

Workspace trust and directory boundaries — what the agent can read versus write

Read and write must be separate decisions. A triage agent needs to read the repo and write one report file; it does not need to write source. Express the boundary as a path set, not a sentiment:

// policy/paths.js — explicit, reviewable, boring on purpose.
export const agentPolicy = {
  read: ["src/**", "test/**", "docs/**", "package.json", "package-lock.json"],
  write: ["artifacts/agent/**"],
  deny: [".env*", "**/*.pem", "**/.ssh/**", ".git/config"],
};

Two properties make this useful. Deny is evaluated before allow, so a broad glob cannot accidentally re-permit a secret. And the write set is a single directory, which means git status after a run tells you immediately whether the agent stayed in bounds — a cheap, high-signal invariant.

Command execution: deny by default, allowlist by exact command, no shell interpolation from model output

The failure mode to design against is not a malicious command; it is a benign-looking command assembled from model output and handed to a shell. Quoting mistakes and shell metacharacters do the rest.

// Allowlist by exact argv, never by substring match, never via a shell.
const ALLOWED = new Map([
  ["npm-ci", ["npm", ["ci", "--ignore-scripts"]]],
  ["unit-tests", ["npm", ["test", "--", "--runInBand"]]],
]);

export function runAllowed(name) {
  const entry = ALLOWED.get(name);
  if (!entry) throw new Error(`command not allowed: ${name}`);
  const [cmd, args] = entry;
  return new Promise((resolve, reject) => {
    execFile(cmd, args, { timeout: 120_000, shell: false }, (err, stdout, stderr) => {
      if (err) return reject(Object.assign(err, { stdout, stderr }));
      resolve({ stdout, stderr });
    });
  });
}

execFile with shell: false and a fixed argv means model output can select a command name from a closed set, but cannot contribute a token to the argv. That is the difference between an agent that can run tests and an agent that can run arbitrary code.

Keeping secrets out of model context and out of logs (env scoping, redaction before logging, no credentials in fetched pages)

Two leaks to prevent, with different fixes.

Into the model: scope the environment the task inherits. Do not hand the agent a shell that already has your cloud credentials exported. A scheduled task should inherit a minimal environment, explicitly constructed.

Into the logs: redact at write time, not read time, because logs get committed, copied, and shipped.

const RULES = [
  [/\b(AKIA|ASIA)[A-Z0-9]{16}\b/g, "[redacted:aws-key-id]"],
  [/\bgh[pousr]_[A-Za-z0-9]{36,}\b/g, "[redacted:github-token]"],
  [/\beyJ[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\.[A-Za-z0-9_-]{10,}\b/g, "[redacted:jwt]"],
  [/(?<=(?:password|token|secret|api[_-]?key)\s*[=:]\s*)\S+/gi, "[redacted:value]"],
];

export const redact = (line) => RULES.reduce((out, [re, tag]) => out.replace(re, tag), String(line));

The lookbehind rule is the workhorse and also the one that will embarrass you eventually: it catches password= style assignments and nothing semantically richer. Redaction is a seatbelt, not a boundary — the real control is never putting the secret in the transcript.

Human checkpoints that survive rubber-stamping — diff review, not "approve run"

"Approve run" is a button that trains people to click. It asks for consent before the evidence exists. Diff review asks after.

Design the checkpoint to present what changed, with the run record attached, and make the default action "do nothing". If a change is not visible as a diff, the checkpoint is theatre.

Building an audit log you can actually rely on

Minimum per-run record: task id, trigger time, model identifier, prompt hash, tool calls, file diffs, exit status, duration

The minimum viable record. Each field earns its place:

FieldQuestion it answers
taskIdWhich schedule produced this?
triggeredAt / durationMsWas it on time, and did it hang?
model and router.reasonWho processed the input, and why them?
promptSha256Was the input the same as last time?
toolCalls[]What did it actually do?
writes[] with hashesWhat changed on disk?
exitGreen or not — but read it last, not first

promptSha256 is the field people skip and then regret. It is how you separate "the model behaved differently" from "the input was different", and those two have completely different fixes.

Where to capture it when the editor does not emit it — wrapper script, git hooks, local HTTP proxy, filesystem watch

Assume no native structured log. Four capture points, in order of preference:

  1. Wrapper script around whatever the schedule invokes. Cheapest, most complete for process-level facts.
  2. Git hooks — pre-commit and post-commit for write attribution.
  3. Local HTTP proxy with HTTPS_PROXY set for the child process, for egress and provider attribution.
  4. Filesystem watch on the write set, as a backstop.
// scripts/audited-run.mjs — wraps a task, emits one JSON record per run.
const taskId = process.argv[2] ?? "unnamed";
const id = `${Date.now()}-${randomUUID().slice(0, 8)}`;
const startedAt = new Date().toISOString();
const t0 = process.hrtime.bigint();

const promptPath = process.env.AGENT_PROMPT_FILE;
const promptSha256 = promptPath
  ? createHash("sha256").update(await readFile(promptPath)).digest("hex")
  : null;

const child = spawn(process.execPath, ["scripts/task.mjs", taskId], {
  stdio: ["ignore", "pipe", "pipe"],
  env: { PATH: process.env.PATH, NODE_ENV: "production" }, // minimal env, no inherited secrets
});

let stdout = "";
child.stdout.on("data", (c) => (stdout += c));
child.stderr.on("data", (c) => (stdout += c));

child.on("close", async (code) => {
  await mkdir(".agent-audit", { recursive: true });
  const record = {
    taskId,
    runId: id,
    triggeredAt: startedAt,
    durationMs: Number((process.hrtime.bigint() - t0) / 1_000_000n),
    promptSha256,
    exit: code,
    outputTail: redact(stdout.slice(-4000)),
  };
  await writeFile(`.agent-audit/${id}.json`, JSON.stringify(record, null, 2));
  process.exitCode = code ?? 1;
});
$ node scripts/audited-run.mjs nightly-lint-triage; echo "exit=$?"
exit=0
$ ls .agent-audit
1763546042118-3f9c1a2b.json

Again illustrative, not a captured 1.147 run. Note the deliberate omission: git state, tool calls, and model resolution are not in this record, because a wrapper around an opaque editor process cannot see inside it. Closing that gap needs a native log or the proxy-plus-hook combination.

Redaction and retention: what must never land in a log file that gets committed

Never commit: raw prompt bodies containing customer data, full HTTP bodies from authenticated sessions, environment dumps, browser cookies or storage state, and any token that survived redaction.

Retention: audit records are evidence, so treat them like evidence — append-only, timestamped, with a defined expiry. If .agent-audit/ is inside the repo, add it to .gitignore and ship records to a store that is not your working tree.

printf '.agent-audit/\nartifacts/agent/\n' >> .gitignore
git check-ignore -v .agent-audit/2026-11-19.json
## .gitignore:1:.agent-audit/	.agent-audit/2026-11-19.json

Run that git check-ignore rather than assuming. Ignore rules interact in ways that surprise people, and a committed audit log with a leaked token is a worse outcome than no log.

Logging the router decision, not just the model, so substitutions are visible after the fact

Log requested, resolved, and reason together. resolved alone tells you where the request went; reason tells you whether that was your policy or a fallback. When a task's output quality drops, look first at the distribution of router.reason across recent runs. A shift from policy to provider-unavailable is a change in your effective data-handling posture, and it will not announce itself anywhere else.

Making scheduled runs reproducible

Pinning inputs: lockfiles, container image, clean checkout, pinned model and router config

Reproducibility starts with removing the easy variables. Pin the toolchain and the dependency tree, run from a clean checkout, and pin the model — not just the router policy.

git status --porcelain            # must be empty before a reproducibility run
npm ci --ignore-scripts           # lockfile-exact, no lifecycle scripts
node --version                    # record it; the record is the point

If git status --porcelain prints anything, stop. Comparing two runs on a dirty tree compares two different inputs and teaches you nothing.

The hard part — non-deterministic text output; assert on side effects and diffs instead of prose

Model output text will differ between runs even with identical input, temperature zero, and a pinned model, because batching and hardware are not part of your contract. Asserting on prose produces a test that fails for uninteresting reasons until someone deletes it.

Assert on what is actually contractual:

  • which files changed, and their content hashes
  • which commands were invoked, in what order
  • exit status and duration bounds
  • the set of tool calls, not their arguments verbatim
  • for browser tasks, the set of URLs contacted

Ordinary testing advice applied to an unusual subject: test the observable behaviour, not the implementation's prose.

A small Node harness that normalizes two runs and diffs the structured record (CodeBlock example)

harness/compare-runs.mjs
import { readFile } from "node:fs/promises";

const normalize = (r) => ({
taskId: r.taskId,
exit: r.exit,
promptSha256: r.promptSha256,
router: { requested: r.router?.requested, resolved: r.router?.resolved },
toolCalls: (r.toolCalls ?? []).map((c) => `${c.tool}:${c.target ?? ""}`).sort(),
writes: (r.writes ?? []).map((w) => `${w.path}:${w.sha256}`).sort(),
urls: [...new Set(r.urls ?? [])].sort(),
});

const [a, b] = await Promise.all(
process.argv.slice(2, 4).map(async (p) => normalize(JSON.parse(await readFile(p, "utf8")))),
);

const keys = new Set([...Object.keys(a), ...Object.keys(b)]);
const diffs = [];
for (const k of keys) {
const left = JSON.stringify(a[k] ?? null);
const right = JSON.stringify(b[k] ?? null);
if (left !== right) diffs.push({ field: k, left: a[k] ?? null, right: b[k] ?? null });
}

if (diffs.length === 0) {
console.log("STABLE: no normalized-field differences");
} else {
console.table(diffs.map((d) => ({ field: d.field, runA: JSON.stringify(d.left), runB: JSON.stringify(d.right) })));
process.exitCode = 1;
}

Reading the harness output: a stable pair versus a drifting pair

The shape you want on a stable pair:

$ node harness/compare-runs.mjs .agent-audit/a.json .agent-audit/b.json
STABLE: no normalized-field differences

And on a drifting pair, something like this — constructed to show the failure shape, not captured from a real run:

$ node harness/compare-runs.mjs .agent-audit/a.json .agent-audit/b.json
┌─────────┬───────────┬──────────────────────────────────┬──────────────────────────────────┐
│ (index) │ field     │ runA                             │ runB                             │
├─────────┼───────────┼──────────────────────────────────┼──────────────────────────────────┤
│ 0       │ 'router'  │ '{"requested":"auto","resolved":"small-classifier"}' │ '{"requested":"auto","resolved":"large-general"}' │
│ 1       │ 'writes'  │ '["artifacts/agent/report.json:9f2c…"]' │ '["artifacts/agent/report.json:4a71…","src/index.js:1b02…"]' │
│ 2       │ 'urls'    │ '["http://127.0.0.1:8080/"]'      │ '["http://127.0.0.1:8080/","https://example.com/help"]' │
└─────────┴───────────┴──────────────────────────────────┴──────────────────────────────────┘
exit=1

Read that table as a diagnosis. Row 0 is a router substitution — the run was no longer comparing like with like. Row 1 is an out-of-bounds write: a file outside the declared write set was modified, the highest-severity signal here. Row 2 is unplanned navigation, the browser leaving the allowlist. Any one of these should fail the run; together they mean the task is not under control yet.

Two-run comparison table: what a stable run looks like versus a drifting one

SignalStableDriftingLikely cause
promptSha256equalequalNot an input problem — look at the environment
router.resolvedequaldiffersFallback or policy change
writes[] setequalextra pathAgent escaped its write set
writes[].sha256equal for generated artifactsdiffersModel nondeterminism, or changed tool versions
toolCalls setequalextra toolCapability creep or injection
urls setequal, loopback onlyexternal hostInjection or an unpinned fetch
exit0/00/0Green does not mean stable — this is the trap
durationMswithin band10xRetry loop or hung browser task

The exit row is the one to internalise. Two green runs can differ in every other row, which is why the harness normalizes on side effects rather than status.

Failure modes to test before trusting a schedule

Injection via issue bodies, commit messages, and fetched documentation

Test it deliberately, in a disposable repo, with a harmless payload: an issue body that instructs the agent to write a file outside the write set, and a README that instructs it to fetch an external URL. Success criteria are the observable ones — did a file appear outside artifacts/agent/, did a request go to a host outside the allowlist. Do not evaluate success by asking the model whether it was tricked; that measures the model's self-report, not your controls.

Runaway loops and cost amplification when a retry policy meets a paid model router

Retry-on-failure is correct engineering for a flaky network call and catastrophic for a paid inference call in a loop. The amplification path is short: a task fails a validation check, retries with a larger context, fails again, and every retry is a full-price request with growing input. Three defences, all boring:

  • a hard per-run token and cost ceiling, enforced by the wrapper
  • a hard wall-clock timeout with process kill
  • a circuit breaker: three consecutive failures pause the schedule and require a human

If the router does not expose cost per request, estimate from token counts and log the estimate. An approximate ceiling beats no ceiling.

Silent partial success — the green run that changed nothing

The inverse failure: the task exits 0, the report says "no issues found", and it found nothing because it read an empty directory or a stale checkout. Detection is simple if you design for it — assert that expected inputs were non-empty and that the run produced a non-empty artifact within a plausible size range.

Failure mode table: symptom, detection method, mitigation

Failure modeSymptomDetectionMitigation
Indirect injectionOut-of-set write or external fetchNormalized diff on writes/urlsDeny-by-default write set, egress allowlist
Runaway costDuration and token count climbWrapper ceilings, per-run recordHard timeout, circuit breaker
Silent partial successGreen run, empty artifactSize/emptiness assertionsFail on empty inputs
Router substitutionQuality shift, no errorrouter.reason distributionPin model, disable fallback
Uncommitted-work clobberDeveloper loses local editsgit status before/afterRun in clean checkout or container
Secret in logToken appears in audit filePre-write redaction + git check-ignoreRedact at write time, ignore audit dir
Assertion drift in UI testTest passes, behaviour wrongBrowser trace reviewKeep traces, review hosts
State bleed between runsSecond run diverges on clean inputTwo-run harnessFresh profile and clean checkout per run

Editor agents versus pipeline agents

Where the editor ends and the runner begins; what should never run on a developer laptop

The split I would draw: the editor is for authoring and interactive assistance; the runner is for unattended execution. Anything that runs on a schedule, holds credentials, and can write should run where the filesystem is ephemeral and credentials are scoped per job.

Never on a laptop, unattended: anything with push access, anything with production credentials, anything that drives the browser profile you also use for email, anything that can rewrite source without human review.

Ephemeral credentials and secret scoping in CI

The CI property worth copying is not the machine — it is the credential lifetime. A token minted for one job, with a one-hour TTL, a read-only scope, and an audit trail, is a fundamentally different risk object from a long-lived personal access token in a keychain. When you move a task off the laptop, move the credential model with it; otherwise you kept the blast radius and added latency.

Gate design: the agent proposes a diff, CI verifies it, a human merges it

The gate that actually works is a pipeline, not a button:

  1. Agent runs in CI, in a container, on a clean checkout.
  2. Agent emits a patch or a report artifact. It does not push.
  3. CI runs the tests against the patched tree.
  4. A human reviews the diff with the audit record attached.
  5. Merge, or discard.

Every step is ordinary software delivery. The only novel element is that step 2's author is a model, and the answer to that is not a new process — it is the existing one, applied without exception.

Exit cost and portability — what survives when the editor changes

What stays portable — git history, scripts, containers, test suites

If a scheduled agent produces a commit, that commit outlives the tool that made it. So do your scripts, containers, and tests. These are the assets that survive editor changes, vendor changes, and model changes.

What does not — task definitions, router policy, and browser automation config living in editor settings

Task definitions in user settings do not port. Router policy in editor settings does not port. Browser automation configuration in editor settings does not port, and worse, is invisible to review. This is the concrete lock-in mechanism in the 1.147 story — not that the features are proprietary, but that their configuration naturally lands in a per-user file nobody diffs.

Practical rule: keep task definitions in-repo as plain files so the schedule is reviewable in a pull request

The rule I would enforce: if a schedule can modify the repository, its definition belongs in the repository. Plain files, reviewed like code, versioned with the code they act on. Then "why did the nightly task start writing to src/" has a git log answer.

If 1.147 stores definitions in user settings, mirror them into the repo and add a consistency check. Imperfect, but it makes drift visible, and visible drift gets fixed.

Rollout checklist for enabling scheduled agents

Pre-enable checks, first-week monitoring, and the conditions that should pause the schedule

Pre-enable: read the settings schema, determine the browser profile, define the write set, define the cost ceiling, and get one green audited run in a disposable environment.

First week: compare every run against the harness, review the router.reason distribution daily, and read the browser traces for any task that touches the network.

Pause immediately if: a run writes outside the declared set, an unplanned host appears in the urls set, router.resolved changes without a corresponding policy change, cost exceeds the daily ceiling, or three consecutive runs fail.

Checklist table with owner, evidence required, and pass criteria

CheckOwnerEvidencePass criteria
Settings schema read and defaults recordedTask ownerSchema dump in the repoDefaults documented, not assumed
Browser profile identifiedSecurity reviewerProfile path + a test showing no prod sessionsAutomation profile only
Write set definedTask ownerPolicy file in PRSingle output directory, deny list present
Read set scopedTask ownerPolicy file in PRNo .env, keys, or .git/config
Command allowlistTask ownerWrapper code in PRFixed argv, shell: false
Cost and time ceilingsTask ownerWrapper configHard kill on exceed
Audit record emittedPlatformSample recordAll minimum fields present
Redaction verifiedSecurity reviewerTest with a planted fake tokenToken absent from record
Router resolution loggedPlatformSample recordrequested, resolved, reason
Two-run stabilityTask ownerHarness outputSTABLE on normalized fields
Injection testSecurity reviewerDiff showing blocked write/fetchNo out-of-set effects
Diff-review gateTeam leadPR templateAgent cannot push
Pause conditions documentedTask ownerRunbookNamed human and escalation path

Conclusion — audit the agent like a process, not a feature

The report dated 2026-11-18 is thin, and I have not pretended otherwise. What it describes is enough to establish the direction: scheduling, browser control, and routing turn an assistant into an unattended process. Whether 1.147 implements any of it well is a question I cannot answer from here — which is precisely why the audit path matters more than the feature list.

Restate the position and the two things that must be true before scheduling anything

The position: an editor agent is a credential-bearing process and should be audited like one. Two preconditions follow, and I would hold the line on both:

  1. You can reconstruct the run afterwards — task, trigger, prompt hash, model and routing reason, tool calls, file writes with hashes, exit status, duration.
  2. The agent's output is a reviewable artifact, not a side effect — a diff or a report a human merges, never a direct push and never a silent write into a working tree.

Everything else in this post is a mechanism for satisfying one of those two.

What to watch for in the next release that would settle the open questions

Four specifics would resolve most of the uncertainty, in descending order of value: the permission model for scheduled tasks (per-task approval, workspace trust integration, or nothing); the browser profile scope used by browser control; whether task definitions live in workspace files or user settings; and whether the router exposes its resolution and supports disabling fallback.

If the permission model is per-task and the router logs its decisions, this becomes a genuinely useful automation feature with an auditable trail. If neither is true, the correct deployment is CI-only, with a wrapper doing the logging the editor does not — which is what this post builds.

Further Reading

Primary context and official documentation

📝

I could not verify the 1.147 release notes, settings schema, or telemetry documentation at the time of writing. If you are making a deployment decision, treat the official release notes as the primary source and this post as the audit scaffolding to apply to them.

Share this post

More posts

Loading comments…