
Testing Inkling 2 Mini’s Laptop Coding-Agent Claim with a Node.js Harness
Three headlines in November 2026 said Thinking Machines shipped Inkling 2 Mini: open weights, a "looped" architecture, 26B parameters, laptop operation, and — per Dealroom — a lead in agentic coding among open models under 100B. Those are the claims. What I could not get hold of at the time of writing: a weight file, a model card, a license, or a benchmark methodology.
So this post does two things. First, it walks through the Node.js harness I built to test the laptop coding-agent claim properly — task completion, peak memory, end-to-end latency, and energy per successful task, measured against a conventional local model and a pinned hosted API under the same agent scaffold. Second, it works through the arithmetic behind whether "26B parameters on a laptop" is physically plausible at all. I'm not going to publish numbers for a model I couldn't load, so the results table keeps an Inkling 2 Mini row that says unavailable instead of quietly holding a plausible-looking median.
On this page
Test the Laptop Coding-Agent Claim, Not Just Token Generation
The engineering position: laptop suitability requires correct patches within explicit memory, latency, and energy budgets
Tokens per second is the wrong metric for a coding agent. One that emits 40 tok/s but needs 40 round trips to fix an off-by-one is slower to use than one that emits 15 tok/s and lands the patch in six. So the unit of measurement has to be the task: a repo state goes in, a patch comes out, and the patch either passes acceptance tests or it does not.
"Runs on a laptop" then decomposes into four budgets that must all hold at once:
- Memory — peak resident bytes for weights plus KV cache plus runtime, with headroom, because swap turns latency into a cliff.
- Latency — end-to-end wall time for a completed task, including every tool round trip.
- Energy — joules per successful task, because a cheap model that fails three times burns more battery than an expensive one that works.
- Correctness — hidden acceptance tests passing, with no scope violations.
A model can hit three of these and still be unusable. That's the reason to build a harness instead of reading a leaderboard.
Evaluation scope: one reproducible Node.js agent harness, three inference backends, and a bounded suite of repository tasks
The design is one scaffold, three backends, one task suite:
| Slot | Purpose | Status in this post |
|---|---|---|
inkling-2-mini (local) | The claim under test | Artifacts unavailable — no row |
local-conventional | Is the harness's local path itself slow? | Pinned placeholder, documented |
hosted-pinned | Ceiling check, network included | Pinned placeholder, documented |
mock | Harness validation, deterministic | Exercised |
The mock backend isn't a joke entry. It's how you prove the harness's accounting works before you spend a single GPU-second. More on that below.
Establish What Is Actually Known About Inkling 2 Mini
Reported timeline: Dealroom and VentureBeat on November 21, 2026, then CASRAI on November 22
The supplied coverage gives three timestamps, and they matter because secondary reporting tends to drift on details like parameter counts:
| Published (UTC) | Publisher | Claim in the headline |
|---|---|---|
| 2026-11-21 15:16 | Dealroom | 26B open-weights looped model, runs on a laptop, tops open models under 100B at agentic coding |
| 2026-11-21 18:29 | VentureBeat | "looped transformer model that runs locally on laptops" |
| 2026-11-22 01:17 | CASRAI | "open-weights looped model for laptops" |
Dealroom is the earliest of the three and the only one carrying the 26B figure and the under-100B coding ranking. The other two headlines, as supplied, do not repeat the parameter count.
What the coverage actually claims: open weights, a looped architecture, 26B parameters, laptop operation, and an under-100B coding lead
To be precise about what is being reported and by whom:
- Open weights — stated by all three publishers. This is a licensing claim, and licensing is exactly the kind of thing secondary coverage gets wrong. I'm not treating it as confirmed.
- Looped architecture — stated by all three. "Looped" is presented as an architecture description, not a spec. Loop count, where the loop is applied, and whether it is configurable at inference time are all unreported in the supplied material.
- 26B parameters — stated by Dealroom only.
- Runs locally on laptops — stated by all three, with no hardware qualification: no RAM floor, no quantization, no context length.
- Tops open models under 100B at agentic coding — stated by Dealroom only, with no harness, no pass@k, no budget, and no methodology in the supplied snippet.
A headline is not a specification. Every one of those five items is a claim from secondary reporting here, not a verified fact, and two of them appear in exactly one outlet.
Verification gate: locate official weights, model card, license, runtime support, and benchmark methodology before treating secondary reports as confirmed
Before any measurement, these five gates need to open:
- Weights — a downloadable artifact with a published checksum, so a revision can be pinned.
- Model card — parameter count broken down as total vs. active, layer count, attention head configuration, vocabulary size, context limit, and the evaluation methodology behind the coding claim.
- License — commercial use, redistribution, derivative naming, and whether quantized redistributions are permitted.
- Runtime support — this is the gate people skip. A looped architecture is not the standard transformer graph. Whether llama.cpp's GGUF path, or any mainstream inference server, can load it is an open question until a conversion exists and someone publishes the kernel work.
- Benchmark methodology — which agent scaffold, which tool set, what step budget, how many attempts, and whether the "under 100B" cohort was evaluated in the same harness.
Gate 4 is the one I'd bet causes trouble. "Open weights" and "open runnable runtime" are different products, and a novel architecture usually lands weights first and efficient kernels later. That's my inference — untested, and it gets settled the moment a conversion script and a supported-runtime note appear.
Separate reported claims, verified documentation, local measurements, and unresolved questions without inventing release details
A four-column ledger keeps this straight:
| Reported | Verified | Measured locally | Unresolved |
|---|---|---|---|
| 26B params (Dealroom) | — | — | Total or active parameters? |
| Looped architecture | — | — | Loop count, placement, configurability |
| Runs on laptops | — | — | Minimum RAM, quantization, context |
| Under-100B agentic coding lead | — | — | Harness, task set, budget, attempts |
| Open weights | — | — | License terms, redistribution |
I leave the Verified column empty until I have artifacts. That's deliberate.
Translate the Architecture Claim into Testable Resource Questions
Explain weight storage with a parameter-count and precision calculation, explicitly conditional on the reported 26B count
This is the one place the 26B figure is genuinely useful, because weight storage is arithmetic and arithmetic is checkable. Assume 26 × 10⁹ parameters and disregard runtime overhead entirely:
| Precision | Bytes/param | Weight storage | In GiB |
|---|---|---|---|
| FP32 | 4.0 | 104.0 GB | 96.9 |
| BF16 / FP16 | 2.0 | 52.0 GB | 48.4 |
| INT8 (Q8) | 1.0 | 26.0 GB | 24.2 |
| 4-bit (~Q4_K_M) | ~0.55 | ~14.3 GB | ~13.3 |
| 4-bit with embedding/output kept higher | ~0.6 | ~15.6 GB | ~14.5 |
## the arithmetic, so you can redo it for a different parameter count
node -e 'const p=26e9; for (const [q,b] of Object.entries({fp32:4,fp16:2,int8:1,q4:0.55})) console.log(q, (p*b/1e9).toFixed(1)+" GB", (p*b/2**30).toFixed(1)+" GiB")'
Read that table against real laptop memory. A 16 GB unified-memory machine has roughly 10–12 GB usable for a model after the OS and a browser. At 4-bit, a 26B model sits at or slightly over that line before any KV cache. A 32 GB machine has room. A 64 GB machine has comfortable room.
So "runs on a laptop" holds only with a quantization qualifier and a RAM floor attached. The claim isn't wrong; it's underspecified in a way that decides whether it runs on your laptop.
Account for runtime overhead: quantization metadata, KV cache, activations, context length, and CPU or accelerator allocation
Weights are the floor, not the peak. The rest of the budget:
- KV cache — grows linearly with context and batch. The formula is
2 × layers × kv_heads × head_dim × context × bytes. For a hypothetical 48-layer model with 8 KV heads at 128 dims, 32k context, FP16:2 × 48 × 8 × 128 × 32768 × 2≈ 6.0 GiB. I made that configuration up for illustration — the real numbers need the model card. - Activations and workspace — modest at batch 1, but not zero, and larger for long prompts.
- Quantization metadata and dequant buffers — small but nonzero; mixed-precision recipes keep embeddings and output layers at higher precision.
- Runtime allocator behaviour — mmap'd weights count toward resident memory once touched. A runtime that pre-faults the whole file hits peak immediately; one that pages lazily spikes unpredictably.
- Context limit as configured, not as advertised — an agent harness will happily push 20k tokens of file content into a prompt if you let it. Cap it deliberately.
An agent makes all of this worse than a chat session because each step re-sends a growing transcript. Step 12 carries every prior tool result.
Distinguish parameter reuse from inference work: inspect documented loop controls rather than assuming smaller weights imply faster execution
This is where "smaller weights imply faster" breaks.
In a looped (weight-tied, recurrent-depth) design, the same block of parameters is applied repeatedly. Two consequences follow directly:
- Memory is bounded by the parameter count, not by the loop count.
- Compute scales roughly with the loop count, not the parameter count.
So a 26B looped model with 8 loop iterations at inference time does work comparable to a much larger dense model per token, while occupying the memory of a 26B model. That's the trade: you buy memory headroom with compute time. On a laptop, where memory bandwidth and thermals are the binding constraints, that trade can go either way.
The testable questions are therefore not "how big is it" but:
- Is the loop count fixed at inference or configurable?
- Does increasing the loop count improve task success enough to justify the linear latency cost?
- What are the measured prefill and decode rates at each loop setting?
If the loop count is not documented or not exposed, question 1 answers "unknown" and questions 2–3 collapse. Record that in the manifest rather than guessing a value.
Define the Experiment Before Running the Coding Agent
Register exact model revisions, quantization formats, runtime versions, hosted API identifiers, and any documented loop settings
Everything that could change a number goes in one file, written before the first run.
{
"suiteVersion": "1.0.0",
"budget": {
"maxSteps": 12,
"maxToolCalls": 20,
"stepTimeoutMs": 120000,
"suiteTimeoutMs": 600000,
"maxPromptTokens": 24000
},
"fixtures": [
{ "id": "async-cancel-retry", "dir": "tasks/async-cancel-retry" },
{ "id": "validate-input", "dir": "tasks/validate-input" },
{ "id": "cross-file-config", "dir": "tasks/cross-file-config" }
],
"backends": [
{
"id": "mock",
"adapter": "mock",
"revision": "fixture-1",
"quantization": "n/a",
"runtime": "node"
},
{
"id": "local-conventional",
"adapter": "llamacpp",
"revision": "<pin: gguf sha256>",
"quantization": "<pin: e.g. Q4_K_M>",
"runtime": "<pin: server build id>",
"loopSettings": "n/a"
},
{
"id": "hosted-pinned",
"adapter": "openai-compatible",
"revision": "<pin: snapshot id, not a floating alias>",
"quantization": "provider-managed",
"runtime": "provider",
"loopSettings": "n/a"
},
{
"id": "inkling-2-mini-local",
"adapter": "TBD",
"revision": "UNAVAILABLE",
"quantization": "UNAVAILABLE",
"runtime": "UNAVAILABLE",
"loopSettings": "UNDOCUMENTED"
}
]
}The last entry is in the manifest on purpose. A backend that is declared and cannot be run is a recorded fact; a backend that gets quietly dropped is a gap someone will mistake for a result.
Never pin a hosted model to a floating alias like latest. Providers rotate snapshots underneath aliases, and a rotating backend invalidates paired comparisons silently.
Compare Inkling 2 Mini, one verified conventional local model, and one pinned hosted model under the same agent scaffold
The three-way comparison separates "this model is good" from "this scaffold is slow":
- Local conventional baseline answers whether the harness itself is the bottleneck. If a well-supported local model also feels awful through this scaffold, the scaffold is the problem.
- Hosted pinned answers what the ceiling looks like when latency includes network. If hosted is also 4 minutes per task, the harness is over-budgeting steps.
- Inkling 2 Mini local answers whether the claim survives contact with this machine.
The rule that makes the comparison valid: identical system prompt, identical tool schemas, identical step and tool-call budgets, identical fixtures. Backend-specific tuning is allowed, but each change gets its own labelled row.
Record laptop hardware, OS, power mode, thermal conditions, context limits, and background workload
A template that goes in the run event header, not in a notebook:
{
"hardware": { "model": "<machine id>", "cpuCores": 0, "gpuCores": 0, "unifiedMemoryGiB": 0 },
"os": { "name": "", "version": "", "kernel": "" },
"power": { "mode": "ac | battery", "charging": false, "lowPowerMode": false },
"thermal": { "ambientC": 0, "stateBefore": "", "stateAfter": "" },
"contextLimit": 0,
"background": { "browser": false, "indexer": false, "other": "" },
"notes": ""
}
Power mode and charging state are not pedantic. On battery, many machines cap sustained package power, which silently changes decode throughput by a large factor. Charging while measuring energy makes gross energy meaningless unless you account for charge going into the battery.
Define task success, timeout, tool-call budget, retry policy, and failure categories before inspecting results
Predeclare them, because post-hoc categories always flatter the model:
| Category | Definition | Counts as |
|---|---|---|
passed | Hidden suite exits 0 AND scope check clean | Success |
failed_tests | Agent declared done, hidden suite fails | Failure |
budget_exhausted | Steps or tool calls hit the cap | Failure |
timeout | Step or suite wall clock exceeded | Failure |
protocol_error | Malformed tool call, unparseable arguments | Failure |
runtime_error | Backend crashed, unreachable, OOM | Failure, environment-attributed |
scope_violation | Patch touches protected files | Failure |
Retry policy: each fixture runs once per backend per trial. A retried run is a new run with a new ID and gets included in the denominator. No silent retries of flaky runs — that's the single easiest way to fake a benchmark.
Separate equal-budget comparisons from backend-specific tuning and label every configuration change
Two run families:
parity— identical budgets everywhere. The headline comparison. Any backend that needs more than 12 steps to do a three-line fix is telling you something real.tuned— per-backend settings, changed for a stated reason, recorded in the manifest diff. Useful for "what is the best this model can do", useless for cross-model claims.
Never mix rows from the two families in one table without a column that says which family the row came from.
Build a Small Coding Suite with Hidden Acceptance Tests
Create pinned JavaScript fixtures covering asynchronous control flow, input validation, and cross-file changes
Three fixtures, each targeting a failure mode that's common for agents:
| Fixture | Bug class | Why it is here |
|---|---|---|
async-cancel-retry | Async control flow | Requires reasoning about ordering, not just syntax |
validate-input | Input validation | Edge cases live outside the happy path in the bug report |
cross-file-config | Multi-file change | Requires touching two files consistently |
Three tasks is an anecdote, not a benchmark. Saying so up front keeps the results section from overreaching later.
Walk through one complete fixture: visible bug report, baseline failure, hidden edge cases, and exact success criterion
The async-cancel-retry fixture, in full:
Visible bug report (tasks/async-cancel-retry/ISSUE.md), visible to the agent:
fetchWithRetrykeeps retrying after the caller aborts. When a user cancels the upload, we still see three more 503 retries in the log, and the promise rejects with the server error instead ofAbortError.
Visible test — covers the happy path only: a 503 followed by a 200 resolves with two attempts.
Baseline — the fixture is written so the visible test fails before any patch, because the shipped implementation calls fetch without forwarding signal:
$ node --test test/visible.test.js
not ok 1 - fetchWithRetry retries once after a 503
error: 'expected attempts to equal 2, got 1'
## pass 0
## fail 1
(That's the baseline the fixture is designed to produce. If your run prints something else, the fixture is stale, not the model.)
Hidden tests (tasks/async-cancel-retry/.hidden/acceptance.test.js), never mounted into the agent's workspace:
import assert from "node:assert/strict";
test("abort during backoff rejects with AbortError and stops retrying", async () => {
const controller = new AbortController();
let attempts = 0;
const fakeFetch = async (_url, init) => {
attempts += 1;
assert.equal(init?.signal, controller.signal, "signal must be forwarded");
if (attempts === 1) {
setTimeout(() => controller.abort(), 5);
const err = new Error("service unavailable");
err.status = 503;
throw err;
}
return { ok: true, json: async () => ({ done: true }) };
};
await assert.rejects(
() => fetchWithRetry("/upload", { fetchImpl: fakeFetch, signal: controller.signal, retries: 3 }),
(err) => err.name === "AbortError",
);
assert.equal(attempts, 1, "aborted call must not issue further attempts");
});Plus three more: abort before the first attempt; retry cap honoured exactly (not off by one); and a non-abort error still surfaces with its status code.
Exact success criterion — all of:
.hidden/acceptance.test.jsexits 0,test/visible.test.jsexits 0,git diff --name-onlyis a subset of{"src/fetch-with-retry.js"},- no test file is modified, deleted, or skipped,
package.jsondependency list unchanged.
Prevent false passes by protecting acceptance tests and checking patch scope as well as test exit status
Test exit status is a weak signal on its own. Agents have been known to edit the test, add it.skip, tighten a timeout, or vendor a helper that guts the assertion. Two cheap defences:
- Mount the hidden tests after the agent stops. They never exist in the workspace while the agent is running, so they cannot be read or edited.
- Diff the scope before trusting the exit code. Cheap, and it catches the embarrassing cases.
git -C "$WS" diff --name-only | sort > /tmp/touched.txt
comm -23 /tmp/touched.txt <(printf '%s\n' "${ALLOWED[@]}" | sort) > /tmp/violations.txt
if [ -s /tmp/violations.txt ]; then echo "SCOPE_VIOLATION"; exit 3; fi
Also worth grepping the patch for process.exit, --test-force-exit, .only(, and .skip(, and treating any of them as a scope_violation.
Reset each workspace from a clean snapshot and describe the suite's limits for broader coding-agent claims
Every run starts from a fresh git clone of a pinned commit into a temp directory, then npm ci from a lockfile, then nothing else. No carry-over from the previous run, no warm node_modules shared between fixtures unless the manifest declares it.
Limits, stated plainly:
- Three fixtures cannot rank models. They can only say whether a model is usable on this machine for this class of bug.
- Single-language, single-repo-shape, small codebase. Cross-language and large-repo behaviour are untested.
- The hidden tests were written by the same person who wrote the bug, so they share a blind spot.
- No test of long-horizon work — a 12-step cap excludes anything resembling real refactors.
None of that invalidates the harness. It just bounds what a pass or fail means.
Implement the Node.js Agent Harness
Show the project layout, run manifest, CLI arguments, and adapter contract without assuming an undocumented Inkling API
harness/
package.json
run-manifest.json
src/
cli.mjs
agent-loop.mjs
events.mjs
classify.mjs
adapters/
mock.mjs
llamacpp.mjs
openai-compatible.mjs
index.mjs
tools/
fs.mjs
test.mjs
registry.mjs
measure/
latency.mjs
memory.mjs
energy.mjs
tasks/
async-cancel-retry/
validate-input/
cross-file-config/
runs/ # git-ignored; one directory per run
The adapter contract is deliberately narrow, and it does not assume an Inkling 2 Mini endpoint. If a runtime appears, it gets a new adapter behind the same interface; if it never appears, the interface still worked for two other backends.
/**
* @typedef {Object} Adapter
* @property {string} id
* @property {() => Promise<{ revision: string, quantization: string,
* contextLimit: number, loopSettings: string|"n/a"|"UNDOCUMENTED" }>} describe
* @property {(req: ChatRequest, opts: { signal: AbortSignal }) => Promise<ChatResponse>} chat
*
* ChatRequest: { messages, tools, maxTokens }
* ChatResponse: {
* message: { role: "assistant", content: string|null,
* toolCalls: Array<{ id: string, name: string, arguments: string }> },
* usage: { promptTokens: number|null, completionTokens: number|null,
* source: "backend"|"estimated"|"unavailable" },
* unsupported: string[] // e.g. ["tool_choice", "parallel_tool_calls"]
* }
*/CLI:
node src/cli.mjs run \
--manifest run-manifest.json \
--backend mock \
--fixtures async-cancel-retry,validate-input,cross-file-config \
--trials 1 \
--out runs/$(date -u +%Y%m%dT%H%M%SZ)-mock
Normalize messages and tool results while preserving backend-specific token accounting and unsupported features
Normalize the shapes; never normalize the numbers into existence. If a backend does not report prompt tokens, the usage record says source: "unavailable" rather than an estimate silently blended with real counts.
export function normalizeUsage(raw, backendId) {
const p = raw?.prompt_tokens ?? raw?.usage?.input_tokens ?? null;
const c = raw?.completion_tokens ?? raw?.usage?.output_tokens ?? null;
return {
promptTokens: p,
completionTokens: c,
source: p === null || c === null ? "unavailable" : "backend",
};
}
Tool calls get flattened into a common { id, name, arguments } shape regardless of whether the backend returned arguments as a JSON string or an object. unsupported carries capability gaps forward so the results table can say "this backend had no structured tool calls" instead of pretending the model chose plain text.
Implement the bounded agent loop: request, validate tool call, execute, append result, and stop on completion or budget exhaustion
The loop must be boring. Every exit is explicit, every exit is logged, and no path silently continues.
export async function runAgent({ adapter, task, tools, budget, emit, runId, clock }) {
const messages = [
{ role: "system", content: task.systemPrompt },
{ role: "user", content: task.prompt },
];
let toolCalls = 0;
const finish = (status, extra = {}) => {
emit({ type: "run_end", runId, taskId: task.id, status, toolCalls, ...extra });
return { status, messages };
};
for (let step = 1; step <= budget.maxSteps; step++) {
emit({ type: "step_start", runId, taskId: task.id, step, at: clock() });
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), budget.stepTimeoutMs);
let res;
try {
res = await adapter.chat({ messages, tools: tools.schema, maxTokens: budget.maxTokens },
{ signal: controller.signal });
} catch (err) {
emit({ type: "step_error", runId, taskId: task.id, step,
code: classify(err), message: String(err?.message ?? err) });
return finish(classify(err) === "aborted" ? "timeout" : "runtime_error");
} finally {
clearTimeout(timer);
}
emit({ type: "model_response", runId, taskId: task.id, step,
usage: res.usage, unsupported: res.unsupported,
contentBytes: res.message.content?.length ?? 0 });
messages.push(res.message);
if (res.message.toolCalls.length === 0) {
return finish("declared_done");
}
for (const call of res.message.toolCalls) {
if (++toolCalls > budget.maxToolCalls) return finish("budget_exhausted");
const check = tools.validate(call);
if (!check.ok) {
emit({ type: "tool_rejected", runId, taskId: task.id, step,
callId: call.id, code: check.code });
messages.push({ role: "tool", toolCallId: call.id,
content: "ERROR: " + check.code + ". Fix the call and retry." });
continue;
}
emit({ type: "tool_call", runId, taskId: task.id, step, callId: call.id,
name: call.name, args: check.args });
const result = await tools.execute(call.name, check.args);
emit({ type: "tool_result", runId, taskId: task.id, step, callId: call.id,
ok: result.ok, bytes: result.output.length, truncated: result.truncated });
messages.push({ role: "tool", toolCallId: call.id, content: result.output });
}
}
return finish("budget_exhausted");
}Note the protocol-error path: a rejected tool call is fed back as an error message and counted against the tool-call budget. Malformed calls cost the model steps. That's the honest accounting — a model that can't emit valid JSON should not get free retries.
Provide scoped file and test tools with path validation, output limits, and explicit handling of malformed calls
Two tools only: read_file / write_file (scoped to the workspace) and run_tests (a fixed command, no arguments from the model). Fewer tools means fewer protocol failures, and protocol failures are a large share of agent failures.
import path from "node:path";
export class ToolError extends Error {
constructor(code) { super(code); this.code = code; }
}
export function resolveInside(workspace, rel) {
if (typeof rel !== "string" || rel.length === 0) throw new ToolError("bad_path");
if (rel.includes("\0")) throw new ToolError("bad_path");
if (path.isAbsolute(rel)) throw new ToolError("absolute_path");
const abs = path.resolve(workspace, rel);
const prefix = workspace.endsWith(path.sep) ? workspace : workspace + path.sep;
if (!abs.startsWith(prefix)) throw new ToolError("path_escape");
return abs;
}
// reject symlinked components: resolve the nearest existing ancestor
export async function assertNoSymlink(workspace, abs) {
let cur = abs;
for (;;) {
try {
const st = await fs.lstat(cur);
if (st.isSymbolicLink()) throw new ToolError("symlink_component");
break;
} catch (err) {
if (err.code !== "ENOENT") throw err;
const parent = path.dirname(cur);
if (parent === cur) break;
cur = parent;
}
}
if (!cur.startsWith(workspace)) throw new ToolError("path_escape");
}Output limits: cap read_file at 64 KiB and tool output at 32 KiB with an explicit truncated: true marker. Truncation has to be visible to the model and to the event log, otherwise a model that failed because it never saw the end of a file looks like a model that ignored it.
Run generated code in a disposable sandbox with no secrets, restricted network access, and externally enforced resource limits
Four layers, none of them optional:
- Disposable root — a temp directory per run, deleted after artifact collection.
- No secrets in env — pass a minimal env; do not inherit the harness's environment.
- Node's permission model — check your build first:
node --help | grep -i permission
If --permission (or --experimental-permission on older builds) is listed, use it with --allow-fs-read and --allow-fs-write scoped to the workspace and no --allow-child-process or --allow-net. If it isn't listed, your build predates the flag — do not silently skip the restriction, change the runtime.
4. External resource limits — timeouts and memory ceilings enforced by the parent, not by the code under test. ulimit -v plus a hard SIGKILL after a wall-clock cap.
The agent's shell is not a trust boundary you control. Treat anything the model produces as hostile: no network by default, no inherited credentials, no writes outside the temp root, and no reliance on the model to enforce any of it.
Write append-only JSONL events containing run IDs, task IDs, model revisions, monotonic timestamps, and terminal status
import fs from "node:fs";
export function makeEmitter(file) {
const stream = fs.createWriteStream(file, { flags: "a" });
const t0 = process.hrtime.bigint();
let seq = 0;
return (event) => {
const record = {
schema: 1,
seq: seq++,
monoMs: Number((process.hrtime.bigint() - t0) / 1000000n),
wall: new Date().toISOString(),
...event,
};
stream.write(JSON.stringify(record) + "\n");
};
}monoMs from process.hrtime.bigint() is monotonic and immune to NTP steps; wall is for humans. Both go in every record. The terminal run_end event carries the status from the loop, so an interrupted run has no terminal event at all — which is itself a detectable signal that a run was killed rather than completed.
Every record also carries runId, taskId, backendId, and the manifest's revision string, so a results table can be regenerated from the logs alone.
Measure End-to-End Latency and Peak Memory
Time model loading, inference, tool execution, and final acceptance testing separately with a monotonic clock
Four separate clocks, because they diagnose different problems:
| Phase | Measured | What a bad number means |
|---|---|---|
load | Backend ready → first token | Disk-bound weights or slow pre-faulting |
inference | Sum of per-step request durations | Compute or bandwidth bound |
tools | Sum of tool executions | Scaffold overhead, not the model |
acceptance | Hidden suite run | Fixture cost, excluded from agent latency |
const t0 = process.hrtime.bigint();
// ... do work ...
const elapsedMs = Number(process.hrtime.bigint() - t0) / 1e6;
Do not use Date.now() for phase timing. A clock step during a 3-minute agent run will produce a negative duration and you will spend an hour not believing it.
Report cold-start and warm-run latency independently, including median and tail behavior across repeated runs
Cold start includes reading weights off disk; warm start does not. They differ by an order of magnitude on laptops, so a blended average describes neither. Report both, plus p50 and p90 across trials — with three fixtures, p90 is the maximum, and the table should say so rather than implying statistical weight.
per-task latency, ms (illustrative shape, not a measurement)
backend trial task load infer tools total
...
Monitor the inference server and agent process tree rather than mistaking Node.js heap usage for total model memory
process.memoryUsage().rss for the Node process tells you nothing about a model running in a separate server. Sum the tree:
## total RSS of the inference server and the agent (Linux)
tree_rss_kb() {
local root=$1 total=0
for pid in "$root" $(pgrep -P "$root" | tr '\n' ' '); do
[ -r "/proc/$pid/status" ] || continue
total=$(( total + $(awk '/^VmRSS:/ {print $2}' "/proc/$pid/status") ))
done
echo "$total"
}
But sampling a process tree misses transients between samples. Better, if the kernel supports it: put the whole run in a cgroup v2 and read memory.peak, which the kernel maintains exactly.
## exact high-water mark for everything in the group, no sampling gap
cat /sys/fs/cgroup/hackyjs-run/memory.peak
memory.peak in cgroup v2 requires a reasonably recent kernel (5.19+). Verify with ls /sys/fs/cgroup/<group>/memory.peak before relying on it — if the file is absent, you are back to sampling, and you must say so.
Explain RSS, shared pages, unified memory, accelerator counters, sampling intervals, and missed transient peaks
Definitions that decide whether two numbers are comparable:
- RSS counts pages resident in the process's address space, including shared pages counted once per process. Summing RSS across processes double-counts shared libraries.
- mmap'd weights only count once touched. A model file opened lazily shows low RSS until the load completes.
- Unified memory (Apple Silicon) blurs the GPU/CPU split: "VRAM" and "RAM" are the same pool, so a model's footprint competes with the OS directly.
- Discrete accelerator counters report their own allocator's view, which typically excludes driver overhead and can exceed system RAM through shared memory fallback.
- Sampling interval is a resolution limit. A 200 ms sample over a 400 ms allocation spike can miss the peak entirely; only a kernel-maintained high-water mark or a cgroup counter is safe.
If your peak-memory number came from sampling, label it "sampled at N ms" — because it's a lower bound.
Treat hosted memory as unavailable and hosted latency as including network and provider-side delays
For the hosted backend:
- Memory — unavailable. Not "estimated", not inferred from a parameter count. You do not know the precision, the batching, or the tensor-parallel sharding.
- Latency — includes DNS, TLS, queueing, and provider-side scheduling. It is a legitimate number for "how long do I wait", and an illegitimate one for "how fast is this model".
Name the network path in the results table so nobody subtracts it later and calls the difference model speed.
Measure Energy Without Claiming Access to Hidden Infrastructure
Define the boundary: local device energy is measurable, but hosted-provider inference energy is not available from client timing
Local energy is a device measurement: package power over the run, minus or including idle, as declared. Hosted energy is not derivable from client-side timing — you'd need the provider's hardware, utilization, and PUE. The results table gets unavailable for hosted energy, and any claim about it is out of scope.
Document the selected platform counter or external power meter, its units, permissions, sampling rate, and coverage
| Platform | Source | Units | Notes |
|---|---|---|---|
| Linux (Intel/AMD) | RAPL energy_uj under /sys/class/powercap/ | microjoules, monotonic counter | Often root-only; wraps at max_energy_range_uj |
| Linux (whole machine) | External meter at the wall | watts | Only option that includes display and charging |
| macOS | powermetrics (requires root) | mW per subsystem | Coarse sampling; CPU+GPU+ANE coverage varies by chip |
| Windows | powercfg /energy (report, not realtime) | — | Not suitable for per-task integration |
Record the source, units, sampling rate, permission level, and coverage — package-only versus whole-machine is a large difference on a laptop where the display and wireless draw real power.
RAPL counters wrap. Read max_energy_range_uj and handle the rollover explicitly, or a long run produces a negative energy delta and a nonsense result.
Integrate power over time, retain gross energy, and label any idle-adjusted estimate and its assumptions
export function integrateJoules(samples) {
// samples: [{ tMs, watts }] ascending by tMs
let joules = 0;
for (let i = 1; i < samples.length; i++) {
const dt = (samples[i].tMs - samples[i - 1].tMs) / 1000;
const pAvg = (samples[i].watts + samples[i - 1].watts) / 2;
joules += pAvg * dt;
}
return joules;
}
Trapezoidal integration over sampled power. Keep the gross joules as the primary number, because it is what the battery actually gave up. An idle-adjusted figure (gross minus baseline draw × duration) is also useful, but it depends on the baseline being measured on the same machine in the same power mode, and it has to be labelled as an estimate with the baseline duration stated.
Control battery charging and thermal drift, batch short tasks when necessary, and include failed attempts in energy per successful task
- Pin the power mode and record it. A machine that silently caps package power on battery will report lower energy and lower throughput — the ratio can look improved.
- Do not measure while charging, or account for charge into the battery explicitly.
- Short tasks integrate badly at coarse sampling rates. If a task runs for 8 seconds and the meter samples every 1 second, either batch several tasks into one measurement window or report the number with its resolution limit attached.
- The headline metric is joules per successful task:
total_joules / tasks_passed. Failed attempts stay in the numerator. A model that burns 40% less energy per token but needs twice the attempts is not the efficient choice.
Execute a Controlled Run and Validate the Harness
Provide installation and execution commands after verifying the actual runtime, model artifacts, and adapter configuration
node --version
node --help | grep -i permission # confirm the sandbox flag on this build
cd harness
npm ci
## 1) harness validation, no model involved
node src/cli.mjs run --manifest run-manifest.json --backend mock \
--fixtures async-cancel-retry,validate-input,cross-file-config \
--trials 1 --out runs/mock-smoke
## 2) only after 1 passes: a real backend, pinned in the manifest
node src/cli.mjs run --manifest run-manifest.json --backend local-conventional \
--fixtures async-cancel-retry,validate-input,cross-file-config \
--trials 3 --randomize-order --out runs/local-t1
If the runtime reports the model artifacts are missing, the correct output of step 2 is a runtime_error run record — not a skipped step and not a manual retry.
Use deterministic mock-adapter smoke tests with expected traces before spending real inference time
The mock adapter emits a scripted sequence per fixture: read the file, write a known-good patch, run tests, declare done. Because the patch is authored by the fixture, the hidden suite must pass — so a failure here is a harness bug, not a model finding.
Expected trace for async-cancel-retry:
{"seq":0,"monoMs":0,"type":"run_start","runId":"r-001","taskId":"async-cancel-retry","backendId":"mock","revision":"fixture-1"}
{"seq":1,"type":"step_start","step":1}
{"seq":2,"type":"tool_call","step":1,"name":"read_file","args":{"path":"src/fetch-with-retry.js"}}
{"seq":3,"type":"tool_result","step":1,"ok":true,"bytes":812,"truncated":false}
{"seq":4,"type":"tool_call","step":2,"name":"write_file","args":{"path":"src/fetch-with-retry.js"}}
{"seq":5,"type":"tool_result","step":2,"ok":true}
{"seq":6,"type":"tool_call","step":3,"name":"run_tests","args":{}}
{"seq":7,"type":"tool_result","step":3,"ok":true}
{"seq":8,"type":"run_end","runId":"r-001","taskId":"async-cancel-retry","status":"declared_done","toolCalls":3}
{"type":"acceptance","taskId":"async-cancel-retry","scopeClean":true,"exitCode":0,"status":"passed"}
That trace comes from the mock, which is deterministic by construction. The point is that it proves path validation accepts a legal write, the tool-call counter increments, acceptance runs after the agent stops, and the terminal event carries a status. It proves nothing about any model, and the results table must not blur the two.
Exercise timeout, sandbox violation, malformed output, crashed server, and failed-test paths to verify honest accounting
Five failure injections, each with an expected status:
| Injection | Expected status | Check |
|---|---|---|
Adapter sleeps past stepTimeoutMs | timeout | Wall time bounded; no partial patch accepted |
Tool call with ../../etc/passwd | tool_rejected: path_escape | Rejection logged, run continues |
| Tool arguments as invalid JSON | tool_rejected: bad_arguments | Counted against tool budget |
Adapter throws ECONNREFUSED | runtime_error | Attributed to environment, not the model |
| Mock patch that leaves visible test failing | failed_tests | Acceptance not skipped |
Every one of those must appear in the JSONL with a terminal status. If any produces a missing run_end, the accounting is broken.
Run paired repeated trials in randomized backend order and record seeds where supported without promising determinism
Paired trials mean every fixture runs on every backend within the same session, so thermal drift affects both. Randomize backend order per fixture to spread drift rather than letting it land on whichever backend ran last.
--trials 3 --randomize-order --order-seed 20261122
Record the seed. Do not promise reproducibility: hosted backends rotate, local servers warm differently, and sampling may not be exposed. Three trials across three fixtures is nine runs per backend — enough to notice a 3× latency gap, not enough to estimate a pass rate.
Report Results Without Manufacturing Evidence
Define the results table: exact backend configuration, completed tasks, peak local memory, end-to-end latency, and measured energy
| Backend | Config | Passed | Peak local mem | E2E latency p50 / p90 | J / successful task |
|---|---|---|---|---|---|
mock | fixture-1 | 3/3 (by construction) | n/a | n/a | n/a |
inkling-2-mini-local | UNAVAILABLE | unavailable | unavailable | unavailable | unavailable |
local-conventional | not pinned | unavailable | unavailable | unavailable | unavailable |
hosted-pinned | not pinned | unavailable | unavailable | unavailable | unavailable |
That's the table. Three of four rows are empty because the artifacts were not verified and I will not publish a number I did not produce. The mock row is labelled so it cannot be mistaken for a result.
Publish task counts and individual outcomes alongside aggregates; distinguish expected smoke-test output from actual model observations
An aggregate hides the interesting part. Per-task rows, always:
| Task | mock | inkling-2-mini-local | local-conventional |
|---|---|---|---|
async-cancel-retry | passed (fixture) | unavailable | unavailable |
validate-input | passed (fixture) | unavailable | unavailable |
cross-file-config | passed (fixture) | unavailable | unavailable |
When real runs exist, the individual failure categories matter more than the counts — a model that fails cross-file-config by budget_exhausted is telling you something different from one that fails it by failed_tests.
Quantify uncertainty with task-level paired comparisons and clearly acknowledge small samples and correlated retries
For paired data with binary outcomes, the honest summary is the per-task pair, not an aggregate rate:
task backend A backend B outcome
async-cancel-retry pass pass tie
validate-input pass fail A
cross-file-config fail fail tie
With three tasks, one discordant pair is the entire signal. Report the pair count and say the confidence interval spans everything from "no difference" to "one model is much better". Retries are correlated — the same temperature, the same prompt, the same seed produce the same failure — so treating three trials as three independent samples overstates confidence.
Inspect patches and failure traces to separate incorrect reasoning, tool-protocol failures, budget exhaustion, and runtime incompatibility
Four diagnoses that require looking at the diff, not the exit code:
- Incorrect reasoning — plausible-looking patch, hidden edge case fails. Look for the case the model did not consider.
- Tool-protocol failure —
tool_rejectedevents, malformed JSON, wrong argument names. The model lost to the interface, not the problem. - Budget exhaustion — steps ran out mid-exploration. Often a scaffolding problem: a 12-step cap is tight.
- Runtime incompatibility — the backend never produced a token. Attributed to the environment column, never to the model.
Conflating these is how an unrun model acquires a reputation. A model that never loaded has no coding ability to assess; it has an integration problem.
Mark missing measurements as unavailable and avoid presenting an unrun experiment as an independent validation
Two temptations to refuse:
- Filling in a plausible number from a parameter count and a quantization. That's arithmetic, not measurement, and the two belong in different sections with different labels.
- Downgrading "unavailable" to "not tested" in prose while leaving a number in the table. The table is what people quote.
If you publish the harness before you can run the model, others can run it and contribute logs in the same schema. An empty row with a pinned format is more useful than a filled row with no provenance.
Decide Whether Inkling 2 Mini Fits the Laptop Workflow
Apply a predeclared acceptance envelope for correctness, memory headroom, turnaround time, and energy
Set the envelope before the first run, so you can't move the goalposts after seeing results:
| Dimension | Threshold | Why this number |
|---|---|---|
| Correctness | ≥ 2/3 hidden suites, zero scope violations | Small suite; one failure is survivable, two is a pattern |
| Memory headroom | peak ≤ 70% of usable unified memory | Leaves room for the OS, editor, and browser; avoids swap |
| Turnaround | p50 ≤ 90 s/task, p90 ≤ 180 s/task | Above this, I stop using the agent and edit by hand |
| Energy | ≤ 1.5× the local conventional baseline per successful task | Battery life is a real constraint on a laptop |
These thresholds are mine and they are arbitrary. The point is that they were stated before the measurements, and a reader with different constraints can substitute their own.
Diagnose measured bottlenecks before changing quantization, context limits, concurrency, or documented loop settings
A fixed order, because the cheap fixes are the ones people reach for first and they usually make accuracy worse:
- Confirm the artifact loads at all with a single non-agent prompt. If that fails, nothing downstream is measurable.
- Split prefill from decode. Long prefill with slow decode is a different problem from slow prefill.
- Check the loop setting if the runtime exposes it. If increasing loop count doubles latency without improving hidden-test pass rate, the extra compute is not buying accuracy on your tasks.
- Then reduce context — agent transcripts balloon, and a 24k cap with summarization of tool output is often the cheapest win.
- Then quantize. Quantization is an accuracy risk on the exact tasks you are measuring; do it last and re-run the suite.
- Then consider concurrency, which on a single laptop mostly means contention.
Changing two of these at once and reporting one number is how quantization gets blamed for a context-length bug.
State the narrow conclusion supported by the experiment without promoting a small local suite into confirmation of a leaderboard claim
The narrow conclusion I can support right now: "26B parameter model runs on a laptop" is physically plausible at 4-bit precision on a 32 GB unified-memory machine, and plausibly marginal on 16 GB once KV cache is included. That's arithmetic, conditional on Dealroom's 26B figure, and it's the strongest statement the available evidence supports.
The narrow conclusion I cannot support is anything about coding ability. Not because the claim is wrong — because I have no artifacts, no model card, no methodology, and therefore no measurement. A leaderboard claim about "agentic coding among open models under 100B" is only checkable against the harness, task set, step budget, and attempt policy it was produced under, and none of those were in the supplied coverage.
The harness exists so that when a weight file, a license, and a supported runtime show up, answering the question takes an afternoon instead of an argument. Until then, the results table has an empty row. The empty row is what I actually know.
Further Reading and Reproduction Artifacts
Secondary reporting used here, plus the official references to check for once they exist
Secondary reporting used for the timeline and claims above — all three are news references, not primary sources:
- Dealroom: 26B open-weights looped model, laptop operation, under-100B agentic-coding claim (published 2026-11-21)
- VentureBeat: looped transformer that runs locally on laptops (published 2026-11-21)
- CASRAI: open-weights looped model for laptops (published 2026-11-22)
I could not locate a verified Thinking Machines model card, license page, or official benchmark write-up for Inkling 2 Mini while writing this, so no official link appears here. That absence is the verification gate described above — add the link only once you have opened it.
Link the harness, task fixtures, pinned manifests, raw event logs, and platform-specific measurement documentation
Runtime and measurement documentation used by the harness design:
- Node.js
perf_hookshigh-resolution time — the monotonic clock used for every phase measurement - Node.js
child_process— sandboxed test execution with externally enforced timeouts - Node.js
process.resourceUsage()— per-process counters, explicitly not total model memory - Node.js permission model — filesystem scoping for the sandbox; verify the flag exists on your build first
- cgroup v2 documentation —
memory.peakand the exact accounting it provides - turbostat(8) — package power and frequency counters on x86
- JSON Lines — the event log format
- llama.cpp and the Ollama API reference — the two local runtimes the adapters target
Reproduction artifacts to publish alongside any real run: run-manifest.json with every revision pinned, the task fixtures including hidden tests with a checksum, the raw events.jsonl per run, and the machine profile block from each run header. Without those four, a number in a table is just a claim with better formatting.
More posts

Where Bun's 3x Speedup Disappears: A Reproducible Benchmark Harness for Node.js Workloads
A reproducible benchmark harness reveals where Bun's claimed 3x speedup over Node.js disappears in real-world workloads, examining methodology, compatibility gaps, and migration costs.

Testing Inkling 2 Mini on a 32GB Mac: Verified JavaScript Fixes, Peak Memory, and Swap
A hands-on deep technical walkthrough testing Thinking Machines' Inkling 2 Mini on a 32GB Mac, measuring peak memory and swap while evaluating tool-call reliability, latency, cost per verified JavaScript fix, and the impact of its looped transformer design.

Windows Update vs. Third-Party Update Channels: Where Emergency Patch Delay Comes From
An analysis of why Microsoft can ship emergency Windows fixes in days while third-party apps face longer patch delays, examining update channel control, signing, rollout gates, and release engineering tradeoffs.
Loading comments…