Testing Inkling 2 Mini’s Laptop Coding-Agent Claim with a Node.js Harness

Testing Inkling 2 Mini’s Laptop Coding-Agent Claim with a Node.js Harness

pr0h0••36 min read•ai
AI Usage (95%)

Three headlines in November 2026 said Thinking Machines shipped Inkling 2 Mini: open weights, a "looped" architecture, 26B parameters, laptop operation, and — per Dealroom — a lead in agentic coding among open models under 100B. Those are the claims. What I could not get hold of at the time of writing: a weight file, a model card, a license, or a benchmark methodology.

So this post does two things. First, it walks through the Node.js harness I built to test the laptop coding-agent claim properly — task completion, peak memory, end-to-end latency, and energy per successful task, measured against a conventional local model and a pinned hosted API under the same agent scaffold. Second, it works through the arithmetic behind whether "26B parameters on a laptop" is physically plausible at all. I'm not going to publish numbers for a model I couldn't load, so the results table keeps an Inkling 2 Mini row that says unavailable instead of quietly holding a plausible-looking median.

On this page

Test the Laptop Coding-Agent Claim, Not Just Token Generation

The engineering position: laptop suitability requires correct patches within explicit memory, latency, and energy budgets

Tokens per second is the wrong metric for a coding agent. One that emits 40 tok/s but needs 40 round trips to fix an off-by-one is slower to use than one that emits 15 tok/s and lands the patch in six. So the unit of measurement has to be the task: a repo state goes in, a patch comes out, and the patch either passes acceptance tests or it does not.

"Runs on a laptop" then decomposes into four budgets that must all hold at once:

  • Memory — peak resident bytes for weights plus KV cache plus runtime, with headroom, because swap turns latency into a cliff.
  • Latency — end-to-end wall time for a completed task, including every tool round trip.
  • Energy — joules per successful task, because a cheap model that fails three times burns more battery than an expensive one that works.
  • Correctness — hidden acceptance tests passing, with no scope violations.

A model can hit three of these and still be unusable. That's the reason to build a harness instead of reading a leaderboard.

Evaluation scope: one reproducible Node.js agent harness, three inference backends, and a bounded suite of repository tasks

The design is one scaffold, three backends, one task suite:

SlotPurposeStatus in this post
inkling-2-mini (local)The claim under testArtifacts unavailable — no row
local-conventionalIs the harness's local path itself slow?Pinned placeholder, documented
hosted-pinnedCeiling check, network includedPinned placeholder, documented
mockHarness validation, deterministicExercised

The mock backend isn't a joke entry. It's how you prove the harness's accounting works before you spend a single GPU-second. More on that below.

Establish What Is Actually Known About Inkling 2 Mini

Reported timeline: Dealroom and VentureBeat on November 21, 2026, then CASRAI on November 22

The supplied coverage gives three timestamps, and they matter because secondary reporting tends to drift on details like parameter counts:

Published (UTC)PublisherClaim in the headline
2026-11-21 15:16Dealroom26B open-weights looped model, runs on a laptop, tops open models under 100B at agentic coding
2026-11-21 18:29VentureBeat"looped transformer model that runs locally on laptops"
2026-11-22 01:17CASRAI"open-weights looped model for laptops"

Dealroom is the earliest of the three and the only one carrying the 26B figure and the under-100B coding ranking. The other two headlines, as supplied, do not repeat the parameter count.

What the coverage actually claims: open weights, a looped architecture, 26B parameters, laptop operation, and an under-100B coding lead

To be precise about what is being reported and by whom:

  • Open weights — stated by all three publishers. This is a licensing claim, and licensing is exactly the kind of thing secondary coverage gets wrong. I'm not treating it as confirmed.
  • Looped architecture — stated by all three. "Looped" is presented as an architecture description, not a spec. Loop count, where the loop is applied, and whether it is configurable at inference time are all unreported in the supplied material.
  • 26B parameters — stated by Dealroom only.
  • Runs locally on laptops — stated by all three, with no hardware qualification: no RAM floor, no quantization, no context length.
  • Tops open models under 100B at agentic coding — stated by Dealroom only, with no harness, no pass@k, no budget, and no methodology in the supplied snippet.
⚠️

A headline is not a specification. Every one of those five items is a claim from secondary reporting here, not a verified fact, and two of them appear in exactly one outlet.

Verification gate: locate official weights, model card, license, runtime support, and benchmark methodology before treating secondary reports as confirmed

Before any measurement, these five gates need to open:

  1. Weights — a downloadable artifact with a published checksum, so a revision can be pinned.
  2. Model card — parameter count broken down as total vs. active, layer count, attention head configuration, vocabulary size, context limit, and the evaluation methodology behind the coding claim.
  3. License — commercial use, redistribution, derivative naming, and whether quantized redistributions are permitted.
  4. Runtime support — this is the gate people skip. A looped architecture is not the standard transformer graph. Whether llama.cpp's GGUF path, or any mainstream inference server, can load it is an open question until a conversion exists and someone publishes the kernel work.
  5. Benchmark methodology — which agent scaffold, which tool set, what step budget, how many attempts, and whether the "under 100B" cohort was evaluated in the same harness.

Gate 4 is the one I'd bet causes trouble. "Open weights" and "open runnable runtime" are different products, and a novel architecture usually lands weights first and efficient kernels later. That's my inference — untested, and it gets settled the moment a conversion script and a supported-runtime note appear.

Separate reported claims, verified documentation, local measurements, and unresolved questions without inventing release details

A four-column ledger keeps this straight:

ReportedVerifiedMeasured locallyUnresolved
26B params (Dealroom)——Total or active parameters?
Looped architecture——Loop count, placement, configurability
Runs on laptops——Minimum RAM, quantization, context
Under-100B agentic coding lead——Harness, task set, budget, attempts
Open weights——License terms, redistribution

I leave the Verified column empty until I have artifacts. That's deliberate.

Translate the Architecture Claim into Testable Resource Questions

Explain weight storage with a parameter-count and precision calculation, explicitly conditional on the reported 26B count

This is the one place the 26B figure is genuinely useful, because weight storage is arithmetic and arithmetic is checkable. Assume 26 × 10⁹ parameters and disregard runtime overhead entirely:

PrecisionBytes/paramWeight storageIn GiB
FP324.0104.0 GB96.9
BF16 / FP162.052.0 GB48.4
INT8 (Q8)1.026.0 GB24.2
4-bit (~Q4_K_M)~0.55~14.3 GB~13.3
4-bit with embedding/output kept higher~0.6~15.6 GB~14.5
## the arithmetic, so you can redo it for a different parameter count
node -e 'const p=26e9; for (const [q,b] of Object.entries({fp32:4,fp16:2,int8:1,q4:0.55})) console.log(q, (p*b/1e9).toFixed(1)+" GB", (p*b/2**30).toFixed(1)+" GiB")'

Read that table against real laptop memory. A 16 GB unified-memory machine has roughly 10–12 GB usable for a model after the OS and a browser. At 4-bit, a 26B model sits at or slightly over that line before any KV cache. A 32 GB machine has room. A 64 GB machine has comfortable room.

So "runs on a laptop" holds only with a quantization qualifier and a RAM floor attached. The claim isn't wrong; it's underspecified in a way that decides whether it runs on your laptop.

Account for runtime overhead: quantization metadata, KV cache, activations, context length, and CPU or accelerator allocation

Weights are the floor, not the peak. The rest of the budget:

  • KV cache — grows linearly with context and batch. The formula is 2 × layers × kv_heads × head_dim × context × bytes. For a hypothetical 48-layer model with 8 KV heads at 128 dims, 32k context, FP16: 2 × 48 × 8 × 128 × 32768 × 2 ≈ 6.0 GiB. I made that configuration up for illustration — the real numbers need the model card.
  • Activations and workspace — modest at batch 1, but not zero, and larger for long prompts.
  • Quantization metadata and dequant buffers — small but nonzero; mixed-precision recipes keep embeddings and output layers at higher precision.
  • Runtime allocator behaviour — mmap'd weights count toward resident memory once touched. A runtime that pre-faults the whole file hits peak immediately; one that pages lazily spikes unpredictably.
  • Context limit as configured, not as advertised — an agent harness will happily push 20k tokens of file content into a prompt if you let it. Cap it deliberately.

An agent makes all of this worse than a chat session because each step re-sends a growing transcript. Step 12 carries every prior tool result.

Distinguish parameter reuse from inference work: inspect documented loop controls rather than assuming smaller weights imply faster execution

This is where "smaller weights imply faster" breaks.

In a looped (weight-tied, recurrent-depth) design, the same block of parameters is applied repeatedly. Two consequences follow directly:

  • Memory is bounded by the parameter count, not by the loop count.
  • Compute scales roughly with the loop count, not the parameter count.

So a 26B looped model with 8 loop iterations at inference time does work comparable to a much larger dense model per token, while occupying the memory of a 26B model. That's the trade: you buy memory headroom with compute time. On a laptop, where memory bandwidth and thermals are the binding constraints, that trade can go either way.

The testable questions are therefore not "how big is it" but:

  1. Is the loop count fixed at inference or configurable?
  2. Does increasing the loop count improve task success enough to justify the linear latency cost?
  3. What are the measured prefill and decode rates at each loop setting?

If the loop count is not documented or not exposed, question 1 answers "unknown" and questions 2–3 collapse. Record that in the manifest rather than guessing a value.

Define the Experiment Before Running the Coding Agent

Register exact model revisions, quantization formats, runtime versions, hosted API identifiers, and any documented loop settings

Everything that could change a number goes in one file, written before the first run.

harness/run-manifest.json
{
"suiteVersion": "1.0.0",
"budget": {
  "maxSteps": 12,
  "maxToolCalls": 20,
  "stepTimeoutMs": 120000,
  "suiteTimeoutMs": 600000,
  "maxPromptTokens": 24000
},
"fixtures": [
  { "id": "async-cancel-retry", "dir": "tasks/async-cancel-retry" },
  { "id": "validate-input", "dir": "tasks/validate-input" },
  { "id": "cross-file-config", "dir": "tasks/cross-file-config" }
],
"backends": [
  {
    "id": "mock",
    "adapter": "mock",
    "revision": "fixture-1",
    "quantization": "n/a",
    "runtime": "node"
  },
  {
    "id": "local-conventional",
    "adapter": "llamacpp",
    "revision": "<pin: gguf sha256>",
    "quantization": "<pin: e.g. Q4_K_M>",
    "runtime": "<pin: server build id>",
    "loopSettings": "n/a"
  },
  {
    "id": "hosted-pinned",
    "adapter": "openai-compatible",
    "revision": "<pin: snapshot id, not a floating alias>",
    "quantization": "provider-managed",
    "runtime": "provider",
    "loopSettings": "n/a"
  },
  {
    "id": "inkling-2-mini-local",
    "adapter": "TBD",
    "revision": "UNAVAILABLE",
    "quantization": "UNAVAILABLE",
    "runtime": "UNAVAILABLE",
    "loopSettings": "UNDOCUMENTED"
  }
]
}

The last entry is in the manifest on purpose. A backend that is declared and cannot be run is a recorded fact; a backend that gets quietly dropped is a gap someone will mistake for a result.

⚠️

Never pin a hosted model to a floating alias like latest. Providers rotate snapshots underneath aliases, and a rotating backend invalidates paired comparisons silently.

Compare Inkling 2 Mini, one verified conventional local model, and one pinned hosted model under the same agent scaffold

The three-way comparison separates "this model is good" from "this scaffold is slow":

  • Local conventional baseline answers whether the harness itself is the bottleneck. If a well-supported local model also feels awful through this scaffold, the scaffold is the problem.
  • Hosted pinned answers what the ceiling looks like when latency includes network. If hosted is also 4 minutes per task, the harness is over-budgeting steps.
  • Inkling 2 Mini local answers whether the claim survives contact with this machine.

The rule that makes the comparison valid: identical system prompt, identical tool schemas, identical step and tool-call budgets, identical fixtures. Backend-specific tuning is allowed, but each change gets its own labelled row.

Record laptop hardware, OS, power mode, thermal conditions, context limits, and background workload

A template that goes in the run event header, not in a notebook:

{
  "hardware": { "model": "<machine id>", "cpuCores": 0, "gpuCores": 0, "unifiedMemoryGiB": 0 },
  "os": { "name": "", "version": "", "kernel": "" },
  "power": { "mode": "ac | battery", "charging": false, "lowPowerMode": false },
  "thermal": { "ambientC": 0, "stateBefore": "", "stateAfter": "" },
  "contextLimit": 0,
  "background": { "browser": false, "indexer": false, "other": "" },
  "notes": ""
}

Power mode and charging state are not pedantic. On battery, many machines cap sustained package power, which silently changes decode throughput by a large factor. Charging while measuring energy makes gross energy meaningless unless you account for charge going into the battery.

Define task success, timeout, tool-call budget, retry policy, and failure categories before inspecting results

Predeclare them, because post-hoc categories always flatter the model:

CategoryDefinitionCounts as
passedHidden suite exits 0 AND scope check cleanSuccess
failed_testsAgent declared done, hidden suite failsFailure
budget_exhaustedSteps or tool calls hit the capFailure
timeoutStep or suite wall clock exceededFailure
protocol_errorMalformed tool call, unparseable argumentsFailure
runtime_errorBackend crashed, unreachable, OOMFailure, environment-attributed
scope_violationPatch touches protected filesFailure

Retry policy: each fixture runs once per backend per trial. A retried run is a new run with a new ID and gets included in the denominator. No silent retries of flaky runs — that's the single easiest way to fake a benchmark.

Separate equal-budget comparisons from backend-specific tuning and label every configuration change

Two run families:

  • parity — identical budgets everywhere. The headline comparison. Any backend that needs more than 12 steps to do a three-line fix is telling you something real.
  • tuned — per-backend settings, changed for a stated reason, recorded in the manifest diff. Useful for "what is the best this model can do", useless for cross-model claims.

Never mix rows from the two families in one table without a column that says which family the row came from.

Build a Small Coding Suite with Hidden Acceptance Tests

Create pinned JavaScript fixtures covering asynchronous control flow, input validation, and cross-file changes

Three fixtures, each targeting a failure mode that's common for agents:

FixtureBug classWhy it is here
async-cancel-retryAsync control flowRequires reasoning about ordering, not just syntax
validate-inputInput validationEdge cases live outside the happy path in the bug report
cross-file-configMulti-file changeRequires touching two files consistently

Three tasks is an anecdote, not a benchmark. Saying so up front keeps the results section from overreaching later.

Walk through one complete fixture: visible bug report, baseline failure, hidden edge cases, and exact success criterion

The async-cancel-retry fixture, in full:

Visible bug report (tasks/async-cancel-retry/ISSUE.md), visible to the agent:

fetchWithRetry keeps retrying after the caller aborts. When a user cancels the upload, we still see three more 503 retries in the log, and the promise rejects with the server error instead of AbortError.

Visible test — covers the happy path only: a 503 followed by a 200 resolves with two attempts.

Baseline — the fixture is written so the visible test fails before any patch, because the shipped implementation calls fetch without forwarding signal:

$ node --test test/visible.test.js
not ok 1 - fetchWithRetry retries once after a 503
  error: 'expected attempts to equal 2, got 1'
## pass 0
## fail 1

(That's the baseline the fixture is designed to produce. If your run prints something else, the fixture is stale, not the model.)

Hidden tests (tasks/async-cancel-retry/.hidden/acceptance.test.js), never mounted into the agent's workspace:

hidden acceptance — abort semantics
import assert from "node:assert/strict";



test("abort during backoff rejects with AbortError and stops retrying", async () => {
const controller = new AbortController();
let attempts = 0;
const fakeFetch = async (_url, init) => {
  attempts += 1;
  assert.equal(init?.signal, controller.signal, "signal must be forwarded");
  if (attempts === 1) {
    setTimeout(() => controller.abort(), 5);
    const err = new Error("service unavailable");
    err.status = 503;
    throw err;
  }
  return { ok: true, json: async () => ({ done: true }) };
};

await assert.rejects(
  () => fetchWithRetry("/upload", { fetchImpl: fakeFetch, signal: controller.signal, retries: 3 }),
  (err) => err.name === "AbortError",
);
assert.equal(attempts, 1, "aborted call must not issue further attempts");
});

Plus three more: abort before the first attempt; retry cap honoured exactly (not off by one); and a non-abort error still surfaces with its status code.

Exact success criterion — all of:

  1. .hidden/acceptance.test.js exits 0,
  2. test/visible.test.js exits 0,
  3. git diff --name-only is a subset of {"src/fetch-with-retry.js"},
  4. no test file is modified, deleted, or skipped,
  5. package.json dependency list unchanged.

Prevent false passes by protecting acceptance tests and checking patch scope as well as test exit status

Test exit status is a weak signal on its own. Agents have been known to edit the test, add it.skip, tighten a timeout, or vendor a helper that guts the assertion. Two cheap defences:

  • Mount the hidden tests after the agent stops. They never exist in the workspace while the agent is running, so they cannot be read or edited.
  • Diff the scope before trusting the exit code. Cheap, and it catches the embarrassing cases.
git -C "$WS" diff --name-only | sort > /tmp/touched.txt
comm -23 /tmp/touched.txt <(printf '%s\n' "${ALLOWED[@]}" | sort) > /tmp/violations.txt
if [ -s /tmp/violations.txt ]; then echo "SCOPE_VIOLATION"; exit 3; fi

Also worth grepping the patch for process.exit, --test-force-exit, .only(, and .skip(, and treating any of them as a scope_violation.

Reset each workspace from a clean snapshot and describe the suite's limits for broader coding-agent claims

Every run starts from a fresh git clone of a pinned commit into a temp directory, then npm ci from a lockfile, then nothing else. No carry-over from the previous run, no warm node_modules shared between fixtures unless the manifest declares it.

Limits, stated plainly:

  • Three fixtures cannot rank models. They can only say whether a model is usable on this machine for this class of bug.
  • Single-language, single-repo-shape, small codebase. Cross-language and large-repo behaviour are untested.
  • The hidden tests were written by the same person who wrote the bug, so they share a blind spot.
  • No test of long-horizon work — a 12-step cap excludes anything resembling real refactors.

None of that invalidates the harness. It just bounds what a pass or fail means.

Implement the Node.js Agent Harness

Show the project layout, run manifest, CLI arguments, and adapter contract without assuming an undocumented Inkling API

harness/
  package.json
  run-manifest.json
  src/
    cli.mjs
    agent-loop.mjs
    events.mjs
    classify.mjs
    adapters/
      mock.mjs
      llamacpp.mjs
      openai-compatible.mjs
      index.mjs
    tools/
      fs.mjs
      test.mjs
      registry.mjs
    measure/
      latency.mjs
      memory.mjs
      energy.mjs
  tasks/
    async-cancel-retry/
    validate-input/
    cross-file-config/
  runs/            # git-ignored; one directory per run

The adapter contract is deliberately narrow, and it does not assume an Inkling 2 Mini endpoint. If a runtime appears, it gets a new adapter behind the same interface; if it never appears, the interface still worked for two other backends.

src/adapters/index.mjs — the contract
/**
* @typedef {Object} Adapter
* @property {string} id
* @property {() => Promise<{ revision: string, quantization: string,
*   contextLimit: number, loopSettings: string|"n/a"|"UNDOCUMENTED" }>} describe
* @property {(req: ChatRequest, opts: { signal: AbortSignal }) => Promise<ChatResponse>} chat
*
* ChatRequest:  { messages, tools, maxTokens }
* ChatResponse: {
*   message: { role: "assistant", content: string|null,
*              toolCalls: Array<{ id: string, name: string, arguments: string }> },
*   usage: { promptTokens: number|null, completionTokens: number|null,
*            source: "backend"|"estimated"|"unavailable" },
*   unsupported: string[]   // e.g. ["tool_choice", "parallel_tool_calls"]
* }
*/

CLI:

node src/cli.mjs run \
  --manifest run-manifest.json \
  --backend mock \
  --fixtures async-cancel-retry,validate-input,cross-file-config \
  --trials 1 \
  --out runs/$(date -u +%Y%m%dT%H%M%SZ)-mock

Normalize messages and tool results while preserving backend-specific token accounting and unsupported features

Normalize the shapes; never normalize the numbers into existence. If a backend does not report prompt tokens, the usage record says source: "unavailable" rather than an estimate silently blended with real counts.

export function normalizeUsage(raw, backendId) {
  const p = raw?.prompt_tokens ?? raw?.usage?.input_tokens ?? null;
  const c = raw?.completion_tokens ?? raw?.usage?.output_tokens ?? null;
  return {
    promptTokens: p,
    completionTokens: c,
    source: p === null || c === null ? "unavailable" : "backend",
  };
}

Tool calls get flattened into a common { id, name, arguments } shape regardless of whether the backend returned arguments as a JSON string or an object. unsupported carries capability gaps forward so the results table can say "this backend had no structured tool calls" instead of pretending the model chose plain text.

Implement the bounded agent loop: request, validate tool call, execute, append result, and stop on completion or budget exhaustion

The loop must be boring. Every exit is explicit, every exit is logged, and no path silently continues.

src/agent-loop.mjs — bounded loop with typed termination
export async function runAgent({ adapter, task, tools, budget, emit, runId, clock }) {
const messages = [
  { role: "system", content: task.systemPrompt },
  { role: "user", content: task.prompt },
];
let toolCalls = 0;

const finish = (status, extra = {}) => {
  emit({ type: "run_end", runId, taskId: task.id, status, toolCalls, ...extra });
  return { status, messages };
};

for (let step = 1; step <= budget.maxSteps; step++) {
  emit({ type: "step_start", runId, taskId: task.id, step, at: clock() });

  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), budget.stepTimeoutMs);
  let res;
  try {
    res = await adapter.chat({ messages, tools: tools.schema, maxTokens: budget.maxTokens },
                             { signal: controller.signal });
  } catch (err) {
    emit({ type: "step_error", runId, taskId: task.id, step,
           code: classify(err), message: String(err?.message ?? err) });
    return finish(classify(err) === "aborted" ? "timeout" : "runtime_error");
  } finally {
    clearTimeout(timer);
  }

  emit({ type: "model_response", runId, taskId: task.id, step,
         usage: res.usage, unsupported: res.unsupported,
         contentBytes: res.message.content?.length ?? 0 });

  messages.push(res.message);

  if (res.message.toolCalls.length === 0) {
    return finish("declared_done");
  }

  for (const call of res.message.toolCalls) {
    if (++toolCalls > budget.maxToolCalls) return finish("budget_exhausted");

    const check = tools.validate(call);
    if (!check.ok) {
      emit({ type: "tool_rejected", runId, taskId: task.id, step,
             callId: call.id, code: check.code });
      messages.push({ role: "tool", toolCallId: call.id,
                      content: "ERROR: " + check.code + ". Fix the call and retry." });
      continue;
    }

    emit({ type: "tool_call", runId, taskId: task.id, step, callId: call.id,
           name: call.name, args: check.args });
    const result = await tools.execute(call.name, check.args);
    emit({ type: "tool_result", runId, taskId: task.id, step, callId: call.id,
           ok: result.ok, bytes: result.output.length, truncated: result.truncated });
    messages.push({ role: "tool", toolCallId: call.id, content: result.output });
  }
}

return finish("budget_exhausted");
}

Note the protocol-error path: a rejected tool call is fed back as an error message and counted against the tool-call budget. Malformed calls cost the model steps. That's the honest accounting — a model that can't emit valid JSON should not get free retries.

Provide scoped file and test tools with path validation, output limits, and explicit handling of malformed calls

Two tools only: read_file / write_file (scoped to the workspace) and run_tests (a fixed command, no arguments from the model). Fewer tools means fewer protocol failures, and protocol failures are a large share of agent failures.

src/tools/fs.mjs — containment before write
import path from "node:path";


export class ToolError extends Error {
constructor(code) { super(code); this.code = code; }
}

export function resolveInside(workspace, rel) {
if (typeof rel !== "string" || rel.length === 0) throw new ToolError("bad_path");
if (rel.includes("\0")) throw new ToolError("bad_path");
if (path.isAbsolute(rel)) throw new ToolError("absolute_path");
const abs = path.resolve(workspace, rel);
const prefix = workspace.endsWith(path.sep) ? workspace : workspace + path.sep;
if (!abs.startsWith(prefix)) throw new ToolError("path_escape");
return abs;
}

// reject symlinked components: resolve the nearest existing ancestor
export async function assertNoSymlink(workspace, abs) {
let cur = abs;
for (;;) {
  try {
    const st = await fs.lstat(cur);
    if (st.isSymbolicLink()) throw new ToolError("symlink_component");
    break;
  } catch (err) {
    if (err.code !== "ENOENT") throw err;
    const parent = path.dirname(cur);
    if (parent === cur) break;
    cur = parent;
  }
}
if (!cur.startsWith(workspace)) throw new ToolError("path_escape");
}

Output limits: cap read_file at 64 KiB and tool output at 32 KiB with an explicit truncated: true marker. Truncation has to be visible to the model and to the event log, otherwise a model that failed because it never saw the end of a file looks like a model that ignored it.

Run generated code in a disposable sandbox with no secrets, restricted network access, and externally enforced resource limits

Four layers, none of them optional:

  1. Disposable root — a temp directory per run, deleted after artifact collection.
  2. No secrets in env — pass a minimal env; do not inherit the harness's environment.
  3. Node's permission model — check your build first:
node --help | grep -i permission

If --permission (or --experimental-permission on older builds) is listed, use it with --allow-fs-read and --allow-fs-write scoped to the workspace and no --allow-child-process or --allow-net. If it isn't listed, your build predates the flag — do not silently skip the restriction, change the runtime. 4. External resource limits — timeouts and memory ceilings enforced by the parent, not by the code under test. ulimit -v plus a hard SIGKILL after a wall-clock cap.

⚠️

The agent's shell is not a trust boundary you control. Treat anything the model produces as hostile: no network by default, no inherited credentials, no writes outside the temp root, and no reliance on the model to enforce any of it.

Write append-only JSONL events containing run IDs, task IDs, model revisions, monotonic timestamps, and terminal status

src/events.mjs — append-only JSONL
import fs from "node:fs";

export function makeEmitter(file) {
const stream = fs.createWriteStream(file, { flags: "a" });
const t0 = process.hrtime.bigint();
let seq = 0;
return (event) => {
  const record = {
    schema: 1,
    seq: seq++,
    monoMs: Number((process.hrtime.bigint() - t0) / 1000000n),
    wall: new Date().toISOString(),
    ...event,
  };
  stream.write(JSON.stringify(record) + "\n");
};
}

monoMs from process.hrtime.bigint() is monotonic and immune to NTP steps; wall is for humans. Both go in every record. The terminal run_end event carries the status from the loop, so an interrupted run has no terminal event at all — which is itself a detectable signal that a run was killed rather than completed.

Every record also carries runId, taskId, backendId, and the manifest's revision string, so a results table can be regenerated from the logs alone.

Measure End-to-End Latency and Peak Memory

Time model loading, inference, tool execution, and final acceptance testing separately with a monotonic clock

Four separate clocks, because they diagnose different problems:

PhaseMeasuredWhat a bad number means
loadBackend ready → first tokenDisk-bound weights or slow pre-faulting
inferenceSum of per-step request durationsCompute or bandwidth bound
toolsSum of tool executionsScaffold overhead, not the model
acceptanceHidden suite runFixture cost, excluded from agent latency
const t0 = process.hrtime.bigint();
// ... do work ...
const elapsedMs = Number(process.hrtime.bigint() - t0) / 1e6;

Do not use Date.now() for phase timing. A clock step during a 3-minute agent run will produce a negative duration and you will spend an hour not believing it.

Report cold-start and warm-run latency independently, including median and tail behavior across repeated runs

Cold start includes reading weights off disk; warm start does not. They differ by an order of magnitude on laptops, so a blended average describes neither. Report both, plus p50 and p90 across trials — with three fixtures, p90 is the maximum, and the table should say so rather than implying statistical weight.

per-task latency, ms (illustrative shape, not a measurement)
backend              trial  task                    load  infer  tools  total
...

Monitor the inference server and agent process tree rather than mistaking Node.js heap usage for total model memory

process.memoryUsage().rss for the Node process tells you nothing about a model running in a separate server. Sum the tree:

## total RSS of the inference server and the agent (Linux)
tree_rss_kb() {
  local root=$1 total=0
  for pid in "$root" $(pgrep -P "$root" | tr '\n' ' '); do
    [ -r "/proc/$pid/status" ] || continue
    total=$(( total + $(awk '/^VmRSS:/ {print $2}' "/proc/$pid/status") ))
  done
  echo "$total"
}

But sampling a process tree misses transients between samples. Better, if the kernel supports it: put the whole run in a cgroup v2 and read memory.peak, which the kernel maintains exactly.

## exact high-water mark for everything in the group, no sampling gap
cat /sys/fs/cgroup/hackyjs-run/memory.peak

memory.peak in cgroup v2 requires a reasonably recent kernel (5.19+). Verify with ls /sys/fs/cgroup/<group>/memory.peak before relying on it — if the file is absent, you are back to sampling, and you must say so.

Explain RSS, shared pages, unified memory, accelerator counters, sampling intervals, and missed transient peaks

Definitions that decide whether two numbers are comparable:

  • RSS counts pages resident in the process's address space, including shared pages counted once per process. Summing RSS across processes double-counts shared libraries.
  • mmap'd weights only count once touched. A model file opened lazily shows low RSS until the load completes.
  • Unified memory (Apple Silicon) blurs the GPU/CPU split: "VRAM" and "RAM" are the same pool, so a model's footprint competes with the OS directly.
  • Discrete accelerator counters report their own allocator's view, which typically excludes driver overhead and can exceed system RAM through shared memory fallback.
  • Sampling interval is a resolution limit. A 200 ms sample over a 400 ms allocation spike can miss the peak entirely; only a kernel-maintained high-water mark or a cgroup counter is safe.

If your peak-memory number came from sampling, label it "sampled at N ms" — because it's a lower bound.

Treat hosted memory as unavailable and hosted latency as including network and provider-side delays

For the hosted backend:

  • Memory — unavailable. Not "estimated", not inferred from a parameter count. You do not know the precision, the batching, or the tensor-parallel sharding.
  • Latency — includes DNS, TLS, queueing, and provider-side scheduling. It is a legitimate number for "how long do I wait", and an illegitimate one for "how fast is this model".

Name the network path in the results table so nobody subtracts it later and calls the difference model speed.

Measure Energy Without Claiming Access to Hidden Infrastructure

Define the boundary: local device energy is measurable, but hosted-provider inference energy is not available from client timing

Local energy is a device measurement: package power over the run, minus or including idle, as declared. Hosted energy is not derivable from client-side timing — you'd need the provider's hardware, utilization, and PUE. The results table gets unavailable for hosted energy, and any claim about it is out of scope.

Document the selected platform counter or external power meter, its units, permissions, sampling rate, and coverage

PlatformSourceUnitsNotes
Linux (Intel/AMD)RAPL energy_uj under /sys/class/powercap/microjoules, monotonic counterOften root-only; wraps at max_energy_range_uj
Linux (whole machine)External meter at the wallwattsOnly option that includes display and charging
macOSpowermetrics (requires root)mW per subsystemCoarse sampling; CPU+GPU+ANE coverage varies by chip
Windowspowercfg /energy (report, not realtime)—Not suitable for per-task integration

Record the source, units, sampling rate, permission level, and coverage — package-only versus whole-machine is a large difference on a laptop where the display and wireless draw real power.

📝

RAPL counters wrap. Read max_energy_range_uj and handle the rollover explicitly, or a long run produces a negative energy delta and a nonsense result.

Integrate power over time, retain gross energy, and label any idle-adjusted estimate and its assumptions

export function integrateJoules(samples) {
  // samples: [{ tMs, watts }] ascending by tMs
  let joules = 0;
  for (let i = 1; i < samples.length; i++) {
    const dt = (samples[i].tMs - samples[i - 1].tMs) / 1000;
    const pAvg = (samples[i].watts + samples[i - 1].watts) / 2;
    joules += pAvg * dt;
  }
  return joules;
}

Trapezoidal integration over sampled power. Keep the gross joules as the primary number, because it is what the battery actually gave up. An idle-adjusted figure (gross minus baseline draw × duration) is also useful, but it depends on the baseline being measured on the same machine in the same power mode, and it has to be labelled as an estimate with the baseline duration stated.

Control battery charging and thermal drift, batch short tasks when necessary, and include failed attempts in energy per successful task

  • Pin the power mode and record it. A machine that silently caps package power on battery will report lower energy and lower throughput — the ratio can look improved.
  • Do not measure while charging, or account for charge into the battery explicitly.
  • Short tasks integrate badly at coarse sampling rates. If a task runs for 8 seconds and the meter samples every 1 second, either batch several tasks into one measurement window or report the number with its resolution limit attached.
  • The headline metric is joules per successful task: total_joules / tasks_passed. Failed attempts stay in the numerator. A model that burns 40% less energy per token but needs twice the attempts is not the efficient choice.

Execute a Controlled Run and Validate the Harness

Provide installation and execution commands after verifying the actual runtime, model artifacts, and adapter configuration

node --version
node --help | grep -i permission   # confirm the sandbox flag on this build

cd harness
npm ci

## 1) harness validation, no model involved
node src/cli.mjs run --manifest run-manifest.json --backend mock \
  --fixtures async-cancel-retry,validate-input,cross-file-config \
  --trials 1 --out runs/mock-smoke

## 2) only after 1 passes: a real backend, pinned in the manifest
node src/cli.mjs run --manifest run-manifest.json --backend local-conventional \
  --fixtures async-cancel-retry,validate-input,cross-file-config \
  --trials 3 --randomize-order --out runs/local-t1

If the runtime reports the model artifacts are missing, the correct output of step 2 is a runtime_error run record — not a skipped step and not a manual retry.

Use deterministic mock-adapter smoke tests with expected traces before spending real inference time

The mock adapter emits a scripted sequence per fixture: read the file, write a known-good patch, run tests, declare done. Because the patch is authored by the fixture, the hidden suite must pass — so a failure here is a harness bug, not a model finding.

Expected trace for async-cancel-retry:

{"seq":0,"monoMs":0,"type":"run_start","runId":"r-001","taskId":"async-cancel-retry","backendId":"mock","revision":"fixture-1"}
{"seq":1,"type":"step_start","step":1}
{"seq":2,"type":"tool_call","step":1,"name":"read_file","args":{"path":"src/fetch-with-retry.js"}}
{"seq":3,"type":"tool_result","step":1,"ok":true,"bytes":812,"truncated":false}
{"seq":4,"type":"tool_call","step":2,"name":"write_file","args":{"path":"src/fetch-with-retry.js"}}
{"seq":5,"type":"tool_result","step":2,"ok":true}
{"seq":6,"type":"tool_call","step":3,"name":"run_tests","args":{}}
{"seq":7,"type":"tool_result","step":3,"ok":true}
{"seq":8,"type":"run_end","runId":"r-001","taskId":"async-cancel-retry","status":"declared_done","toolCalls":3}
{"type":"acceptance","taskId":"async-cancel-retry","scopeClean":true,"exitCode":0,"status":"passed"}

That trace comes from the mock, which is deterministic by construction. The point is that it proves path validation accepts a legal write, the tool-call counter increments, acceptance runs after the agent stops, and the terminal event carries a status. It proves nothing about any model, and the results table must not blur the two.

Exercise timeout, sandbox violation, malformed output, crashed server, and failed-test paths to verify honest accounting

Five failure injections, each with an expected status:

InjectionExpected statusCheck
Adapter sleeps past stepTimeoutMstimeoutWall time bounded; no partial patch accepted
Tool call with ../../etc/passwdtool_rejected: path_escapeRejection logged, run continues
Tool arguments as invalid JSONtool_rejected: bad_argumentsCounted against tool budget
Adapter throws ECONNREFUSEDruntime_errorAttributed to environment, not the model
Mock patch that leaves visible test failingfailed_testsAcceptance not skipped

Every one of those must appear in the JSONL with a terminal status. If any produces a missing run_end, the accounting is broken.

Run paired repeated trials in randomized backend order and record seeds where supported without promising determinism

Paired trials mean every fixture runs on every backend within the same session, so thermal drift affects both. Randomize backend order per fixture to spread drift rather than letting it land on whichever backend ran last.

--trials 3 --randomize-order --order-seed 20261122

Record the seed. Do not promise reproducibility: hosted backends rotate, local servers warm differently, and sampling may not be exposed. Three trials across three fixtures is nine runs per backend — enough to notice a 3× latency gap, not enough to estimate a pass rate.

Report Results Without Manufacturing Evidence

Define the results table: exact backend configuration, completed tasks, peak local memory, end-to-end latency, and measured energy

BackendConfigPassedPeak local memE2E latency p50 / p90J / successful task
mockfixture-13/3 (by construction)n/an/an/a
inkling-2-mini-localUNAVAILABLEunavailableunavailableunavailableunavailable
local-conventionalnot pinnedunavailableunavailableunavailableunavailable
hosted-pinnednot pinnedunavailableunavailableunavailableunavailable

That's the table. Three of four rows are empty because the artifacts were not verified and I will not publish a number I did not produce. The mock row is labelled so it cannot be mistaken for a result.

Publish task counts and individual outcomes alongside aggregates; distinguish expected smoke-test output from actual model observations

An aggregate hides the interesting part. Per-task rows, always:

Taskmockinkling-2-mini-locallocal-conventional
async-cancel-retrypassed (fixture)unavailableunavailable
validate-inputpassed (fixture)unavailableunavailable
cross-file-configpassed (fixture)unavailableunavailable

When real runs exist, the individual failure categories matter more than the counts — a model that fails cross-file-config by budget_exhausted is telling you something different from one that fails it by failed_tests.

Quantify uncertainty with task-level paired comparisons and clearly acknowledge small samples and correlated retries

For paired data with binary outcomes, the honest summary is the per-task pair, not an aggregate rate:

task                  backend A   backend B   outcome
async-cancel-retry    pass        pass        tie
validate-input        pass        fail        A
cross-file-config     fail        fail        tie

With three tasks, one discordant pair is the entire signal. Report the pair count and say the confidence interval spans everything from "no difference" to "one model is much better". Retries are correlated — the same temperature, the same prompt, the same seed produce the same failure — so treating three trials as three independent samples overstates confidence.

Inspect patches and failure traces to separate incorrect reasoning, tool-protocol failures, budget exhaustion, and runtime incompatibility

Four diagnoses that require looking at the diff, not the exit code:

  • Incorrect reasoning — plausible-looking patch, hidden edge case fails. Look for the case the model did not consider.
  • Tool-protocol failure — tool_rejected events, malformed JSON, wrong argument names. The model lost to the interface, not the problem.
  • Budget exhaustion — steps ran out mid-exploration. Often a scaffolding problem: a 12-step cap is tight.
  • Runtime incompatibility — the backend never produced a token. Attributed to the environment column, never to the model.

Conflating these is how an unrun model acquires a reputation. A model that never loaded has no coding ability to assess; it has an integration problem.

Mark missing measurements as unavailable and avoid presenting an unrun experiment as an independent validation

Two temptations to refuse:

  1. Filling in a plausible number from a parameter count and a quantization. That's arithmetic, not measurement, and the two belong in different sections with different labels.
  2. Downgrading "unavailable" to "not tested" in prose while leaving a number in the table. The table is what people quote.
💪

If you publish the harness before you can run the model, others can run it and contribute logs in the same schema. An empty row with a pinned format is more useful than a filled row with no provenance.

Decide Whether Inkling 2 Mini Fits the Laptop Workflow

Apply a predeclared acceptance envelope for correctness, memory headroom, turnaround time, and energy

Set the envelope before the first run, so you can't move the goalposts after seeing results:

DimensionThresholdWhy this number
Correctness≥ 2/3 hidden suites, zero scope violationsSmall suite; one failure is survivable, two is a pattern
Memory headroompeak ≤ 70% of usable unified memoryLeaves room for the OS, editor, and browser; avoids swap
Turnaroundp50 ≤ 90 s/task, p90 ≤ 180 s/taskAbove this, I stop using the agent and edit by hand
Energy≤ 1.5× the local conventional baseline per successful taskBattery life is a real constraint on a laptop

These thresholds are mine and they are arbitrary. The point is that they were stated before the measurements, and a reader with different constraints can substitute their own.

Diagnose measured bottlenecks before changing quantization, context limits, concurrency, or documented loop settings

A fixed order, because the cheap fixes are the ones people reach for first and they usually make accuracy worse:

  1. Confirm the artifact loads at all with a single non-agent prompt. If that fails, nothing downstream is measurable.
  2. Split prefill from decode. Long prefill with slow decode is a different problem from slow prefill.
  3. Check the loop setting if the runtime exposes it. If increasing loop count doubles latency without improving hidden-test pass rate, the extra compute is not buying accuracy on your tasks.
  4. Then reduce context — agent transcripts balloon, and a 24k cap with summarization of tool output is often the cheapest win.
  5. Then quantize. Quantization is an accuracy risk on the exact tasks you are measuring; do it last and re-run the suite.
  6. Then consider concurrency, which on a single laptop mostly means contention.

Changing two of these at once and reporting one number is how quantization gets blamed for a context-length bug.

State the narrow conclusion supported by the experiment without promoting a small local suite into confirmation of a leaderboard claim

The narrow conclusion I can support right now: "26B parameter model runs on a laptop" is physically plausible at 4-bit precision on a 32 GB unified-memory machine, and plausibly marginal on 16 GB once KV cache is included. That's arithmetic, conditional on Dealroom's 26B figure, and it's the strongest statement the available evidence supports.

The narrow conclusion I cannot support is anything about coding ability. Not because the claim is wrong — because I have no artifacts, no model card, no methodology, and therefore no measurement. A leaderboard claim about "agentic coding among open models under 100B" is only checkable against the harness, task set, step budget, and attempt policy it was produced under, and none of those were in the supplied coverage.

The harness exists so that when a weight file, a license, and a supported runtime show up, answering the question takes an afternoon instead of an argument. Until then, the results table has an empty row. The empty row is what I actually know.

Further Reading and Reproduction Artifacts

Secondary reporting used here, plus the official references to check for once they exist

Secondary reporting used for the timeline and claims above — all three are news references, not primary sources:

I could not locate a verified Thinking Machines model card, license page, or official benchmark write-up for Inkling 2 Mini while writing this, so no official link appears here. That absence is the verification gate described above — add the link only once you have opened it.

Runtime and measurement documentation used by the harness design:

Reproduction artifacts to publish alongside any real run: run-manifest.json with every revision pinned, the task fixtures including hidden tests with a checksum, the raw events.jsonl per run, and the machine profile block from each run header. Without those four, a number in a table is just a claim with better formatting.

Share this post

More posts

Loading comments…