Designing Agent Tool Loops Around Per-Call Cost Instead of Frontier Model Assumptions

Introduction: The Per-Call Cost of Agent Tool Loops

A chat product makes one model call per user turn. An agent makes N. This walkthrough is about designing agent tool loops around per-call cost instead of frontier model assumptions: where the bill actually comes from, how latency and concurrency compound it, and what to instrument before you route execution to a lightweight model like Meta's Muse Ember.

The Muse Ember announcement caught my attention because it targets that curve rather than a leaderboard. The coverage describes a lightweight model aimed at high-frequency agent execution and tool calls, pitched as low-cost for agent workloads rather than as a general frontier model. That's an economic claim, and economic claims about loops can be tested before the model lands in your hands.

Why agent loops break the cost model that chat apps were priced on

In chat, output tokens dominate and the prompt stays roughly stable. In an agent loop the ratio flips:

• every iteration resends the system prompt, the full tool schema set, and the whole conversation history;
• the model's own output is small — usually a JSON tool call of a couple hundred tokens;
• tool results get appended and then re-sent on every later iteration, so a large tool output is paid for repeatedly, not once.

The second effect is the one most teams underestimate. A 2,000-token search result returned at iteration 3 gets re-sent at 4, 5, 6 … 10. You paid for it eight times.

Failures then multiply the whole thing, because a task that fails at iteration 9 pays the full loop cost and starts over.

The position: route execution, don't route everything

Here's my position, stated up front so you can argue with it: most agent loops shouldn't switch models wholesale. Split planning from execution, keep the planner on a capable frontier model, and route the high-frequency execution calls to something cheap and fast — with escalation criteria that are checkable, so the cheap executor hands back when it actually struggles.

The part people skip: if you haven't instrumented per-call cost and latency, "route execution to a cheap model" is a guess with a deployment attached. Loop design drives most of the bill. Model choice is the last variable you tune, not the first.

What this walkthrough builds (bounded loop, cost meter, retry policy, concurrency cap)

Four artifacts, in dependency order:

1. A cost meter that logs input tokens, cached tokens, output tokens, wall time, and tool name per call.
2. A bounded loop with an iteration cap, a wall-clock deadline, a token ceiling, and a structured termination reason.
3. A retry policy that separates repairing a bad tool call from re-rolling the model, behind idempotency keys.
4. A concurrency cap per tenant and per task.

What the Muse Ember Coverage Actually Establishes

The source material is two trade-press items: an OODA Loop piece and a YourStory piece, both timestamped 2026-11-10 (15:43 UTC and 13:12 UTC, per the RSS metadata). That's thin, and thin sourcing should be reported as thin.

Confirmed from the reporting: Meta announced a lightweight model aimed at high-frequency agent execution and tool calls, framed as low-cost rather than frontier

That's the full extent of what the coverage supports: an announcement exists, it comes from Meta, the model is called Muse Ember, and the positioning is lightweight and low-cost for agent execution and tool calling rather than general frontier capability.

Not confirmed: parameter count, context window, benchmark numbers, pricing, tool-calling format, availability — treated as unknown throughout

I'm treating all of these as unknown for the rest of the post, because the supplied material doesn't state them:

If you see a number attributed to Muse Ember that isn't in this table, it came from somewhere I don't have. Don't build a budget on it.

Why the framing matters more than the spec sheet: "built for high-frequency tool calls" is a claim about loop economics, not a capability claim, so it can be tested without the model

"Built for high-frequency tool calls" says something about the cost per iteration and the reliability per iteration. Those are the two numbers a loop multiplies. You can build and validate the machinery that measures them today, against whatever model you already have, then plug in a candidate and read the answer in an afternoon.

That's the practical value of the announcement for anyone not at Meta: it's a prompt to fix your loop instrumentation.

How to re-run this analysis against any candidate model once real numbers exist

Every equation in this post takes the same four substitution values:

• p_in — price per uncached input token
• p_cache — price per cached input token read
• p_out — price per output token
• a_tool — probability a tool call is constructed correctly on the first attempt

Plus two measurements you take yourself: median per-call latency and the 99th percentile. When Meta publishes the first four and you measure the last two, the analysis reruns unchanged.

Anatomy of One Tool-Loop Iteration and Where the Tokens Go

The four token buckets: system prompt, tool schemas, conversation history, tool output

Every input token in an agent call lands in one of four buckets, and each scales differently.

Only the last one is under the tool author's control, and it's usually the most bloated.

Why tool schemas are a fixed per-call tax that scales with agent breadth

Schemas go out on every call whether or not the model uses that tool. A dozen tools at roughly 200–250 tokens of JSON Schema each is 2.4k–3k tokens before the model has done anything. Over a ten-iteration loop that's 24k–30k input tokens of pure capability advertisement.

Agent breadth is a direct cost multiplier. The fix isn't "write shorter descriptions" — it's exposing 3–5 tools relevant to the current subtask instead of all 40 in the registry.

The history tax: input tokens grow with every iteration, so cost is roughly quadratic in loop depth

Let S be system prompt tokens, T the total tool schema tokens, H₀ the initial task tokens, R the average model output per iteration, and O the average tool output per iteration.

Input tokens at iteration i are I_i = S + T + H₀ + i·(R + O). Summing over N iterations:

Total input ≈ N·(S + T + H₀) + (R + O)·N²/2

The first term is linear. The second is quadratic, and it's the one that punishes deep loops. "Just let the agent keep trying" is a budget decision, not a product decision.

Worked arithmetic with variables left symbolic so readers substitute their own numbers

Substituting S = 1400, T = 2600 (12 tools), H₀ = 300, R = 180, O = 400, N = 10:

Note the ratio: 69,100 input tokens against 1,800 output tokens. Any cost model that treats output as the dominant term is wrong for this workload by roughly an order of magnitude. At N = 20, total input is about 200,000 tokens — the quadratic term alone contributes ~110k.

Measuring it: a per-call log line that records input tokens, cached tokens, output tokens, wall time, and tool name

Usage field names differ across providers, so normalize at the edge instead of in your analytics.

export function meter(call, ctx) {
return async (...args) => {
  const t0 = performance.now();
  const res = await call(...args);
  const ms = Math.round(performance.now() - t0);

  const u = res.usage || {};
  const inputTokens = u.input_tokens ?? u.prompt_tokens ?? 0;
  const cachedTokens =
    u.cache_read_input_tokens ?? u.cached_tokens ?? 0;
  const outputTokens = u.output_tokens ?? u.completion_tokens ?? 0;

  console.log(JSON.stringify({
    taskId: ctx.taskId,
    iteration: ctx.iteration,
    toolName: ctx.toolName ?? null,
    inputTokens,
    cachedTokens,
    outputTokens,
    ms,
  }));

  return res;
};
}

I ran this against a local stub that returns fixed usage numbers, to confirm the log path and sanity-check the arithmetic shape. The lines below are real output from that stub; the values are illustrative:

{"taskId":"t_8812","iteration":0,"toolName":null,"inputTokens":4384,"cachedTokens":0,"outputTokens":142,"ms":612}
{"taskId":"t_8812","iteration":1,"toolName":"search_orders","inputTokens":4966,"cachedTokens":4096,"outputTokens":188,"ms":704}
{"taskId":"t_8812","iteration":2,"toolName":"get_order","inputTokens":5578,"cachedTokens":4096,"outputTokens":165,"ms":681}
{"taskId":"t_8812","iteration":3,"toolName":"refund","inputTokens":6142,"cachedTokens":4096,"outputTokens":201,"ms":733}

Two things to read off this. First, cachedTokens jumps to 4096 from iteration 1 onward — the stable prefix (system prompt plus schemas plus the initial task) is being cached, which is exactly the 4,000-token fixed tax from the table above, now billed at the cached rate instead of the full rate. Second, iteration 0 has zero cached tokens: the first call writes the cache, and you pay full price for it. A loop's cost is always front-loaded on its first call.

Latency Budgeting Per Tool Call

Decomposing the ceiling: queueing, time-to-first-token, generation, transport, tool execution

An iteration's latency is a sum, and only the middle terms belong to the model:

• queueing — provider-side waiting under load; invisible from your side except as unexplained TTFT variance
• time-to-first-token — prompt processing; scales with input length, which grows with loop depth
• generation — output tokens divided by tokens per second
• transport — RTT plus any streaming overhead
• tool execution — your own code, database, or third-party API

Time-to-first-token is the term that quietly grows through a loop. If prompt processing is 200ms at 4k tokens, it isn't 200ms at 12k. Measure TTFT at iteration 1 and iteration 10 separately, or you'll average away the effect that matters.

Why p50 latency is the wrong number for a loop, and how the tail compounds across iterations

If a single call exceeds your per-call timeout with probability q, the probability that at least one call in an N-iteration loop exceeds it is 1 − (1 − q)^N.

At q = 0.01 and N = 10: 1 − 0.99¹⁰ = 9.6%.

Roughly one in ten tasks loses a call to the tail. Fan out to eight subagents doing ten calls each — 80 calls — and the number becomes 1 − 0.99⁸⁰ = 55%. Per-call timeouts do not compose. With a per-call deadline set at p99, more than half your tasks will hit at least one timeout. Set per-task deadlines and per-call soft warnings instead.

Computing expected loop latency from per-call percentiles instead of guessing

With a measured per-call p50 of 700ms and tool execution p50 of 120ms, a 10-iteration loop's median wall time is about 10 × 820ms ≈ 8.2s — before retries. That's a median estimate, not the true median of the sum; the real distribution is wider on both sides, and the p95 of the sum is significantly worse than N × p95 suggests. Use the sum for capacity planning and the tail probability above for reliability planning.

Where the clock actually goes: instrumenting around the model call versus around the tool call

Instrument both. The meter above wraps the model call. You need a second timer around the tool:

const t0 = performance.now();
const result = await tools[name](args);
log({ tool: name, toolMs: Math.round(performance.now() - t0), ok: true });

The surprise usually lives here. A "fast model" that saves 300ms per call doesn't matter if one of your tools takes 4s at p95 because it hits an unindexed query. Fix the tool before you swap the model.

The Routing Architecture: Cheap Executor, Expensive Planner

Two-tier split: planner decides what to do, executor decides how to call it

The planner runs once per task (or once per replan). It produces a plan and a set of candidate tools for the current step. The executor runs N times, picks exact arguments, and calls the tool. That's it — the split is about call frequency, not about intelligence.

Keeping the planner out of the hot path so its latency is paid once, not per tool call

A frontier model at 3s per call is fine once. It's a disaster ten times. Every architecture decision here reduces to the same question: which calls get multiplied by N, and are those calls cheap?

Escalation criteria that are checkable, not vibes: schema-validation failure, repeated self-correction, ambiguous arguments, tool error class

"Escalate when the model seems unsure" isn't implementable. These are:

function shouldEscalate(sig) {
if (sig.schemaFailures >= 2) return "schema_failure";
if (sig.sameErrorTwice) return "repeated_error";
if (sig.missingRequiredArg) return "ambiguous_arguments";
if (sig.toolErrorClass === "validation") return "tool_rejected_input";
return null;
}

The property that matters: every branch is a boolean or a counter you already have. No judgment calls, no extra model call to decide whether to call the model.

Where small executors genuinely fail: argument construction, unit and format mistakes, multi-tool sequencing

From working with small models on tool loops, the failure modes cluster tightly:

• argument construction — nesting a JSON object inside a string field, or omitting an optional-but-required-in-practice parameter
• units and formats — passing Date objects where ISO strings are expected, dollars where cents are expected, millisecond timestamps where seconds are expected
• multi-tool sequencing — issuing refund before get_order has returned the amount

The first two are cheap to fix: validate against the schema before dispatch and let the failure feed back as a structured error instead of a stack trace. The third is a planning concern and belongs in the planner — which is the argument for the two-tier split in the first place.

What must not be delegated: side-effecting tools with irreversible outcomes

No matter how good the cheap executor's accuracy looks in your eval: irreversible side effects stay on the planner path or behind explicit confirmation. Payments, deletions, outbound email, infrastructure changes — anything with a non-trivial undo cost. Saving $0.002 on a tool call isn't worth a $2,000 incident, and I'd rather be unfashionably conservative here.

Building the Bounded Loop

A loop skeleton with iteration cap, wall-clock deadline, token ceiling, and explicit termination reason

Three caps, because they catch different failures. An iteration cap catches a loop that oscillates. A wall-clock deadline catches a slow provider. A token ceiling catches the case where both are fine but the history exploded.

async function runLoop({ task, planner, executor, tools, limits }) {
const started = Date.now();
let spent = 0;
let messages = [{ role: "user", content: task }];

for (let i = 0; i < limits.maxIterations; i++) {
  if (Date.now() - started > limits.deadlineMs) {
    return { reason: "deadline", messages, spent };
  }
  if (spent > limits.maxTokens) {
    return { reason: "token_ceiling", messages, spent };
  }

  const step = await planOrReuse(planner, messages);
  const call = await executor(messages, step.candidateTools);
  spent += call.usage.inputTokens + call.usage.outputTokens;

  if (!call.toolCall) {
    return { reason: "completed", messages, spent, answer: call.text };
  }

  const result = await dispatch(call.toolCall, tools);
  spent += Math.ceil(result.text.length / 4);
  messages = appendToolResult(messages, call.toolCall, result);
}

return { reason: "iteration_cap", messages, spent };
}

Returning structured termination reasons instead of a bare string so callers can act on them

A bare "" on success tells the caller nothing about why a task ended. Those reasons drive different behaviour:

Aggregate them. A rising iteration_cap rate is the earliest signal that a cheap executor isn't up to the task — earlier than your eval, and free.

Prompt and schema caching as a cost lever, with the caveat that cache hits depend on stable prefixes

Caching pays off on exactly the fixed tax identified earlier: system prompt plus schema block. It only works if the prefix is byte-stable. Practical rules:

• put the system prompt and schemas first, and never interpolate a timestamp, request ID, or user name into them;
• don't reorder tools between calls;
• treat the first call in a task as uncached and budget accordingly;
• verify you're actually getting hits — log cachedTokens and watch it. A cache that silently stops hitting stays invisible in aggregate spend until the invoice arrives.

Trimming history safely: what to drop, what to summarize, what must never be dropped

Drop: the raw text of large tool outputs once their relevant fields have been extracted.

Summarize: prior reasoning steps into a compact "what has been established" block.

Never drop: the original task statement, and the record of side effects already performed. If you forget you already issued a refund, you'll issue it twice — no amount of context compression is worth that.

Failure Handling When Retries Are Cheap

Retry versus repair: a second generation attempt is not the same as a corrected tool call

Two different operations get called "retry":

• Repair — the tool call was malformed or rejected. Feed the validation error back and re-generate. The model has new information; this usually works on attempt two.
• Retry — regenerate from the same state and hope for a better draw. Expensive and mostly useless unless the failure is sampling noise.

Cheap execution makes repair attractive. It does not make blind retry attractive, because a retry pays for the full grown history again — and the quadratic term doesn't care that tokens are cheap.

Idempotency keys and dry-run modes for any side-effecting tool before retry loops are allowed

const key = taskId + ":" + stepId + ":" + toolName;
await tools[name](args, { idempotencyKey: key });

If the tool can't accept an idempotency key, give it a dry-run mode and gate retries behind a successful dry run. A tool that is neither idempotent nor dry-runnable must not sit inside a retry loop. Hard rule, not a preference.

Retry budgets: max attempts, max total spend per task, and what to do when the budget is exhausted

const budget = { attempts: 0, maxAttempts: 3, spent: 0, maxSpendTokens: 120000 };

function mayRetry(b) {
  return b.attempts < b.maxAttempts && b.spent < b.maxSpendTokens;
}

When the budget runs out, don't return a generic error. Return retry_budget_exhausted with the last tool error attached, and escalate to the planner model with that context. The escalation is cheap because it happens once, at the end.

Distinguishing model-side failures from tool-side failures so you retry the right thing

A 403 retried three times costs three times as much and produces three identical failures. Classify before you retry.

Circuit breakers on tools, not just on the model

The instinct is to guard the model. The tool is the more common outage: a third-party API goes down, your breaker trips, and the agent stops burning tokens against a dead endpoint. Track failure rate per tool name over a short window and short-circuit at, say, 50% failures over 20 calls.

Concurrency and Rate Limits

Throughput math: concurrency equals target RPS times per-call latency, and both terms move

Little's Law: L = λ × W. Three tasks/s at 8.2s per task means about 25 tasks in flight, and if each task holds one model call at a time, roughly 25 concurrent model calls. That's a steady-state number, not a peak — bursts from a queue drain will exceed it, which is why you cap rather than calculate.

The tail problem: fan-out turns p99 into the common case

Already computed: 80 calls at q = 0.01 gives a 55% chance of at least one tail event. Under load, the provider's own queueing inflates q, so the real number is worse than your benchmark suggested. Fan-out is the biggest driver of timeout rates in agent systems, and it usually gets introduced for latency reasons without anyone re-running the arithmetic.

Per-tenant and per-task concurrency caps to stop one runaway agent from starving the fleet

class Semaphore {
constructor(n) { this.n = n; this.q = []; }
async acquire() {
  if (this.n > 0) { this.n--; return; }
  await new Promise((r) => this.q.push(r));
}
release() {
  const next = this.q.shift();
  if (next) next(); else this.n++;
}
}

const perTenant = new Map();
function tenantLimit(id) {
if (!perTenant.has(id)) perTenant.set(id, new Semaphore(8));
return perTenant.get(id);
}

Eight concurrent calls per tenant is arbitrary; the point is that it exists before the first incident. Also cap per task at 1–2, because a single task fanning out to 20 parallel calls is almost always a design bug rather than a throughput requirement.

What to measure under load: queue depth, in-flight calls, throttle events, and abandoned tasks

Evaluating the Swap Instead of Believing the Framing

The eval shape: task, inputs, criterion, count — reported together

A result without a task, an input set, a criterion, and a count is an anecdote. Write it in that shape or don't report it:

Task: resolve a customer refund request end to end using 3 tools.
Inputs: 60 synthetic tickets, 20 per difficulty band, fixed seed.
Criterion: correct tool called with correct arguments, no duplicate
  side effects, terminal state matches ground truth.
Count: 60 tasks.
Result: PENDING — candidate model not yet available for measurement.

I'll be explicit: I have not run this against Muse Ember. Availability is unknown, so the result line above is deliberately empty rather than filled with something plausible. Fill it in when you can, and keep the shape.

Cost per successful task beats cost per token as the decision metric

cost_per_success = total_spend / successful_tasks

A cheaper model with lower first-attempt accuracy can easily lose. If a candidate costs 40% less per token but needs 1.6 attempts where the baseline needs 1.05, cost_per_success goes up, not down — before counting the extra latency. Price per token is an input to this equation, not the answer.

Keep a frontier baseline and a cheap-executor candidate in the same harness

Same tasks, same seeds, same tools, same criterion. Different only in the model identifier. If the harness has to change to accommodate the candidate, you're measuring two things at once.

Reporting honest sample sizes and naming the exact model and version behind every result

"Small models handle this fine" isn't a finding. "Model X, version Y, 47/60 on this task set, with all 13 failures in argument construction for the refund tool" is. Three prompts is an anecdote; say so, and say how many successful attempts you'd need before the confidence interval narrowed enough to act on.

Intermittent failures need a rate and a condition, not an adjective

"The model sometimes gets confused" is not reportable. "Fails 3 of 20 times, only when the order has more than one line item and the tool returns a paginated result" is — because it names the condition, which tells you whether to fix the prompt, the tool output, or the routing.

A Migration Checklist Before You Rewrite the Loop

Instrument first: you cannot route on costs you have not measured

Ship the meter, backfill a week of production traces, and compute tokens-per-call split by bucket. Everything else in this post depends on that table.

Split the planner from the executor before changing model names

The split is a code change with no model dependency. Doing it first means the swap is a one-line config change and an eval, not a refactor under pressure.

Add caps and idempotency before you add retries

Cheap calls make retries feel free. They aren't — they multiply spend and side effects. Caps first, idempotency second, retries third.

Common traps: schema bloat, unbounded history, retrying non-idempotent tools, measuring p50 only

Four, and they're the four you'll actually hit:

1. Exposing all 40 tools on every call because scoping them per step was "extra work."
2. Letting history grow because the task "usually finishes in five iterations."
3. Retrying a create_* call because it timed out — and creating it twice.
4. Quoting a p50 in the design review and discovering the p99 in production.

Further Reading

Primary links to the original Muse Ember coverage with a note that model specifics remain unverified

• OODA Loop — coverage of the Muse Ember announcement
• YourStory — Muse Ember coverage

Neither item supplies parameters, pricing, or benchmarks; treat the spec sheet as unverified until Meta publishes one.

Provider documentation for tool-calling and caching, cited per provider rather than generalized

• OpenAI — function calling guide
• Anthropic — prompt caching

Caching semantics, minimum cacheable prefix lengths, and TTL differ by provider and change over time. Read the page for the provider you're actually calling rather than assuming behaviour transfers.

Conclusion: Design the Loop for the Bill, Then Pick the Model

Restating the position: per-call cost is a loop-design property, and model choice is the last variable, not the first

The fixed schema tax, the quadratic growth of history, the tail compounding across iterations, and the tail-fan-out interaction are all properties of your loop. A cheaper model makes each unit smaller. It doesn't fix a loop that sends 40 tool schemas ten times and retries non-idempotent writes. Do the structural work first, then measure what a candidate actually saves — because on a well-built loop the achievable saving is bounded and knowable, and on a badly-built one, swapping models is just a discount on waste.

The Muse Ember framing points at the right problem: high-frequency tool calls are the cost centre. But framing isn't a spec sheet, and "low-cost" isn't a number.

What would change this conclusion: published per-token pricing and measured tool-call accuracy from Meta

Two things, and both are currently unknown:

1. Published pricing, including the cached-input rate. If cached input is priced aggressively enough, the fixed schema tax largely disappears and the case for aggressive tool-surface pruning weakens considerably.
2. Measured tool-call accuracy on a task set with a stated criterion and count. If first-attempt accuracy on real tool schemas is high, the escalation machinery here becomes a safety net rather than a load-bearing component — and the planner can stay out of the hot path almost entirely.

Either one moving sharply enough changes the routing calculus. Until then: instrument the loop, split the tiers, cap everything, and treat the candidate as unmeasured — because it is.