Measuring LFM2.5-24B-A2B Throughput Against Real Agent Turn Latency

Introduction: throughput claims versus real agent turn latency

Liquid AI's LFM2.5-24B-A2B is reported as a 24B-parameter mixture-of-experts model that "runs agents on-device at 184 tokens per second." That figure leads every headline, and it is the least useful number in the release. This post shows how to measure what actually decides whether an on-device agent loop is usable: p95 agent turn latency at your real context size, with your real tool calls, on your real hardware.

184 tokens/s is decode throughput: how fast the model emits tokens once it has started, under whatever conditions the measurement used. An agent turn is not a decode benchmark. It is prefill of the accumulated context, plus decode, plus one or more tool round-trips, plus orchestration overhead, plus retries when the model gets the tool arguments wrong. Decode is frequently the smallest of those terms.

The argument here is narrow and testable: on-device agent viability is decided by p95 turn latency at your real context size, not by peak decode throughput. A 24B mixture-of-experts model that does 184 tokens/s but takes four seconds to prefill an 8k-token context on your laptop tells you almost nothing about whether the agent loop feels usable.

So this post builds the harness to measure that, works through the arithmetic showing how far the headline sits from the wall clock, and stays explicit about what I ran and did not run. What I did not run is any inference. There is no measured table below, because I did not have the weights, the device, or a verified model card in front of me. What follows is a measurement plan plus the numbers that fall out of the latency equation — arithmetic, labelled as arithmetic.

What the LFM2.5-24B-A2B release actually confirms

Confirmed: parameters, MoE architecture, and the 184 tokens/s claim

From the available coverage, dated 15–16 October 2026:

• 24B total parameters. Stated consistently across both reports.
• Mixture-of-experts with a smaller active set. The MoE framing is stated; "A2B" in the name strongly suggests a per-token active-parameter budget in the ~2B range.
• On-device agent workloads are the target. Both reports frame it as an agent-inference model for client hardware, positioned against per-token cloud costs.
• 184 tokens/s is the reported figure. It appears in the release coverage as the headline throughput.

Not confirmed: context window, quantization, and the hardware behind 184 tokens/s

This is where the press claim stops being actionable:

• Expert count and per-token active parameters. Not stated in the material I have. The ~2B reading is my inference from the "A2B" suffix, not a confirmed spec.
• Context window. Unknown. For agent use, this matters more than anything else in the list.
• Quantization used for the 184 tokens/s figure. Unknown. An FP16 number and a 4-bit number are not comparable and get quoted interchangeably all the time.
• Hardware used for the 184 tokens/s figure. Unknown — the gap between an M-series laptop, a phone NPU, and a desktop GPU is a multiple, not a percentage.
• Output length and prompt length behind the figure. Unknown. Throughput on 50-token outputs with a warm cache is a different measurement from throughput on a 500-token agent turn.

Confirmed versus unknown claims at a glance

The accountable reading of "184 tokens/s" right now is: someone measured decode throughput on some hardware, at some quantization, with some prompt and output length. Treat it as a direction of travel, not a spec.

Why tokens per second is the wrong headline metric for agent turns

Prefill versus decode: two different bottlenecks inside inference

Two different bottlenecks hide inside "inference."

Prefill is the forward pass over the input context. It is compute-bound and parallel, so it scales roughly linearly with prompt length. It produces time-to-first-token (TTFT).

Decode is the autoregressive loop, one token per step. It is memory-bandwidth-bound and serial, so it scales with output length and is largely indifferent to prompt length — until the KV cache gets big enough that reading it dominates.

TTFT ≈ input_tokens / prefill_rate. Decode_time ≈ output_tokens / decode_rate.

A MoE model improves the second term, because each decode step only reads the active experts' weights from memory. It does very little for the first term, because prefill still touches the full model to route through it.

The turn-latency equation with worked numbers

turn = prefill(input_tokens) + decode(output_tokens) + tools + orchestration + retries

A worked example, with the assumptions out in the open: reported decode of 184 tokens/s, a prefill rate of 900 tokens/s that I picked purely to make the arithmetic concrete, 400 ms per local tool call, and 50 ms of harness glue per turn:

Turn 1: 2,000 in, 120 out, 1 tool
  prefill  = 2000 / 900  = 2,222 ms
  decode   =  120 / 184  =   652 ms
  tools    =                 400 ms
  overhead =                  50 ms
  total    =               3,324 ms

Across a five-turn tool-using run, with context growing the way agent transcripts do:

Totals: ≈58.0 s wall clock, of which ≈4.0 s is decode and ≈52.6 s is prefill. Decode accounts for about 7% of the run.

That is the whole case against the headline. The metric the release leads with covers roughly a fourteenth of the time a user actually waits. And nothing in the table is exotic — 21k input tokens on the final turn is an ordinary agent transcript.

Two caveats, both real. First, prefix caching changes the picture completely: if the server reuses the KV for the unchanged prefix, prefill per turn drops to the new suffix (a few hundred tokens), and total prefill falls from ≈52 s to under 2 s. That is why "cache the system prompt" is lever number two below, and why a benchmark that doesn't state its cache policy is not reproducible. Second, my prefill rate is an assumption. Substitute the measured number for your device; the shape of the result will not change.

MoE mechanics that still tax agent turns

Sparse activation is a genuine decode win: fewer weight bytes per step means the loop runs closer to memory bandwidth limits instead of being capped by them. But three things still tax agents specifically.

Routing overhead is per-token, not amortised. Every decode step runs the router before the experts. On small batches and short outputs — exactly the agent-turn shape — that fixed cost is a larger fraction of the step than it is in a batch-32 throughput benchmark.

The KV cache grows with the transcript, not with the active set. Sparse experts do nothing for a KV cache that is now 21k tokens long and re-read on every step. Long-context decode degrades on MoE models just as it does on dense ones.

Routing quality is a capability question, not a latency question. It is untested here, and that is a gap: a model that routes badly produces malformed tool calls, which produce retries, which show up in your p95 as latency. A throughput chart cannot see this.

The harness: timing turn latency on a local OpenAI-compatible server

Workload definition: five deterministic tool-using turns

Five turns, deterministic, no LLM-judge in the timing path:

1. Turn 1: system prompt + user request. Model must call read_file with a path.
2. Turn 2: tool result injected. Model must call grep with a pattern.
3. Turn 3: tool result injected. Model must call summarise with a fixed argument.
4. Turn 4: tool result injected. Model produces an intermediate plan.
5. Turn 5: model produces the final answer.

Three tools, all local and deterministic, each returning a fixed payload so tool time is measurable independently of network jitter. Temperature 0. The same prompt set for every run — no sampling of user inputs, because input variance is a confound you cannot control away later.

Instrumentation: timing the client call, not the server log

Time the client call, not the server log. The user experiences the client.

import { performance } from "node:perf_hooks";

const ENDPOINT = process.env.ENDPOINT ?? "http://127.0.0.1:8080/v1/chat/completions";

export async function timedTurn(messages, { maxTokens = 512, tools } = {}) {
const t0 = performance.now();
const res = await fetch(ENDPOINT, {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    model: process.env.MODEL ?? "local",
    messages,
    max_tokens: maxTokens,
    temperature: 0,
    stream: true,
    ...(tools ? { tools } : {}),
  }),
});

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buf = "";
let ttft = null;
let chunks = 0;

while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += decoder.decode(value, { stream: true });

  const lines = buf.split("
");
  buf = lines.pop() ?? "";

  for (const line of lines) {
    if (!line.startsWith("data:")) continue;
    const payload = line.slice(5).trim();
    if (payload === "[DONE]") continue;
    const delta = JSON.parse(payload).choices?.[0]?.delta?.content;
    if (delta) {
      chunks += 1;
      if (ttft === null) ttft = performance.now() - t0;
    }
  }
}

const totalMs = performance.now() - t0;
const decodeMs = ttft === null ? 0 : totalMs - ttft;

return {
  ttftMs: ttft === null ? null : Math.round(ttft),
  outputTokens: chunks,
  decodeTps: decodeMs > 0 ? Number(((chunks - 1) / (decodeMs / 1000)).toFixed(1)) : null,
  totalMs: Math.round(totalMs),
};
}

Two things to know about this wrapper. Counting streamed content deltas approximates token count — if your server emits usage on the final chunk, use that instead, since chat templates split tokens unpredictably. And the decodeTps figure deliberately excludes the first token; including it inflates the number by 1/TTFT.

Wrap the tool call and orchestration separately:

const t1 = performance.now();
const toolResult = await runTool(call);
const toolMs = performance.now() - t1;

Record toolMs per call and sum, rather than deriving it by subtraction. You need to see tool time separately to know whether a slow turn was the model or the filesystem.

Context-growth schedule: 2k, 8k, and 24k input tokens

Run the same task at roughly 2k, 8k, and 24k input tokens by prepending irrelevant-but-plausible context (a repo listing, a log file). Plot TTFT against input tokens. If the relationship is linear, prefill dominates and you should be optimising caching. If it is flat until 16k and then jumps, you are hitting a cache-size cliff and should be capping context instead.

Controls to pin before any throughput number counts

Pin everything that moves:

For a local OpenAI-compatible server, the shape is:

llama-server -m ./lfm2.5-24b-a2b.gguf -c 24576 -t 8 --port 8080

Check llama-server --help for your build — flags shift between releases, and -c must be large enough for your largest context case or you will silently get truncation instead of an error.

Results: arithmetic projections, not measurements

No inference was executed for this post. Every number below is either arithmetic from the turn-latency equation or a field the harness will fill in when you run it. I am not going to publish a table of numbers I did not take.

Per-turn record and how to aggregate p50/p95

The harness emits one record per turn. Field names are real; the nulls are where your measurements go:

{"run":0,"turn":0,"inputTokens":2000,"outputTokens":null,"ttftMs":null,"decodeTps":null,"toolMs":null,"totalMs":null}

Aggregate by turn index and report p50/p95 across runs, not across turns. Five turns is not a distribution. You need at least 20 runs of the same 5-turn task before a p95 means anything, and the honest minimum for a p95 claim is closer to 40.

Raw log excerpt: none, because there was no run

There is none, because there was no run. The projection table in the previous section stands in for it and is labelled as arithmetic. When you run the harness, the projection is falsified the moment your measured prefill rate differs from the 900 tokens/s I assumed — and it will.

Where the 184 tokens/s claim should hold

Predicted, not observed: the figure should hold for short prompts (under ~1k tokens), a warm prefix cache, pure decode with no tool calls, and short outputs. That is the regime every throughput benchmark measures, and it is also the regime agent turns almost never occupy after turn two.

Where it should collapse under agent workloads

Predicted, not observed:

• Long prefill. At 24k input tokens with no prefix cache, prefill alone exceeds 25 s at my assumed rate. Decode speed is irrelevant at that point.
• Growing context. Each turn re-pays for the whole transcript unless the server caches the prefix. Agent transcripts grow monotonically; this is not an edge case.
• Tool round-trips. 400 ms × 2 calls is 800 ms of pure latency that a tokens/s chart never shows.
• Retries. A malformed tool call doubles the turn. On a model with untested routing quality, budget for it.
• Thermal throttling. Over 20 turns, sustained load on a laptop will cost you throughput you measured on turn one.

Comparing LFM2.5-24B-A2B with Qwen fairly

Fair-comparison rules for on-device LLM inference

The coverage reportedly compares LFM2.5-24B-A2B with Qwen. I cannot cite which Qwen variant or which numbers, because the material I have does not specify it. If you do run the comparison, the rules are non-negotiable:

1. Same quantization family — do not compare a 4-bit LFM against an FP16 Qwen.
2. Same hardware, same power profile, same thread count.
3. Same context lengths, at 2k / 8k / 24k, not one convenient number.
4. Same output token budget, since decode time is linear in output length.
5. Report turn latency alongside tokens/s, and report the tool time separately.
6. Match parameter class where you can — a 24B MoE with a small active set is not the same weight class as a dense 7B, and it will not have the same memory footprint.

What a single-device comparison can and cannot show

A single-device run of two models is directional. It tells you which one deserves the next week of your time for your workload. It does not produce a ranking, and it should not be written up as one: device-specific bandwidth, one runtime's kernel quality, and one quantization recipe will dominate the gap more than architecture does.

The vendor's 184 tokens/s remains unverified in this post, and any comparison I could write without running both models would be fiction with a decimal point.

Design implications for on-device agent latency

Six levers, roughly in order of payoff:

1. Cap and summarise context. This attacks prefill, which is 90% of the wall clock in the arithmetic above. Hard-cap the transcript at N tokens and replace the oldest turns with a summary. Measure the quality cost — it is real — but measure it as task success, not as latency.

2. Cache the system prompt and stable prefix. Prefix reuse cuts the ≈52 s prefill figure to under 2 s if the cache holds across turns. Verify it holds: log prefill token counts server-side and confirm turn 2 prefills only the delta. If the count resets to the full context, your cache is not working and you have been measuring the wrong thing.

3. Stream and render TTFT. A 2 s TTFT with streaming feels responsive; 4 s of silence followed by 200 tokens at once does not. Perceived latency is a UI property, and it is free.

4. Parallelise independent tool calls. If turn 2 reads three files, issue them concurrently. Sequential tool calls are pure serial latency added to a turn that already has a prefill bill.

5. Bound retries. One retry on malformed tool arguments, then fail loudly with a schema error to the model. Unbounded retries are how an agent loop turns a 3 s turn into a 30 s one, and how a p95 becomes meaningless.

6. Set a p95 turn-latency SLO and gate on it. Pick a number — 5 s at your median context size is a defensible starting point for interactive use, 2 s for autocomplete-style loops — and treat any configuration that misses it as not shippable, regardless of decode throughput.

Limitations and untested claims

Sample size: zero model runs. Single device: none. No concurrency testing, no quality evaluation beyond the five-turn task definition, no verification of the 184 tokens/s figure, no confirmation of the context window, expert count, active parameter count, or quantization behind the vendor number. The 900 tokens/s prefill rate is an assumption chosen to make the arithmetic legible, not a measurement. The "~2B active" reading is inference from the model name.

What is confirmed is narrow: a 24B-parameter MoE aimed at on-device agents was reported in mid-October 2026 with a headline figure of 184 tokens/s, and the surrounding material does not disclose the conditions under which that figure was obtained.

Conclusion: judge on-device agents on p95 turn latency

The 184 tokens/s number is probably real and probably irrelevant to your agent. In the arithmetic above, decode is 7% of wall-clock time on a five-turn tool-using task — the headline metric covers the one part of the loop that sparse activation already made fast, and says nothing about the part that actually hurts.

Judge on-device agent models on p95 turn latency at your real context size, with your real tool calls, on your real thermal state. Run the harness, publish the prefill number alongside the decode number, and state your cache policy. Until Liquid AI publishes the model card with context window, quantization, and hardware for that figure, treat 184 tokens/s as a marketing unit — a useful one for signalling that on-device agents are getting tractable, and a useless one for deciding whether the loop is fast enough to ship.

Further Reading

• Liquid AI — company site; the official LFM2.5-24B-A2B model card is the primary source for context window, expert count, and benchmark conditions. I did not verify a direct model-card URL at time of writing, so treat this as the entry point rather than the citation.
• Liquid AI LFM2.5-24B-A2B: Specs, Benchmarks vs Qwen — tech-insider.org, the "specs and benchmarks" report referenced above.
• Liquid AI Releases LFM2.5-24B-A2B: 24B-Parameter MoE Model Runs Agents On-Device at 184 Tokens Per Second — news.lavx.hu, the report carrying the 184 tokens/s claim.
• llama.cpp — server docs for the OpenAI-compatible endpoint used by the timing harness, including --jinja, context sizing, and prefix-cache behaviour.