Benchmarking llm-d 1.0 on Kubernetes: p99 Latency, Tokens/sec/GPU, and Cost per Million Tokens

Why llm-d 1.0 Needs a Benchmark You Run Yourself

llm-d 1.0 landed on 20 October 2026: an open-source distributed inference stack aimed at production-scale AI workloads, IBM-backed, launched at PyTorch Conference. TOKENPOST, HPCwire and Unite.AI all ran with the same three notes — new controls, cost efficiency, production scale.

What none of them carries is a measured number tied to a stated workload. That's not a dig at the reporting; it's the normal shape of a launch story. It does mean a team evaluating llm-d for a real deployment starts at zero evidence, and "cost-efficient" is not a claim you can adopt. It's only a claim you can test.

This post is the test. By the end you'll have a benchmark harness for llm-d 1.0 on Kubernetes that produces p99 latency, tokens per second per GPU, autoscaling behaviour under a mid-run load step, and a defensible cost per million tokens — all against a workload contract you write down before the first run.

My position: llm-d is worth evaluating, and most teams should not adopt it on launch evidence. The only thing between those two statements is a benchmark you design yourself, with the metric set frozen before you touch the cluster.

Disclosure: I have not run this sweep against llm-d 1.0. Everything below is either arithmetic you can re-derive, a command you can run, or a claim explicitly marked as untested.

What llm-d 1.0 Actually Ships — and What the Launch Coverage Does Not Say

The four architecture pieces a benchmark has to isolate

The project documents four things a benchmark has to separate rather than blend: vLLM as the model server, prefix-cache-aware routing at the gateway layer, prefill/decode disaggregation, and Kubernetes-native scheduling. Each one moves a different metric.

Benchmark "llm-d vs a single vLLM Deployment" as one A/B and you learn nothing about which piece paid for itself. Turn routing off, measure again. That's the difference between a result and a vibe.

Claim-to-measurement table

The routing row deserves the emphasis because it's where most self-run benchmarks quietly break. Cache-aware routing only wins when requests share prefixes. Feed it uniformly random prompts and the scheduler has nothing to match on — it degrades toward round-robin and the measured benefit goes to zero. If your synthetic workload has no prefix overlap, you benchmarked the fallback path.

Fix the Metric Set First: TTFT, TPOT, Goodput, and $/M Tokens

Four numbers carry the decision: p99 TTFT, p99 TPOT, goodput inside the SLO, and cost per million output tokens. Everything else is a diagnostic.

TTFT, ITL/TPOT, and why the tail is the number that pages you

Time to first token governs perceived responsiveness; inter-token latency (or time per output token) governs whether streaming feels smooth. Report both at p50, p95 and p99.

p50 hides queueing. When a stack saturates, p50 often stays flat while p99 triples — that gap is the queue, and it's exactly what wakes your on-call. A p99 TTFT quoted without the concurrency level it was measured at isn't a number; it's decoration.

Tokens/sec/GPU is meaningless without a workload contract

Throughput per GPU is a function of input:output ratio, prefix overlap and concurrency. A stack hitting 4,000 output tok/s/GPU at 1:1 and concurrency 8 can plausibly drop to a third of that at 4:1 with concurrency 64. Any tokens/sec/GPU figure stated without those three parameters should be discarded on sight.

Goodput and cost-per-million-tokens as the roll-ups

Two numbers survive contact with finance:

• Goodput — requests per second that completed within the SLO. Not total requests served.
• $/M tokens — node-hour cost divided by measured tokens, at the utilization you actually run at.

Everything else is diagnostics.

Build a Reproducible llm-d Test Cluster on Kubernetes

Pin everything

The model-weights row is the one people skip. Pin the revision hash, not the model name.

Install llm-d 1.0 with Helm and verify the routing path

## Pin the tag first; chart paths differ between releases — check the tag's README.
git clone https://github.com/llm-d/llm-d.git
cd llm-d && git checkout v1.0.0

helm install llmd ./charts/llm-d -n llmd --create-namespace \
  -f bench/values-h100-8x.yaml

Verify the shape before benchmarking anything:

kubectl -n llmd get pods
kubectl get crd | grep -i inference

You want the gateway/scheduler pod Running, one vLLM pod per replica at Running 1/1, and — if this release routes through the Gateway API Inference Extension — its CRDs present. Empty CRD list means you're on the fallback routing path, and your cache-aware numbers will be wrong.

Design the Workload So the Numbers Mean Something

Closed-loop vs open-loop load

Closed loop (fixed concurrency, --request-rate inf) makes each client wait for a response before sending the next. Convenient, and it hides queueing: as the server slows, the client backs off on its own, so you never observe the backlog a real traffic mix would create. Use closed loop to find the saturation point. Use open loop — fixed request rate, Poisson arrivals — for any SLO claim.

Synthetic vs replayed traces

Synthetic fixed-length prompts are honest for comparability: the same distribution across two stacks isolates the stack. Traces are honest for absolute numbers. With no production traces, use synthetic random lengths with a stated distribution — and write the distribution into the report. "Random prompts" without lengths isn't a workload.

Run the Concurrency Sweep and Record p99 Latency

The concurrency ladder: 1 to 128, median of p99s

Concurrency 1, 2, 4, 8, 16, 32, 64, 128. Warm up at the target concurrency and throw that run away — the first requests carry cold prefix caches and un-warmed CUDA graphs. Run each rung 3–5 times and report the median of the p99s, never the best run.

Load generation with vllm bench serve

vllm bench serve \
  --backend openai-chat \
  --base-url http://llmd-gateway.llmd.svc.cluster.local \
  --model "$MODEL" \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --num-prompts 1000 --max-concurrency "$C" \
  --request-rate inf \
  --save-result --result-dir "results/c${C}"

Swap --request-rate inf for a fixed rate such as 8 to switch to open loop. The tool prints:

============ Serving Benchmark Result ============
Successful requests:                    <n>
Benchmark duration (s):                 <t>
Mean TTFT (ms):                         <a>
P99 TTFT (ms):                          <b>
Mean TPOT (ms):                         <c>
P99 TPOT (ms):                          <d>
Output token throughput (tok/s):        <e>
Total token throughput (tok/s):         <f>

Record one row per rung:

How to read the knee: throughput climbs roughly linearly until p99 TTFT crosses your SLO while p50 is still flat. That concurrency level is your operating point. Everything past it is throughput you're buying with latency you promised not to spend. The numbers you fill in above are yours; don't borrow mine, because I have none.

Benchmarking Autoscaling Without Lying to Yourself

What to measure

Three timings, all of which usually get omitted:

• Scale-up latency — from metric threshold breach to the new pod Ready.
• Pending backlog — requests queued in the gateway while pods are Pending. This is where the user pain actually lives.
• Cold-start TTFT — TTFT for the first requests hitting a new replica, before its prefix cache and CUDA graphs are warm. Expect this to be materially worse than steady state.

p99 during a scale event is the real production number

Steady-state p99 is the number you report. p99 measured through a scale event is the number production experiences. Include at least one run where load steps up 3× mid-test and the autoscaler has to respond, then report latency during the response window separately.

Cost per Million Tokens, Computed Properly

Worked arithmetic

Inputs — all assumptions, substitute your own:

• 8× H100 80GB node, $2.50/GPU-hour on-demand → $20.00/node-hour
• Measured output throughput at the SLO knee: 1,800 tok/s
• SLO: p99 TTFT ≤ 800 ms, p99 TPOT ≤ 50 ms

Then:

tokens per node-hour = 1,800 × 3,600        = 6,480,000
$/M output tokens    = 20.00 / 6.48         = $3.09

at 60% average utilization (autoscaler reality):
  $/M = 20.00 / (6.48 × 0.60)               = $5.14

A 40-point swing in utilization moves the headline number by 66%. State the utilization you divided by, or the number is unfalsifiable.

The three traps

1. Idle replicas. minReplicas: 2 running at 20% load means the denominator is a fifth of what justifies the numerator. Compute $/M at the average load, not the peak.
2. Overprovisioning for the tail. Sizing for p99 rather than p50 may need 30% more GPU. That's a legitimate cost of the SLO — but it belongs in the $/M figure, not in a footnote.
3. Amortizing burst hardware. Counting a batch job's tokens against a node that idles the remaining 20 hours flatters everything. Keep interactive and batch cost pools separate.

Ways These Benchmarks Get Wrong by Accident

A Paste-Ready Reporting Template

ENVIRONMENT
  k8s: <ver> | gpu-operator: <ver> | driver: <ver> | CUDA: <ver>
  vllm image: <repo>@sha256:<digest>
  llm-d: <app ver> / chart <ver>
  model: <name> @ <revision sha>
  hardware: <n>× <gpu>, <interconnect>
WORKLOAD
  distribution: <synthetic-fixed | synthetic-random | trace>
  input:output ratio: <x:y> | prefix overlap: <pct> | concurrency ladder: <list>
  loop type: <closed | open @ rps> | warmup: <policy> | reps: <n>
RESULTS
  <table of p50/p95/p99 TTFT, p99 TPOT, tok/s, goodput per rung>
COST
  node-hour: <$> | utilization: <pct> | $/M output tokens: <$>
CAVEAT
  These numbers hold for the workload contract above and should not be
  extrapolated to a different input:output ratio or prefix overlap.

That caveat sentence isn't optional. It's the difference between a measurement and a marketing asset.

What to Do With the Results

Three decision rules:

• Adopt now if goodput at your p99 SLO beats a single vLLM Deployment by a margin that survives your utilization assumptions, and cache-aware routing is demonstrably contributing. If the routing-off run matches the routing-on run, you're paying llm-d's operational complexity for vLLM.
• Hold pending evidence if your workload has low prefix overlap, or you can't get a stable p99 at your target concurrency. Both point at the benchmark being the bottleneck, not the stack.
• Stay on a single vLLM deployment if you're under roughly one node of capacity. Disaggregation and distributed scheduling solve a problem you don't have yet, and they add failure modes you'll debug at 3am.

The launch claims may well be true. They just aren't evidence yet, and the sweep above is cheap enough that there's no reason to take them on faith.

Further Reading

• llm-d project site — architecture and release documentation
• llm-d on GitHub — charts, versions and release tags
• vLLM benchmarking documentation — the vllm bench serve harness used above
• Gateway API Inference Extension — the routing and scheduling CRDs
• Launch coverage (secondary sources, 2026-10-20/21): TOKENPOST, HPCwire