vLLM benchmarking

Target audience: operators producing defensible latency/throughput numbers against production or pre-production vLLM deployments, on datacenter GPUs, often in containerized or air-gapped environments.

This skill measures; it does not tune. Once a number is trusted and the verdict is "too slow", the knobs live elsewhere in the vllm plugin: `vllm-performance-tuning` (scheduler, MoE kernels, CUDA graphs, parallelism), `vllm-caching` (KV tiering when the bottleneck is prefill or cache capacity), `vllm-nvidia-hardware` (the SKU's own ceiling). Measure → change one thing → re-measure with the same methodology; a tuning change compared against a differently-shaped benchmark run is not evidence.

What a bad benchmark costs

A wrong number is acted on with the same confidence as a right one. Three failure modes, by what they cause — the flags that produce each are in Critical pitfalls below:

1. Wrong methodology → GPUs bought to fix a latency problem, or a latency

regression shipped because total throughput looked fine.

1. Wrong workload → caching wins that do not survive contact with production.
2. No warmup, or a mismatched tokenizer → cold-start numbers, and token counts

that are fiction.

Decision tree — which subcommand

Most production questions route to vllm bench serve. Reach for the others only when the question is specifically kernel-level (latency), offline-batch (throughput), or SLO auto-tuning (sweep).

The two methodologies operators actually need

Methodology A: health check / SLO validation

Question: "Does my running deployment meet the latency SLO under realistic load?"

• Fixed --max-concurrency matching the production in-flight ceiling (NOT --request-rate).
• Realistic input/output length distribution — ideally replayed from production logs via --dataset-name custom with a JSONL file.
• Sustained 10+ minute run to cover warmup + steady state.
• Report: P50/P95/P99 TTFT, P95/P99 ITL, throughput (tok/s, req/s).
• Compare against the SLO. Pass/fail.

vllm bench serve \
  --model <served-model> \
  --base-url http://<endpoint> \
  --dataset-name custom \
  --dataset-path /data/captured-prod-prompts.jsonl \
  --max-concurrency 32 \
  --num-prompts 2000 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95,99 \
  --save-result --output-json health-check.json

Methodology B: change comparison / A/B

Question: "Does config change X make it faster, and at what cost?"

• Request-rate sweep, not a single rate: e.g. 1, 2, 4, 8, 16, 32, inf req/s.
• Plot throughput vs P99 latency — the knee of the curve is the usable operating point. A config that shifts the knee right is a win.
• Same seeds, same --num-prompts (≥500), same dataset on both sides.
• Run A and B back-to-back on the same hardware in the same session to avoid thermal/neighbor noise.
• Done when every rate has an output JSON on both sides with failed == 0 (re-run any rate with failures), each side's knee is named as the highest rate whose P99 TTFT and P99 ITL stay within SLO, and the report lists rate, achieved RPS, P50/P99 TTFT and ITL, and tok/s for every rate on both sides.

See scripts/bench-sweep.sh for a parametrized sweep runner that emits one JSON file per rate for plotting. Every rate replays the same seeded prompts, so it resets the server's prefix/MM/encoder caches before each rate; start the server with VLLM_SERVER_DEV_MODE=1 or every rate after the first runs on a warm prefix cache.

Critical pitfalls

1. No warmup. First 30–60 s hit cold CUDA graphs / torch.compile caches. vllm bench serve does not auto-warm (v0.11–v0.32; --num-warmups default re-read as 0 at v0.32.0) — pre-flight the server with a few requests, or set --num-prompts large enough (≥500) to amortize. latency does warm up via --num-iters-warmup (default 10).
2. Wrong tokenizer. --tokenizer defaults to --model, but if they differ (e.g., served via a local path while benching with a HF ID), every token count in the output is fiction. Always specify explicitly.
3. `--dataset-name random` as a proxy for production traffic. Random has zero prefix structure, overstates prefill work, understates prefix-cache hit rate, makes chunked prefill look worse than reality. For anything involving caching claims, use custom with a real-traffic JSONL, or prefix_repetition for synthetic prefix-heavy tests.
4. `--request-rate inf` alone. Measures saturation throughput, not the latency regime users experience. For serving comparisons run the Methodology B request-rate sweep (a --max-concurrency sweep only for closed-loop client pools).
5. `--endpoint-type` is removed (v0.11.0+). Use --backend; vllm-chat is a bench throughput backend only, rejected by serve. Value set: commands.md. The output JSON still carries endpoint_type as an alias of backend (output-schema.md).
6. Conflating tok/s with req/s. High total-tokens/sec can coexist with terrible TTFT. Always report both plus P99 ITL.
7. Noisy neighbor. Shared GPU, unrelated container load, MIG partition changes mid-run — check nvidia-smi dmon for unrelated activity before trusting numbers.
8. `latency` subcommand disables prefix caching by default (to keep numbers clean). If benchmarking prefix-cache behavior, use serve with the prefix_repetition dataset.
9. Do not compare streaming TTFT/E2EL across v0.29 → v0.30.0 — re-baseline. v0.30.0 fixed streaming TTFT timing and stopped counting zero-chunk streams as successes with ttft=0, so older runs read low (troubleshooting.md).

For the full flag reference for each subcommand, see references/commands.md. For the dataset catalog and when to use each, see references/datasets.md.

Air-gapped environments

Operators who can't reach huggingface.co have three working patterns:

1. Reroute to a mirror — set HF_ENDPOINT=https://hf-mirror.com (or an internal reverse-proxy URL). huggingface_hub treats it transparently.
2. ModelScope — set VLLM_USE_MODELSCOPE=True plus trust_remote_code=True. Historical gap: LoRA adapter loading through ModelScope (vLLM issue #32841, closed 2026-01-23). Re-verify on your vLLM version before relying on LoRA-via-ModelScope; issue closure without a linked PR means status is unclear — test first.
3. Fully offline with pre-seeded cache — HF_HUB_OFFLINE=1 + TRANSFORMERS_OFFLINE=1, HF_HOME pointing at a pre-populated directory (NFS, PVC, or JuiceFS/S3).

For benchmark datasets specifically: random and prefix_repetition are synthetic — never download. sonnet was removed in v0.32.0 (--dataset-name sonnet is rejected); use it only on the ≤v0.31 version that produced an old baseline. sharegpt must be pre-staged: wget the JSON on a connected host, rsync into the enclave, point --dataset-path at it.

For the full air-gapped recipe (HF proxy setup, gated model tokens, MinIO-as-HF-cache, transformer cache warming), see references/air-gapped.md.

Measuring the outcomes that matter

Metrics (default --percentile-metrics is ttft,tpot,itl for generative models, e2el for pooling; add e2el explicitly):

• TTFT — time-to-first-token. User-facing responsiveness. Dominated by prefill.
• TPOT — time-per-output-token (averaged across decode). Steady-state perceived speed.
• ITL — inter-token latency (per-step). Catches stalls that TPOT averages away.
• E2EL — end-to-end request latency. Only one that matters for pooling/embedding models.

Reporting guideline: always P50 and P99 together. Either in isolation is misleading. Add P95 if ITL has a long tail. --metric-percentiles defaults to 99 only — pass 50,95,99 explicitly.

Goodput SLO — --goodput KEY:VALUE (milliseconds) tracks requests that completed within an SLO budget. Example: --goodput ttft:500 itl:50. Goodput is what actually matters in production; raw throughput that violates SLO is useless.

For methodology detail (warmup protocols, sweep design, SLO-constrained auto-tune, how to capture real-traffic prompts for replay), see references/methodology.md.

When numbers look wrong or a run crashes

See references/troubleshooting.md for the failure modes: tokenizer mismatch (numbers off 20–40%), cold-cache contamination (suspiciously fast), air-gapped hang (incomplete HF_HUB_OFFLINE setup), goodput=0 (unit error), noisy-neighbor ITL variance, and the full "what to include in a bug report" checklist.

Parsing the output JSON

See references/output-schema.md for the field layout in --output-json — top-level fields (request_throughput, output_throughput, total_token_throughput), the mean_/median_/std_/p<N>_<metric>_ms pattern, speculative decoding fields, and which names are stable across versions vs renamed.

External references

• vLLM bench CLI docs: https://docs.vllm.ai/en/latest/benchmarking/cli/
• vllm bench serve reference: https://docs.vllm.ai/en/stable/cli/bench/serve/
• Performance dashboard (nightly reference numbers): https://docs.vllm.ai/en/latest/benchmarking/dashboard/
• In-tree benchmarks dir: https://github.com/vllm-project/vllm/tree/main/benchmarks
• Air-gapped discussion thread: https://discuss.vllm.ai/t/setting-up-vllm-in-an-airgapped-environment/916
• vLLM env vars (including VLLM_USE_MODELSCOPE): https://docs.vllm.ai/en/stable/configuration/env_vars/
• Blog: Anatomy of a High-Throughput LLM Inference System (2025-09-05): https://vllm.ai/blog/2025-09-05-anatomy-of-vllm
• Blog: Large Scale Serving — DeepSeek @ 2.2k tok/s/H200 (2025-12-17): https://vllm.ai/blog/2025-12-17-large-scale-serving