vLLM benchmarking

Target audience: operators producing defensible latency/throughput numbers against production or pre-production vLLM deployments, on datacenter GPUs, often in containerized or air-gapped environments.

Why this matters

Bad benchmarks are worse than no benchmarks — they drive the wrong decisions with false confidence. The three common failure modes:

1. Wrong methodology. --request-rate inf answers "saturation throughput," not "TTFT my users see." Mixing those up leads to buying GPUs to solve a latency problem, or shipping a latency regression because total throughput looked fine.
2. Wrong workload. --dataset-name random has zero prefix structure. Real coding-agent or RAG traffic has heavy prefix reuse. Benchmarking caching wins on random produces numbers that don't survive contact with prod.
3. No warmup / wrong tokenizer. First N requests hit cold CUDA graphs. Token counts are fiction unless --tokenizer matches the served model exactly.

The cost of getting this right is small; the cost of getting it wrong is buying the wrong hardware.

Decision tree — which subcommand

Most production questions route to vllm bench serve. Reach for the others only when the question is specifically kernel-level (latency), offline-batch (throughput), or SLO auto-tuning (sweep).

The two methodologies operators actually need

Methodology A: health check / SLO validation

Question: "Does my running deployment meet the latency SLO under realistic load?"

• Fixed --max-concurrency matching the production in-flight ceiling (NOT --request-rate).
• Realistic input/output length distribution — ideally replayed from production logs via --dataset-name custom with a JSONL file.
• Sustained 10+ minute run to cover warmup + steady state.
• Report: P50/P95/P99 TTFT, P95/P99 ITL, throughput (tok/s, req/s).
• Compare against the SLO. Pass/fail.

vllm bench serve \
  --model <served-model> \
  --base-url http://<endpoint> \
  --dataset-name custom \
  --dataset-path /data/captured-prod-prompts.jsonl \
  --max-concurrency 32 \
  --num-prompts 2000 \
  --percentile-metrics ttft,tpot,itl,e2el \
  --metric-percentiles 50,95,99 \
  --save-result --output-json health-check.json

Methodology B: change comparison / A/B

Question: "Does config change X make it faster, and at what cost?"

• Request-rate sweep, not a single rate: e.g. 1, 2, 4, 8, 16, 32, inf req/s.
• Plot throughput vs P99 latency — the knee of the curve is the usable operating point. A config that shifts the knee right is a win.
• Same seeds, same --num-prompts (≥500), same dataset on both sides.
• Run A and B back-to-back on the same hardware in the same session to avoid thermal/neighbor noise.

See scripts/bench-sweep.sh for a parametrized sweep runner that emits one JSON file per rate for plotting.

Critical pitfalls

1. No warmup. First 30–60 s hit cold CUDA graphs / torch.compile caches. vllm bench serve does not auto-warm (as of v0.21) — pre-flight the server with a few requests, or set --num-prompts large enough (≥500) to amortize. latency does warm up via --num-iters-warmup (default 10).
2. Wrong tokenizer. --tokenizer defaults to --model, but if they differ (e.g., served via a local path while benching with a HF ID), every token count in the output is fiction. Always specify explicitly.
3. `--dataset-name random` as a proxy for production traffic. Random has zero prefix structure, overstates prefill work, understates prefix-cache hit rate, makes chunked prefill look worse than reality. For anything involving caching claims, use custom with a real-traffic JSONL, or prefix_repetition for synthetic prefix-heavy tests.
4. `--request-rate inf` alone. Measures saturation throughput, not the latency regime users experience. Always include a concurrency sweep for serving comparisons.
5. `--endpoint-type` is removed. Deprecated in v0.11.0, now gone. Use --backend. Current full value set (docs.vllm.ai, verified 2026-04-24): openai, openai-chat, openai-audio, openai-embeddings, openai-embeddings-chat, openai-embeddings-clip, openai-embeddings-vlm2vec, vllm, vllm-chat, vllm-pooling, vllm-rerank, infinity-embeddings, infinity-embeddings-clip.
6. Conflating tok/s with req/s. High total-tokens/sec can coexist with terrible TTFT. Always report both plus P99 ITL.
7. Noisy neighbor. Shared GPU, unrelated container load, MIG partition changes mid-run — check nvidia-smi dmon for unrelated activity before trusting numbers.
8. `latency` subcommand disables prefix caching by default (to keep numbers clean). If benchmarking prefix-cache behavior, use serve with the prefix_repetition dataset.

For the full flag reference for each subcommand, see references/commands.md. For the dataset catalog and when to use each, see references/datasets.md.

Air-gapped environments

Operators who can't reach huggingface.co have three working patterns:

1. Reroute to a mirror — set HF_ENDPOINT=https://hf-mirror.com (or an internal reverse-proxy URL). huggingface_hub treats it transparently.
2. ModelScope — set VLLM_USE_MODELSCOPE=True plus trust_remote_code=True. Historical gap: LoRA adapter loading through ModelScope (vLLM issue #32841, closed 2026-01-23). Re-verify on your vLLM version before relying on LoRA-via-ModelScope; issue closure without a linked PR means status is unclear — test first.
3. Fully offline with pre-seeded cache — HF_HUB_OFFLINE=1 + TRANSFORMERS_OFFLINE=1, HF_HOME pointing at a pre-populated directory (NFS, PVC, or JuiceFS/S3).

For benchmark datasets specifically: sonnet is in-tree at vllm/benchmarks/sonnet.txt — never downloads. random is synthetic — never downloads. sharegpt must be pre-staged: wget the JSON on a connected host, rsync into the enclave, point --dataset-path at it.

For the full air-gapped recipe (HF proxy setup, gated model tokens, MinIO-as-HF-cache, transformer cache warming), see references/air-gapped.md.

Measuring the outcomes that matter

Default metrics (--percentile-metrics ttft,tpot,itl,e2el):

• TTFT — time-to-first-token. User-facing responsiveness. Dominated by prefill.
• TPOT — time-per-output-token (averaged across decode). Steady-state perceived speed.
• ITL — inter-token latency (per-step). Catches stalls that TPOT averages away.
• E2EL — end-to-end request latency. Only one that matters for pooling/embedding models.

Reporting guideline: always P50 and P99 together. Either in isolation is misleading. Add P95 if ITL has a long tail.

Goodput SLO — --goodput KEY:VALUE (milliseconds) tracks requests that completed within an SLO budget. Example: --goodput ttft:500 itl:50. Goodput is what actually matters in production; raw throughput that violates SLO is useless.

For methodology detail (warmup protocols, sweep design, SLO-constrained auto-tune, how to capture real-traffic prompts for replay), see references/methodology.md.

When numbers look wrong or a run crashes

See references/troubleshooting.md for the failure modes: tokenizer mismatch (numbers off 20–40%), cold-cache contamination (suspiciously fast), air-gapped hang (incomplete HF_HUB_OFFLINE setup), goodput=0 (unit error), noisy-neighbor ITL variance, and the full "what to include in a bug report" checklist.

Parsing the output JSON

See references/output-schema.md for the field layout in --output-json — top-level fields (request_throughput, output_throughput, total_token_throughput), the mean_/median_/std_/p<N>_<metric>_ms pattern, speculative decoding fields, and which names are stable across versions vs renamed.

External references

• vLLM bench CLI docs: https://docs.vllm.ai/en/latest/benchmarking/cli/
• vllm bench serve reference: https://docs.vllm.ai/en/stable/cli/bench/serve/
• Performance dashboard (nightly reference numbers): https://docs.vllm.ai/en/latest/benchmarking/dashboard/
• In-tree benchmarks dir: https://github.com/vllm-project/vllm/tree/main/benchmarks
• Air-gapped discussion thread: https://discuss.vllm.ai/t/setting-up-vllm-in-an-airgapped-environment/916
• vLLM env vars (including VLLM_USE_MODELSCOPE): https://docs.vllm.ai/en/stable/configuration/env_vars/
• Blog: Anatomy of a High-Throughput LLM Inference System (2025-09-05): https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html
• Blog: Large Scale Serving — DeepSeek @ 2.2k tok/s/H200 (2025-12-17): https://blog.vllm.ai/2025/12/17/large-scale-serving.html