AIPerf — NVIDIA generative-AI inference benchmarking

Use for defensible latency / throughput / goodput numbers against any OpenAI-compatible server (vLLM, SGLang, TensorRT-LLM, NVIDIA Dynamo, NIM, Triton, HF TGI, Ollama), and for extending AIPerf with plugins. AIPerf is the genai-perf successor; its numbers differ for reasoning models (pitfall 2). For a vLLM-only target driven by the in-tree vllm bench toolchain, use the vllm-benchmarking skill; use AIPerf when the target is non-vLLM or the workload is trace-driven, multi-modal, multi-server, or needs goodput.

Version-gated capabilities — check aiperf --version before using

Behaviour changes that make results non-comparable across versions:

• v0.11.0 warmup fix — warmup no longer leaks prefix-cache reuse into profiling. Baselines from ≤v0.10.x with warmup on are not comparable; re-run them.
• v0.13.0 sends `X-Session-Affinity` (the session's X-Correlation-ID value) on every request. A router that honours it pins a session's turns to one replica, so multi-turn results against such a router differ from v0.12.0. Disable: AIPERF_HTTP_X_SESSION_AFFINITY_FROM_CORRELATION_ID=false. Router-specific affinity is opt-in: AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID=true (Dynamo --router-session-affinity-ttl-secs), AIPERF_HTTP_X_SMG_ROUTING_KEY_FROM_CORRELATION_ID=true (SGLang Model Gateway manual routing).

Versions

• Stable on PyPI: v0.13.0 (2026-09-24), requires-python >=3.11,<3.14. pip install aiperf. A nightly wheel is published alongside as aiperf-nightly (v0.10.0+).
• Repo `main` at https://github.com/ai-dynamo/aiperf: post-v0.13.0.
• Python: ≥3.11, <3.14 — the floor moved at v0.12.0 (PR #1107); v0.11.0 and earlier accept 3.10. Pin to v0.11.0 if the runner is stuck on 3.10. Uses uvloop on Linux/macOS, falls back to default asyncio on Windows — Windows is a first-class port with blocking CI since v0.11.0 (PR #1007), and Windows-on-ARM install/run was added at v0.12.0 (PR #1068).
• Source of truth for flags: aiperf profile --help. CLI options doc is auto-generated via make generate-cli-docs. If the doc disagrees with --help on a flag spelling, trust --help.

Decision tree — which subcommand

The four scheduling modes — pick exactly one

aiperf profile schedules requests in one of four mutually exclusive modes. Picking the wrong one is the single most common source of misleading numbers. See references/timing-modes.md for the full compatibility matrix and validation errors.

A stop condition is required: --request-count, --num-sessions, or --benchmark-duration. Duration mode also reads --benchmark-grace-period (default 30s; user-centric defaults to ∞).

Two methodologies operators actually need

A — Health check / SLO validation

Question: Does this deployment meet the SLO under realistic load?

aiperf profile \
  --model Qwen/Qwen3-0.6B \
  --url http://endpoint:8000 \
  --endpoint-type chat \
  --streaming \
  --tokenizer Qwen/Qwen3-0.6B \
  --input-file prod-trace.jsonl --custom-dataset-type mooncake_trace --fixed-schedule \
  --goodput "time_to_first_token:250 inter_token_latency:10 request_latency:2000" \
  --warmup-duration 30 \
  --benchmark-duration 600 --benchmark-grace-period 60 \
  --artifact-dir artifacts/health-$(date -Iseconds)

The operator reports goodput-req/s and the offending tail percentiles. Goodput=0 with high throughput means the SLO is failing across the board.

B — A/B comparison / before-after

Question: Did config change X help, and at what cost?

Run a concurrency or request-rate sweep, not a single point. The knee of the throughput-vs-tail-latency curve is the usable operating point. A change that moves the knee right is a win.

aiperf profile \
  --model Qwen/Qwen3-0.6B --url http://endpoint:8000 \
  --endpoint-type chat --streaming \
  --tokenizer Qwen/Qwen3-0.6B \
  --concurrency 10,50,100,200,500 --request-count 1000 \
  --isl 1000 --osl 500 \
  --num-dataset-entries 1000 --random-seed 42 \
  --artifact-dir artifacts/pareto
aiperf plot --paths artifacts/pareto --dashboard   # finds every point's run dir recursively

A comma-separated --concurrency (also --request-rate, --isl, --osl; --sweep-type grid|zip) runs the sweep natively and gives each point its own seed (base + index) by default — do not write a shell loop with one fixed seed. To answer "highest throughput that still meets the SLA" directly, use a recipe instead of a grid: --search-recipe max-throughput-ttft-sla --ttft-sla-ms 200 (others: max-goodput-under-slo, concurrency-ramp, pareto-sweep; aiperf plugins search_recipe lists them).

Same seed list, same hardware, same tokenizer for config A and config B. Synthetic prompts are a 100-entry pool by default (`--num-dataset-entries`), reused across requests, and a given `--random-seed` regenerates the identical pool. Against a prefix-caching server (vLLM APC, SGLang radix) a 1000-request run with the default pool, or a second run with the same seed, is served from cache — TTFT and throughput become cache measurements. So: size --num-dataset-entries ≥ requests + warmup, keep a distinct seed per sweep point (the native sweep does; --parameter-sweep-same-seed turns it off), and restart the server (or --reset-kv-cache, v0.13.0+) before switching config. Multi-run confidence (--num-profile-runs 3 --confidence-level 0.95) yields CI bands when each run is short enough that re-running 3× is feasible; its trials share one seed by default, so on a prefix-caching server add --vary-seed-per-trial or trials 2..N measure a warm cache.

Critical pitfalls

1. Tokenizer mismatch. Default tokenizer is whatever HF resolves from --model. If the served model and the HF ID differ (custom path, fine-tune, gated repo), every token-count metric is fiction. Always pass --tokenizer <hf-id> explicitly. For zero-network use --tokenizer builtin (tiktoken o200k_base, GPT-4o-class). Set --tokenizer-trust-remote-code for tokenizers that ship custom Python (Kimi K2.5, some DeepSeek). Pin via --tokenizer-revision <sha> to avoid silent drift on model-card updates.

1. Reasoning-model TTFT/TTFO/OSL semantics. AIPerf's TTFT counts reasoning tokens (reasoning_content SSE field); genai-perf's TTFT did not. To compare apples-to-apples to historical genai-perf numbers, read TTFO (time-to-first-output-token) instead. Same trap on OSL: aiperf OSL = output + reasoning, aiperf output_token_count = output only. Prerequisite: the server must surface reasoning_content as a separate SSE field (vLLM --reasoning-parser deepseek_r1 / qwen3 / gpt_oss etc.; SGLang --reasoning-parser deepseek-r1) — without it, aiperf can't separate streams and TTFO collapses to TTFT, reasoning_token_count stays 0. Verify with curl ... | jq '.choices[0].delta.reasoning_content' before trusting numbers. See references/migration-from-genai-perf.md.

1. `--request-rate` alone, no concurrency cap. The benchmark will queue indefinitely on slow servers and the timing manager will fall behind, producing a curve that looks like a system collapse but is really client-side queue buildup. Always pair with a sane --concurrency ceiling (e.g. 2–3× expected steady-state inflight).

1. `--fixed-schedule` with synthetic data. Fixed-schedule needs timestamped traces (mooncake / bailian / burst_gpt). It is auto-enabled for trace dataset types. Combining it with --request-rate or --user-centric-rate raises a validation error.

1. `--isl-block-size` = the block size the trace was recorded at — never the server's. Loader defaults: 512 mooncake_trace (the FAST25 trace's hash_ids are 512-token blocks), 16 bailian_trace; no effect on baseten_trace. Do not lower it to "match" the server (e.g. 64 for SGLang --page-size 64): aiperf then rebuilds every prompt proportionally shorter (~8×) and the prefix structure breaks — symptom: ISL far below the trace, TTFT too good, near-zero cache hits. Server alignment needs only that the server block size divides 512 (vLLM 16, SGLang 64 do). Confirm what the trace expects with aiperf analyze-trace --input-file trace.jsonl --block-size 512. Runs made with a lowered block size measured a different workload — discard and re-run; never keep them as a baseline.

1. Skipping warmup on cold pods. First N requests hit cold CUDA graphs / torch.compile caches / un-pinned KV pages. Warmup is opt-in: --warmup-duration 30 or --warmup-request-count 100. Multi-run confidence has --profile-run-disable-warmup-after-first to amortize.

1. Ignoring `--use-server-token-count` for OpenAI-compat servers. When set, AIPerf reads token counts from the server's usage field instead of re-tokenizing client-side. Auto-enables stream_options.include_usage for streaming. This avoids client/server tokenizer drift entirely. Use this whenever the server is trustworthy. The --tokenizer flag is still required (for input-shape generation) but tokenizer.encode() is not called for metrics.

1. `--export-level raw` on a long run. Writes every request/response payload to disk. Fine for 1k requests, catastrophic for 1M. Default is records (per-request metrics, no payloads); only escalate when actually debugging. On v0.13.0 a raw export that takes more than ~20 s ends in exit 1 with RequiredServiceReaped although every request completed — the heartbeat watchdog reaps services during export; set AIPERF_SERVICE_HEARTBEAT_MISSED_THRESHOLD=24. Exit 0 does not prove the raw file is complete: a record processor force-killed at shutdown truncates it. Check wc -l profile_export_raw.jsonl against the request count before recomputing metrics from it.

1. Async video generation polling. endpoint-type video_generation polls /v1/videos/{job_id} until the job completes. Tune AIPERF_HTTP_VIDEO_POLL_INTERVAL (default 0.1 s) — faster polling burns server resources, slower polling inflates request latency. With --download-video-content, request latency includes the bytes download.

1. `--goodput` is per-request, not per-percentile. The flag asks "is this individual request below the threshold?" and reports good_request_count / benchmark_duration as goodput. There is no built-in p99_time_to_first_token:400 syntax. To express "p99 TTFT must be ≤ 400 ms", set --goodput "time_to_first_token:400" and check goodput / request_throughput ≥ 0.99 post-run — that ratio is the fraction of requests meeting all SLOs, which for a single threshold is exactly the "X% under the budget" measure operators usually want. Combine multiple thresholds and the same ratio still works as joint compliance.

Done = every row checked before a number leaves the run

What to read next — and when

The upstream repo at https://github.com/ai-dynamo/aiperf is the most authoritative reference — docs/cli-options.md, docs/metrics-reference.md, docs/environment-variables.md, and docs/tutorials/*.md are all auto-generated or hand-curated from the same code. When this skill disagrees with the repo, trust the repo (and update this skill).

Quick recipes

Smoke test a vLLM endpoint

aiperf profile --model Qwen/Qwen3-0.6B --url http://localhost:8000 \
  --endpoint-type chat --streaming --tokenizer Qwen/Qwen3-0.6B \
  --concurrency 10 --request-count 100 --isl 1000 --osl 500

ShareGPT against an OpenAI-compatible server

aiperf profile --model my-model --url http://endpoint:8000 \
  --endpoint-type chat --streaming --tokenizer my-tokenizer \
  --public-dataset sharegpt --num-sessions 200 --concurrency 50

Mooncake trace replay with goodput

curl -O https://raw.githubusercontent.com/kvcache-ai/Mooncake/refs/heads/main/FAST25-release/arxiv-trace/mooncake_trace.jsonl
aiperf profile --model my-model --url http://endpoint:8000 \
  --endpoint-type chat --streaming --tokenizer my-tokenizer \
  --input-file mooncake_trace.jsonl --custom-dataset-type mooncake_trace --fixed-schedule \
  --goodput "time_to_first_token:400 request_latency:2000"

NIM embeddings benchmark

aiperf profile --model nvidia/nv-embedqa-e5-v5 --url http://nim:8000 \
  --endpoint-type nim_embeddings --tokenizer intfloat/e5-large-v2 \
  --concurrency 32 --request-count 5000 --prompt-batch-size 16

Custom multi-turn JSONL with shared system prompt

aiperf profile --model my-model --url http://endpoint:8000 \
  --endpoint-type chat --streaming --tokenizer my-tokenizer \
  --input-file conversations.jsonl --custom-dataset-type multi_turn \
  --shared-system-prompt-length 1000 --user-context-prompt-length 200 \
  --num-sessions 50 --concurrency 25

Goodput per concurrency point with multi-run CI

aiperf profile --model my-model --url http://endpoint:8000 \
  --endpoint-type chat --streaming --tokenizer my-tokenizer \
  --concurrency 100 --request-count 2000 --isl 1000 --osl 500 \
  --num-dataset-entries 2000 --vary-seed-per-trial \
  --num-profile-runs 5 --confidence-level 0.95 \
  --profile-run-cooldown-seconds 30 \
  --goodput "time_to_first_token:300 inter_token_latency:8"

MMLU accuracy benchmark

aiperf profile --model my-model --url http://endpoint:8000 \
  --endpoint-type chat --streaming --tokenizer my-tokenizer \
  --accuracy-benchmark mmlu --accuracy-n-shots 5 --accuracy-grader multiple_choice

For the full flag catalogue, validation rules, and every endpoint/dataset/metric option, drill into the references/ files.