vllm-performance-tuning — independently scanned and version-tracked by SaferSkills.
SaferSkills independently audited vllm-performance-tuning (Agent Skill) and scored it 100/100 (green). The audit ran 55 deterministic rules across Security, Supply Chain, Maintenance, Transparency, and Community; it found 0 high-severity and 0 lower-severity findings. The full rule-by-rule trace and per-finding evidence are below. Free, methodology-open.
Findings & checks · 0 flagged
Every scanned point with the score it earned and what moved between them.
First recorded scan — no prior version to compare against.
The primary manifest — the file an agent reads to learn what this artifact does.
Target: operators deploying models on new hardware, chasing throughput / latency / goodput SLOs, or diagnosing perf regressions. Current through v0.21.0 stable (2026-05-15); v0.20.x stable since 2026-04-27. Last freshened 2026-05-28.
Companion skills: vllm-benchmarking (measure), vllm-caching (KV), vllm-nvidia-hardware (GPU/GEMM), vllm-configuration (env vars), vllm-observability (metrics).
Always first — characterize the workload. ISL / OSL / req/s / concurrency / SLO (P95 TTFT, P95 TPOT, P95 ITL). "Goodput" = tok/s/GPU under SLO, not raw tok/s. Everything below is keyed off these numbers.
Parallelism + MoE kernels (biggest single wins):
references/moe-and-ep.md) — model-fits-1-GPU → TP=1 + replicas (DP); MoE MLA (DeepSeek/Kimi-K2) → DP-attn + EP; multi-node → TP intra + PP inter OR Wide-EP.E=*,N=*,device_name=*.json configs. Without tuned configs vLLM logs "Using default MoE config. Performance might be sub-optimal!" = 20-40% throughput loss.--enable-expert-parallel --enable-eplb --enable-dbo) for DeepSeek/Qwen3/Kimi-K2 at ≥16 GPUs.Throughput / batching:
benchmarks/auto_tune/) sweeps max_num_seqs × max_num_batched_tokens.--max-num-batched-tokens (default 2048 since PR #10544) if TTFT > SLO; lower if ITL > SLO.Latency / graph + compile:
FULL_AND_PIECEWISE (default); align --cuda-graph-sizes with max_num_seqs*2.$VLLM_CACHE_ROOT/torch_compile_cache on a representative pod; mount as PVC / bake into OCI layer.Distributed (last resort, only when the lever above is exhausted):
NCCL_IB_HCA, NCCL_IB_GID_INDEX, NCCL_NET_GDR_LEVEL. Never NCCL_CUMEM_ENABLE=0 on GB200.From Red Hat's 5-step triage (2026-03-09):
| Symptom | Look at | Common cause |
|---|---|---|
| TTFT high, queue empty | compute-bound prefill | chunked-prefill budget too low, no prefix cache, bad parallelism |
| TTFT high, queue growing | capacity | raise replicas, raise max_num_seqs, check preemption rate |
| TPOT high, TTFT fine | decode-bound | MoE kernel not tuned, wrong attention backend, async sched off |
| ITL spikes | CUDA-graph miss | batch sizes fall outside captured buckets |
| Preemptions climbing | KV thrashing | raise --swap-space, lower --max-num-seqs, or add replicas |
num_running < configured concurrency | scheduler stall | check async-sched blockers, multimodal path, structured output |
DCGM signals (not GPU_UTIL): DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE. Low tensor-core active on a GEMM-bound workload = memory-bound.
MoE tuning + expert parallelism + DeepEP + EPLB + parallelism decision matrix → references/moe-and-ep.md
Scheduler knobs + CUDA graphs + torch.compile + compile cache → references/scheduler-and-compile.md
NCCL / InfiniBand / DCGM + PD disaggregation (Nixl/Mooncake/LMCache) → references/distributed.md
Known regressions + vendor quirks (AMD / Ascend / XPU) → references/regressions.md
Full citation anchors → references/sources.md
Using default MoE config. Performance might be sub-optimal! in logs. Fix: python benchmarks/kernels/benchmark_moe.py --model <moe-model> --tp-size <N> --enable-expert-parallel --tune --save-dir ./configs, then export VLLM_TUNED_CONFIG_FOLDER=./configs. Expected: 20-40% throughput recovery.num_heads % TP == 0 AND hidden_size % TP == 0.DP=8, EP=8, TP=1. (DeepSeek-V3.2 recipe)(experts_per_token / total_experts) > 2%.d83f3f7 and 5a84b76 the DeepGEMM MoE M<128 restriction was removed; H200 DeepSeek-R1 EP at concurrency ≤8 regressed 1.5× TTFT. [#28882](https://github.com/vllm-project/vllm/issues/28882) closed 2026-04-21 — upgrade to v0.19.1+ or current main. For pre-v0.19.1 deployments, the workaround was VLLM_MOE_USE_DEEP_GEMM=0 + FlashInfer FP8 for low-concurrency decode; re-benchmark after upgrading before removing the override.$VLLM_CACHE_ROOT/torch_compile_cache on one pod, mount as PVC / OCI layer. Llama-4 specifically needs VLLM_DISABLE_COMPILE_CACHE=1 — stale-cache bug.NCCL_NET_GDR_C2C=1. PR #16992 fixed vLLM's defaults.# Step 1 — run the tuner (uses Ray to parallelize across local GPUs)
python benchmarks/kernels/benchmark_moe.py \
--model deepseek-ai/DeepSeek-V3 \
--tp-size 8 --enable-expert-parallel --dtype fp8_w8a8 \
--tune --save-dir ./moe_configs
# Step 2 — point vLLM at the configs
export VLLM_TUNED_CONFIG_FOLDER=./moe_configs
# Step 3 — serve + verify no "default MoE config" warning in logs
vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8 \
--enable-expert-parallel --enable-eplb --enable-dbo \
--gpu-memory-utilization 0.85| Model family | Small-scale | Large-scale (≥16 GPUs) |
|---|---|---|
| Dense (Llama, Qwen3-dense) | TP=N, DP=replicas | TP=8 intra-node + PP=nodes OR TP=8 + DP=N |
| MoE non-MLA (Mixtral, Qwen3-MoE) | TP=N, EP off | TP + EP: EP = E / TP |
| MoE MLA (DeepSeek-V3/R1, Kimi-K2) | DP=N + EP | Wide-EP: DP-attn + EP-MoE, --enable-eplb |
| Llama-4-Maverick (0.78% density) | TP only | TP only (EP hurts) |
| DeepSeek-V3.2 (FlashMLA-Sparse) | DP=8, EP=8, TP=1 | same |
Concurrency crossover (8× MI300X benchmarks): ≤128 concurrent → TP wins, ≥512 → DP wins, 256-512 mixed. (AMD MoE playbook)
| Scenario | max_num_batched_tokens | max_num_seqs | Other |
|---|---|---|---|
| Throughput-heavy (batch decode) | 4096-16384 | 256-512 | async sched on |
| Latency-heavy (chat) | 1024-2048 | 64-128 | async sched on, --stream-interval 1 |
| Long-context RAG | 8192-16384 | 32-64 | --enable-prefix-caching, --long-prefill-token-threshold |
| Wide-EP DeepSeek | 8192 | 256 | --enable-expert-parallel --enable-eplb --enable-dbo, FULL_AND_PIECEWISE |
| Flag | Effect |
|---|---|
-O0 | No compile, no CUDA graphs (= --enforce-eager) |
-O1 | Simple compile + PIECEWISE graphs |
| `-O2` | default — full compile + FULL_AND_PIECEWISE + fusions (AllReduce+RMSNorm +15%, SP+Async-TP +10%, Attention+Quant FP8 +7%) |
-O3 | reserved (currently = -O2) |
| Metric | Value |
|---|---|
Default max_num_batched_tokens (since PR #10544) | 2048 (was 512) |
Default max_num_seqs | 256 |
| Default CUDA-graph sizes | [1,2,4] + range(8,256,8) + range(256,max,16), cap min(max_num_seqs*2, 512) |
| H200 Wide-EP DeepSeek-R1 throughput | 2.2k tok/s/GPU vs ~1.5k baseline (vllm.ai/blog/large-scale-serving) |
| GB200 Wide-EP DeepSeek-R1 | 26.2K TPGS prefill, 10.1K TPGS decode, 3-5× H200 (vllm.ai/blog/dsr1-gb200-part1) |
| MLPerf v5.1 Blackwell Ultra | 5,842 tok/s/GPU offline, 2,907 server (NVIDIA blog) |
| DeepEP dispatch (FP8, Azure H100 IB400) | 45.9 GB/s RDMA, 149.8 GB/s NVLink (Azure blog) |
| Activation density cutoff for EP win | > 2% (below: TP wins) |
All claims cite file:line, release-note PR refs, or issue IDs. Full anchor list + vendor-specific sources in references/sources.md. Compiled 2026-04-18 against v0.19.0; freshened 2026-05-28 against v0.21.0 stable (#28882 closed 2026-04-21, #27679 closed 2025-12-29, #31679 closed 2026-01-07, #34641 closed 2026-05-28, ascend #4649 closed 2026-03-13). Next refresh when v0.22.x stable ships or when remaining key regressions (#35048, #31475) close.
~30 seconds. Free. No account. Every finding cites a rule and a line of evidence.