vLLM quantization — operator skill

Registries, counts and flag values below are re-derived from the v0.32.0 tree (2026-10-18); verification dates and citations live in references/sources.md.

For production vLLM operators on H100 / H200 / B200 / B300 / GB200 / GB300 fleets deciding which quantization format fits a given target model, producing a checkpoint vLLM will actually load, wiring the right KV-cache dtype, diagnosing accuracy or throughput regressions after an upgrade, and composing quantization with speculative decoding / LoRA / MoE.

Pointer-map format: this SKILL.md picks the format and CLI; the files in references/ hold the per-format deep dives, exact source pointers, and troubleshooting cards. Follow the link, don't paraphrase from memory — the quantization layer moves faster than any other subsystem in vLLM (six formats landed in v0.19 alone).

When quantization wins, when it doesn't

Quantization trades weight precision for memory + compute:

• KV-capacity bound (long context, high concurrency) — FP8 or NVFP4 **KV

cache** gives a 2×/4× KV-capacity multiplier; weight format matters much less than getting --kv-cache-dtype right. Measure kv_cache_usage_perc.

• Memory-bandwidth bound (small batch, decode-heavy, 70B+ on < 8 GPUs) —

weight quantization (NVFP4 / FP8 / W4A16) reduces HBM traffic per token, giving 1.5–3× decode throughput on a well-matched target+kernel.

• Compute bound (prefill, large batch, small model) — quantization may

not help; Blackwell FP4 Tensor Cores are the first architecture where W4A4 actually beats FP8 in compute-bound regimes. On Hopper, W4A16 is memory-only — MMA still runs FP16.

• Multi-node EP / disaggregated serving — NVFP4 reduces all-to-all by 4×

vs BF16. DeepSeek-R1 / V3.2 on GB200/GB300 gets most of its throughput from NVFP4 over the fabric, not from per-GPU compute (see vLLM WideEP blog).

Quantized models are not equivalent to the BF16 original. Always eval on actual traffic. Stock NVFP4 checkpoints recover ~99 % at 70B+, ~95–98 % at 7B–14B (Red Hat / NVIDIA numbers). Code / math / agentic workloads hit harder. For the eval itself use the vllm-benchmarking skill (vLLM's own harness, serving-side accuracy + latency) or aiperf (vendor-neutral, trace replay and accuracy benchmarks against any OpenAI-compatible endpoint).

Format selection — pick once per hardware

Cross-hardware rule of thumb: produce one NVFP4 checkpoint per model. It loads on Blackwell natively and on Hopper via emulation (PR #35733, v0.19). A separate fp8 checkpoint is still worth keeping for older Hopper nodes where the NVFP4 emulation path is slower.

The --quantization flag values (30 at v0.32.0)

Ground truth is the QuantizationMethods = Literal[...] block in vllm/model_executor/layers/quantization/__init__.py (get_quantization_config() dispatches below it) — grep the symbol, not a line range. Full catalog with file paths, min-capability, kernel map, and notes: references/formats.md.

Production formats (keep in head):

Deprecated / legacy / narrow: fbgemm_fp8 and fp_quant are the only two in vLLM's own DEPRECATED_QUANTIZATION_METHODS list. awq / awq_marlin / auto_awq all resolve to AutoAWQConfig and gptq / gptq_marlin / auto_gptq to AutoGPTQConfig: the flag name does not pick the kernel (Marlin on CUDA when the shape allows, else Triton; pin with --linear-backend). Also avoid for new work: experts_int8 (use int8_per_channel_weight_only), moe_wna16, inc (Intel Neural Compressor / AutoRound packing — no longer Intel-only compute: v0.30.0 added CUDA support for AutoRound's 2/3/5/6/7-bit widths via a Humming-kernel fallback, #52890), torchao.

`bitsandbytes` left the tree in v0.28.0 (#43529, listed under Breaking Changes). It is now the out-of-tree vllm-bnb-plugin, so on v0.28.0+ an existing --quantization bitsandbytes deployment needs uv pip install vllm-bnb-plugin rather than a flag change. This is the same shape as gguf moving to vllm-gguf-plugin — check the plugin before assuming the flag still resolves.

`VLLM_TEST_FORCE_FP8_MARLIN` was removed in v0.29.0 (#52182), superseded by --linear-backend / --moe-backend. VLLM_ROCM_USE_AITER_FP4_ASM_GEMM was removed in v0.29.0 (#53141) and restored in v0.31.0, default off (#57055) — set it to 1 to opt back into the AITER W4A4 ASM GEMM. v0.28.0 also removed the deprecated calculate_kv_scales runtime KV-scale calculation (#49389) and override_attention_dtype (#48684).

v0.30.0 removed the `use_fp4_indexer_cache` alias (#55353): set indexer_kv_dtype directly on AttentionConfig for the DSA sparse-attention indexer K cache (values auto / bf16 / fp8 / mxfp4 / nvfp4) — use_fp4_indexer_cache = True used to map to mxfp4. The field is gone from AttentionConfig, not just ignored — a config that still sets it is no longer accepted.

Gone from the in-tree flag list — do not offer them:

• gguf — migrated out-of-tree to vllm-gguf-plugin. pip install vllm-gguf-plugin first, then vllm serve <repo>:Q4_K_M --tokenizer <base-model>. Still experimental; not a datacenter path.
• cpu_awq — folded into `awq_marlin` (PR #43841, merged 2026-05-28). Use awq_marlin on CPU.

Newer values not in older skill copies: humming (Humming mixed-precision, also a --linear-backend / --moe-backend value), deepseek_v4_fp8 (DeepSeek-V4 checkpoint config), fp8_per_channel and nvfp4_per_token (online shorthands; nvfp4_per_token added v0.26.0, #48538).

KV-cache dtypes (19 at v0.32.0)

Single dispatch: vllm/config/cache.py — grep the CacheDType = Literal[...] block, not a line range; it has grown every minor.

• auto — match model weight dtype, except where the model overrides it: DeepSeek-V3.2 and (v0.32.0+, SM100 only) GLM-5.x GlmMoeDsaForCausalLM resolve auto to FP8. float16 / bfloat16 — pin an explicit unquantized KV dtype; set bfloat16 to opt out of those defaults.
• fp8, fp8_e4m3, fp8_e5m2 — the production path. E4M3 is default; E5M2 only for ROCm-specific setups.
• fp8_inc (Intel), fp8_ds_mla (DeepSeek MLA variant), nvfp4_ds_mla (sparse-MLA NVFP4 NoPE + FP8 RoPE, SM100 only).
• ultraquant_4bit — ROCm-only 4-bit KV (FP4 E2M1 + UE8M0 per-32 scales, Hadamard-rotated K), ~2× FP8 density (v0.32.0).
• int4_per_token_head, int8_per_token_head, fp8_per_token_head — dynamic per-(token,head) scales computed in-kernel. No checkpoint scales needed. Added in PR #34281, v0.17.
• turboquant_k8v4, turboquant_4bit_nc, turboquant_k3v4_nc, turboquant_3bit_nc — Hadamard-rotated 2-4 bit KV (v0.19, PR #38479).
• nvfp4 — shipped, not roadmap. #32220 closed COMPLETED 2026-05-04; v0.25.0 landed NVFP4 KV with skip-layers sliding window (#42890). FlashInfer-only: SM100 through TRTLLM; v0.32.0 added SM80/SM90/SM12x through the FA2 path (not at head size 64, not with attention sinks). nvfp4_4over6 (4-over-6 scale selection) stays SM100-only. See references/kv-cache.md.

--calculate-kv-scales was deprecated in v0.19 (PR #37201). Use pre-calibrated scales (LLM Compressor produces them) or let per-token-head scales be computed dynamically.

Producing a checkpoint

vLLM doesn't quantize — a separate tool does, then vLLM loads the result. Two production paths exist:

1. [llm-compressor](https://github.com/vllm-project/llm-compressor) (vLLM-project) — outputs compressed-tensors format. Preferred for the open ecosystem. Covered in references/llm-compressor.md.
2. [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer) — outputs ModelOpt HF format, also consumable by TRT-LLM and SGLang. Preferred for NVFP4 on Blackwell. Covered in references/modelopt.md.

Quick picker:

• FP8 Hopper, no calibration wanted → llm-compressor FP8_DYNAMIC (data-free, ~15 min for a 70B on H100).
• W4A16 INT4 (AWQ or GPTQ) with best accuracy → llm-compressor, AWQModifier or GPTQModifier with 256–512 ultrachat samples.
• NVFP4 on Blackwell → ModelOpt NVFP4_DEFAULT_CFG or llm-compressor NVFP4A16 / NVFP4 scheme (v0.10+; current release 0.14.0, and note the project ships parallel maintenance lines — see references/llm-compressor.md).
• MXFP4 MoE for GPT-OSS-style models → use the vendor checkpoint as-is, or ModelOpt MXFP4.
• KV cache FP8 scales for MLA → llm-compressor kv_cache_scheme block with strategy: tensor (per-tensor is stable; per-head is still the experimental path).

Both tools output a HF directory vLLM serves with --quantization compressed-tensors (llm-compressor) or --quantization modelopt / --quantization modelopt_fp4 (ModelOpt).

Speculative decoding drafters

llm-compressor does not train drafters. ModelOpt does: modelopt/torch/speculative/{eagle,dflash,medusa,plugins}/, examples in examples/speculative_decoding/. Recipes: modelopt_recipes/general/speculative_decoding/{eagle3,dflash}.yaml.

Train the drafter against the BF16 base, never the quantized checkpoint. ModelOpt recipes are BF16-target only; an NVFP4 target is untested (the base runs under torch.no_grad()) — treat it as unsupported. An existing NVFP4/FP8 checkpoint of that same base is kept and served as-is — spec-dec never requires re-quantizing or discarding it. Verdict on "you can't train speculators on a quantized model, so re-quantize everything": the first half is right, the second is wrong. The order is:

1. Train drafter on the BF16 base (ModelOpt, ~4-12h on 8×H100)
2. Export drafter HF dir          (scripts/export_hf_checkpoint.py)
3. PTQ target from that BF16 base (ModelOpt or llm-compressor) — skip if a
   quantized checkpoint of that same base already exists
4. (Optional) PTQ drafter too     (small, minimal accuracy cost)
5. Serve both in vLLM:
   vllm serve <nvfp4-target> --quantization modelopt_fp4 \
     --speculative-config '{"method":"dflash","model":"<drafter-dir>","num_speculative_tokens":7}'
   # EAGLE-3: '{"method":"eagle3","model":"<drafter-dir>"}'

Pick the drafter — state the pick, don't leave it to the user:

Medusa / MTP cannot be trained post-hoc — MTP heads are part of the pretraining (DeepSeek V3, Qwen3-Next, GLM-4.5 MoE, etc.). Full details + exact commands: references/modelopt.md § speculative-decoding.

For spec-dec runtime tuning (acceptance rate metrics, method selection per target family, chunked-prefill composability) use the separate vllm-speculative-decoding skill — don't duplicate here.

Online quantization

Introduced by the v0.14 redesign (PR #37776). Quantizes a BF16 checkpoint at load time, no pre-quantization step. Trade-off: peak load memory is BF16 size.

# Per-tensor FP8 (static scales, simplest)
vllm serve meta-llama/Llama-3.1-70B --quantization fp8_per_tensor

# Per-block FP8 (dynamic per-token activation scales)
vllm serve meta-llama/Llama-3.1-70B --quantization fp8_per_block

# Weight-only INT8
vllm serve meta-llama/Llama-3.1-70B --quantization int8_per_channel_weight_only

# Fine-grained: per-layer-kind override via --quantization-config (JSON or dotted keys)
vllm serve meta-llama/Llama-3.1-70B \
  --quantization fp8_per_tensor \
  --quantization-config '{"moe":{"activation":"mxfp8"},"ignore":["lm_head"]}'

The advanced schema is {linear: {weight, activation}, moe: {weight, activation}, ignore: [...]}; linear / moe also accept a bare shorthand string. Names come from QUANT_KEY_NAMES in vllm/config/quantization.py. There is no --quantization-config-file flag and no global_scheme key.

v0.30.0 added `targets` for per-layer online overrides — a dict of {pattern: shorthand} (exact name, re:-prefixed regex, or fnmatch, mapping to fp8_per_tensor / fp8_per_block / fp8_per_channel / mxfp8 / int8_per_channel_weight_only / nvfp4_per_token), mutually exclusive with linear/moe. A layer matching no pattern keeps its checkpoint dtype (#51285):

vllm serve Qwen/Qwen3.5-35B-A3B --quantization online \
  --quantization-config '{"targets":{"model.layers.0.self_attn.o_proj":"fp8_per_tensor","*mlp.experts*":"mxfp4"}}'

v0.30.0 also let online quantization target the unquantized layers of an already partially pre-quantized checkpoint, regardless of its own quant_method (modelopt, compressed-tensors, quark) — the checkpoint's own method still owns its already-quantized layers (#51392):

# Adds MXFP8 to the dense linear layers of a Quark checkpoint that only
# ships quantized MoE experts.
vllm serve amd/Qwen3.5-35B-A3B-MXFP4 --quantization-config.linear mxfp8

v0.32.0 can re-quantize already-quantized linear layers at load time (dequantize, then apply the online target) — linear only, and only from a serialized ModelOpt MXFP8 source; other pairs raise (#55684): vllm serve <mxfp8-ckpt> --quantization-config.linear fp8_per_channel. Checkpoint exclusions are not honored here — exclude via targets / ignore.

v0.31.0 removed Quark's silent online MXFP4 quantization (#51800): request it through --quantization-config instead.

Known gotchas — #34129 (doesn't split MoE across EP, closed NOT_PLANNED — won't-fix), #19020 / #32029 / #32412 (multiple active RFCs). Dropped bias weights (#39663) is fixed in v0.21.0 (PR #41424) — below it, a bias-ed target needs a pre-quantized checkpoint; at or above it, never name bias loss as the cause of an online-FP8 quality drop. For MoE models, still prefer a pre-quantized checkpoint.

Online FP8 quality drop that switching granularity or `ignore` does not move: granularity is not the cause (FP8 W8A8 on a dense model is near-lossless). Stop tuning online flags. A/B the online run against an offline FP8 checkpoint of the same model, same eval and TP — the publisher's -FP8 build or llm-compressor FP8_DYNAMIC (data-free, minutes) — and serve the offline checkpoint if it recovers.

The operator-pain-point shortlist

Internalize these before debugging accuracy / throughput regressions:

1. `--kv-cache-dtype fp8` on MLA models → garbage on multi-turn (#38652) — fixed in v0.19.0 (PR #37054, KV-scale inconsistency in FP8 MLA). On v0.19.0+ do not exclude FP8 KV for DeepSeek / GLM / Kimi; measure it. See references/kv-cache.md items 1 and 5.
2. Gemma 4 FP8-block → logit saturation / repetitive garbage (#39407, #39049). Use non-block FP8 or FP16. Both issues and the fix PR (#40391) were stale-closed unfixed — closed is not fixed here.
3. NVFP4 on Qwen3-Next / hybrid-attention models silently corrupted output when quantization_config.ignore missed linear_attn layers (#40252, fixed + closed 2026-04-20). The underlying pattern still applies to any new hybrid-attention model: always audit the ignore list when quantizing non-standard architectures.
4. Dynamic FP8 + LoRA-merged model on B200 → non-deterministic degenerate output (#39662). Pin static FP8.
5. SM120 (RTX 5090, 6000 Pro) is not a datacenter NVFP4 MoE target (#35065, #31085) — full kernel set is SM100 / SM103 only. Desktop Blackwell is production only for fp8.
6. CUTLASS NVFP4 / MXFP4 MoE under expert parallelism → silently wrong output before v0.32.0: the CUTLASS FP4 MoE path ignored the EP expert_map (#59042). Upgrade to v0.32.0; on older releases with --enable-expert-parallel, pin a non-CUTLASS --moe-backend.

The hardware-/version-gated traps (B300 TRTLLM hang, ModelOpt-vs-compressed-tensors export drift, MXFP4-linear-falls-back-to-BF16, A100 TurboQuant crash, Qwen3.5 v0.18 KV regression, MXFP8+DeepGEMM pre-v0.19 crash) live in the full triage playbook with symptoms → PR → workaround: references/troubleshooting.md.

Version gates an upgrade acts on

Before any upgrade, read every hop between the installed and target release in references/version-gates.md. Gates not covered elsewhere in this file:

What to read next

• references/formats.md — per-format deep dive: kernels, config JSON shapes, min-capability, known caveats.
• references/llm-compressor.md — recipe cookbook: FP8_DYNAMIC / W4A16 / AWQ / NVFP4A16 / KV-cache FP8 / model-free PTQ commands with exact calibration budgets and output layouts.
• references/modelopt.md — ModelOpt PTQ (hf_ptq.py) + speculative-decoding training (EAGLE-3, dflash, MTP constraints) + vLLM loader compatibility.
• references/kernels.md — kernel × format × SM dispatch map (Marlin / CUTLASS / DeepGEMM / FlashInfer / TRTLLM / Qutlass / Machete / Triton / Exllamav2).
• references/kv-cache.md — KV-cache quantization: dtypes, per-token-head scales, attention-backend compatibility, calibration.
• references/troubleshooting.md — symptom → known-issue → fix playbook.
• references/version-gates.md — release-by-release quantization changes, v0.14 → v0.32.

External references

Load source, not paraphrase:

• vLLM docs: FP8 W8A8, Quantized KV Cache, AMD Quark.
• vLLM recipes: index.
• llm-compressor docs: index, NVFP4 W4A4, Qwen3.5 NVFP4 MoE.
• compressed-tensors spec: quant_scheme.py, overview.
• NVIDIA: Introducing NVFP4, NVFP4 KV cache, MoE perf leaps on Blackwell.
• Red Hat: Accelerating LLMs with NVFP4, LLM Compressor 0.9, vLLM FP8 foundational.
• vLLM blog: GPT-OSS on Blackwell, DeepSeek-R1 WideEP on GB200, DeepSeek-V3.2 on GB300.
• AMD: FP8 with Quark for vLLM, MXFP4 Llama3.3 with Quark.

When in doubt, read the vLLM source — vllm/model_executor/layers/quantization/ is the ground truth, and the quantization layer churns fast enough that cached knowledge rots inside a release cycle.