vLLM tiered KV caching

Target audience: operators running vLLM on H100/H200-class datacenter GPUs in production. Assumes CUDA 12/13, Kubernetes or bare container deployment, multi-GPU tensor parallel.

Where this sits. Cache flags are one slice of a serving config: the manifest that carries them (cache mounts, probes, image tags) is `vllm-deployment`, the non-cache throughput knobs are `vllm-performance-tuning`, and the numbers that justify either are `vllm-benchmarking` — all in the vllm plugin. For the same problem on other stacks, see `sglang-hicache` (SGLang's tiering) and `lmcache-mp` (LMCache as a standalone server rather than in-process).

Version gates — check these FIRST

Confirm the vLLM version before recommending any config.

Latest stable: v0.32.0 (2026-10-18). Confirm the deployed tag before recommending a config; per-tag evidence lives in references/sources.md.

Two-step bundling verification (build flag + runtime import)

Build flag = "we tried to install it." Runtime import = "the package actually loads." Different things — the torch-conflict era of mid-2025 had cases where the build flag said yes but import lmcache failed at runtime.

Step 1 — build-flag check (no pull, ~1s):

${CLAUDE_SKILL_DIR}/scripts/inspect-vllm-image.sh <tag>

Prints LMCache/NIXL/Mooncake: YES (built with INSTALL_KV_CONNECTORS=true) if the build layer ran the kv_connectors install.

Step 2 — runtime-import check (pull if needed, ~30s after pull):

~/.claude/skills/lmcache-mp/scripts/verify-bundling.sh <tag>

Starts a sleep-overridden container, exec's a Python probe that confirms lmcache, nixl, mooncake import cleanly; checks the LMCache MP adapter classes and the ParallelStrategy version-hazard symbol; lists all registered KV connectors; loads each connector class through the factory's own thunk. Exit 0 = the connectors are usable on that tag (bundling only; the security floor is separate). The four connectors it loads are the KV-offload set; the factory registers more (16 at v0.29.0), and its list is the only place the rest appear.

lmcache version compatibility: vLLM main imports ParallelStrategy from the lmcache MP adapter. The symbol was added in lmcache 0.4.4 and is absent in 0.4.3 (verified against the v0.4.3 tag), so images built around that era raise ImportError: cannot import name 'ParallelStrategy' when their bundled lmcache is mixed with vLLM main. The v0.29.0 image ships 0.5.4 and the symbol is present — the hazard is now confined to pinning an older image, not to the current one. Fix by pinning the pair or pip install -U lmcache>=0.4.4 inside the container.

Run step 2 on a GPU node. With lmcache 0.5.6 (the v0.32.0 image) the MP adapter classes load libcuda.so.1 at import, so on a host or container without the NVIDIA driver the script prints MP adapter classes FAIL ImportError: libcuda.so.1 and exits 1 although every package imports. Treat that line alone as a probe-environment failure, not a bundling failure.

For other diagnostics (LMCache logs, gdscheck, Prometheus metrics), see references/diagnostics.md.

Decision tree — which backend

Ask these in order:

1. Single node, CPU DRAM tier only, no disk? → --kv-offloading-backend native. Zero extra deps, included since v0.11.1. Start here unless there is a concrete reason to add complexity.
2. Single node, want NVMe (or S3, or a peer's DRAM) as a third tier? → on v0.22.0+, prefer the native multi-tier framework — OffloadingConnector with spec_name: TieringOffloadingSpec and a secondary_tiers list. Registered tier types are fs (local filesystem/NVMe), obj (object store), p2p (peer vLLM instances over NIXL), kvcr (v0.30.0+, see references/connectors.md), and example. Zero extra pods, no second cache implementation to keep in version-sync. LMCache in-process via LMCacheConnectorV1 + LMCACHE_LOCAL_DISK remains the pre-v0.22 answer and is still the right call when a deploy needs LMCache-specific features (CacheBlend, its own controller, cross-engine sharing with SGLang).
3. Multiple vLLM pods on the same node want a SHARED KV cache, OR cache CPU work should not contend with the inference GIL, OR cache memory needs to scale independently of GPU pods? → LMCache MP mode (separate-pod LMCache server, vLLM connects via ZMQ using LMCacheMPConnector). Defer to the `lmcache-mp` skill for the DaemonSet+Deployment pattern, image pair, ZMQ protocol, L2 adapter cascade.
4. Disaggregated prefill across nodes (separate prefill and decode pods)? → NixlConnector via --kv-transfer-config. Tunes TTFT and ITL independently.
5. RDMA-backed KV transfer between nodes, high-throughput datacenter fabric? → MooncakeConnector.
6. NIXL between prefill/decode pods AND local CPU overflow on each? → MultiConnector, composing NixlConnector + OffloadingConnector.
7. Sparse-MLA decode model (e.g. DeepSeek-style), want a host-resident tier for cold KV pages? → `HiSparseConnector`, v0.30.0+, experimental. See references/connectors.md § HiSparseConnector.

Do not reach for LMCache or NIXL or Mooncake just because they exist. Native offload handles the 80% case with zero operational surface area. LMCache MP adds an extra pod and another image — only justified by the multi-pod-shared-cache or GIL-isolation cases above.

For concrete config recipes for any of the above, see `references/connectors.md`. Load that reference once a backend has been selected — it contains copy-paste-ready invocations with all required env vars, GDS host prerequisites (cuda-keyring, open kernel modules, Secure Boot), and the MultiConnector JSON format.

NIXL deep-dive — NIXL itself (transfer library, 13 backend plugins, agent API, telemetry, ETCD/side-channel metadata, plugin authoring) lives in the dedicated `nvidia-nixl` skill. This skill covers vLLM-side wiring of NixlConnector / LMCache-P2P-over-NIXL only. Reach for nvidia-nixl when picking transports (UCX_TLS, GDS, Mooncake, libfabric…), tuning UCX, debugging nixl_agent directly, or writing custom plugins.

LMCache MP deep-dive — the standalone-server multiprocess mode (LMCacheMPConnector, lmcache server, lmcache/standalone:nightly, DaemonSet+Deployment, L2 adapters: nixl_store / fs / mooncake_store / s3) is its own operational shape. The dedicated `lmcache-mp` skill covers it.

Sizing math — READ BEFORE RECOMMENDING A NUMBER

`--kv-offloading-size` is TOTAL across all TP ranks, in GiB. This is the OPPOSITE of SGLang's equivalent flag (which is per-rank). Teams migrating from SGLang routinely under-allocate by a factor of TP.

If an SGLang config was 200 GB per GPU × 8 GPUs = 1600 GB, then on vLLM use --kv-offloading-size 1600. Not 200.

For per-token KV formulas, worked slot-math examples, Dell XE9680 hardware reference numbers, and NVMe-vs-DRAM-vs-prefill load-time comparisons, see `references/hardware-sizing.md`.

If Model Runner V2 was opted into before v0.29.0, KV auto-sizing did not reserve CUDA graph memory. MRV1 always did; MRV2 gained it in v0.29.0 (#53306), explicitly to unblock making MRV2 the default. Since the default flip ships in that same release, a deployment that never set VLLM_USE_V2_MODEL_RUNNER=1 is unaffected. An early opt-in on v0.27.x or v0.28.0 is the exposed case: auto-sizing over-allocated KV against memory the graphs then needed. Suspect it before re-deriving a sizing formula on a box that OOMs only after capture.

`--mamba-cache-mode all` is removed in v0.32.0 (#58997) — it now fails config validation at startup (accepted values: align, none). On v0.30.0–v0.31.x it was deprecated (#55041): it warned and forced MRV1. vLLM sets align itself when prefix caching is on (vllm/model_executor/models/config.py), so only a deployment that explicitly passes all is affected — delete the flag before upgrading to v0.32.0.

When using LMCache as the backend, keep LMCACHE_MAX_LOCAL_CPU_SIZE and --kv-offloading-size consistent. LMCache env vars win on the backend side; inconsistency leads to scheduler miscounting available slots and either over-subscribing or under-using the cache.

Critical pitfalls

--disable-hybrid-kv-cache-manager — required only for connectors that lack HMA support

This changed in v0.23.0 (#41847) and the old advice is now actively harmful. disable_hybrid_kv_cache_manager is tri-state, defaulting to None (auto): vLLM turns HMA off by itself, with a warning, only when the selected connector does not subclass SupportsHMA. Passing the flag by reflex on a modern build costs hybrid-model support and sliding-window performance for nothing.

Which connectors declare SupportsHMA at v0.32.0 (re-verified 2026-10-18 against kv_connector/v1/base.py and the connector classes; HiSparseConnector, MoRIIOConnector and SimpleCPUOffloadConnector also declare it):

On vLLM ≤ v0.22.x the flag was mandatory with OffloadingConnector, and the startup error named it:

ValueError: Connector OffloadingConnector does not support HMA but HMA is enabled.
Please set `--disable-hybrid-kv-cache-manager`.

If a deploy still hits that error on v0.23.0+, the cause is a connector genuinely lacking HMA support, not a missing flag — read the warning line vLLM emits, which names the connector.

Hybrid models (Qwen3.5, Gemma-4, Mamba+attention)

Pick in this order: native offload → native multi-tier (disk/object tier) → LMCache MP → never the in-process LMCacheConnectorV1.

`--prefix-cache-retention-interval` (SWA and SSM models only). From v0.29.0 the default is 0: only the latest replayable prompt boundary is retained (the system prompt is still kept); a positive N retains every N tokens.

• Set it as the CLI flag. VLLM_PREFIX_CACHE_RETENTION_INTERVAL is removed from v0.30.0 and does nothing.
• Hybrids running EAGLE or MTP get dense retention automatically; they need no flag.
• Agentic multi-turn traffic without spec-decode is where 0 bites. If prefix hit rate drops after an upgrade to v0.29.0+, set a positive interval and re-measure before changing anything else.
• From v0.30.0 the interval also gates which SWA/SSM chunks OffloadingConnector offloads (native and multi-tier). It must be a multiple of the SWA group's tokens_per_chunk, or startup raises ValueError.

v0.29.0+ also cuts Mamba prefix-caching TTFT by 9–25% with no flag; count it when weighing an upgrade for a Mamba or hybrid deploy.

Pure-transformer models (Qwen3-14B, Llama-3, Mistral-7B) are unaffected; the standard offload recipes work on any v0.14.0+ tag.

Recheck the hybrid state at a new tag:

gh issue view 3106 --repo LMCache/LMCache --json state,updatedAt   # LMCacheConnectorV1 hybrid block
# which connectors declare HMA support at a given tag:
git -C <vllm-clone> grep -n "SupportsHMA)" v0.32.0 -- vllm/distributed/kv_transfer/kv_connector/v1/

--cpu-offload-gb is NOT the same as --kv-offloading-size

Never recommend --cpu-offload-gb for a KV-tiering question. They are different subsystems with opposite throughput implications.

FP8 KV cache without shipped scales

--calculate-kv-scales is gone from v0.28.0 (#49389); through v0.27.1 it only warned and had no effect. vLLM always loads scales from the checkpoint. On v0.32.0, `--kv-cache-dtype auto` selects `fp8_e4m3` for GLM-5.x (`GlmMoeDsaForCausalLM`) on SM100 (#60140, #60281) — pass --kv-cache-dtype bfloat16 to keep BF16 KV there. If --kv-cache-dtype fp8 is set on a model whose checkpoint doesn't ship calibrated k_scale/v_scale, vLLM defaults to scale=1.0, which can clip pathological activations — especially on long code contexts where specific tokens produce large activations in specific layers.

Symptoms: subtle quality degradation, often only past 128k context. "Usually works fine" is the common operator experience because most activations fit, but the risk is real.

Recommend: stay on BF16 KV and use offload to claw back the memory. The prefill savings from offload hits dwarf whatever would be saved by moving KV from BF16 to FP8.

NVSwitch doesn't accelerate host-to-GPU offload

Operators new to tiered caching sometimes assume NVSwitch/NVLink helps. It doesn't — those interconnects are GPU↔GPU only. KV offload traffic is CPU DRAM → PCIe Gen5 x16 → GPU HBM, one link per GPU. See references/hardware-sizing.md for the bandwidth numbers on XE9680-class hardware.

NVMe tier is NOT a hot path

Loading a 100k-token BF16 context from NVMe takes ~5–7 seconds (Gen4 U.2, buffered) — still beats a ~10 second re-prefill, but DRAM beats NVMe by 20×. Size the DRAM tier to hold the hot working set; treat NVMe as overflow for sessions from "earlier today" that might come back.

Offload "doesn't help" — root causes

Check in order, most common first:

1. Concurrency is already at the memory wall, not the cache. On an 8× H200 with a 300–500 GB weight footprint serving 200k+ contexts, 1–2 live slots is physics — compute jumped ~6× over the previous generation but memory bandwidth only ~1.6×. Adding offload doesn't raise live concurrency; it raises the warm-set size (sessions that skip re-prefill on return). Confirm the expectation is "more warm sessions," not "more simultaneous running."
2. Hit rate is low. No prefix reuse in the workload → offload gains nothing. Check prefix_cache_hits_total / prefix_cache_queries_total. Non-agentic, non-RAG traffic often has low prefix locality.
3. Prefix caching is off. It is on by default (enable_prefix_caching: bool = True in vllm/config/cache.py at v0.32.0); check the serve args for --no-enable-prefix-caching. With it off, offload has nothing to hit against.
4. Flag sized as per-GPU (SGLang habit). Check vllm:cache_config_info — the number reported is what vLLM actually allocated. If it's 1/TP of what was expected, this is the cause.
5. Cold cache. First N requests pay full prefill cost while filling the cache. Measure steady-state, not ramp-up.
6. Max context set higher than needed. If --max-model-len is 200k but typical requests are 8k, the scheduler reserves capacity for 200k; effective concurrency is starved. Tune --max-model-len to the 95th or 99th percentile of real traffic, not the theoretical maximum.
7. Actually compute-bound. Short prompts + many tokens out = decode-dominated. Offload can't help; more GPUs or a smaller model is the fix.

Known bugs to check before recommending offload

States verified 2026-10-18. Before authoring an offload deploy, recheck each row with gh issue view <N> --repo <repo> --json state,stateReason.

Fixed at or below the v0.22.0 security floor: segfault with --decode-context-parallel-size + offload (#40259, v0.21.0), Qwen3.5 native-offload start failure (#36463, v0.21.0), SimpleCPUOffloadScheduler TOCTOU AssertionError: Expected N hit tokens (#39702, v0.22.0). On an older tag, upgrade rather than work around them.

Other long-context knobs worth tuning alongside offload

Offload is necessary but not sufficient for stable long-context serving. Tune these alongside it:

• `--block-size 32` (default 16) — larger KV blocks reduce internal fragmentation at very long contexts. Meaningful win past ~128k; usually neutral or slight loss below 32k.
• `--max-num-batched-tokens <N>` — caps how many prefill tokens vLLM will batch in one step. Without it, a burst of long-prompt arrivals can starve decode and spike tail TTFT. Good starting value: 4096–8192 on H200-class.
• `--load-format fastsafetensors` — direct-mapped safetensors loader. Bundled in NVIDIA Dockerfile from v0.20.0 (#38950); also present in cu130-nightly. `fastsafetensors` is a CLI flag, not an env var — VLLM_USE_FASTSAFETENSOR=1 does NOT exist. On consumer GPUs (no GDS) it auto-falls back to non-GDS mode with a GDS is not supported in this platform warning; loader still ~3× faster than default safetensors path (3.7 s for Qwen3-4B BF16 on RTX 4060 Ti). Pair with HF_HUB_ENABLE_HF_TRANSFER=1 for first-pull download speed.

Do not enable --enforce-eager as a fragmentation workaround — it disables CUDA graphs and hurts steady-state throughput by more than fragmentation costs. Verified live: dropping --enforce-eager on Qwen3-4B (pure transformer, RTX 4060 Ti) cut prefix-cache-hit turn from 2.4 s → 0.64 s (-73 %). Only keep it when the architecture genuinely needs it (hybrid DeltaNet+Attention models like Qwen3.5).

Validating that offload is actually helping

After enabling offload, the canonical workload for stressing the CPU tier is `vllm bench serve --dataset-name prefix_repetition` — it generates N distinct shared prefixes and cycles requests through them, forcing KV eviction once aggregate prefix footprint exceeds GPU capacity. Knobs:

vllm bench serve \
  --backend openai-chat \
  --base-url http://<endpoint> \
  --endpoint /v1/chat/completions \
  --model <served-name> \
  --tokenizer <hf-repo> \
  --dataset-name prefix_repetition \
  --prefix-repetition-prefix-len 8000 \
  --prefix-repetition-suffix-len 200 \
  --prefix-repetition-num-prefixes 16 \
  --prefix-repetition-output-len 64 \
  --num-prompts 128 --max-concurrency 4 --seed 42 \
  --save-result --result-filename /tmp/bench-prefix-repetition.json

Size prefix_len × num_prefixes so aggregate exceeds 2× num_gpu_blocks × block_size. That guarantees evictions during the run. If num_gpu_blocks=1458 and block_size=32 (46.6 K KV slots), 8000 × 16 = 128 K overshoots by ~2.7× — CPU hits start landing within seconds.

But do NOT overshoot the CPU tier — aggregate > CPU capacity thrashes LRU and the hit rate collapses (verified on RTX 4060 Ti: 48 K aggregate fitting a 6 GB tier hit 9.4 %, while 160 K aggregate at 4× capacity dropped to 2.1 % despite moving 2.5× the bytes). Size the bench to the CPU tier's unique-prefix-token budget — the kv_bytes_per_token formula and the worked Qwen3-4B cap (~41 K tokens for 6 GB) are in references/hardware-sizing.md.

After the run, diff metrics:

diff <(curl -s .../metrics-pre.txt | grep vllm:external_prefix_cache) \
     <(curl -s .../metrics-post.txt | grep vllm:external_prefix_cache)

Must see external_prefix_cache_hits_total increase. Also scan the pod log for the bidirectional transfer line:

KV Transfer metrics: GPU_to_CPU_total_bytes=N  GPU_to_CPU_total_time=Ts
                     CPU_to_GPU_total_bytes=M  CPU_to_GPU_total_time=Us

A non-zero CPU_to_GPU_total_bytes = offload tier served a cache hit back to GPU. That's the physical proof the offload path works end-to-end.

Compare two runs on identical traffic with and without --kv-offloading-size set. Look at P50/P99 TTFT and GPU prefix cache hit rate. On agentic workloads with 100k+ contexts, expect TTFT P50 to drop several seconds on returning sessions.

Other built-in workloads relevant to cache:

• --dataset-name random with --random-prefix-len N — simpler baseline
• LMCache repo benchmarks/multi_round_qa/multi-round-qa.py — stateful multi-turn chat, closer to agentic pattern
• LMCache CLI lmcache bench engine --workload long-doc-permutator — 5-axis stress (blended context boundaries, eviction, vocab, prefix domination, concurrency). Requires lmcache CLI on PATH (bundled in v0.14.0+ images but check — on CUDA 13 images the CLI may fail to import c_ops if built against CUDA 12, see LMCache #2843).

Done when — authoring or auditing a KV-cache config

The recipe is finished only when every item holds for the exact tag deployed:

1. Every cache-related flag and env var in the serve args is accounted for: available at that tag (version gates), not removed (--calculate-kv-scales, --mamba-cache-mode all, VLLM_PREFIX_CACHE_RETENTION_INTERVAL), and not the weights flag --cpu-offload-gb standing in for KV offload.
2. --kv-offloading-size is the TP-wide total in GiB; with LMCache, LMCACHE_MAX_LOCAL_CPU_SIZE agrees with it.
3. --disable-hybrid-kv-cache-manager appears only for a connector without SupportsHMA or MoRI-IO WRITE mode on a hybrid.
4. The image tag passed both bundling steps and is ≥ v0.22.0.
5. Every row of "Known bugs" was rechecked and its avoidance applied.
6. A validation run shows external_prefix_cache_hits_total rising and a non-zero CPU_to_GPU_total_bytes.

External references

• vLLM source: vllm/v1/kv_offload/ (native), vllm/distributed/kv_transfer/kv_connector/v1/ (all connectors), requirements/kv_connectors.txt (bundled backend pins)
• docs/features/disagg_prefill.md — overview of the disaggregation connector types (the doc says 9 at v0.32.0)
• LMCache config reference: https://docs.lmcache.ai/api_reference/configurations.html
• NVIDIA GPU Operator GDS: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-rdma.html

See references/sources.md for verification dates and probe notes.