vLLM deployment (Kubernetes first, Docker lab, OpenShift sidebar)

Target audience: platform engineers bringing up vLLM on production Kubernetes (H100/H200/B200/B300 fleets), and individual researchers running 1-to-2-node Docker / Podman setups in a lab.

This skill is a pointer map. It points to the canonical sources — in the vLLM repo, in docs.vllm.ai, in the ecosystem repos, and to the load-bearing blog posts — rather than paraphrasing them. Paraphrase rots; pointers survive.

Decision guide — pick the path

The three load-bearing facts

1. `/dev/shm` is the single most common cause of silent multi-GPU failure on k8s. On vanilla k8s there is no --ipc=host. Without a shared-memory volume, torch.distributed segfaults on the first all-reduce of a TP>1 pod. Mount an emptyDir with medium: Memory and sizeLimit: 10Gi at /dev/shm. Documented in vLLM's own k8s guide — see `vllm repo: docs/deployment/k8s.md:210,290. Since **v0.27.0 (#48879)** a pre-flight free-space check turns the *undersized* case into an actionable startup RuntimeError instead of a later crash — but the trap it replaces is worth knowing, because a 64 MiB /dev/shm is big enough to boot (see references/pod-shape.md`).
2. Multi-node vLLM on k8s is Ray-on-LWS, not headless Service. The parallel-config + pure headless-Service path exists but is not the vLLM-endorsed recipe. Use LeaderWorkerSet (kubernetes-sigs/lws) as the gang-scheduling primitive and examples/ray_serving/multi-node-serving.sh (which bootstraps Ray head/worker) as the entrypoint. Since Nov 2025 the ray symmetric-run pattern replaces the old head/worker split — see https://blog.vllm.ai/2025/11/22/ray-symmetric-run.html.
3. The upstream `vllm/vllm-openai` image runs as root. On OpenShift (and any k8s cluster with restricted-v2 PSA), that is a deploy-time failure. Either rebuild with chgrp -R 0 /root /tmp && chmod -R g=u /root /tmp, or use the Red Hat RHAIIS images (registry.redhat.io/rhaiis/vllm-cuda-rhel9:3.3.0), which are UID-agnostic by construction.

Minimum viable pod shape

# Deployment essentials — not a complete manifest. Full annotated template in references/pod-shape.md.
spec:
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:<pinned-tag>          # do NOT use :latest
        args: ["--model", "$(MODEL)", "--tensor-parallel-size", "8",
               "--disable-access-log-for-endpoints", "/health,/metrics,/ping"]
        env:
        - {name: VLLM_HOST_IP, valueFrom: {fieldRef: {fieldPath: status.podIP}}}
        - {name: HF_HOME, value: /models/.cache}      # pre-warmed PVC or ModelCar
        - {name: VLLM_NO_USAGE_STATS, value: "1"}     # disable telemetry
        - {name: VLLM_DO_NOT_TRACK, value: "1"}
        # Multi-NIC (SR-IOV/RDMA): pin NCCL_SOCKET_IFNAME/NCCL_IB_HCA — see references/pod-shape.md
        ports: [{containerPort: 8000, name: http}]
        readinessProbe: {httpGet: {path: /health, port: http}, periodSeconds: 5,  failureThreshold: 3}
        livenessProbe:  {httpGet: {path: /health, port: http}, periodSeconds: 10, failureThreshold: 3, initialDelaySeconds: 600}
        resources:
          limits: {nvidia.com/gpu: 8}
        volumeMounts:
        - {name: dshm, mountPath: /dev/shm}           # LOAD-BEARING
        - {name: models, mountPath: /models}
      volumes:
      - name: dshm
        emptyDir: {medium: Memory, sizeLimit: 10Gi}   # LOAD-BEARING
      - name: models
        persistentVolumeClaim: {claimName: vllm-model-cache}
      nodeSelector: {nvidia.com/gpu.product: NVIDIA-H200}

The initialDelaySeconds: 600 on the liveness probe is not excessive — cold model loads on a 405B FP8 take 8–12 min. A 30 s default makes the pod liveness-kill before it ever becomes ready. See VLLM_ENGINE_READY_TIMEOUT_S (default 600 s) in `vllm repo: vllm/envs.py`.

Full annotated manifest (all env vars, all probes, PVC vs ModelCar choice, nodeSelector per SM, RuntimeClass for nvidia), plus serve-args review (--enforce-eager trade-off, MoE --enable-expert-parallel/"ep2 dp2" layout checks) and compile-cache survival (VLLM_CACHE_ROOT), in references/pod-shape.md.

Reviewing a manifest — done when every row is marked

For "review / audit this manifest", "cross-check flags", "caches saved?": mark every row pass, fail or n/a, citing the manifest line it rests on. Then run the smoke test below against the live pod for the runtime half.

Sibling skill boundaries

This skill owns the pod/container/topology layer. It does not own:

• Metrics, alerts, SLO, PromQL, Grafana, OTLP, DCGM pairing — that is vllm-observability. This skill points autoscaling at the metric names; vllm-observability owns their semantics and pitfalls.
• KV cache sizing, LMCache nvme/cpu/gds tiers, offloading backend choice — that is vllm-caching. This skill covers which pod topology supports cross-pod KV transfer; vllm-caching covers how to size the tiers.
• Performance tuning, MoE fused-kernel autotune, TP/EP/DP decision trees, async scheduler, CUDA graph modes — that is vllm-performance-tuning. This skill gets the pod running; that skill makes it fast.
• Benchmarking methodology, `vllm bench`, request-rate-vs-concurrency semantics, goodput SLO — that is vllm-benchmarking.
• Env-var and YAML-config semantics, air-gapped HF mirror setup, ModelScope, trust_remote_code — that is vllm-configuration. This skill shows where the env vars go in the pod spec; that one explains what they do.
• NVIDIA hardware SKU selection, HBM/power/NVLink, Blackwell gotchas per SM — that is vllm-nvidia-hardware.

Structure of this skill

• `references/pod-shape.md` — complete annotated Deployment manifest; env vars catalogue; probes; compile-cache survival; serve-args review; parser-plugin ConfigMap mount; nodeSelector per SM generation; PVC vs ModelCar trade-off; image tag discipline; vllm-openai entrypoint contract.
• `references/multi-node.md` — LWS vs KubeRay; ray symmetric-run; NCCL on k8s (shm, SR-IOV, RoCE, InfiniBand, NCCL_SOCKET_IFNAME/NCCL_IB_HCA); the in-repo multi-node-serving.sh/run_cluster.sh; known issue list.
• `references/ecosystem.md` — llm-d, vllm-production-stack, AIBrix, NVIDIA Dynamo, KServe vLLM runtime, vllm-semantic-router, Agent Router (formerly Envoy AI Gateway) — what each one is, current version, when to pick.
• `references/routing.md` — Gateway API Inference Extension (InferencePool, EPP), production-stack router, semantic-router, kgateway/Istio/NGF, OCP Route SSE timeout gotcha.
• `references/autoscaling.md` — KEDA on vllm:num_requests_waiting, cooldown discipline (cooldownPeriod: 360), HPA with custom metrics, scale-to-zero, llm-d KEDA-on-EPP (WVA deprecated).
• `references/disagg.md` — cross-pod PD with NixlConnector, Mooncake, LMCache, MORI-IO; Dynamo's relation to vLLM's own connectors; topology recipes.
• `references/openshift.md` — RHAIIS, RHOAI ServingRuntime templates, SCC, arbitrary UID, Routes 60 s timeout, NVIDIA GPU Operator on OCP, user-workload monitoring, air-gapped (oc-mirror v2, IDMS, ModelCar).
• `references/docker-lab.md` — docker run canonical flags, --shm-size vs --ipc=host, --gpus, MIG strings, Podman/podman-compose, rootless friction, 2-node disagg compose template.
• `references/upgrading.md` — manifest- and rollout-affecting changes from v0.28.0 through v0.32.0; read before bumping the image tag.

The vLLM in-repo deployment artifacts (cheat sheet)

All paths relative to vLLM repo root (https://github.com/vllm-project/vllm).

Operator smoke test — is this pod observable and multi-node-capable?

One-shot smoke test covering all critical checks:

${CLAUDE_SKILL_DIR}/scripts/deployment-smoke.sh <pod-name> [namespace]

The script validates pod health, /health, /v1/models, /dev/shm sizing, /metrics surface, NCCL env on multi-GPU pods, usage-stats opt-out, and image-tag discipline. Output is color-coded pass/warn/fail; exits non-zero on critical failure. If any check fails, the corresponding reference file has a diagnostic flow.

Critical pitfalls (the short list — full treatment in references)

1. No `/dev/shm` emptyDir. Silent NCCL segfault on first all-reduce. See references/pod-shape.md.
2. Default liveness probe. initialDelaySeconds: 30 vs 8–12 min cold load → pod liveness-kill loop. Fix: initialDelaySeconds: 600, or cleaner, a startupProbe with a 15-min budget — both in references/pod-shape.md.
3. `:latest` image tag. Breaks on every vLLM release. Pin to a version tag and roll forward deliberately.
4. Root-UID image on OpenShift. Use RHAIIS images or rebuild. See references/openshift.md.
5. KEDA threshold 1–2 on `num_requests_waiting`. Thrashing. Use 5–10 per replica and cooldownPeriod: 360. See references/autoscaling.md.
6. OCP Route 60 s idle timeout. Kills long SSE streams. Annotate haproxy.router.openshift.io/timeout: 10m.
7. Missing `NCCL_SOCKET_IFNAME` on multi-NIC hosts. NCCL picks the wrong interface and hangs on bootstrap. Pin explicitly.
8. Using the in-tree `chart-helm` (`chart-vllm` 0.0.2) in production. It is an example under examples/, not a supported release artifact. For production, use vllm-production-stack Helm or llm-d Helm.
9. Telemetry to `stats.vllm.ai`. Opt out with VLLM_NO_USAGE_STATS=1 VLLM_DO_NOT_TRACK=1 — especially in regulated/air-gapped environments.
10. Assuming Gateway API is GA on every OCP. It is GA on OCP 4.19+, dev-preview on 4.17. Check the cluster version.
11. Liveness that runs inference, or `failureThreshold: 1`. Refuse it and give the /health probe set — not the requested YAML with caveats. A generation probe (/v1/completions, /v1/chat/completions) queues behind real traffic and times out on a busy pod, so kubelet restarts healthy pods under load, each a full cold boot. Keep liveness on /health: it returns 503 once the engine core is dead (EngineDeadError); the API server exits ~5 s after engine death unless VLLM_KEEP_ALIVE_ON_ENGINE_DEATH=1. Dead engine still Ready → that env is set, readiness is not on /health, or the engine is hung, not dead. Keep readiness (drops the pod from Service endpoints) and liveness (restarts it) separate. Catch hung-but-alive with a /metrics alert (requests running > 0, generation-token rate flat), not a probe.
12. Short `terminationGracePeriodSeconds`, or a long one alone. On SIGTERM vLLM aborts every in-flight request unless --shutdown-timeout N is set (default 0 = abort; N>0 drains up to N s and rejects new requests). Set N to the longest generation served (60–120 s) and the pod grace to N + ~30 s for teardown. Never cut the pod grace to speed eviction: a terminating pod leaves Service endpoints at once, and the grace applies to every rollout, drain and scale-down. For a fast kill on liveness failure only, set livenessProbe.terminationGracePeriodSeconds.

Upgrading the image tag

Current release: v0.32.0 (2026-10-18). Write vllm/vllm-openai:v0.32.0 (or the tag the operator runs) into every command and manifest, never :latest or a <tag> placeholder.

Before bumping the image tag, read references/upgrading.md: the v0.28.0–v0.32.0 changes that alter a manifest or its rollout (CORS closed by default, a removed env var that now silently does nothing, deprecated python -m entrypoints, Model Runner V2 default, a YaRN --max-model-len that must be recomputed).

External references

Canonical entry: https://docs.vllm.ai/en/stable/ (there is no /deployment/ index page — deployment/k8s/ and deployment/docker/ are direct) — topic URLs live in the reference files (references/ecosystem.md, references/multi-node.md, references/openshift.md).

Sibling skills: vllm-observability, vllm-caching, vllm-performance-tuning, vllm-benchmarking, vllm-configuration, vllm-nvidia-hardware, helm, openshift-app.

Also in the vllm plugin: vllm-quantization decides the weight format the manifest's image tag and memory budget have to match (pick it before sizing gpu_memory_utilization, not after), and vllm-gemma-4-31b is a worked operating point for one model — useful as a filled-in example of the flags this skill leaves generic.