vLLM chat templates — operator triage

Target audience: operators deploying vLLM in production. Assumes OpenAI-API-compatible frontend (/v1/chat/completions or /v1/responses), multi-GPU, mid-2024 through 2026 model families.

Scope — three sibling skills, three layers

Chat-template bugs span three layers. This skill owns the prompt-rendering layer; route to a sibling when the problem is output-side.

When in doubt: if the operator complains about what the server receives, it's this skill; if about what the client receives, it's one of the parser skills. Template and parser bugs often present the same symptom ("reasoning_content is null"), so all three skills name each other in diagnostics. This file stays on the Jinja side.

Why this matters

Chat templates are the Jinja layer between structured messages and the raw prompt string the model sees. Tool-calling and reasoning models are extremely sensitive to this layer — a one-character whitespace difference in a template can send tool-call success rate from 99% to 0%, silently.

The hardest class of bug in vLLM deployments is template silent failure: the server starts, requests return 200s, but the model emits garbled tool calls, or reasoning_content is null, or enable_thinking=false has no effect. No exception, no log line, just wrong outputs. Operators often chase parser bugs when the real fix is a template override (or vice versa — override a template when the real fix is a parser flag).

This skill's job: given a symptom, identify whether it's a template, a parser, a kwarg, or an API-endpoint issue, and point at the known fix.

Template resolution precedence (memorize this)

vLLM picks a chat template in this order. First hit wins. Source: vllm/renderers/hf.py:96-145 (resolve_chat_template()).

1. `--chat-template <path-or-name>` CLI flag — highest. Path to a .jinja file, literal Jinja string, or a named template ("tool_use") that the tokenizer knows.
2. AutoProcessor chat template — only consulted when the request has tools=None. Multimodal models often expose their template via the processor, not the tokenizer. Source: hf.py:110-116.
3. AutoTokenizer `chat_template` — the one shipped in tokenizer_config.json. Source: hf.py:119-126.
4. Bundled fallback in `vllm/transformers_utils/chat_templates/` — vLLM ships defaults for ~10 vision/OCR model families (blip-2, chameleon, clip, colpali, deepseek_ocr/ocr2/vl_v2, fuyu, minicpmv, paligemma, qwen, siglip/siglip2). Source: registry.py:64-75.

If none resolves, `ChatTemplateResolutionError` raised at hf.py:477 with: "As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one."

Consequences operators miss

• Passing `--chat-template` does NOT override the tokenizer's template if the path is invalid — vLLM raises "The supplied chat template ... appears path-like, but doesn't exist!" (chat_utils.py:1105-1109). Check ls first.
• Tool-calling disables the AutoProcessor step. If a multimodal model has a processor template but the request has tools, vLLM skips it and falls through to tokenizer. This is intentional — tool templates must be explicit.
• `--tokenizer-mode mistral` silently ignores `--chat-template`. Warning only; see Mistral section in references/model-families.md. Issue vllm-project/vllm#25401 — closed as COMPLETED 2025-10-09. Behavior is at least louder in current vLLM; verify on your version before assuming silent-drop. Verified 2026-04-24.
• Some templates (e.g. gpt-oss) appear hard-coded — operator-supplied --chat-template has no effect (vllm-project/vllm#23015, closed NOT_PLANNED 2026-04-20). Intentional: gpt-oss goes through the Harmony path via /v1/responses; --chat-template is not the intended escape hatch. Verified 2026-04-24.

Triage: symptom → suspect

Use this table first. Load references/model-families.md for the specific model bug + flags.

Template, parser (output), and kwarg (chat_template_kwargs) are orthogonal layers — diagnose one at a time, don't change multiple at once.

The flag matrix (common case)

Per-family flag matrix lives in references/flags-matrix.md. Quick cheat-sheet for the production-critical ones:

Commit-gated models (Kimi-K2) require minimum HF revision — check references/model-families.md before recommending.

Top 15 patterns that silently break production

Consult this list for any "broken chat template" symptom. Most reports reduce to one of these.

1. Chat template silently ignored. Mistral in --tokenizer-mode mistral (#25401), reportedly gpt-oss (#23015). Warning-only. Symptom: edit template, nothing changes. Fix: switch to HF tokenizer mode, or accept that the built-in Mistral template is canonical.

1. `skip_special_tokens=true` strips control tokens before reasoning parser. Gemma-4 (#38855), Qwen3 reasoning, gpt-oss harmony. Fix: extra_body={"skip_special_tokens": false} per request.

1. `<think>` opening tag prepended to prompt → never appears in output → `reasoning_content=null`. DeepSeek-R1 (HF #144, vLLM #12999). Template forces thinking by inserting <think>\n in the prefill; parser sees only </think>. Fix: post-process prepend <think>, or use patched template.

1. Empty string `''` wrapped to `[{'type':'text','text':''}]`. Broke Kimi-K2 templates expecting strings. Any template that doesn't type-check message.content is at risk. Fix: updated chat template or vLLM version.

1. Tool-call ID regex mismatch. Mistral requires 9 digits; Kimi-K2 required functions.X:idx exactly. vLLM generates longer IDs → IndexError → all tool calls silently dropped. Fix: Kimi-K2 fixed in PR #27565; Mistral template truncates to last 9 digits.

1. `add_generation_prompt` dropped by kwarg allowlist. PR #25794 restricted **kwargs forwarding for security; tokenizers relying on non-standard params broke. Fix: PR #27622 whitelists standard HF params — requires vLLM ≥ v0.11.1.

1. Jinja template from disk triggers `_try_extract_ast()` exception. #14884: \n escape sequences double-converted when shell-interpolated. Fix: pass the file path, not the contents; don't shell-escape.

1. `tool_calls=[]` in historical assistant message crashes template. #13978 (Llama). Templates assume nonempty if present. vLLM now drops empty tool_calls (chat_utils.py:1586-1589).

1. `tool_choice="auto"` vs `"required"` divergence. auto = free-form generation + regex parse (schema-free, brittle). required/named = structured-outputs FSM (schema-valid). Many "tool calling is broken" reports vanish when switching to required.

1. Multi-turn whitespace accumulation. DeepSeek V3.1 (#28804, closed NOT_PLANNED 2026-03-23) — leading whitespace grows each turn. Look for this when assistant replies get progressively indented. Upstream not planning a fix; apply client-side mitigation (strip leading whitespace on each turn) or use deepseek_v31 parser commits post-issue report. Verified 2026-04-24.

1. `reasoning_effort="none"` silently breaks output. gpt-oss #37909. Fix: omit or use "low".

1. Concurrent requests → `<pad>` tokens. Gemma-4 #39392. Not reproducible sequentially. Fix: version bump; interim workaround --max-num-seqs 1.

1. `--chat-template-content-format auto` misroutes tool results. GLM-5.1 #39614 (CLOSED/COMPLETED 2026-04-25): tool result as {"type":"text","text":"..."} hits an "else" branch in a template checking .name → renders <tools></tools> instead of result. Fixed upstream; the --chat-template-content-format openai/string workaround is only needed on vLLM before the fix.

1. `/v1/chat/completions` vs `/v1/responses` divergence. gpt-oss tool calling works only on /v1/responses (#22578). GLM-5.1 (#39611, CLOSED/COMPLETED 2026-04-12) — tool results were ignored on chat-completions but work on completions; fixed upstream, so on patched vLLM tool results now render on /v1/chat/completions.

1. API field name mismatch. reasoning (vLLM older) vs reasoning_content (DeepSeek/OpenAI-style). Clients that hard-code one break on version bump. vLLM settled on reasoning_content (#28472).

What chat_template_kwargs accepts

Threaded as extra_body={"chat_template_kwargs": {...}} in OpenAI client. Source: vllm/renderers/params.py:70-125, hf.py:407-435.

• Any Jinja variable referenced in the resolved template — detected via AST parse at hf.py:429.
• HF `apply_chat_template` params auto-detected from tokenizer signature (hf.py:387-404).
• add_generation_prompt (bool)
• continue_final_message (bool) — mutually exclusive with add_generation_prompt=True
• documents (list of {title, contents}) for RAG templates
• enable_thinking (Qwen3 and some others)
• Reserved (will raise `ValueError`): chat_template, tokenize (hf.py:416-421).
• Unknown kwargs silently filtered (hf.py:434-435) — if enable_thinking doesn't appear in the detected Jinja vars, it's dropped without warning. This is a confusing failure mode: operator sets it, sees no effect, can't find an error.

Merging: server defaults (serving.py:111) ← request kwargs (params.py:28-40) — request wins.

Reasoning parser vs chat template (they're confused a lot)

• Chat template (this skill) controls the prompt — where <think> tags go going in to the model, and how tool definitions get rendered.
• Reasoning parser (vllm-reasoning-parsers) extracts <think> content from output into reasoning_content. Template-side causes of null reasoning_content are pattern #3 above (opening <think> prepended in prefill); all other root causes live in the parser skill.
• Tool-call parser (vllm-tool-parsers) extracts tool_calls[] from output. Template-side causes of empty tool_calls are wrong tool template or tool_choice="auto" vs "required" (pattern #9); parser-selection and streaming state-machine debugging live in that skill.

Raw-output capture (--disable-log-requests=False + no --tool-call-parser/--reasoning-parser) determines which layer is at fault: if the model emits the expected tags/JSON, the prompt is fine and the problem is parser-side; if not, the template is misrendering.

Multimodal template notes

Models use model-specific placeholders (<image>, <|image_start|>, etc.) — vLLM's internal placeholder is <##IMAGE##> / <##AUDIO##> / <##VIDEO##> (chat_utils.py:96-100).

• If request has images but template has no placeholder, vLLM prepends them to final prompt (chat_utils.py:1233-1240). Some models tolerate this; others produce garbage.
• If template has more placeholders than media items: ValueError: "Found more '<image>' placeholders in input prompt than actual multimodal data items." (chat_utils.py:1226-1229).
• interleave_strings=True (via multimodal_config.interleave_mm_strings) enables in-order substitution; default is prepend-all.

For Pixtral, Mistral-Small-3.1 multimodal, and other MM-tool-calling combinations, see references/model-families.md.

When nothing works — full debugging workflow

Load references/debugging.md. The short version:

1. Log the actual prompt. --disable-log-requests=False + log level DEBUG. Compare against model card example. If they differ, template is wrong.
2. Bypass vLLM. Load the same tokenizer in a Python REPL, call tokenizer.apply_chat_template(messages, add_generation_prompt=True), compare. If same bad output, template is wrong. If different, vLLM's kwarg forwarding is eating something.
3. Capture raw model output. Set --tool-call-parser to none, read what the model actually emits. Then work backward to pick the right parser.
4. Pin versions. Record vllm --version, transformers.__version__, model revision SHA. Many template bugs track a specific tokenizer_config.json commit — mention in bug reports.

Code locations (for reading vLLM source)

• vllm/entrypoints/chat_utils.py — message parsing, multimodal placeholder injection, ChatTemplateResolutionError.
• vllm/renderers/hf.py:96-145 — resolve_chat_template() precedence.
• vllm/renderers/hf.py:460-505 — safe_apply_chat_template() + error surface.
• vllm/renderers/hf.py:407-435 — resolve_chat_template_kwargs() allowlist.
• vllm/renderers/params.py:70-125 — ChatParams dataclass.
• vllm/reasoning/__init__.py:22-103 — registered reasoning parsers.
• vllm/transformers_utils/chat_templates/registry.py:64-75 — bundled fallback lookup.
• vllm/entrypoints/openai/serving_chat.py:91-182 — parser instantiation at server init.
• examples/tool_chat_template_*.jinja (27 files) — ready-made tool templates per model family.

Per-family deep-dive

When the operator names a model family, load references/model-families.md — it has per-family flag recipes, minimum commit SHAs, known bugs with issue numbers, and recovery strategies.

Further reference material

• references/model-families.md — Qwen3, DeepSeek, GPT-OSS, Kimi-K2, Llama-4, Mistral, Gemma, Phi-4, GLM, Hermes/Granite/Hunyuan — with inline issue URLs.
• references/debugging.md — stepwise debugging workflow + full error-message catalog.
• references/flags-matrix.md — quick CLI-flag lookup per model.
• references/sources.md — verification log for external URLs/issues/PRs cited in this skill, with Last verified dates.

Last verified: 2026-05-28 (see references/sources.md for per-ref status).