vLLM — embeddings, reranking, speech-to-text, OCR

Embedding and reranking run on the pooling runner; STT and OCR run on the generate runner.

The mental model — one flag rules the surface

vLLM decides what a model does from the combination of three flags:

--runner {auto|generate|pooling|draft}      # what kind of workload
--convert {auto|none|embed|classify}        # adapt a generative LM to a pooler
--pooler-config '{...}'                     # override pool type, dimensions, etc.

The pair (runner, convert) replaced --task, which no longer parses (pitfall 1). Canonical today:

Scoring API is automatic. There is no --enable-scoring-api. The /score + /rerank endpoints light up whenever the loaded model's Pooler.get_supported_tasks() includes classify (with num_labels==1), embed, or token_embed (late-interaction). Nothing for the operator to toggle.

Quick-answer router

scripts/probe-endpoint.sh checks a running vLLM whether it exposes /v1/embeddings, /rerank, /v1/audio/transcriptions, etc., so an operator can confirm the right endpoints are live before pointing a client at it.

Operator cheat sheet — the common cases inline

Embedding

# Qwen3-Embedding (causal LM, last-token pooling — auto-detected)
vllm serve Qwen/Qwen3-Embedding-0.6B --runner pooling

# BGE-M3 (XLM-Roberta, CLS pooling — native embedding model)
vllm serve BAAI/bge-m3 --runner pooling

# Jina v3 (needs trust-remote-code; only text-matching LoRA is merged)
vllm serve jinaai/jina-embeddings-v3 --runner pooling --trust-remote-code

# Jina v4 — use the pre-merged retrieval variant
vllm serve jinaai/jina-embeddings-v4-vllm-retrieval --runner pooling \
  --pooler-config '{"pooling_type":"ALL"}' --dtype float16
# Normalization happens client-side (vector is multi-vector per token).

# Mean-pool override (Sentence-Transformers config is broken for this model)
vllm serve ssmits/Qwen2-7B-Instruct-embed-base --runner pooling \
  --pooler-config '{"pooling_type":"MEAN"}'

Client request format is standard OpenAI: client.embeddings.create(...).

Matryoshka dimensions. Gated on is_matryoshka: true in the model's config.json (or matryoshka_dimensions). If the config is missing it, force-enable:

--hf-overrides '{"is_matryoshka": true}'
# or pin specific dimensions:
--hf-overrides '{"matryoshka_dimensions":[256,512,768]}'

Request-side: client.embeddings.create(model=..., input=..., dimensions=512). Passing dimensions to a non-MRL model (BGE-M3, older BGE) returns a 400 by design — not a bug.

`dimensions` above the model's `hidden_size` is rejected as of v0.24.0 (#46313). Before that, an MRL model with no explicit matryoshka_dimensions list validated only dimensions >= 1 and then sliced [..., :d] — so an oversized value silently returned a `hidden_size`-length vector instead of erroring. If a client has been over-asking, it was never getting the width it requested; upgrading turns that into a visible 400. Pin the list explicitly via --hf-overrides to make the valid set unambiguous.

`/v2/embed` (Cohere v2 compat) adds input_type prompt prefixing, output_dimension (server-side MRL), truncate=END|START|NONE, and embedding_types=["float","binary","ubinary","base64"]. Use it when a client expects Cohere v2's shape.

Reranking / scoring

Three serving modes; same endpoints, picked automatically:

# Cross-encoder (classify, num_labels==1)
vllm serve BAAI/bge-reranker-v2-m3 --runner pooling

# Cross-encoder with instruction-aware score template
vllm serve Qwen/Qwen3-Reranker-0.6B --runner pooling --convert classify \
  --chat-template examples/pooling/score/template/qwen3_reranker.jinja \
  --hf-overrides '{"architectures":["Qwen3ForSequenceClassification"],
                    "classifier_from_token":["no","yes"],
                    "is_original_qwen3_reranker":true}'

# Late-interaction (ColBERT family) — MaxSim over token embeddings
vllm serve jinaai/jina-colbert-v2 --runner pooling --trust-remote-code

# Multimodal reranker (ColPali / ColQwen)
vllm serve vidore/colpali-v1.3-hf --runner pooling

Client:

# /rerank (Cohere + Jina compat)
resp = requests.post("http://localhost:8000/rerank", json={
    "query": "what is vLLM",
    "documents": ["text 1", "text 2"],
    "top_n": 3,
    "max_tokens_per_doc": 512,  # added v0.20.0 (PR #38827)
})

# /score (bi-encoder cosine, or cross-encoder logit)
resp = requests.post("http://localhost:8000/score", json={
    "text_1": ["query"],
    "text_2": ["doc A", "doc B"],
})

Three score_types served through the same routes:

jinaai/jina-reranker-v3 is listwise ("last but not late interaction") — JinaForRanking, not MaxSim.

Speech-to-text

# Whisper large-v3-turbo (base)
vllm serve openai/whisper-large-v3-turbo

# Red Hat production quants (fit on smaller cards, validated)
vllm serve RedHatAI/whisper-large-v3-turbo-FP8-dynamic
vllm serve RedHatAI/whisper-large-v3-turbo-quantized.w8a8
vllm serve RedHatAI/whisper-large-v3-turbo-quantized.w4a16

# Voxtral (Mistral)
vllm serve mistralai/Voxtral-Mini-3B-2507

Client:

curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -F "[email protected]" \
  -F "model=openai/whisper-large-v3-turbo" \
  -F "language=en"

Chunking >30 s audio is server-side (energy-aware split at min_energy_split_window_size). Beam-search transcription arrived in v0.18.

Audio decode backend (v0.30.0+). Selectable via --media-io-kwargs '{"audio": {"audio_backend": "torchcodec"}}' — soundfile, pyav, or torchcodec. auto (default) tries torchcodec first from v0.32.0 (#58364), then soundfile, then pyav; on v0.30.x–v0.31.x it was soundfile → torchcodec → pyav (#51826, #55642). Only a missing backend (or a container soundfile cannot read) falls through — other decode errors raise. Default resampler switched from PyAV to torchaudio in the same release (#52598); torchaudio ships in the standard CUDA/ROCm/CPU/XPU requirement files, no extra install. Details: references/stt.md §6.

OOM on 24 GB with Whisper (issue #15216) is a known sharp edge — Whisper allocates aggressively for its encoder KV, despite the 1.6 GB checkpoint. Production path is one of the RedHatAI quants above, or raising --gpu-memory-utilization past 0.9 with eager mode if memory is truly tight; unquantized large-v3 / large-v3-turbo wants ≥32 GB for comfortable batch sizes.

OCR (DeepSeek-OCR)

Canonical recipe from docs.vllm.ai/projects/recipes:

vllm serve deepseek-ai/DeepSeek-OCR \
  --logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
  --no-enable-prefix-caching \
  --mm-processor-cache-gb 0

Three non-obvious flags:

• NGramPerReqLogitsProcessor is required — without it, table-token

generation degrades. Send whitelist_token_ids={128821,128822}, ngram_size=30, window_size=90 per request in extra_args — the processor reads them from there (references/ocr.md §3).

• Disable prefix caching. OCR per-request inputs don't share prefixes;

the cache bookkeeping is pure overhead.

• `--mm-processor-cache-gb 0` — the multimodal processor cache isn't

useful for one-off document images.

DeepSeek reports ~2500 tok/s per A100-40 GB, ~200 k pages/day per GPU. Mode is hard-coded to GUNDAM (base=1024, image=640, crop=True); Tiny/Small/Base/ Large are not selectable (still hard-coded at v0.32.0).

Invocation is still plain /v1/chat/completions with image URLs — there is no dedicated /ocr endpoint.

Top pitfalls

1. `--task embed` is dead, not deprecated. No --task in

vllm/engine/arg_utils.py and no task field in ModelConfig at v0.32.0 — passing it is an unrecognized-argument error. Use --runner pooling. The score and encode pooling task names are removed too and raise VLLMValidationError (vllm/tasks.py): use --convert classify on a num_labels==1 model to light up /score + /rerank, and token_embed / token_classify in place of encode.

1. Pooling runs on PIECEWISE CUDA graphs, not full graphs. That's

deliberate (pooling models have variable-shape outputs). Don't force --enforce-eager for production as older cheat sheets suggest — you lose the piecewise graph win without gaining anything.

1. Qwen3-Reranker needs a score template AND hf-overrides. It's an

instruction-tuned causal LM masquerading as a cross-encoder — skipping any of the three extras (see the reranking cheat-sheet command above) gives random-looking scores, not errors. Full recipe: references/reranking.md §2. Second cause of the same symptom, below v0.26.0: #48901 — any LAST-pooling model returns wrong scores once a query+doc pair is long enough to be chunk-prefilled, under torch.compile (the default). Rule out the version before re-checking the overrides. v0.30.0+ warns at startup if the original Qwen3 reranker (is_original_qwen3_reranker=true) is served without a chat template — check the logs before assuming silent wrong scores (#56017).

1. Late-interaction kernel regression sniff test. ColBERT / ColPali

throughput jumped ~14% in v0.17–0.19 from MaxSim optimisations. If those models feel slow, check --enable-flash-late-interaction (default true) wasn't disabled by an old config. From v0.32.0 the same flag also gates the engine-side flash-maxsim Triton kernel (#40337).

Version gates — check the running version first

Each row changes what a request returns or whether it is accepted. Release history and the PRs behind each row: references/sources.md.

Paired skills

• vllm-configuration → environment variables, cache paths, telemetry opt-out.
• vllm-observability → metrics exposition, Prometheus endpoints.
• vllm-nvidia-hardware → SM-level platform support for pooling + FP8 paths.

Sources

First-party docs: <https://docs.vllm.ai/en/stable/models/pooling_models/>. Every citation, its verification date and the refresh triggers: references/sources.md.