vLLM

vLLM 0.32.0 introduces a built-in POST /v1/systemone endpoint for structured choice/score queries, a generic hf tool/reasoning parser that reads a model's response_template without needing model-specific parsers, and expands low-precision KV cache support (NVFP4, FP8, 4-bit) across SM8x-SM100 GPUs and ROCm, reducing memory use while increasing throughput. This release drops Python 3.10 support, requires transformers>=5.16.1, removes torch.compile sequence parallelism/AsyncTP passes and cascade attention, restricts default CORS origins, and renames Responses API reasoning stream events, so deployments must upgrade dependencies and update any code relying on the removed or renamed features.

Matters for: Teams self-hosting GLM-5.x, MiniMax-M3, or other large open-weight models across mixed NVIDIA (SM8x-SM100) and ROCm GPU fleets.

This release adds a weight-cache preload daemon and CRIU-based engine snapshots that let vLLM restart serving engines without reloading or reinitializing weights, plus dedicated FlashMLA/DeepGEMM kernel paths, Mega-Gate fusion, and fused decoder boundary operations optimized for DeepSeek-V4.1-Flash on SM100/SM103 GPUs. The release also removes tokenizer_mode="slow" and the AllSpark INT8 W8A16 backend, renames --enable-mamba-fine-grained-prefix-cache to --enable-mamba-shared-prefix-checkpoint, replaces quantization="fp8" with the fp8_per_tensor shorthand, and now requires --trust-request-mm-kwargs to accept per-request multimodal kwargs, so deployment configs and client requests using these options must be updated.

Matters for: Relevant to teams running large-scale multi-GPU inference clusters that need fast engine restarts, and to those serving DeepSeek-V4.1-Flash or other MoE models on NVIDIA SM100/SM103 hardware.

vLLM now supports new sparse-MLA models with GPU weight caching for near-instant engine restarts (Fast Start), a host-memory tier for sparse-MLA decode KV pages (HiSparse), watermarked generation compatible with speculative decoding, and numerous throughput improvements for models like Kimi K3 and Qwen3.8-Flash-Next. This release breaks compatibility: scale-out endpoints now require the explicit --enable-scale-out flag instead of the VLLM_ENABLE_SCALE_OUT_ENDPOINTS env var, GPTQ g_idx activation ordering is removed, several 0.29-deprecated env vars and the 'all' Mamba cache mode are removed, the gRPC entrypoint module is deprecated in favor of vllm serve --grpc, and YaRN scaling behavior changed to align with Transformers.

Matters for: Teams self-hosting large sparse-MLA models like DeepSeek-V4 and Kimi K3 at multi-node, multi-GPU scale who need fast engine restarts and disaggregated KV caching.

vLLM 0.29.0 makes Model Runner V2 the default execution path for all models, adding CUDA graph memory profiling, batch-sharded sampling, prompt embeds, and speculative decoding support, while adding new models like Hy4-preview, Qwen3.8-Flash-Next, and Kimi K3, plus performance work for Kimi-K3 and DeepSeek V4 inference; Model Runner V1 is now deprecated with removal targeted for v0.32.Ten deprecated model architectures were removed, FlexOlmo/Olmo3/Hunyuan V1/VL moved to the Transformers backend, the PyAV video decoder backend was removed, python -m vllm.entrypoints.openai.api_server is deprecated in favor of vllm serve, and two environment variables (VLLM_TEST_FORCE_FP8_MARLIN, VLLM_ROCM_USE_AITER_FP4_ASM_GEMM) were removed, so users relying on these must migrate to their replacements before upgrading.

Matters for: Teams self-hosting large MoE models like DeepSeek V4 or Kimi K3 across multi-GPU or multi-node clusters using tensor/expert parallelism.