{"url":"https://api.github.com/repos/sgl-project/sglang/releases/401467591","assets_url":"https://api.github.com/repos/sgl-project/sglang/releases/401467591/assets","upload_url":"https://uploads.github.com/repos/sgl-project/sglang/releases/401467591/assets{?name,label}","html_url":"https://github.com/sgl-project/sglang/releases/tag/v0.5.21","id":401467591,"author":{"login":"Fridge003","id":56809903,"node_id":"MDQ6VXNlcjU2ODA5OTAz","avatar_url":"https://avatars.githubusercontent.com/u/56809903?v=4","gravatar_id":"","url":"https://api.github.com/users/Fridge003","html_url":"https://github.com/Fridge003","followers_url":"https://api.github.com/users/Fridge003/followers","following_url":"https://api.github.com/users/Fridge003/following{/other_user}","gists_url":"https://api.github.com/users/Fridge003/gists{/gist_id}","starred_url":"https://api.github.com/users/Fridge003/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Fridge003/subscriptions","organizations_url":"https://api.github.com/users/Fridge003/orgs","repos_url":"https://api.github.com/users/Fridge003/repos","events_url":"https://api.github.com/users/Fridge003/events{/privacy}","received_events_url":"https://api.github.com/users/Fridge003/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOLCAjRs4X7ejH","tag_name":"v0.5.21","target_commitish":"main","name":"v0.5.21","draft":false,"immutable":false,"prerelease":false,"created_at":"2026-10-01T06:22:41Z","updated_at":"2026-10-02T19:42:55Z","published_at":"2026-10-02T01:09:04Z","assets":[],"tarball_url":"https://api.github.com/repos/sgl-project/sglang/tarball/v0.5.21","zipball_url":"https://api.github.com/repos/sgl-project/sglang/zipball/v0.5.21","body":"# Highlights\r\n\r\n*779 PRs from 227 contributors.*\r\n\r\n## New models\r\n\r\n| Model | Type | Cookbook |\r\n|---|---|---|\r\n| DeepSeek-V4.1 Flash | LLM / VLM | [link](https://docs.sglang.io/cookbook/autoregressive/DeepSeek/DeepSeek-V4_1) |\r\n| GigaChat 3.5 | LLM / VLM | [link](https://docs.sglang.io/cookbook/autoregressive/GigaChat/GigaChat3.5) |\r\n| IQuest-Q1 | LLM / VLM | [link](https://docs.sglang.io/cookbook/autoregressive/IQuestLab/IQuest-Q1) |\r\n| MiMo-V2.6 / MiMo-V2.6-Pro | LLM / VLM | [link](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.6) |\r\n| Ling-3.0-flash-VL | LLM / VLM | [link](https://docs.sglang.io/cookbook/autoregressive/InclusionAI/Ling-3.0-flash-VL) |\r\n| DiffusionGemma | Diffusion | [link](https://docs.sglang.io/cookbook/autoregressive/Google/DiffusionGemma) |\r\n| Qwen-Image 2.1 | Diffusion | [link](https://docs.sglang.io/cookbook/diffusion/Qwen-Image/Qwen-Image-2.1) |\r\n| Anima Base v1.0 | Diffusion | [link](https://docs.sglang.io/cookbook/diffusion/CircleStone/Anima) |\r\n| Ming-Image 0.1 Design / Design-Layer | Diffusion | [link](https://docs.sglang.io/cookbook/diffusion/inclusionAI/Ming-Image) |\r\n| FLUX 3 Action | Diffusion | [link](https://docs.sglang.io/cookbook/vla/FLUX/FLUX-3-Action) |\r\n\r\n## Key features\r\n\r\n* PD instances can switch between prefill and decode on the fly, no restart needed ([#28403](https://github.com/sgl-project/sglang/pull/28403)).\r\n* The prefix cache now runs on a Rust core by default ([#39627](https://github.com/sgl-project/sglang/pull/39627)).\r\n* DeepSeek-V4.1 gets 22% faster first token on long prompts ([#40352](https://github.com/sgl-project/sglang/pull/40352)).\r\n* Kimi K3 gets 20.6% higher prefill throughput in PD serving ([#40045](https://github.com/sgl-project/sglang/pull/40045)).\r\n* GLM-5.3-Flash now runs on AMD MI355X, with FP8 / MXFP4 MoE and MTP speculative decoding ([#38546](https://github.com/sgl-project/sglang/pull/38546), [#39341](https://github.com/sgl-project/sglang/pull/39341), [#39778](https://github.com/sgl-project/sglang/pull/39778)).\r\n* More accurate results under PP, DP attention, and CP, with layer communication now handled by SGLang ([guide](https://docs.sglang.io/docs/developer_guide/layer_boundary), [#41557](https://github.com/sgl-project/sglang/pull/41557)).\r\n* New Decisions API (`/v1/decisions`) turns an LLM / VLM into a low-latency classifier and scorer ([docs](https://docs.sglang.io/docs/supported-models/decision_models), [#41208](https://github.com/sgl-project/sglang/pull/41208)).\r\n* Score API (`/v1/score`) scores all candidates in one request ([#38965](https://github.com/sgl-project/sglang/pull/38965), [#41188](https://github.com/sgl-project/sglang/pull/41188)).\r\n* Run MiniMax-H3 with SGLang Diffusion inside ComfyUI ([#35990](https://github.com/sgl-project/sglang/pull/35990)).\r\n\r\n## To upgrade\r\n\r\n```bash\r\nuv pip install --prerelease=allow sglang==0.5.21\r\n```\r\n\r\n| Platform | Docker |\r\n|---|---|\r\n| NVIDIA (CUDA 13) | [`docker pull lmsysorg/sglang:v0.5.21`](https://hub.docker.com/r/lmsysorg/sglang/tags?name=v0.5.21) |\r\n| AMD MI35x | [`docker pull lmsysorg/sglang:v0.5.21-rocm10-mi35x`](https://hub.docker.com/r/lmsysorg/sglang/tags?name=v0.5.21-rocm10-mi35x) |\r\n| AMD MI30x | [`docker pull lmsysorg/sglang:v0.5.21-rocm10-mi30x`](https://hub.docker.com/r/lmsysorg/sglang/tags?name=v0.5.21-rocm10-mi30x) |\r\n| Intel GPU | [`docker pull lmsysorg/sglang:v0.5.21-xpu`](https://hub.docker.com/r/lmsysorg/sglang/tags?name=v0.5.21-xpu) |\r\n| Intel CPU | [`docker pull lmsysorg/sglang:v0.5.21-xeon`](https://hub.docker.com/r/lmsysorg/sglang/tags?name=v0.5.21-xeon) |\r\n\r\n# Full Release Notes\r\n\r\n## Speculative Decoding\r\n\r\n* Pipeline parallelism x speculative decoding (EAGLE/MTP) compatibility: [#30775](https://github.com/sgl-project/sglang/pull/30775)\r\n* Support XQA backend for SpecDec verify: [#32269](https://github.com/sgl-project/sglang/pull/32269)\r\n* [Spec] Windowed draft-decode attention for built-in EAGLE / MTP drafts: [#32673](https://github.com/sgl-project/sglang/pull/32673)\r\n* Avoid materializing GDN QKV tensors during target verification: [#33778](https://github.com/sgl-project/sglang/pull/33778)\r\n* [Spec] Fix CDF boundary handling in `TreeSpeculativeSamplingTargetOnly`: [#35798](https://github.com/sgl-project/sglang/pull/35798)\r\n* [Spec] Add LiLiCorr: a candidate-lattice reranker for DFlash drafts: [#37462](https://github.com/sgl-project/sglang/pull/37462)\r\n* Add out-of-tree DFlash extension points: [#38740](https://github.com/sgl-project/sglang/pull/38740)\r\n* [Spec] Add explicit prefill shared-read capability for plugins: [#39502](https://github.com/sgl-project/sglang/pull/39502)\r\n* [Fix] Don't write conv state from the fused KDA verify kernel: [#39524](https://github.com/sgl-project/sglang/pull/39524)\r\n* [Spec] Model-agnostic last-stage draft embedding under pipeline parallelism: [#39643](https://github.com/sgl-project/sglang/pull/39643)\r\n* Use runtime token widths for Triton speculative verification: [#39859](https://github.com/sgl-project/sglang/pull/39859)\r\n* avoid host sync in DSpark prefill slot expansion: [#40111](https://github.com/sgl-project/sglang/pull/40111)\r\n* [Experimental] Preserve speculative decoding during prefill across DP ranks: [#40118](https://github.com/sgl-project/sglang/pull/40118)\r\n* [Spec][PP] Launch extend microbatches before the spec output exchange: [#40499](https://github.com/sgl-project/sglang/pull/40499)\r\n* [KDA] Enable ReplaySSM for GLM-5.3 Flash: [#40517](https://github.com/sgl-project/sglang/pull/40517)\r\n* [LFM2-VL] Add DSpark speculative decoding (1.66x to 2.56x speedup at batch 1): [#40651](https://github.com/sgl-project/sglang/pull/40651)\r\n* [Spec] Support DFLASH for Kimi K3: [#40794](https://github.com/sgl-project/sglang/pull/40794)\r\n* [Fix] Reduce Nemotron MTP attention outputs once: [#40800](https://github.com/sgl-project/sglang/pull/40800)\r\n* [Fix] Capture complete Nemotron auxiliary hidden states: [#40801](https://github.com/sgl-project/sglang/pull/40801)\r\n* [Fix] Skip the DCP target-verify MLA kernel during FlashInfer autotune: [#41138](https://github.com/sgl-project/sglang/pull/41138)\r\n* Fix mixed chunk prefill with DP speculative coordination: [#41179](https://github.com/sgl-project/sglang/pull/41179)\r\n* [Fix] Plan NextN / MTP draft layers as one-layer models and fix the Bailing V2 NextN draft: [#41194](https://github.com/sgl-project/sglang/pull/41194)\r\n\r\n## Piecewise & Breakable CUDA Graph\r\n\r\n* [Fix] Preserve model runner contracts in prefill CUDA graphs: [#35452](https://github.com/sgl-project/sglang/pull/35452)\r\n* [DSV4][BCG] Optimize the heavy memory use of C4 Indexer when BCG is enabled (capture memory down 58% to 18 GB): [#36534](https://github.com/sgl-project/sglang/pull/36534)\r\n* [Fix] Don't free the multi-CTAs KV counter the decode graphs captured: [#39175](https://github.com/sgl-project/sglang/pull/39175)\r\n* [CPU] Avoid prefill CP predicates during decode graph capture: [#39690](https://github.com/sgl-project/sglang/pull/39690)\r\n* [Runtime] Add decode CUDA graph hooks for eager logits processing: [#40222](https://github.com/sgl-project/sglang/pull/40222)\r\n* [Kimi K3] Fix CUDA graph stream explosion: [#40640](https://github.com/sgl-project/sglang/pull/40640)\r\n* [DSpark] Fix draft CUDA graph stream explosion: [#40658](https://github.com/sgl-project/sglang/pull/40658)\r\n* [Fix] Give the full prefill CUDA graph replay view the captured bucket's input_ids: [#40851](https://github.com/sgl-project/sglang/pull/40851)\r\n* [DSA] Fix the pooled-indexer breakable prefill bridge under DP attention: [#41311](https://github.com/sgl-project/sglang/pull/41311)\r\n\r\n## Attention Backends\r\n\r\n* Allow attention layers to opt out of the prefill wrapper: [#40683](https://github.com/sgl-project/sglang/pull/40683)\r\n* [Fix] Use cached prefix lengths for FlashInfer full-attention ragged prefill: [#40796](https://github.com/sgl-project/sglang/pull/40796)\r\n\r\n## MoE & Expert Parallelism\r\n\r\n* [3/N] elastic-ep: Recapture decode CUDA graphs after scale-up: [#33723](https://github.com/sgl-project/sglang/pull/33723)\r\n* Fix int4 MoE tuner config filename: [#35260](https://github.com/sgl-project/sglang/pull/35260)\r\n* [MegaMoE] Wire Qwen MoE blocks to DeepGEMM MegaMoE (MXFP4 and NVFP4 experts): [#38080](https://github.com/sgl-project/sglang/pull/38080)\r\n* [Feature] Support BF16 and batch-invariant inference with DeepEP v2: [#38160](https://github.com/sgl-project/sglang/pull/38160)\r\n* [Perf] Optimize w4a8 MoE for glm5.2 on H200: [#38220](https://github.com/sgl-project/sglang/pull/38220)\r\n* [Fix] Guard FlashInfer CUTLASS MoE against 0-token inputs: [#38780](https://github.com/sgl-project/sglang/pull/38780)\r\n* [Kernel] Add H20 block-FP8 MoE configs for GLM-5.3-Flash EP4/EP8: [#38913](https://github.com/sgl-project/sglang/pull/38913)\r\n* Accept MXFP8 dispatch in FlashInfer A2A TRT-LLM MoE: [#39613](https://github.com/sgl-project/sglang/pull/39613)\r\n* [MoE] Honor swiglu_limit clamped activation in flashinfer_cutlass runner: [#39939](https://github.com/sgl-project/sglang/pull/39939)\r\n* Support MXFP8 and deferred route weighting in DeepEP v2: [#40030](https://github.com/sgl-project/sglang/pull/40030)\r\n* [MoE] Disable FlashInfer fused finalize by default for numerical accuracy: [#40105](https://github.com/sgl-project/sglang/pull/40105)\r\n* [Fix] Fix top-1 MoE routing with non-unit scaling: [#40187](https://github.com/sgl-project/sglang/pull/40187)\r\n* [DeepEP v2] support GLM-5.3-Flash (Glm5NextForConditionalGeneration): [#40466](https://github.com/sgl-project/sglang/pull/40466)\r\n* [Fix] Decide the MoE padded-row bound from the layer scatter mode: [#40672](https://github.com/sgl-project/sglang/pull/40672)\r\n* [DeepEP v2] Let a model package supply its per-rank prefill dispatch bound: [#41201](https://github.com/sgl-project/sglang/pull/41201)\r\n\r\n## Quantization\r\n\r\n* [MXFP8-KV] Skip writes to the reserved CUDA-graph padding slot: [#35351](https://github.com/sgl-project/sglang/pull/35351)\r\n* feat(kv-cache): support SM100 NVFP4 GenMHA and speculative decoding (1.355x faster decode attention kernel at 1M context): [#36340](https://github.com/sgl-project/sglang/pull/36340)\r\n* [Quant] ModelOpt mixed precision: dispatch block-FP8 MoE experts and derive the block size: [#38726](https://github.com/sgl-project/sglang/pull/38726)\r\n* fix(modelopt): dispatch NVFP4 MoE on the cached backend, not the live global: [#38932](https://github.com/sgl-project/sglang/pull/38932)\r\n* [Quant] Serve 32-wide-K ue8m0 block-FP8 linears through the FlashInfer MXFP8 GEMMs: [#40039](https://github.com/sgl-project/sglang/pull/40039)\r\n* [ModelOpt][PP] Keep BF16 shared experts out of the NVFP4 fusion so TP1 pipeline stages can load: [#40628](https://github.com/sgl-project/sglang/pull/40628)\r\n\r\n## Parallelism & Disaggregation\r\n\r\n* [PD] Introduce runtime role switching between prefill and decode: [#28403](https://github.com/sgl-project/sglang/pull/28403)\r\n* disaggregation: Fix FakeKVSender queue accumulation: [#28652](https://github.com/sgl-project/sglang/pull/28652)\r\n* [PD] Keep EAGLE DP graph and token metadata consistent: [#32196](https://github.com/sgl-project/sglang/pull/32196)\r\n* [NVIDIA][comm] Merge EP+MoE-TP post-experts all-reduces into one _TP reduction: [#32963](https://github.com/sgl-project/sglang/pull/32963)\r\n* Add 8-node AllReduce/AllGather and MNVLS algorithm support to MSCCL++: [#37442](https://github.com/sgl-project/sglang/pull/37442)\r\n* [KV-Shard 2/4] Sharded pools: [#37615](https://github.com/sgl-project/sglang/pull/37615)\r\n* [KV-Shard 3/4] Enable Control plane: [#38468](https://github.com/sgl-project/sglang/pull/38468)\r\n* [KV-Shard 3/4] Enable Control Plane B: [#39964](https://github.com/sgl-project/sglang/pull/39964)\r\n* Fix device context during NIXL backend initialization: [#38774](https://github.com/sgl-project/sglang/pull/38774)\r\n* [PD] Do not admit intake-rejected requests to a PD handoff: [#38935](https://github.com/sgl-project/sglang/pull/38935)\r\n* Reduce decode bootstrap latency with request-owned speculative KV (mean TTFT 19.4 to 15.1 s on DeepSeek-V4-Pro 1P1D, MI355X, concurrency 256): [#38978](https://github.com/sgl-project/sglang/pull/38978)\r\n* Fix disagg PP MTP for GLM-5.2: [#39378](https://github.com/sgl-project/sglang/pull/39378)\r\n* [PD] Add optional KV transfer checksums: [#39500](https://github.com/sgl-project/sglang/pull/39500)\r\n* [DCP] Use logical token capacity for PD admission and load reporting: [#39731](https://github.com/sgl-project/sglang/pull/39731)\r\n* Refactor the Cute-DSL AR fusion to support DeepseekV2 archs (GLM-5.3, etc.): [#39816](https://github.com/sgl-project/sglang/pull/39816)\r\n* [PD] Validate Mooncake EFA allocator compatibility: [#39973](https://github.com/sgl-project/sglang/pull/39973)\r\n* [PD] Skip singleton transfer-status all-reduces: [#40003](https://github.com/sgl-project/sglang/pull/40003)\r\n* [PD] Enable optimistic prefill with buffer-only L3 write-through HiCache: [#40043](https://github.com/sgl-project/sglang/pull/40043)\r\n* [Kimi K3][PD] support pp prefill + dcp decode with dspark: [#40045](https://github.com/sgl-project/sglang/pull/40045)\r\n* [PD] Add decode host receive for custom transfer backends: [#40238](https://github.com/sgl-project/sglang/pull/40238)\r\n* [PD] Allow decode radix cache and HiCache L1/L2 with DCP: [#40263](https://github.com/sgl-project/sglang/pull/40263)\r\n* [PD] Enter the custom mem pool once when allocating DCP pack buffers: [#40284](https://github.com/sgl-project/sglang/pull/40284)\r\n* [PD] Bound cached-prefix DCP transfers by pack capacity (cached-prefix throughput about 10.9x higher at concurrency 8 on Kimi Linear, 8x B200): [#40376](https://github.com/sgl-project/sglang/pull/40376)\r\n* [PD] Pack draft KV head slices for DCP transfers: [#40500](https://github.com/sgl-project/sglang/pull/40500)\r\n* [PD] Preserve abort ACKs until in-flight KV transfers drain: [#40645](https://github.com/sgl-project/sglang/pull/40645)\r\n* Fix NIXL transfer of MXFP8 KV block scales: [#40792](https://github.com/sgl-project/sglang/pull/40792)\r\n* [PD] Honor gracefully_exit in disaggregation event loops and keep non-zero-rank launchers alive on SIGTERM: [#40793](https://github.com/sgl-project/sglang/pull/40793)\r\n* [DP Attention] Publish DP buffer sizes from a ForwardBatch: [#40858](https://github.com/sgl-project/sglang/pull/40858)\r\n* [PD] Enable deferred decode-side KV release by default: [#41023](https://github.com/sgl-project/sglang/pull/41023)\r\n* [PD] Add a `none` decode retraction backup and subclass seams in the PD queues: [#41103](https://github.com/sgl-project/sglang/pull/41103)\r\n* [PD] fix: cache resumed decode-radix requests from root instead of an unlocked re-match: [#41261](https://github.com/sgl-project/sglang/pull/41261)\r\n* [PD] Fan drain abort ACKs out to every decode peer of the room: [#41402](https://github.com/sgl-project/sglang/pull/41402)\r\n* [PD] Defer decode KV release on every transfer failure, not only decode-initiated aborts: [#41404](https://github.com/sgl-project/sglang/pull/41404)\r\n* [Fix] Stop deferring the last layer's FFN all-reduce in five models: [#40868](https://github.com/sgl-project/sglang/pull/40868)\r\n* [Fix] Complete the deferred FFN all-reduce before a pipeline-parallel send: [#41079](https://github.com/sgl-project/sglang/pull/41079)\r\n* [Fix] Complete the deferred FFN all-reduce before deepstack addition and aux hidden-state capture: [#41080](https://github.com/sgl-project/sglang/pull/41080)\r\n* [Fix] Step-3.5: stop dense layers from summing their output twice under DP attention: [#41082](https://github.com/sgl-project/sglang/pull/41082)\r\n* [Fix] Broadcast requests along attention CP before attention TP: [#41083](https://github.com/sgl-project/sglang/pull/41083)\r\n* [Fix] Complete the all-reduce when the flashinfer fused norm declines a batch: [#41193](https://github.com/sgl-project/sglang/pull/41193)\r\n* [Fix] Stop counting a deferred FFN sum more than once: replicated TP1 shared expert, dense reduce_scatterv: [#41195](https://github.com/sgl-project/sglang/pull/41195)\r\n* [Fix] Keep one copy of CP-replicated rows in the DP gather: [#41422](https://github.com/sgl-project/sglang/pull/41422)\r\n* [Fix] Gather a dense FFN's input across attention DP and CP in one DP sum: [#41423](https://github.com/sgl-project/sglang/pull/41423)\r\n* [Fix] Run MoE layers under attention DP and GQA prefill CP on the declared DP × CP gather: [#41432](https://github.com/sgl-project/sglang/pull/41432)\r\n* [Fix] Falcon-H1: count the Mamba mixer's output once under tensor parallelism: [#41433](https://github.com/sgl-project/sglang/pull/41433)\r\n* [Fix] LongCat-Flash under attention DP: branch and merge the dense FFNs through the communicators: [#41436](https://github.com/sgl-project/sglang/pull/41436)\r\n* [Refactor] Group communicator fusion and CP adapters: [#41547](https://github.com/sgl-project/sglang/pull/41547)\r\n* [Refactor] Centralize decoder output access: [#41548](https://github.com/sgl-project/sglang/pull/41548)\r\n* [Refactor] Carry residual state across stage boundaries: [#41549](https://github.com/sgl-project/sglang/pull/41549)\r\n* [Refactor] Capture auxiliary states at residual reads: [#41550](https://github.com/sgl-project/sglang/pull/41550)\r\n* [Refactor] Select reduction fusion at the consumer: [#41551](https://github.com/sgl-project/sglang/pull/41551)\r\n* [Refactor] Construct independent decoder stage boundaries: [#41552](https://github.com/sgl-project/sglang/pull/41552)\r\n* [Refactor] Migrate specialized decoder and overlap boundaries: [#41553](https://github.com/sgl-project/sglang/pull/41553)\r\n* [Refactor] Retire the layer facade and simplify boundary internals: [#41554](https://github.com/sgl-project/sglang/pull/41554)\r\n* [Refactor] Rename the module to layer_boundary: [#41555](https://github.com/sgl-project/sglang/pull/41555)\r\n* [Refactor] Group layer boundary unit tests: [#41556](https://github.com/sgl-project/sglang/pull/41556)\r\n* [Refactor] Document layer boundary contracts and integration: [#41557](https://github.com/sgl-project/sglang/pull/41557)\r\n\r\n## Scheduler & Runtime\r\n\r\n* feat: support custom OTLP trace service name: [#35802](https://github.com/sgl-project/sglang/pull/35802)\r\n* [RL] Release the weight-checker snapshot once compare passes: [#37284](https://github.com/sgl-project/sglang/pull/37284)\r\n* [Runtime] Let out-of-tree platforms provide full graph backends: [#37969](https://github.com/sgl-project/sglang/pull/37969)\r\n* Fix: post-load staging regression breaks offload meta/sharded_gpu modes: [#38779](https://github.com/sgl-project/sglang/pull/38779)\r\n* feat: add kv hint envelope to request transport: [#38891](https://github.com/sgl-project/sglang/pull/38891)\r\n* [Observability] Fix negative queue_time for retracted requests: [#39312](https://github.com/sgl-project/sglang/pull/39312)\r\n* Fix first-token metadata and reused attention-layer indexing: [#39328](https://github.com/sgl-project/sglang/pull/39328)\r\n* perf(engine): avoid timed waits for Engine responses (+37.7% output throughput in a Dynamo TCP push benchmark at the default batch_notify_size): [#39486](https://github.com/sgl-project/sglang/pull/39486)\r\n* [Misc] Fix tool-call index, graph padded-row count, and prefill-graph input_embeds refresh: [#39574](https://github.com/sgl-project/sglang/pull/39574)\r\n* [Misc] Merge FlashInfer autotune caches across spec workers, pad MXFP4 TP shards, drop dead ngram attrs: [#39678](https://github.com/sgl-project/sglang/pull/39678)\r\n* [Observability] Expose python/rust frontend identity in `/server_info`: [#39993](https://github.com/sgl-project/sglang/pull/39993)\r\n* [Metrics] Propagate idle gaps across all scheduler loops: [#40004](https://github.com/sgl-project/sglang/pull/40004)\r\n* [Scheduler] Count complete prefill bursts and their tokens: [#40006](https://github.com/sgl-project/sglang/pull/40006)\r\n* [Scheduler] Add shortest-prefill-first scheduling: [#40024](https://github.com/sgl-project/sglang/pull/40024)\r\n* [Profiler] Ignore PREBUILT batches in profile-by-stage: [#40098](https://github.com/sgl-project/sglang/pull/40098)\r\n* Enable optimistic prefill for Mamba radix-cache models: [#40184](https://github.com/sgl-project/sglang/pull/40184)\r\n* [Perf] Fork-safe import: no CUDA context at import time, lighter argument parsing: [#40201](https://github.com/sgl-project/sglang/pull/40201)\r\n* [Feature] Support --tokenizer-worker-num > 1 in the offline Engine API: [#40260](https://github.com/sgl-project/sglang/pull/40260)\r\n* Clean up startup logging and streamline log audits: [#40526](https://github.com/sgl-project/sglang/pull/40526)\r\n* [Fix] Run KV canary hooks for context-parallel prefill: [#40642](https://github.com/sgl-project/sglang/pull/40642)\r\n* [RL] Fix Kimi K3 expert-count lookup for routed-expert capture: [#40700](https://github.com/sgl-project/sglang/pull/40700)\r\n* [RL] Add RL weight-update sessions and support updating spec draft runners: [#40777](https://github.com/sgl-project/sglang/pull/40777)\r\n* [RL] Keep pause_generation and weight updates from deadlocking each other: [#40779](https://github.com/sgl-project/sglang/pull/40779)\r\n* [Misc] Remove deprecated endpoints, env vars and aliases past two releases: [#40795](https://github.com/sgl-project/sglang/pull/40795)\r\n* [Metrics] Log forward and forward+idle occupancy over total wall time: [#40802](https://github.com/sgl-project/sglang/pull/40802)\r\n* [RL] Keep DSA cuda-graph state and the graph pool intact across TMS pause/resume: [#40804](https://github.com/sgl-project/sglang/pull/40804)\r\n* [Fix] Recover from stale torch extension locks in every `cpp_extension` loader: [#40989](https://github.com/sgl-project/sglang/pull/40989)\r\n* [Perf] Lazy-load built-in model definitions and nixl_ep at startup: [#41061](https://github.com/sgl-project/sglang/pull/41061)\r\n* [Model Loader] Stop checkpoint prefetch after iterator completion: [#41588](https://github.com/sgl-project/sglang/pull/41588)\r\n* [Benchmark] Limit warmup concurrency in serving benchmark: [#39398](https://github.com/sgl-project/sglang/pull/39398)\r\n* Support LongBench v2 in one-batch server benchmarks: [#39874](https://github.com/sgl-project/sglang/pull/39874)\r\n* [Benchmark] Take each request's prompt length from the server: [#39889](https://github.com/sgl-project/sglang/pull/39889)\r\n* [Benchmark] Add agentic rollout simulator and offline explorer: [#40034](https://github.com/sgl-project/sglang/pull/40034)\r\n* [Benchmark] Optionally clear HiCache storage between cases: [#40659](https://github.com/sgl-project/sglang/pull/40659)\r\n\r\n## Sampling\r\n\r\n* fix(sampling): validate sampling_seed is an int within int64 range: [#28960](https://github.com/sgl-project/sglang/pull/28960)\r\n* perf(sampling): avoid GPU syncs when applying custom logit processors: [#39234](https://github.com/sgl-project/sglang/pull/39234)\r\n* Use pinned memory for asynchronous sampling metadata transfers: [#39777](https://github.com/sgl-project/sglang/pull/39777)\r\n* [Logprob] Borrow graph-pool memory for input logprob logits construction: [#40007](https://github.com/sgl-project/sglang/pull/40007)\r\n* [Logprob] Serve input-logprob temporaries from CUDA-graph-pool dead space: [#40038](https://github.com/sgl-project/sglang/pull/40038)\r\n* [Sampling] Add selected/support sampling logprob modes: [#40932](https://github.com/sgl-project/sglang/pull/40932)\r\n* [Sampling] Stream sampling masks as per-request arrays: [#40986](https://github.com/sgl-project/sglang/pull/40986)\r\n* [PD] Keep the sampling mask of a replayed rebootstrap token: [#41235](https://github.com/sgl-project/sglang/pull/41235)\r\n\r\n## HiCache & Radix Cache\r\n\r\n* [Radix Cache] Sync Rust TreeCore and make it the default: [#39627](https://github.com/sgl-project/sglang/pull/39627)\r\n* Add Agentic-Aware Tail-Optimized LRU eviction to the unified radix cache (opt-in; p99 ITL down 43.9% at concurrency 32 on DeepSeek-V4-Pro FP4, 4x B300, agentic trace): [#34012](https://github.com/sgl-project/sglang/pull/34012)\r\n* Feat: Add TensorCast storage as a new HiCache backend: [#27265](https://github.com/sgl-project/sglang/pull/27265)\r\n* [HiCache] fix: resolve Mooncake local_hostname per node for runtime attach: [#29668](https://github.com/sgl-project/sglang/pull/29668)\r\n* [Fix] fix(hicache): wait for decode offload before retraction: [#30899](https://github.com/sgl-project/sglang/pull/30899)\r\n* [HiSparse] Add MHA hisparse support for MiniMax M3: [#31446](https://github.com/sgl-project/sglang/pull/31446)\r\n* [Mooncake] Fix silent SSD offload corruption when TP/PP ranks share ssd_offload_path: [#31926](https://github.com/sgl-project/sglang/pull/31926)\r\n* [Scheduler] Align `RadixCache` no-insert cleanup with `kv_len_to_handle`: [#35204](https://github.com/sgl-project/sglang/pull/35204)\r\n* [PP + HiCache] Add PP Prefetch Tickets for eager cross-stage storage prefetch: [#36700](https://github.com/sgl-project/sglang/pull/36700)\r\n* Use a shared byte budget for unified hybrid-SWA memory: [#36729](https://github.com/sgl-project/sglang/pull/36729)\r\n* Gate mamba extra-buffer predicates on uses_mamba_radix_cache: [#37474](https://github.com/sgl-project/sglang/pull/37474)\r\n* [Unified Memory] Hierarchical cache for every unified pool shape: [#37507](https://github.com/sgl-project/sglang/pull/37507)\r\n* [HiCache] Fix sparse hybrid transfer layer IDs: [#37870](https://github.com/sgl-project/sglang/pull/37870)\r\n* [Unified Memory] fix: preserve FP8 dtype in unified MHA pool: [#38133](https://github.com/sgl-project/sglang/pull/38133)\r\n* [HiCache] Yield idle scheduler so storage workers can drain: [#38504](https://github.com/sgl-project/sglang/pull/38504)\r\n* [LMCache] Support lmcache unified radix cache: [#38652](https://github.com/sgl-project/sglang/pull/38652)\r\n* [Unified Cache] Dedup replicated MLA/DSA KV in the UMBP direct linker: [#38778](https://github.com/sgl-project/sglang/pull/38778)\r\n* [Perf] Fuse SWA page lookup and mapping clear: [#38948](https://github.com/sgl-project/sglang/pull/38948)\r\n* [HiCache][Perf] fix: batch HiCache D2H submits per step for hybrid pools: [#39050](https://github.com/sgl-project/sglang/pull/39050)\r\n* [HiCache] Back up MXFP8 KV scales in the host pool: [#39089](https://github.com/sgl-project/sglang/pull/39089)\r\n* [Mamba] Fix checkpoint depth for prefixes that end off the radix page: [#39115](https://github.com/sgl-project/sglang/pull/39115)\r\n* [HiCache] Label radix-cache metrics per rank and split the \"shrunk\" prefetch reason: [#39280](https://github.com/sgl-project/sglang/pull/39280)\r\n* bugfix:fix unifiedcache c128 radix cache management: [#39426](https://github.com/sgl-project/sglang/pull/39426)\r\n* Support unified memory page-envelope transfers in PD: [#39477](https://github.com/sgl-project/sglang/pull/39477)\r\n* Support unified memory decode host pools: [#39478](https://github.com/sgl-project/sglang/pull/39478)\r\n* [Unified Cache][9/N] add opt-in MLA load deduplication for Mooncake Linker (TTFT 3.85 to 2.23 s with a 256K cached prefix, GLM-5.2 on 8x H20): [#39565](https://github.com/sgl-project/sglang/pull/39565)\r\n* [HiCache] Forward prefix metadata to v2 storage calls: [#39567](https://github.com/sgl-project/sglang/pull/39567)\r\n* [HiCache] Keep hybrid transfer layer maps stage-local under PP: [#39699](https://github.com/sgl-project/sglang/pull/39699)\r\n* [HiCache] Add the page-unified KV load-back JIT kernel: [#39726](https://github.com/sgl-project/sglang/pull/39726)\r\n* [2/N][Kernel] Fuse padding-preserving HiSparse slot translation: [#39837](https://github.com/sgl-project/sglang/pull/39837)\r\n* [Unified Tree] fix: exempt host-locked aux nodes from the sanity_check host-LRU check: [#39980](https://github.com/sgl-project/sglang/pull/39980)\r\n* [HiCache] Read the in-flight buffer backup's node id from its snapshot in sanity_check: [#40013](https://github.com/sgl-project/sglang/pull/40013)\r\n* [HiCache] Stop arming a prefetch retry for a too-short storage span: [#40042](https://github.com/sgl-project/sglang/pull/40042)\r\n* [MemCache] Release up to `owned_kv_len` on radix cache insert: [#40075](https://github.com/sgl-project/sglang/pull/40075)\r\n* [HiCache] Auto-size the host pool to fit available host memory: [#40135](https://github.com/sgl-project/sglang/pull/40135)\r\n* Preallocate HiCache MHA staging before post-capture KV sizing: [#40256](https://github.com/sgl-project/sglang/pull/40256)\r\n* Fix prefetch attempt cleanup on abort: [#40262](https://github.com/sgl-project/sglang/pull/40262)\r\n* [HiCache] TMA-staged host<->device KV transfer kernel (sm_90+): [#40278](https://github.com/sgl-project/sglang/pull/40278)\r\n* [HiCache] Size MHA host pools from device row width: [#40304](https://github.com/sgl-project/sglang/pull/40304)\r\n* [Fix] Missing SWA eviction during decode preallocation: [#40309](https://github.com/sgl-project/sglang/pull/40309)\r\n* [HiCache] fix: bound the controller reset join so a stalled storage thread cannot hang the scheduler: [#40312](https://github.com/sgl-project/sglang/pull/40312)\r\n* Remove swa and mamba radix cache: [#40313](https://github.com/sgl-project/sglang/pull/40313)\r\n* [Fix] Forward SWA prealloc reclaim through the DSV4 HiSparse allocator: [#40354](https://github.com/sgl-project/sglang/pull/40354)\r\n* [HiCache] Give trailing sidecar storage transfers a contiguous prefix_keys chain: [#40456](https://github.com/sgl-project/sglang/pull/40456)\r\n* [HiCache] Make host reclamation independent of transfer order: [#40512](https://github.com/sgl-project/sglang/pull/40512)\r\n* [HiCache] Demote internal-node mamba states on write_back eviction: [#40680](https://github.com/sgl-project/sglang/pull/40680)\r\n* [HiCache] Demote internal-node SWA KV to host on write_back eviction instead of dropping it: [#40712](https://github.com/sgl-project/sglang/pull/40712)\r\n* [MemCache] Remove the experimental C++ radix tree: [#40775](https://github.com/sgl-project/sglang/pull/40775)\r\n* [MemCache] Clean up SWA/Mamba radix cache leftovers and drop SGLANG_ENABLE_UNIFIED_RADIX_TREE: [#40780](https://github.com/sgl-project/sglang/pull/40780)\r\n* [HiCache] Remove the unused HiRadixCache: [#40787](https://github.com/sgl-project/sglang/pull/40787)\r\n* [MemCache] Free the rows below the SWA evict floor on all-SWA request release: [#40798](https://github.com/sgl-project/sglang/pull/40798)\r\n* [HiCache] Batch buffer-only KV backups within each flush: [#40960](https://github.com/sgl-project/sglang/pull/40960)\r\n* [Fix] Derive per-runner hybrid SWA layer ids on ModelLayerInfo instead of mutating ModelConfig: [#40983](https://github.com/sgl-project/sglang/pull/40983)\r\n* [HiCache] fix: Drain pending backups before internal Mamba write-back: [#41092](https://github.com/sgl-project/sglang/pull/41092)\r\n* [Unified Memory] Fix Inkling conv-checkpoint track ids written to virtual slot numbers: [#41144](https://github.com/sgl-project/sglang/pull/41144)\r\n* Let predicate-registered linear-attention models carry the mamba radix-cache leaves: [#41165](https://github.com/sgl-project/sglang/pull/41165)\r\n* [Unified Memory] Honor move gates in float relocation and size auto HiCache from host capacity: [#41248](https://github.com/sgl-project/sglang/pull/41248)\r\n* [MemCache] Unify component eviction cursors and lock receipts: [#41276](https://github.com/sgl-project/sglang/pull/41276)\r\n* [MemCache] Never free the protected prefix on request release: [#41312](https://github.com/sgl-project/sglang/pull/41312)\r\n* Make sliding-window caching and speculative batch padding extensible: [#41325](https://github.com/sgl-project/sglang/pull/41325)\r\n* [MemCache] Fix LMCache component cursors and per-cache backend selection: [#41328](https://github.com/sgl-project/sglang/pull/41328)\r\n\r\n## LoRA\r\n\r\n* [LoRA] Support DP attention in LoRA backends: [#36389](https://github.com/sgl-project/sglang/pull/36389)\r\n* [LoRA] Size dense row/column-parallel LoRA buffers from the base linear's real shard: [#39379](https://github.com/sgl-project/sglang/pull/39379)\r\n\r\n## Multimodal\r\n\r\n* fix(multimodal): return 400 for corrupt image inputs: [#28131](https://github.com/sgl-project/sglang/pull/28131)\r\n* [Multimodal] Avoid CUDA placement on non-CUDA platforms: [#38750](https://github.com/sgl-project/sglang/pull/38750)\r\n* Fix Mistral3 retaining every vision-tower layer to read one: [#39185](https://github.com/sgl-project/sglang/pull/39185)\r\n* perf(multimodal): offload CPU feature hashing with bounded admission: [#39539](https://github.com/sgl-project/sglang/pull/39539)\r\n* Carry deferred attention operands and reuse multimodal shared memory: [#39870](https://github.com/sgl-project/sglang/pull/39870)\r\n* [MM] Skip VMM error gathers for text-only requests: [#40005](https://github.com/sgl-project/sglang/pull/40005)\r\n* [MM] Copy placeholder ids to CUDA asynchronously: [#40010](https://github.com/sgl-project/sglang/pull/40010)\r\n* [MM] Keep scheduler padding in packed token arrays: [#40357](https://github.com/sgl-project/sglang/pull/40357)\r\n* [Fix] Raise on undelivered embeddings in `send_with_url`, fix broken tests: [#40502](https://github.com/sgl-project/sglang/pull/40502)\r\n* Fix multimodal feature offload races: [#40621](https://github.com/sgl-project/sglang/pull/40621)\r\n* [VLM] Introduce FA4 into ViT for SM100/SM103: [#41344](https://github.com/sgl-project/sglang/pull/41344)\r\n\r\n## Model Support & Optimizations\r\n\r\n* [Feature] Gigachat 3.5 support: [#29189](https://github.com/sgl-project/sglang/pull/29189)\r\n* [dLLM] Support DiffusionGemma serving: [#34061](https://github.com/sgl-project/sglang/pull/34061)\r\n* Add Ling-3.0-flash-VL model support: [#38526](https://github.com/sgl-project/sglang/pull/38526)\r\n* dsv4.1: remaining model and runtime integration: [#38798](https://github.com/sgl-project/sglang/pull/38798)\r\n* dsv4.1: standalone kernels and Python wrappers: [#39646](https://github.com/sgl-project/sglang/pull/39646)\r\n* dsv4.1: Top-k kernels and candidate selection: [#39648](https://github.com/sgl-project/sglang/pull/39648)\r\n* dsv4.1: compression, KV I/O, and metadata kernels: [#39652](https://github.com/sgl-project/sglang/pull/39652)\r\n* dsv4.1: communication kernels and wrappers: [#39653](https://github.com/sgl-project/sglang/pull/39653)\r\n* dsv4.1: RoPE and FP4 packing kernels: [#39656](https://github.com/sgl-project/sglang/pull/39656)\r\n* dsv4.1: Hopper FP8 matmul kernels and tuning: [#39657](https://github.com/sgl-project/sglang/pull/39657)\r\n* dsv4.1: mHC computation and compensated projections: [#39664](https://github.com/sgl-project/sglang/pull/39664)\r\n* dsv4.1: chat encoding and tool parsing: [#39665](https://github.com/sgl-project/sglang/pull/39665)\r\n* dsv4.1: Engram module and request history support: [#39666](https://github.com/sgl-project/sglang/pull/39666)\r\n* dsv4.1: vision tower and image preprocessing: [#39668](https://github.com/sgl-project/sglang/pull/39668)\r\n* dsv4.1: candidate indexer library: [#39671](https://github.com/sgl-project/sglang/pull/39671)\r\n* dsv4.1: Rust extension modules for image preprocessing, KV pool names, and PD bootstrap: [#39677](https://github.com/sgl-project/sglang/pull/39677)\r\n* [DSv4.1] Score prefill consumer index layers on candidate blocks with DeepGEMM: [#40352](https://github.com/sgl-project/sglang/pull/40352)\r\n* [Feature] Xiaomi MiMo-V2.6/MiMo-V2.6-Pro day0 support: [#40448](https://github.com/sgl-project/sglang/pull/40448)\r\n* [Model] Add IQuest Q1 support and MTP draft: [#41590](https://github.com/sgl-project/sglang/pull/41590)\r\n* fix(gemma4): set lm_head_is_tied for Gemma4UnifiedForConditionalGeneration: [#35809](https://github.com/sgl-project/sglang/pull/35809)\r\n* [Fix] Preserve YaRN scaling when extending rotary caches: [#38786](https://github.com/sgl-project/sglang/pull/38786)\r\n* [Kimi-K3] O(1) expert weight lookup in load_weights (expert-name matching 21.45 to 0.13 s at load): [#38805](https://github.com/sgl-project/sglang/pull/38805)\r\n* [Model] Serve DeepSeek-OCR-2 with its official 768px local-crop geometry: [#38996](https://github.com/sgl-project/sglang/pull/38996)\r\n* Fix GLM-OCR MTP multimodal embeddings and positions: [#39088](https://github.com/sgl-project/sglang/pull/39088)\r\n* [DSV4] Chunk the indexer MQA logits by query rows under a free-memory budget: [#39095](https://github.com/sgl-project/sglang/pull/39095)\r\n* [Perf] Fuse the glm5_next mHC attn->MLP boundary: [#39200](https://github.com/sgl-project/sglang/pull/39200)\r\n* [DSV4] fix: keep the TileLang JIT cache under SGLANG_CACHE_DIR: [#39364](https://github.com/sgl-project/sglang/pull/39364)\r\n* Fuse GLM-5.3-Flash KDA projections and prefill metadata: [#39688](https://github.com/sgl-project/sglang/pull/39688)\r\n* [GLM-5.3-Flash] Reduce KPool planning synchronization and overlap indexer preparation: [#39695](https://github.com/sgl-project/sglang/pull/39695)\r\n* [Fix] Fix GLM5 mHC PP forward: [#39720](https://github.com/sgl-project/sglang/pull/39720)\r\n* [DSV4] Generalize attention metadata, sparse prefill, and KV pool over compress ratios: [#39921](https://github.com/sgl-project/sglang/pull/39921)\r\n* [Qwen3.8 Next] Fuse Qwen PLE gate and convolution preparation for target verify: [#40041](https://github.com/sgl-project/sglang/pull/40041)\r\n* Support GLM-5.3-Flash hybrid attention CPU offload and PD index mapping: [#40310](https://github.com/sgl-project/sglang/pull/40310)\r\n* [DSV4] fix: size the C4 state ring by the page it is addressed by: [#40337](https://github.com/sgl-project/sglang/pull/40337)\r\n* [Fix] Keep mHC context out of non-V4 compiled MoE forwards: [#40353](https://github.com/sgl-project/sglang/pull/40353)\r\n* [Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp: [#40501](https://github.com/sgl-project/sglang/pull/40501)\r\n* Fix GLM-5.3 forget-gate shape for nvCUTEDSL verify: [#40607](https://github.com/sgl-project/sglang/pull/40607)\r\n* [Fix] Avoid duplicate residual in LongCat MoE shortcut: [#40799](https://github.com/sgl-project/sglang/pull/40799)\r\n* [DSA] Chunk the kpool indexer MQA logits by query rows under a free-memory budget: [#40854](https://github.com/sgl-project/sglang/pull/40854)\r\n* [DSV4] Budget the ratio-2 pair state pool in DSV4PoolConfigurator: [#41048](https://github.com/sgl-project/sglang/pull/41048)\r\n* [DSV4] Size compressed pools from one per-ratio table in DSV4PoolConfigurator: [#41049](https://github.com/sgl-project/sglang/pull/41049)\r\n* [Fix] Add name mapping in load_weights of nvidia/LocateAnything-3B: [#41059](https://github.com/sgl-project/sglang/pull/41059)\r\n* [DSV4] Fix TRTLLM uniform FP8 KV memory budgeting: [#41090](https://github.com/sgl-project/sglang/pull/41090)\r\n* [DSV4] Account for FlashMLA physical KV page padding in memory budgets: [#41091](https://github.com/sgl-project/sglang/pull/41091)\r\n* [Kimi-K3] Merge fused_qkvg_proj into the loader-seeded packed_modules_mapping: [#41164](https://github.com/sgl-project/sglang/pull/41164)\r\n* [Qwen3.8 Next] Fuse small CUDA graph input buffer copies: [#41166](https://github.com/sgl-project/sglang/pull/41166)\r\n\r\n## Kernel Library\r\n\r\n* [Kernel] Share the warp vectorized copy and enforce its alignment: [#36176](https://github.com/sgl-project/sglang/pull/36176)\r\n* fix(kda_prefill): fence shared writes before async proxy reads: [#39124](https://github.com/sgl-project/sglang/pull/39124)\r\n* [Kernel] Coalesce the KDA CuTe DSL decode state transpose: ~3x faster, bit-identical: [#39680](https://github.com/sgl-project/sglang/pull/39680)\r\n* [Kernel] Move CUDA and ROCm speculative kernels to JIT: [#40033](https://github.com/sgl-project/sglang/pull/40033)\r\n* Fix TopK v2 fallback when 16-block cluster capacity is zero: [#40163](https://github.com/sgl-project/sglang/pull/40163)\r\n* [Kernel] Fuse hc_combine_norm for mid-size verify batches (9-96 rows): [#40208](https://github.com/sgl-project/sglang/pull/40208)\r\n* [KDA] Fix missing beta sigmoid in PTX prefill: [#40685](https://github.com/sgl-project/sglang/pull/40685)\r\n* [JIT] Add an occupancy-preserving L1 carveout preference: [#40767](https://github.com/sgl-project/sglang/pull/40767)\r\n* [Perf] Mamba2 selective_state_update up to 2x faster on B200 via 8x1 launch config for dstate 128 (+7.5% Nemotron-3-Super serving): [#41223](https://github.com/sgl-project/sglang/pull/41223)\r\n\r\n## Configuration System\r\n\r\n* Add registration for external model configurations: [#39452](https://github.com/sgl-project/sglang/pull/39452)\r\n\r\n## OpenAI-Compatible API\r\n\r\n* [Fix] Guard conditional top-logprob keys in the completions echo path: [#34776](https://github.com/sgl-project/sglang/pull/34776)\r\n* [Fix] Prevalidate JSON Schema support per grammar backend: [#37839](https://github.com/sgl-project/sglang/pull/37839)\r\n* fix(openai): recover logprobs token bytes from token_id (UTF-8 fragments): [#38604](https://github.com/sgl-project/sglang/pull/38604)\r\n* [Score API] Setwise Scoring Support: [#38965](https://github.com/sgl-project/sglang/pull/38965)\r\n* feat: use XGrammar V4.1 DSML parameter constraints: [#39026](https://github.com/sgl-project/sglang/pull/39026)\r\n* fix(function_call): buffer complete DeepSeek DSML invokes: [#39632](https://github.com/sgl-project/sglang/pull/39632)\r\n* Fix corrupted chat prompts on mistral_common tokenizers (tool_choice auto never fires): [#39773](https://github.com/sgl-project/sglang/pull/39773)\r\n* [Fix] Keep Inkling automatic tool grammar active across the response: [#40468](https://github.com/sgl-project/sglang/pull/40468)\r\n* fix(grpc): expose native response timeout as a server argument: [#40644](https://github.com/sgl-project/sglang/pull/40644)\r\n* [Feature] Add per-item candidate token scoring and calibration: [#40826](https://github.com/sgl-project/sglang/pull/40826)\r\n* Add MiniMax arch fallback to auto parser resolution: [#40930](https://github.com/sgl-project/sglang/pull/40930)\r\n* [Score API] Setwise scoring: CausalLM support (batched + --enable-mis): [#41188](https://github.com/sgl-project/sglang/pull/41188)\r\n* [Feature] Add a System One compatible /v1/systemone route (also ships the `/v1/decisions` route it builds on): [#41208](https://github.com/sgl-project/sglang/pull/41208)\r\n* Fix chat template cache key order: [#41517](https://github.com/sgl-project/sglang/pull/41517)\r\n\r\n## Rust Server\r\n\r\nThe experimental Rust router and frontend (`experimental/sgl-router`, the Rust renderer) keep moving toward parity with the Python server.\r\n\r\n* [Rust Renderer] Standalone preprocessing: [#36718](https://github.com/sgl-project/sglang/pull/36718)\r\n* [SGL Router] Render chat prompts with dynamo-render: [#38983](https://github.com/sgl-project/sglang/pull/38983)\r\n* [Router] Fleet-wide sampling contract 1/3: the config surface: [#39000](https://github.com/sgl-project/sglang/pull/39000)\r\n* [Router] Fleet-wide sampling contract 2/3: enforce and inject per request: [#39001](https://github.com/sgl-project/sglang/pull/39001)\r\n* [Router] Fleet-wide sampling contract 3/3: splice injection without re-serializing: [#39002](https://github.com/sgl-project/sglang/pull/39002)\r\n* [Router] Resolve a wire protocol per worker at registration: [#39004](https://github.com/sgl-project/sglang/pull/39004)\r\n* [Router] Speak cleartext h2c on both edges: serve it inbound, forward it outbound: [#39006](https://github.com/sgl-project/sglang/pull/39006)\r\n* [Router] Count open HTTP exchanges until their response body finishes: [#39014](https://github.com/sgl-project/sglang/pull/39014)\r\n* [Router] Add the shutdown-drain configuration surface: [#39015](https://github.com/sgl-project/sglang/pull/39015)\r\n* [Router] Drain readiness before SIGTERM shutdown so k8s deregisters the pod first: [#39016](https://github.com/sgl-project/sglang/pull/39016)\r\n* [Router] Real-GPU e2e coverage for storage-tier-aware cache routing (3/4): [#39110](https://github.com/sgl-project/sglang/pull/39110)\r\n* [Router] Add a KV storage-tier coverage row to the Grafana dashboard (4/4): [#39111](https://github.com/sgl-project/sglang/pull/39111)\r\n* [Router] Improve SGLang chat render parity: [#39133](https://github.com/sgl-project/sglang/pull/39133)\r\n* [Router] Shard the cache-aware KV tree by chain root: [#39167](https://github.com/sgl-project/sglang/pull/39167)\r\n* [Router] Add --worker-queue-limit: stop sending cache-affinity traffic to a queueing worker: [#39168](https://github.com/sgl-project/sglang/pull/39168)\r\n* [Router] Pin to the prefix owner when the whole fleet is queueing (--saturation-queue-floor): [#39169](https://github.com/sgl-project/sglang/pull/39169)\r\n* [Router] Sample k random candidates for the min-load fallback (--min-load-choices): [#39170](https://github.com/sgl-project/sglang/pull/39170)\r\n* Add external multimodal processors to the Rust frontend: [#39329](https://github.com/sgl-project/sglang/pull/39329)\r\n* [Rust] Extract a transport-neutral frontend core: [#39385](https://github.com/sgl-project/sglang/pull/39385)\r\n* [SGL Router] Forward input_ids only for string content; count tokenize errors only when forwardable: [#39458](https://github.com/sgl-project/sglang/pull/39458)\r\n* [Router] Abort the engine when a client disconnects mid-request: [#39461](https://github.com/sgl-project/sglang/pull/39461)\r\n* [Router] Derive error status from a failure class; preserve the worker's status (1/3): [#39463](https://github.com/sgl-project/sglang/pull/39463)\r\n* [Router] Treat an upstream 503/429 as backpressure, not a breaker fault (2/3): [#39464](https://github.com/sgl-project/sglang/pull/39464)\r\n* [Router] Log every request at one site; derive its outcome from the final status (3/3): [#39465](https://github.com/sgl-project/sglang/pull/39465)\r\n* [SGL Router] Share model-file discovery for chat formatters: [#39485](https://github.com/sgl-project/sglang/pull/39485)\r\n* Rust server unify datapath for mm and generate requests: [#39679](https://github.com/sgl-project/sglang/pull/39679)\r\n* [SGL Router] Add Kimi-K3 rendering with SGLang parity: [#40390](https://github.com/sgl-project/sglang/pull/40390)\r\n* [SGL Router] Bound streaming lifetimes and release guards on idle disconnect: [#40391](https://github.com/sgl-project/sglang/pull/40391)\r\n* Port chat_parsing core: [#40477](https://github.com/sgl-project/sglang/pull/40477)\r\n* [SGL Router] Match DeepSeek V4 rendering to SGLang: [#40530](https://github.com/sgl-project/sglang/pull/40530)\r\n* [SGL Router] Add SGLang-compatible DeepSeek V4.1 Flash rendering: [#40532](https://github.com/sgl-project/sglang/pull/40532)\r\n* [SGL Router] Release cancelled circuit-breaker probes: [#40603](https://github.com/sgl-project/sglang/pull/40603)\r\n* [SGL Router] Fix readiness, IPv6 discovery, logging, and model validation: [#40604](https://github.com/sgl-project/sglang/pull/40604)\r\n* [Router] Give the cache-aware tree a snapshot surface (1/13): [#40687](https://github.com/sgl-project/sglang/pull/40687)\r\n* [Router] Serve the cache-aware tree at /internal/kv_snapshot (2/13): [#40688](https://github.com/sgl-project/sglang/pull/40688)\r\n* [Router] Name a replica's siblings with --kv-peer-selector (3/13): [#40689](https://github.com/sgl-project/sglang/pull/40689)\r\n* [Rust Renderer] decouple renderer sampling from protocols: [#40747](https://github.com/sgl-project/sglang/pull/40747)\r\n* [SGL Router] Launch reorg routing with existing policy options: [#40766](https://github.com/sgl-project/sglang/pull/40766)\r\n* [SGL Router] Track input_ids forwarding outcomes per chat request: [#41185](https://github.com/sgl-project/sglang/pull/41185)\r\n* [SGL Router] Take SGLang's render defaults: --default-chat-template-kwargs and thinking/effort envs: [#41221](https://github.com/sgl-project/sglang/pull/41221)\r\n* [SGL Router] Scope input_ids forwarding by renderer: all text chats for DeepSeek-V4: [#41226](https://github.com/sgl-project/sglang/pull/41226)\r\n* [Rust Frontend] Decode input_ids without untagged buffering (input_ids deserialization about 45% faster): [#41246](https://github.com/sgl-project/sglang/pull/41246)\r\n* [SGL Router] Book the input_ids forwarding outcome only for built bodies: [#41342](https://github.com/sgl-project/sglang/pull/41342)\r\n\r\n## Simulator\r\n\r\n* [Simulator][Compatibility] Adapt to latest KV cache pool interfaces: [#40418](https://github.com/sgl-project/sglang/pull/40418)\r\n\r\n## SGLang-Diffusion\r\n\r\n* [Diffusion] model: support qwen-image-2.1: [#39983](https://github.com/sgl-project/sglang/pull/39983)\r\n* [Diffusion] Add native Anima Base v1.0 support: [#41011](https://github.com/sgl-project/sglang/pull/41011)\r\n* [Diffusion] Support FLUX 3 Action robot policies: [#41066](https://github.com/sgl-project/sglang/pull/41066)\r\n* [Diffusion] Add native Ming-Image Design and Design-Layer support: [#41067](https://github.com/sgl-project/sglang/pull/41067)\r\n* [Diffusion] add  /metrics support: [#19084](https://github.com/sgl-project/sglang/pull/19084)\r\n* [Diffusion] Preserve per-sample rollout trajectories across multi-output merge: [#34416](https://github.com/sgl-project/sglang/pull/34416)\r\n* [Diffusion] Fix AttributeError in grouped forward_batch by installing the residency manager: [#34417](https://github.com/sgl-project/sglang/pull/34417)\r\n* [Diffusion] Apply latent-ids and packing to caller-provided initial latents: [#34418](https://github.com/sgl-project/sglang/pull/34418)\r\n* [Diffusion] MiniMax-H3 Spectrum skip-step + fused RMSNorm/AdaLN: [#35684](https://github.com/sgl-project/sglang/pull/35684)\r\n* [Diffusion] Add MiniMax-H3 to ComfyUI integrated mode: [#35990](https://github.com/sgl-project/sglang/pull/35990)\r\n* [Diffusion] In-place LoRA merge/unmerge under layerwise offload: [#36192](https://github.com/sgl-project/sglang/pull/36192)\r\n* [Diffusion] feature: out of tree platform support: [#37547](https://github.com/sgl-project/sglang/pull/37547)\r\n* [Diffusion] support multiple task types for pipelines: [#38762](https://github.com/sgl-project/sglang/pull/38762)\r\n* [Diffusion] Enable shared RMSNorm dispatch for SenseNova-U1: [#39705](https://github.com/sgl-project/sglang/pull/39705)\r\n* [Diffusion] Preserve explicit attention backends during autotune: [#39882](https://github.com/sgl-project/sglang/pull/39882)\r\n* [Diffusion] Cache-DiT 1.5.1: DMD Calibrator, SVDQuant DQ, etc.: [#40104](https://github.com/sgl-project/sglang/pull/40104)\r\n* [Diffusion][MiniMax-H3] Add SM120 Sage compute for SubBlock sparse attention: [#40116](https://github.com/sgl-project/sglang/pull/40116)\r\n* [Diffusion] attention: add fp8_fa_sm120 FP8 backend for SM120 GPUs: [#40175](https://github.com/sgl-project/sglang/pull/40175)\r\n* [Diffusion] Fuse lossless SenseNova RoPE for 5% faster H200 inference: [#40374](https://github.com/sgl-project/sglang/pull/40374)\r\n* [Diffusion] Fuse rounded SwiGLU for quantized MiniMax-H3 MLPs (FastH3 FP8 worker E2E 1.9% lower on 2x H200): [#40378](https://github.com/sgl-project/sglang/pull/40378)\r\n* [Diffusion] Fuse LongCat GELU+cat and support Edit-Turbo BCG: [#40384](https://github.com/sgl-project/sglang/pull/40384)\r\n* [Diffusion] Accelerate Cosmos3 Edge on Hopper with lossless fusions: [#40386](https://github.com/sgl-project/sglang/pull/40386)\r\n* [Diffusion] Enable lossless SANA-Video eager conv fusions for 12.6% lower latency: [#40388](https://github.com/sgl-project/sglang/pull/40388)\r\n* [Diffusion] Fuse lossless Wan VAE post-ops for LongLive 2 I2V (worker E2E 5.5 to 5.9% lower on one H200): [#40405](https://github.com/sgl-project/sglang/pull/40405)\r\n* [Diffusion] Fuse lossless LingBot World FP32 normalization: [#40425](https://github.com/sgl-project/sglang/pull/40425)\r\n* [Diffusion] Populate CPU weight stores before host registration: [#40439](https://github.com/sgl-project/sglang/pull/40439)\r\n* [Diffusion] Add bounded exact conditioning cache across native models: [#40470](https://github.com/sgl-project/sglang/pull/40470)\r\n* [Diffusion] Enable lossless Cosmos3 Super T2I QK fusion on Hopper TP2 (worker E2E 7.4 to 8.8% lower on 2x H200 at TP2): [#40486](https://github.com/sgl-project/sglang/pull/40486)\r\n* [Diffusion] Fuse lossless Klein packed QK RMSNorm and RoPE on Hopper: [#40490](https://github.com/sgl-project/sglang/pull/40490)\r\n* [Diffusion] Fuse Joy Image Edit QKV concatenation and avoid QK copies: [#40494](https://github.com/sgl-project/sglang/pull/40494)\r\n* [Diffusion] Support MiniMax-H3 PDD(Parallel Decoding Distillation) inference: [#40568](https://github.com/sgl-project/sglang/pull/40568)\r\n* [Diffusion] Separate a use-scoped layerwise release from release_all: [#40590](https://github.com/sgl-project/sglang/pull/40590)\r\n* [Diffusion] feat: allow a component use retain its layerwise resident set: [#40592](https://github.com/sgl-project/sglang/pull/40592)\r\n* [Diffusion] Add a permanent lifetime for layerwise resident layers: [#40599](https://github.com/sgl-project/sglang/pull/40599)\r\n* [Fix] Keep diffusion encoder TP context bindings consistent: [#40646](https://github.com/sgl-project/sglang/pull/40646)\r\n* [Diffusion] Add opt-in SRT prompt enhancement to image and video APIs: [#41095](https://github.com/sgl-project/sglang/pull/41095)\r\n* [Diffusion] fix: LoRA-wrapped linears crash Qwen-Image and MiniMax-H3 inference: [#41272](https://github.com/sgl-project/sglang/pull/41272)\r\n* [Diffusion] Copy small files into overlay materialized trees instead of linking them: [#41298](https://github.com/sgl-project/sglang/pull/41298)\r\n* [KDA+Kimi K3] Speed up SANA-Video residual gate add on H200: [#41305](https://github.com/sgl-project/sglang/pull/41305)\r\n* [Diffusion] Run an all-valid attention mask on the backend's unmasked kernel (Ideogram4 request latency 3.42 to 2.19 s at TP2 on B200): [#41309](https://github.com/sgl-project/sglang/pull/41309)\r\n\r\n## Local & Desktop AI\r\n\r\n* [SM120] Add optional FlashInfer PCIe-IPC all-reduce for switch-free hosts: [#34528](https://github.com/sgl-project/sglang/pull/34528)\r\n* fix(moe): support Llama4 NVFP4 router input weights on SM120: [#35504](https://github.com/sgl-project/sglang/pull/35504)\r\n* [PP][DeepSeek V4] Overlap communication and optimize SM120 prefill: [#38792](https://github.com/sgl-project/sglang/pull/38792)\r\n* [Fix] Include SM121 in DeepGEMM packed-scale selection: [#39482](https://github.com/sgl-project/sglang/pull/39482)\r\n* [Qwen4-Exp] Build the offloaded PLE table on the meta device so --ple-offload-embedding never materialises it on the accelerator: [#39928](https://github.com/sgl-project/sglang/pull/39928)\r\n* Pin triton_kernels num_warps for MXFP4 MoE below Hopper (6x gpt-oss decode on RTX 4090): [#41292](https://github.com/sgl-project/sglang/pull/41292)\r\n\r\n## AMD / ROCm\r\n\r\n* [AMD] Speed up Wan2.2 DiT FP8 attention per-tensor quantization: [#34695](https://github.com/sgl-project/sglang/pull/34695)\r\n* [AMD] Fix DSV4 FP4 dequant path for AITER on ROCm: [#35123](https://github.com/sgl-project/sglang/pull/35123)\r\n* [ROCm][Diffusion] Enable fused qk norm and rope on ROCm: [#35573](https://github.com/sgl-project/sglang/pull/35573)\r\n* [AMD] Integrate Aiter MegaMoEv2 for DeepSeek-V4: [#35619](https://github.com/sgl-project/sglang/pull/35619)\r\n* [AMD] Skip full-vocab softmax in EAGLE topk==1 draft on ROCm: [#35872](https://github.com/sgl-project/sglang/pull/35872)\r\n* [ROCm][Perf] aiter: page-level KV view for gfx950 fp8 page-64 asm prefill: [#36505](https://github.com/sgl-project/sglang/pull/36505)\r\n* MiniMax-M3: run the sparse prefill main attention through AITER Gluon paged attention: [#36546](https://github.com/sgl-project/sglang/pull/36546)\r\n* MiniMax-M3: allocate the lightning-indexer K cache in fp8 on gfx95: [#36549](https://github.com/sgl-project/sglang/pull/36549)\r\n* MoE: small-batch sorting path with fused mxfp8 quantisation: [#36559](https://github.com/sgl-project/sglang/pull/36559)\r\n* MiniMax-M3: wave64 histogram-select decode top-k, and raise kMaxNumBlocks for CUDA graphs: [#36560](https://github.com/sgl-project/sglang/pull/36560)\r\n* MiniMax-M3: MXFP8 dense-only block convert + aiter MXFP8 MoE on gfx950: [#36574](https://github.com/sgl-project/sglang/pull/36574)\r\n* MiniMax-M3: allow shared-experts fusion on ROCm gfx942 and newer: [#36576](https://github.com/sgl-project/sglang/pull/36576)\r\n* [ROCm] Fix EAGLE spec-decode verify silently sampling greedy on HIP: [#37134](https://github.com/sgl-project/sglang/pull/37134)\r\n* [ROCm] Widen the HiCache JIT copy rounds and enable the K-only host pool: [#37152](https://github.com/sgl-project/sglang/pull/37152)\r\n* [AMD] Preserve deterministic inference when Lean Attention is enabled: [#37740](https://github.com/sgl-project/sglang/pull/37740)\r\n* [AMD][Diffusion] FlyDSL fused norm kernels on wave32 targets (gfx1250): [#37751](https://github.com/sgl-project/sglang/pull/37751)\r\n* [AMD] Fix DeepSeek-R1-MXFP4 accuracy with AITER FP8: [#37762](https://github.com/sgl-project/sglang/pull/37762)\r\n* [AMD][DSV4] Enable hicache on deepseek-v4 fp8 unified attn: [#37778](https://github.com/sgl-project/sglang/pull/37778)\r\n* [ROCm][DSV4] Enable breakable CUDA graph prefill: [#37810](https://github.com/sgl-project/sglang/pull/37810)\r\n* [AMD] Enable GLM DSA prefill top-k to the v2 kernel (+4.9% token throughput at 70K input on GLM-5.2-MXFP4, 4x MI355X): [#37889](https://github.com/sgl-project/sglang/pull/37889)\r\n* [AMD][Spec] Enable GDN ReplaySSM target-verify on ROCm: [#38184](https://github.com/sgl-project/sglang/pull/38184)\r\n* [ROCm] Fuse the MLA q absorb into the RoPE + KV-write kernel on gfx950: [#38340](https://github.com/sgl-project/sglang/pull/38340)\r\n* [AMD] Avoid the FP8 wo_a path when the weight is BF16: [#38453](https://github.com/sgl-project/sglang/pull/38453)\r\n* [AMD][GLM-5.3-Flash Day 0] Route mHC through AITER on gfx950: [#38545](https://github.com/sgl-project/sglang/pull/38545)\r\n* [AMD][GLM-5.3-Flash Day 0] Enable FP8 and Quark MXFP4 MoE on gfx950: [#38546](https://github.com/sgl-project/sglang/pull/38546)\r\n* [AMD][GLM-5.3-Flash Day 0] Enable zero-RoPE TileLang DSA on gfx950: [#38547](https://github.com/sgl-project/sglang/pull/38547)\r\n* [AMD][GLM-5.3-Flash Day 0] Honor fused and per-expert names in quark `exclude`: [#39317](https://github.com/sgl-project/sglang/pull/39317)\r\n* [AMD][GLM-5.3-Flash Day 0] Enable zero-RoPE MHA prefill on ROCm: [#39338](https://github.com/sgl-project/sglang/pull/39338)\r\n* [AMD][GLM-5.3-Flash Day 0] Build the fused DSA k-pool top-k JIT kernel on HIP: [#39339](https://github.com/sgl-project/sglang/pull/39339)\r\n* [AMD][GLM-5.3-Flash Day 0] Support non-2048 top-k widths in the DSA page-table transform: [#39340](https://github.com/sgl-project/sglang/pull/39340)\r\n* [AMD][GLM-5.3-Flash Day 0] Enable the k-pool DSA indexer on gfx950: [#39341](https://github.com/sgl-project/sglang/pull/39341)\r\n* [AMD][GLM-5.3-Flash Day 0] Enable speculative decoding (MTP) on ROCm: [#39778](https://github.com/sgl-project/sglang/pull/39778)\r\n* [AMD][GLM-5.3-Flash Day 0] Load the MXFP4 MTP draft layer: [#39779](https://github.com/sgl-project/sglang/pull/39779)\r\n* [AMD] Pad QSA MQA decode Q-heads to 16 for ROCm MFMA: [#38875](https://github.com/sgl-project/sglang/pull/38875)\r\n* [AMD] Add a Triton packed sparse decode path for QSA on ROCm: [#38876](https://github.com/sgl-project/sglang/pull/38876)\r\n* [AMD] Load fused shared experts for Qwen4-Exp and Qwen3.5 MTP: [#38878](https://github.com/sgl-project/sglang/pull/38878)\r\n* [AMD][DSV4] feat: enable DSpark with fp8 unified_kv on gfx950: [#38901](https://github.com/sgl-project/sglang/pull/38901)\r\n* [AMD] Tune Triton sparse MLA on gfx950 and make split-K workspaces graph-safe: [#39059](https://github.com/sgl-project/sglang/pull/39059)\r\n* [ROCm][Fix] Keep quantization for mixed Quark Qwen3.5 MTP checkpoints: [#39064](https://github.com/sgl-project/sglang/pull/39064)\r\n* [AMD][Kimi-K3] Fix deferred KDA gate projection and update DCP cookbook: [#39066](https://github.com/sgl-project/sglang/pull/39066)\r\n* [AMD] GLM-5.2 NextN: cast draft fused MoE to per-channel FP8: [#39155](https://github.com/sgl-project/sglang/pull/39155)\r\n* [AMD] Use exact CU share for gfx950 segment-plan headroom: [#39503](https://github.com/sgl-project/sglang/pull/39503)\r\n* [AMD][Fix] Fix vattn_asm HIP error 709 under CUDA graph capture on ROCm 10: [#39513](https://github.com/sgl-project/sglang/pull/39513)\r\n* [AMD] Fix deferred Kimi-K3 forget gate in fused in-projection: [#39525](https://github.com/sgl-project/sglang/pull/39525)\r\n* [AMD][Fix] Fix DSV4 MTP crash: [#39547](https://github.com/sgl-project/sglang/pull/39547)\r\n* [AMD] Prefer HIP Top-K for GLM-5.x on ROCm (P90 interactivity 96.97 to 106.69 tok/s/user on MI355X at concurrency 8): [#39631](https://github.com/sgl-project/sglang/pull/39631)\r\n* [AMD] Update deepseek-v4 PDI and cache policy setting for agentic workload: [#39702](https://github.com/sgl-project/sglang/pull/39702)\r\n* [AMD] Clamp MORI intranode grid GPUs: [#39763](https://github.com/sgl-project/sglang/pull/39763)\r\n* [ROCm] fix: remove extra bf16 -> fp32 cast in jit grouped topk kernel path: [#39775](https://github.com/sgl-project/sglang/pull/39775)\r\n* [ROCm] feat: enable aiter allreduce fusion for GLM models: [#39790](https://github.com/sgl-project/sglang/pull/39790)\r\n* [AMD][Fix] Fix dsv4 server launch: [#39875](https://github.com/sgl-project/sglang/pull/39875)\r\n* [AMD] Reuse KV gather indices across ASM context prefill layers: [#39901](https://github.com/sgl-project/sglang/pull/39901)\r\n* [AMD] Pack Qwen3.5 GDN input projections on ROCm: [#39902](https://github.com/sgl-project/sglang/pull/39902)\r\n* [AMD][DSV4] Allow moe_a2a_backend='mori' with DSpark + dp attention: [#39910](https://github.com/sgl-project/sglang/pull/39910)\r\n* [AMD] Update ROCm AITER pin to acf8fdf9: [#39965](https://github.com/sgl-project/sglang/pull/39965)\r\n* [AMD] dsv4: pick kv_splits per index stream, not by occupancy alone: [#39968](https://github.com/sgl-project/sglang/pull/39968)\r\n* [AMD] Use Triton softmax routing for Qwen3.5 on gfx950: [#39986](https://github.com/sgl-project/sglang/pull/39986)\r\n* [AMD] Tune Qwen3.5 TP4 GDN recurrent launch on gfx950: [#39987](https://github.com/sgl-project/sglang/pull/39987)\r\n* [AMD][DSV4] fix: drop shadowing local get_exec import that breaks model startup on ROCm: [#40186](https://github.com/sgl-project/sglang/pull/40186)\r\n* [AMD] Small-M MXFP4 fused-MoE kernel for gfx950 (Qwen): [#40204](https://github.com/sgl-project/sglang/pull/40204)\r\n* [AMD] Drop the redundant scale zero-fill before AITER per-tensor FP8 quant: [#40557](https://github.com/sgl-project/sglang/pull/40557)\r\n* [ROCm][DSA] Enable AITER fused FP8 indexer writer: [#40710](https://github.com/sgl-project/sglang/pull/40710)\r\n* [AMD] Critical fix enabling Qwen3.8 FP8: restore dropped fused shared-expert weights: [#40754](https://github.com/sgl-project/sglang/pull/40754)\r\n* [AMD] Add diffusion (Wan2.2) extras to gfx1151 Docker image: [#40844](https://github.com/sgl-project/sglang/pull/40844)\r\n* [AMD][DSV4] fp8 unified_kv decode: wave-aware split count past 40 tokens: [#40878](https://github.com/sgl-project/sglang/pull/40878)\r\n* [AMD] Drop the unreachable vLLM fallback from ROCm FP8 activation quant: [#40879](https://github.com/sgl-project/sglang/pull/40879)\r\n* [AMD][DSV4] moe: enable shared-expert fusion on the grouped-topk path (megamoe): [#40943](https://github.com/sgl-project/sglang/pull/40943)\r\n* [AMD] Add tuned dsv4 shape: [#40996](https://github.com/sgl-project/sglang/pull/40996)\r\n* [AMD] Add .co for deepseek v4 fp8 decode kernel and add group decode opt: [#41120](https://github.com/sgl-project/sglang/pull/41120)\r\n* [AMD] Fix int32 offset overflow in Triton DSv4 KV store kernels: [#41159](https://github.com/sgl-project/sglang/pull/41159)\r\n* [AMD][GLM5] Fuse shared expert into AITER MoE on gfx950: [#41161](https://github.com/sgl-project/sglang/pull/41161)\r\n* [AMD] Honor an explicit triton moe_runner_backend for mxfp8 on ROCm: [#41377](https://github.com/sgl-project/sglang/pull/41377)\r\n* dsv4.1-amd: gfx950 MXFP8 matmul kernels and fp8-grid producers: [#41018](https://github.com/sgl-project/sglang/pull/41018)\r\n* dsv4.1-amd: KV cache layouts, FP4 indexer, compressor and router kernels: [#41019](https://github.com/sgl-project/sglang/pull/41019)\r\n* dsv4.1-amd: gfx950 sparse decode attention and sorted top-k: [#41020](https://github.com/sgl-project/sglang/pull/41020)\r\n* dsv4.1-amd: fused mHC boundary and all-reduce + mHC post kernels: [#41021](https://github.com/sgl-project/sglang/pull/41021)\r\n\r\n## NPU / Ascend\r\n\r\n* feat(hicache): support Ascend Mamba states with FIA and async IO: [#32500](https://github.com/sgl-project/sglang/pull/32500)\r\n* [NPU] decoding procedure optimization on qwen3.5/3.6: [#35958](https://github.com/sgl-project/sglang/pull/35958)\r\n* [NPU] Fix xgrammar apply_vocab_mask device dispatch to use torch.ops.npu: [#36120](https://github.com/sgl-project/sglang/pull/36120)\r\n* [NPU] add chunk gdn kernel and unify ssm state layout for ascend gdn backend: [#36187](https://github.com/sgl-project/sglang/pull/36187)\r\n* fix(npu): fix hybrid KV transfer with PP prefill in PD disaggregation: [#38402](https://github.com/sgl-project/sglang/pull/38402)\r\n* [NPU] Refactor weight processing and add NPUSwigluLimit activation: [#38420](https://github.com/sgl-project/sglang/pull/38420)\r\n* [NPU][Fix] update low latency quantization input and update MXFP8 tests: [#38831](https://github.com/sgl-project/sglang/pull/38831)\r\n* feat(npu): Support returning indexer top-k results: [#39060](https://github.com/sgl-project/sglang/pull/39060)\r\n* [NPU][Diffusion] Optimize SenseNova-U1 batched generation: [#39382](https://github.com/sgl-project/sglang/pull/39382)\r\n* [NPU] Avoid device synchronization in Ascend sampling: [#39404](https://github.com/sgl-project/sglang/pull/39404)\r\n* [NPU] Support nccl backend for --remote-instance-weight-loader: [#39413](https://github.com/sgl-project/sglang/pull/39413)\r\n* [NPU] Adapt hicache for K3 hybrid models: [#39415](https://github.com/sgl-project/sglang/pull/39415)\r\n* dsv4(npu): support prefill context parallelism with interleave and zigzag: [#39427](https://github.com/sgl-project/sglang/pull/39427)\r\n* [NPU][Diffusion] FA MXFP8 and modelslim w4a4f8 and w8a8f8 support for Wan2.2 and FLUX: [#39438](https://github.com/sgl-project/sglang/pull/39438)\r\n* [NPU][Fix] Fix undefined require_attn_tp_allgather: [#39561](https://github.com/sgl-project/sglang/pull/39561)\r\n* [NPU] support kimi k3 on A5 and improve performance: [#39589](https://github.com/sgl-project/sglang/pull/39589)\r\n* [NPU] Support batch invariant FIA graphs for deterministic inference: [#39607](https://github.com/sgl-project/sglang/pull/39607)\r\n* [NPU] fix npu docker workspace directory: [#39732](https://github.com/sgl-project/sglang/pull/39732)\r\n* [NPU][DSV4] dsv4 enable cpp: [#39820](https://github.com/sgl-project/sglang/pull/39820)\r\n* [NPU] Run arch35 block-FP8 dense linears on the native MXFP8 GEMM: [#39823](https://github.com/sgl-project/sglang/pull/39823)\r\n* [NPU] Gate DFlash replay metadata refresh behind spec_algorithm check: [#39879](https://github.com/sgl-project/sglang/pull/39879)\r\n* [NPU] Fuse MXFP4 W4A8 MoE gmm1 + swiglu + requant into one kernel: [#39881](https://github.com/sgl-project/sglang/pull/39881)\r\n* [NPU] Avoid repeated BF16 wo_a weight transposes in DeepSeek-V4 decode: [#39919](https://github.com/sgl-project/sglang/pull/39919)\r\n* [NPU][Fix] Avoid M-RoPE recompilation for variable sequence lengths: [#40371](https://github.com/sgl-project/sglang/pull/40371)\r\n* [Fix][NPU] fix dp-attn hang when pin_mem is True on NPU: [#40446](https://github.com/sgl-project/sglang/pull/40446)\r\n* [NPU] Fix DSV4 hard-coding kv dtype: [#40510](https://github.com/sgl-project/sglang/pull/40510)\r\n* [NPU] Update CANN version to 9.1.0: [#40524](https://github.com/sgl-project/sglang/pull/40524)\r\n* [NPU] support NPU 910C L2 memcache offload: [#41527](https://github.com/sgl-project/sglang/pull/41527)\r\n* [Docs][NPU] Add MiMo-V2.5-Pro FP4 DFlash best practice on Ascend NPU: [#40577](https://github.com/sgl-project/sglang/pull/40577)\r\n\r\n## CPU / Intel / XPU\r\n\r\n* [Intel XPU] Enable fused_moe_triton tuning on XPU and add tuned DeepSeek-OCR-2 configs: [#28723](https://github.com/sgl-project/sglang/pull/28723)\r\n* Speculative Decoding with NGRAM support for XPU: [#31362](https://github.com/sgl-project/sglang/pull/31362)\r\n* [CPU] Add fp8_per_tensor_scaled_mm_cpu kernel: [#32618](https://github.com/sgl-project/sglang/pull/32618)\r\n* [XPU] Enable HiSparse hierarchical sparse KV cache on Intel XPU: [#32792](https://github.com/sgl-project/sglang/pull/32792)\r\n* [CPU][Diffusion] Add fused scale-shift and norm kernels for CPU: [#33452](https://github.com/sgl-project/sglang/pull/33452)\r\n* [Intel][XPU][KVCanary] Enable KV Canary on Intel XPU: [#33520](https://github.com/sgl-project/sglang/pull/33520)\r\n* [Intel][XPU] Enable chunked prefill scnearios for XPU with UT: [#33804](https://github.com/sgl-project/sglang/pull/33804)\r\n* [XPU] Use torch scaled_mm for XPU block FP8 linear: [#35605](https://github.com/sgl-project/sglang/pull/35605)\r\n* [Diffusion] Fix the XPU capability gates that broke the Wan2.2 A14B DiT path: [#36825](https://github.com/sgl-project/sglang/pull/36825)\r\n* [XPU] Qwen3.8-flash-next enablement: [#37213](https://github.com/sgl-project/sglang/pull/37213)\r\n* [CPU] Upgrade torch cpu dependencies to version 2.14.: [#37939](https://github.com/sgl-project/sglang/pull/37939)\r\n* [XPU] weekly simple model enablement 2026/09/14: [#39439](https://github.com/sgl-project/sglang/pull/39439)\r\n* [CPU] Add fused_sigmod_mul_cpu operators to the Meta Muse Glimmer model.: [#39538](https://github.com/sgl-project/sglang/pull/39538)\r\n* [Intel GPU] Xpu/weekly simple model enablement 2026 09 21: [#40664](https://github.com/sgl-project/sglang/pull/40664)\r\n* [XPU] Support compressed-tensors W4A16 by reusing the torch int4pack path: [#40828](https://github.com/sgl-project/sglang/pull/40828)\r\n* [XPU] Bump sglang-kernel-xpu wheel to v0.3.0: [#41220](https://github.com/sgl-project/sglang/pull/41220)\r\n\r\n## Security\r\n\r\n* fix(openai): reject request-supplied chat_template by default: [#28135](https://github.com/sgl-project/sglang/pull/28135)\r\n* Restrict SafeUnpickler standard-library globals: [#39858](https://github.com/sgl-project/sglang/pull/39858)\r\n* fix: restrict SafeUnpickler to explicit globals: [#40259](https://github.com/sgl-project/sglang/pull/40259)\r\n## Cookbook Updates\r\n\r\n* DeepSeek-V4: a PD-disaggregated arm for B200 FP4 agentic serving with DSpark ([#40610](https://github.com/sgl-project/sglang/pull/40610)); MI355X Pro PD pairs with DSpark and UMBP, plus MegaMoE, FP8 KV attention, and breakable CUDA graph recipes: [#41024](https://github.com/sgl-project/sglang/pull/41024), [#41458](https://github.com/sgl-project/sglang/pull/41458)\r\n* GLM-5.2: MI355X MXFP4 recipes move to HIP Top-K and newer images, and the high-throughput cell adds HiCache: [#40148](https://github.com/sgl-project/sglang/pull/40148), [#40570](https://github.com/sgl-project/sglang/pull/40570), [#41109](https://github.com/sgl-project/sglang/pull/41109), [#41597](https://github.com/sgl-project/sglang/pull/41597)\r\n* GLM-5.3 and GLM-5.3-Flash: reasoning and tool-call parsers on by default, and the DCP option is temporarily removed from GLM-5.3-Flash: [#40497](https://github.com/sgl-project/sglang/pull/40497), [#40036](https://github.com/sgl-project/sglang/pull/40036)\r\n* Qwen3.5: GB200 and GB300 configs, and the MI355X HiCache recipe uses the kernel IO backend with the `page_first` layout: [#40770](https://github.com/sgl-project/sglang/pull/40770), [#39572](https://github.com/sgl-project/sglang/pull/39572)\r\n* Qwen3.8-Flash-Next: NVIDIA NVFP4 on B200, B300, and GB300: [#41046](https://github.com/sgl-project/sglang/pull/41046)\r\n* Kimi-K3: an Ascend 950PR/DT recipe: [#40575](https://github.com/sgl-project/sglang/pull/40575)\r\n* Intel XPU: the cookbook now marks XPU-supported models: [#33649](https://github.com/sgl-project/sglang/pull/33649)\r\n\r\n## Dependencies\r\n\r\n* xgrammar 0.2.1 to 0.2.7: [#39026](https://github.com/sgl-project/sglang/pull/39026)\r\n* cache-dit 1.3.0 to 1.5.1: [#40104](https://github.com/sgl-project/sglang/pull/40104)\r\n* sentencepiece pinned to 0.2.1 (0.2.2 rejects null pieces in InternVL tokenizers): [#41777](https://github.com/sgl-project/sglang/pull/41777)\r\n* sgl-eval 0.1.0 to 0.1.2: [#40620](https://github.com/sgl-project/sglang/pull/40620)\r\n* CUDA 13.4 image: `nvidia/cuda:13.4.1-cudnn-devel-ubuntu24.04` base, sgl-kernel 0.4.7, sgl-deep-gemm 0.2.0, sgl-deep-ep 0.1.2: [#40987](https://github.com/sgl-project/sglang/pull/40987); Triton 3.8 compatibility for DeepSeek-V4.1-Flash on Rubin: [#40805](https://github.com/sgl-project/sglang/pull/40805)\r\n* CPU: torch 2.14.0, torchvision 0.29.0, triton 3.8.0: [#37939](https://github.com/sgl-project/sglang/pull/37939)\r\n* XPU: sglang-kernel-xpu 0.2.0 to 0.3.0: [#41220](https://github.com/sgl-project/sglang/pull/41220)\r\n* ROCm: AITER pin moves to `acf8fdf9`: [#39965](https://github.com/sgl-project/sglang/pull/39965)\r\n* NPU: CANN 9.1.0 with Python 3.12 ([#40524](https://github.com/sgl-project/sglang/pull/40524)) and sgl-kernel-npu 2026.9.0.post5: [#40157](https://github.com/sgl-project/sglang/pull/40157)\r\n* Official releases now also publish a versioned renderer image: [#40639](https://github.com/sgl-project/sglang/pull/40639)\r\n\r\n## Breaking Changes & Upgrade Notes\r\n\r\n* **The unified radix tree uses the Rust core by default.** Configurations the Rust core does not support fall back to Python automatically; set `SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python` to force the Python core everywhere: [#39627](https://github.com/sgl-project/sglang/pull/39627)\r\n* **FlashInfer MoE fused finalize is off by default** for numerical accuracy. This costs some W4A4 MoE latency; set `SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=1` to restore the fused atomic finalize: [#40105](https://github.com/sgl-project/sglang/pull/40105)\r\n* **PD decode now defers KV release on abort by default** on the Mooncake and NIXL backends, holding pages until every prefill rank acks (30 s timeout, `SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE_TIMEOUT`). Set `SGLANG_DISAGGREGATION_DEFERRED_DECODE_KV_RELEASE=False` to restore immediate release: [#41023](https://github.com/sgl-project/sglang/pull/41023)\r\n* **Request-supplied chat templates are rejected.** A `chat_template` key inside `chat_template_kwargs` on `/v1/chat/completions` or `/v1/tokenize` now returns an error unless the server starts with `--trust-request-chat-template`; other `chat_template_kwargs` are unchanged: [#28135](https://github.com/sgl-project/sglang/pull/28135)\r\n* **Deprecated endpoints, env vars, and aliases are removed.** Use `/model_info` instead of `/get_weight_version` and `/weight_version`, `/v1/loads` instead of `/get_load`, and `POST /hicache/storage-backend/clear` instead of `/clear_hicache_storage_backend`. `SGLANG_NSA_*` aliases give way to `SGLANG_DSA_*`, `SGLANG_ENABLE_SPEC_V2` is gone, `--elastic-ep-rejoin` becomes `--elastic-ep-join-mode recover`, `--kv-cache-dtype fp4_e2m1` becomes `fp4_mx_block16`, and `python -m sglang.bench_offline_throughput` / `sglang.bench_one_batch` move to `sglang.benchmark.offline_throughput` / `sglang.benchmark.one_batch`: [#40795](https://github.com/sgl-project/sglang/pull/40795)\r\n* **Legacy radix cache implementations are removed.** The experimental C++ radix tree (`SGLANG_EXPERIMENTAL_CPP_RADIX_TREE`) is gone ([#40775](https://github.com/sgl-project/sglang/pull/40775)), and `SGLANG_ENABLE_UNIFIED_RADIX_TREE` no longer does anything, though it still prints a deprecation warning ([#40780](https://github.com/sgl-project/sglang/pull/40780)). Code importing `SWARadixCache`, `MambaRadixCache`, or `HiRadixCache` must move to `UnifiedRadixCache`: [#40313](https://github.com/sgl-project/sglang/pull/40313), [#40787](https://github.com/sgl-project/sglang/pull/40787)\r\n* **Custom logit processors must return full-shape logits.** A `CustomLogitProcessor` that returned a reduced shape such as `(1, vocab)` and relied on broadcasting now raises at decode time. The built-in processors are unaffected: [#39234](https://github.com/sgl-project/sglang/pull/39234)\r\n* **RL weight updates run inside a session.** `update_weights_from_tensor` and `update_weights_from_distributed` must be called between `begin_weight_update()` and `end_weight_update()`. `UpdateWeightsFromTensorReqInput.disable_draft_model` is replaced by `selector=\"target\"`, and `ChecksumInfo.parallelism_info` is now a list with one entry per role: [#40777](https://github.com/sgl-project/sglang/pull/40777)\r\n* **Parallel-state getters are deprecated.** They are no longer re-exported from `sglang.srt.distributed`; read groups through `get_parallel()` (for example `get_parallel().tp_group`) or import from `parallel_state` directly: [#40342](https://github.com/sgl-project/sglang/pull/40342), [#40344](https://github.com/sgl-project/sglang/pull/40344), [#40707](https://github.com/sgl-project/sglang/pull/40707)\r\n* **Ascend A2 (910b) images are no longer published.** Release and nightly images ship for A3 and 950 (A5) only, on CANN 9.1.0 with Python 3.12: [#40524](https://github.com/sgl-project/sglang/pull/40524)\r\n\r\n\r\n## New Contributors\r\n* @fechaMe made their first contribution in https://github.com/sgl-project/sglang/pull/35809\r\n* @renyiming25 made their first contribution in https://github.com/sgl-project/sglang/pull/35802\r\n* @JustinPerlman made their first contribution in https://github.com/sgl-project/sglang/pull/37839\r\n* @ironashram made their first contribution in https://github.com/sgl-project/sglang/pull/31926\r\n* @yl3469 made their first contribution in https://github.com/sgl-project/sglang/pull/34012\r\n* @yuzhouo7 made their first contribution in https://github.com/sgl-project/sglang/pull/35123\r\n* @yekerr made their first contribution in https://github.com/sgl-project/sglang/pull/39680\r\n* @Betelgeu made their first contribution in https://github.com/sgl-project/sglang/pull/38780\r\n* @maithilijoshi20 made their first contribution in https://github.com/sgl-project/sglang/pull/35260\r\n* @gurpreet-dhami made their first contribution in https://github.com/sgl-project/sglang/pull/35573\r\n* @hmalgewatta made their first contribution in https://github.com/sgl-project/sglang/pull/37547\r\n* @vvinniev34 made their first contribution in https://github.com/sgl-project/sglang/pull/38750\r\n* @mindlab-bot made their first contribution in https://github.com/sgl-project/sglang/pull/38604\r\n* @jmydurant made their first contribution in https://github.com/sgl-project/sglang/pull/38792\r\n* @dmtri35 made their first contribution in https://github.com/sgl-project/sglang/pull/40184\r\n* @salexspb made their first contribution in https://github.com/sgl-project/sglang/pull/39837\r\n* @chaijiacheng888 made their first contribution in https://github.com/sgl-project/sglang/pull/38996\r\n* @yuhanchan made their first contribution in https://github.com/sgl-project/sglang/pull/38740\r\n* @chenyang08056032 made their first contribution in https://github.com/sgl-project/sglang/pull/38339\r\n* @YZY00Raiser made their first contribution in https://github.com/sgl-project/sglang/pull/40392\r\n* @harmya made their first contribution in https://github.com/sgl-project/sglang/pull/40217\r\n* @ollybbmonster made their first contribution in https://github.com/sgl-project/sglang/pull/40448\r\n* @divyvasal made their first contribution in https://github.com/sgl-project/sglang/pull/38932\r\n* @xenshinu made their first contribution in https://github.com/sgl-project/sglang/pull/40201\r\n* @Amrutha-M05 made their first contribution in https://github.com/sgl-project/sglang/pull/32792\r\n* @skyler-apdx made their first contribution in https://github.com/sgl-project/sglang/pull/38786\r\n* @kpjeeja made their first contribution in https://github.com/sgl-project/sglang/pull/28652\r\n* @GungnirAP made their first contribution in https://github.com/sgl-project/sglang/pull/29189\r\n* @RuibinCheung made their first contribution in https://github.com/sgl-project/sglang/pull/39775\r\n* @avalliappan-nvidia made their first contribution in https://github.com/sgl-project/sglang/pull/32673\r\n* @zhou-yuhan made their first contribution in https://github.com/sgl-project/sglang/pull/27265\r\n* @zhujianwei-ops made their first contribution in https://github.com/sgl-project/sglang/pull/36120\r\n* @jshn9515 made their first contribution in https://github.com/sgl-project/sglang/pull/40647\r\n* @surajm20061998 made their first contribution in https://github.com/sgl-project/sglang/pull/39312\r\n* @tomjen12 made their first contribution in https://github.com/sgl-project/sglang/pull/39525\r\n* @siliangchen-amd made their first contribution in https://github.com/sgl-project/sglang/pull/40557\r\n* @emre570 made their first contribution in https://github.com/sgl-project/sglang/pull/40175\r\n* @wookjeHan made their first contribution in https://github.com/sgl-project/sglang/pull/40309\r\n* @peilii made their first contribution in https://github.com/sgl-project/sglang/pull/40260\r\n* @jshanson7 made their first contribution in https://github.com/sgl-project/sglang/pull/40468\r\n* @MatsueYu made their first contribution in https://github.com/sgl-project/sglang/pull/35958\r\n* @Apexsf made their first contribution in https://github.com/sgl-project/sglang/pull/39929\r\n* @CjhHa1 made their first contribution in https://github.com/sgl-project/sglang/pull/34417\r\n* @chunxiaozheng made their first contribution in https://github.com/sgl-project/sglang/pull/38652\r\n* @weimin023 made their first contribution in https://github.com/sgl-project/sglang/pull/41150\r\n* @TarzanZhao made their first contribution in https://github.com/sgl-project/sglang/pull/41223\r\n* @arieller made their first contribution in https://github.com/sgl-project/sglang/pull/41292\r\n* @nvcnvn made their first contribution in https://github.com/sgl-project/sglang/pull/39889\r\n* @cuzmi made their first contribution in https://github.com/sgl-project/sglang/pull/41266\r\n* @mrusanovsky made their first contribution in https://github.com/sgl-project/sglang/pull/37462\r\n* @yonigozlan made their first contribution in https://github.com/sgl-project/sglang/pull/40477\r\n* @floatlibai made their first contribution in https://github.com/sgl-project/sglang/pull/39060\r\n* @Denji-kk made their first contribution in https://github.com/sgl-project/sglang/pull/39539\r\n* @Hrithvik-Alex made their first contribution in https://github.com/sgl-project/sglang/pull/41588\r\n* @zldrobit made their first contribution in https://github.com/sgl-project/sglang/pull/41059\r\n\r\n**Full Changelog**: https://github.com/sgl-project/sglang/compare/v0.5.20...v0.5.21","reactions":{"url":"https://api.github.com/repos/sgl-project/sglang/releases/401467591/reactions","total_count":1,"+1":0,"-1":0,"laugh":0,"hooray":1,"confused":0,"heart":0,"rocket":0,"eyes":0},"mentions_count":55}