chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
11,139 Commits 1 Branch 0 Tags
cd4412a4b8ff83a943046ffecff8b76143606197
Commit Graph
1280 Commits
Author SHA1 Message Date
laoyao0822 f8fca72635 Expand prefill CP KV capacity by sharding persistent NSA KV 
Prefill CP previously replicated NSA/MLA persistent KV on every CP rank, so CP8 consumed eight copies of KV memory while exposing only one rank of logical cache capacity. This change splits logical KV locs from per-rank physical storage, shards MLA latent KV and NSA index K/scale by deterministic page ownership, and keeps existing NSA attention kernels working through a full-view runtime materialization layer.

Mooncake PD transfer now sends each prefill CP rank's owned physical pages with explicit logical page positions so non-CP decode can reconstruct full-layout KV. The implementation is guarded by an explicit server flag and startup checks, and the design documentation records the implemented scope, debug environment, and Phase 3 boundary.

Constraint: Phase 2 must preserve existing NSA attention/index kernels via runtime full-view materialization
Constraint: Decode side remains non-CP and receives full KV through Mooncake
Rejected: Shard-aware NSA attention in this change | belongs to Phase 3 because it requires distributed topk/softmax/output contracts
Rejected: Request-contiguous CP ownership | unstable under chunked prefill and tied to attention split mode
Confidence: medium
Scope-risk: broad
Directive: Do not enable round-robin CP shared KV without wiring runtime materialization/PD transfer contracts for that split mode
Directive: Keep SGLANG_DEBUG_CP_SHARED_KV disabled for perf measurements; it intentionally enables CUDA-syncing diagnostics
Tested: Remote py_compile for shared-KV touched Python files in g0034 container
Tested: Remote pytest selected cp_shared/shared_kv/nsa suite: 37 passed, 34 deselected
Not-tested: Full GLM5 multi-node throughput/regression run after final doc update
Not-tested: Phase 3 shard-aware runtime, round-robin CP mode, and non-Mooncake PD backends
 2026-04-26 04:11:17 +08:00
laoyao0822 76c830dda8 docs(cp): plan NSA prefill CP shared KV phase2 2026-04-24 16:31:03 +08:00
laoyao0822 7cc4e2dee8 docs(cp): design NSA prefill CP shared KV phase2 2026-04-24 06:06:31 +08:00
laoyao0822 e7a8c4b52f docs(cp): add NSA prefill CP dual all-gather explanation and Phase 1 plan 
- nsa_prefill_cp_all_gather.md: explains why there are two all-gathers
  in current NSA prefill CP (per-layer KV/index gather vs tail output
  hidden gather), with data flow diagrams and involved file lists
- nsa_prefill_cp_phase1_narrow_output_collection.md: detailed Phase 1
  design plan covering batch-level eligibility, fallback conditions,
  affected modules, KV transfer impact, risks, and test matrix
 2026-04-22 04:55:15 +08:00
laoyao0822 101100e25b feat(eplb): add eplb warmup-only mode 
Freeze expert layout after the initial EPLB warmup rebalances so heavy workloads avoid recurring rebalance memory spikes and OOMs.
 2026-04-08 06:46:26 +08:00
kpham-sglandClaude Opus 4.6 bc4aaab6a1 [Spec][Ngram] 2/N: Rename branch length to max trie depth (#21181) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-22 23:35:25 -07:00
Xiaoyu Zhang 766d225fcc Add SGLang CUDA crash API logging inspired by FlashInfer (#20910) 2026-03-22 16:39:40 +08:00
kpham-sgl 6d160b42bb [Spec][Ngram] 1/N: Reference based Speculative Decoding refactor (#20393) 2026-03-22 00:55:10 -07:00
RabinovichYixiao Zengzhaochenyang20
 		3798a8c88d docs: add out-of-tree model integration guide (#21050) 
Co-authored-by: Yixiao Zeng <yixiao.zeng@xiaopeng.com>
Co-authored-by: zhaochenyang20 <zhaochen20@outlook.com>
 2026-03-20 20:07:46 -07:00
Lianmin Zheng 2d7a262ca3 ci: rename 1/2-gpu-runner labels to 1/2-gpu-h100 (#21008) 2026-03-20 06:04:15 -07:00
Cao E 274581fb77 Add support for more batch sizes in cpu_graph_runner (#13881) 2026-03-19 09:50:56 -07:00
blzhengandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> cbea9f6909 [CPU] improve numa memory binding (#19666) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-03-18 22:15:50 -07:00
Xinyuan TongKangyan-ZhouAlison ShaoMick
 		d1e95af282 Upgrade transformers==5.3.0 (#17784) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Kangyan-Zhou <zky314343421@gmail.com>
Co-authored-by: Alison Shao <alisonshao@mac.lan>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-03-18 13:50:43 -07:00
ishandhananiClaude Opus 4.6Ishan Dhanani
 		8f0f36c64b [1/2] Add ModelExpress coordination for remote instance weight loading - matching TP (#19920) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ishan Dhanani <ishan@dhanani.dev>
 2026-03-18 13:38:32 -07:00
Lianmin ZhengandClaude Opus 4.6 46a392658e Refine RL & Post-Training description in README (#20877) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-18 12:43:42 -07:00
Xiaoyu Zhang 20a23e3173 [SKILL] Refine kernel authoring docs and validate add-jit-kernel / add-sgl-kernel end to end with Codex (#20867) 2026-03-18 23:00:33 +08:00
Ke Bao c42da50289 Update test guide to contribution guide (#20805) 2026-03-18 13:25:16 +08:00
Qiaolin Yu c5d2528bff Revert "[AMD][MORI] Fix MTP crash with FP4/FP8 dispatch and add NEXTN dispatch env vars." (#20797) 2026-03-17 17:28:09 -07:00
Kangyan-Zhou 3d8fc9a0ca Revert "[Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api" (#20792) 2026-03-17 11:59:02 -07:00
Shu Wang d35fea1b2b [Nvidia] Add trtllm mnnvl allreduce with unified flashinfer allreduce fusion api (#12787) 2026-03-17 10:02:45 -07:00
Duyi-Wang 385a35bd11 [AMD][MORI] Fix MTP crash with FP4/FP8 dispatch and add NEXTN dispatch env vars. (#20647) 2026-03-17 01:13:42 -07:00
DefTruth 025691cd9e [diffusion] chore: bump up cache-dit & support quant for diffusers backend (#20361) 2026-03-17 12:51:31 +08:00
amote-i 210d0fbaef Update ascend docs (#20674) 2026-03-16 20:14:26 -07:00
Xiaoyu ZhangandBaizhou Zhang 15097c5c3b Release sglang kernel 0.4.0 (#20440) 
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2026-03-16 20:34:58 +08:00
amote-i da1793f63a update ascend feature docs (#20506) 2026-03-15 20:09:20 -07:00
AnonTokyo e9fae69e5f docs: align environment variable reference with environ defaults (#20419) 2026-03-15 18:07:29 -07:00
Teng Ma 7c498a6538 [DOC] add documents for encoder global mm cache (#20636) 2026-03-15 16:44:21 -07:00
Mook 23c191afb6 fix(docs): correct quantization documentation (#20301) (#20619) 2026-03-15 12:33:12 -04:00
Liangsheng Yin fc7f9c1de7 Rename --stream-output to --incremental-streaming-output (#20614) 2026-03-14 23:22:33 -07:00
Matt Van HornMatt Van HornClaude Opus 4.6gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		6c5bf53a36 [Doc] Clarify that --chat-template is required for Qwen3-Reranker (#20596) 
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-03-14 23:43:48 +00:00
Baizhou Zhang 39008955ff Revert "[AMD][MORI] Fix MTP crash with FP4/FP8 dispatch and add NEXTN dispatch env vars." (#20602) 2026-03-14 12:12:42 -07:00
Duyi-Wang 0eea80bc00 [AMD][MORI] Fix MTP crash with FP4/FP8 dispatch and add NEXTN dispatch env vars. (#20453) 2026-03-13 14:03:17 -07:00
StonyPortqiuxuan.lzwgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		d4e68ead1d [quant] Ignore FP8 quantization layers (#20340) 
Co-authored-by: qiuxuan.lzw <qiuxuan.lzw@alibaba-inc.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-03-13 13:59:39 +08:00
Mook 2720ea2667 [Typo] Fix H200 doc links pointing to H20 section in deepseek_v3.md (#20383) 2026-03-11 13:35:20 -07:00
Matt Van HornMatt Van HornClaude Opus 4.6
 		d093e70067 [Doc] Add DSA/NSA attention backend to support matrix (#20326) 
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-11 13:40:35 -04:00
R0CKSTAR dae5c6cadf [diffusion] doc: add Moore Threads as a supported vendor (#20146) 
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com>
 2026-03-11 10:15:15 -07:00
Thomas 05e40922b3 [Doc] Fix wrong link and cmd description (#20365) 2026-03-11 11:30:45 -04:00
Артем Савкин ed42af99a9 [NPU] [Quantization] w4a4 MoE layer support (#18924) 2026-03-11 16:52:35 +03:00
Yoray ZackandBarak Biber 9991debde3 [Feature] Integrate Elastic NIXL-EP into SGLang (#19248) 
Signed-off-by: Barak Biber <bbiber@nvidia.com>
Signed-off-by: Yoray Zack <yorayz@nvidia.com>
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Co-authored-by: Barak Biber <bbiber@nvidia.com>
 2026-03-11 17:37:43 +08:00
Xuhao ZhangwangtianceTiance Wanggemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>ronnie_zheng
 		57b093dc34 [NPU]MindSpore backend support eagle3 (#17098) 
Co-authored-by: wangtiance <tiancew@qq.com>
Co-authored-by: Tiance Wang <wangtiance@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: ronnie_zheng <zl19940307@163.com>
 2026-03-11 09:11:19 +03:00
Aleksi Vesanto c8bbe5010a [diffusion] feat: add AITER Sage attention backend (#20178) 2026-03-11 12:17:45 +08:00
Polisetty V R K Jyothendra Varma b2dd104ade [Intel GPU] Upgrade pytorch xpu version to 2.10 (#20254) 
Signed-off-by: P V R K Jyothendra Varma <polisetty.v.r.k.jyothendra.varma@intel.com>
 2026-03-10 18:47:25 -07:00
Liangsheng Yin 50953aea8d [Scheduler] Unify idle checks into is_fully_idle() and fix weight update test (#20296) 2026-03-10 17:50:23 -07:00
Ziang Li 76ee4bb98c [FlashInfer v0.6.4] [RL] Integrate FlashInfer mxfp8 gemm, MoE, and routed MoE (#19537) 2026-03-10 15:37:57 -07:00
R0CKSTARandMick db97f193b7 [diffusion][llm] macOS support (#19549) 
Signed-off-by: Xiaodong Ye <yeahdongcn@gmail.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-03-10 13:11:07 -07:00
shuwenn 5a11ae19c1 [CI] fix: notebook ci often OOM (#20199) 2026-03-09 22:32:41 -07:00
Brayden Zhong 591e61245a [Doc] Add smal table for GEMM backends (#20213) 2026-03-09 22:19:57 -07:00
YEJIN KIM 0fd9a57d80 [Doc] Verify and Modify some attention backend specs (#20210) 2026-03-09 23:05:52 +00:00
Baizhou Zhang be63f982b7 [V32/GLM5] Control the threshold of applying dense attention with an environ (#20062) 2026-03-09 14:36:10 -07:00
AMD-yanfeiwangandDuyi-Wang f0153ad225 [AMD][Feature] support fp4 dispatch and fp8 combine in moriep (#19757) 
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
 2026-03-09 12:52:05 -07:00