chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
10,885 Commits 1 Branch 0 Tags
7cc4e2dee8fe4d48d7c8c0f50b6237fecae6f230
Commit Graph

7204 Commits

Author SHA1 Message Date
laoyao0822
bc9a9f128b perf(cp): add narrow output collection for NSA prefill CP 
Replace full hidden all-gather at prefill tail with per-request last-token
hidden collection, reducing communication from total_tokens x hidden_size
to cp_size x bs x hidden_size for both in-seq-split and round-robin modes.

- nsa/utils.py: add cp_collect_last_token_hidden() with mode-specific
  narrow collection helpers that only gather the last token hidden
- deepseek_v2.py: add _should_use_narrow_output_path() gate on
  DeepseekV2Model, fallback to full gather for EAGLE/return_logprob/
  capture_hidden batches
- logits_processor.py: add _is_compact_hidden_states() to bypass
  _get_pruned_states() when hidden is already compact
 2026-04-22 04:55:06 +08:00
laoyao0822
7403f14511 feat(kernel): add quanted silu fusion kernel 2026-04-19 04:23:53 +08:00
wxiwnd
91facd5bb2 perf(disaggregation): fast-path common decode result processing 2026-04-19 04:23:48 +08:00
wxiwnd
14c40f4e7d perf(disaggregation): slice decode waiting queue for prebuilt batches 2026-04-19 04:23:48 +08:00
wxiwnd
6237271fc9 perf(disaggregation): compact decode queues in one pass 2026-04-19 04:23:48 +08:00
wxiwnd
9470d6d8dc perf(disaggregation): compact prefill bootstrap queue in one pass 2026-04-19 04:23:48 +08:00
wxiwnd
09acec1908 perf(disaggregation): cache static buffer metadata 2026-04-19 04:23:48 +08:00
wxiwnd
53a04a9a97 perf(disaggregation): reuse req pool freelists and alloc_extend tensors 2026-04-19 04:23:48 +08:00
laoyao0822
a371eacd86 fix(deepgemm): add nsa init in deepgemm init 2026-04-19 04:23:48 +08:00
laoyao0822
d9b6e90b35 fix(dp_ctrl): fix imbalanced DP rank assignment 
1. feat:update total tokens immediately

fea:record request in da controller local
 2026-04-09 01:37:13 +08:00
laoyao0822
cc11cac77c fix(b_overlap): single batch overlap dim not match 2026-04-09 01:37:13 +08:00
laoyao0822
6efa9b4350 fix(disaggregation): check_status not found case crackdown 2026-04-09 01:37:13 +08:00
laoyao0822
ef607c35c9 fix(spec_v2): support EAGLE with spec_v2 
1. add pass for spec_v2 in base_attn

2. fix: EAGLE with spec_v2 overlap Grammar accept_token failed

Signed-off-by: wxiwnd <wxiwnd@outlook.com>
 2026-04-09 01:37:13 +08:00
laoyao0822
101100e25b feat(eplb): add eplb warmup-only mode 
Freeze expert layout after the initial EPLB warmup rebalances so heavy workloads avoid recurring rebalance memory spikes and OOMs.
 2026-04-08 06:46:26 +08:00
laoyao0822
d5ec233834 feat(mtp): add tbo mtp support 2026-04-08 06:45:27 +08:00
laoyao0822
1c04d6078b feat(token_limit): limit input token < 131072 
4.max_tokens=131072，input＜131072时不报错
5. 输入长度超过131072时，报错，且报错内容含有以下字符串，以便被网关识别
"is longer than the model's context length"
 2026-04-08 06:34:03 +08:00
laoyao0822
7c524ccacc feat(tool_call): changed return type of tool_call 
Update the tool call response format so that all non-first chunks in a tool call have type: null.
 2026-04-08 06:27:12 +08:00
laoyao0822
9b5ecc8c1f feat(drain): graceful shutdown and admin drain 
feat: add admin drain to sglang
 2026-04-08 06:13:42 +08:00
laoyao0822
150d4db41d feat(scheduler): support scheduler recv skip in dp 2026-04-08 06:10:42 +08:00
leavelet
663459d2ef feat(deepgemm): deepgemm jit and init in model_runner 
1. fix and tune deepgemm
 2026-04-08 06:05:06 +08:00
leavelet
2ae0237d3e fix(deepep): tuning and enhance deepep on single node 
feat(deepep): enhance single node deepep
 2026-04-08 05:36:31 +08:00
leavelet
e72880b467 feat(eplb): add eplb warmup 2026-04-08 05:21:54 +08:00
XDaoHong
2d288ba8c9 [Bugfix] fix npu get kv_item_lens in PD separation when use ASCEND_US… (#15852) 
Co-authored-by: ZhengdQin <zhengdqin@gmail.com>
 2026-03-23 15:56:47 +08:00
kpham-sgl
59cb9a9da6 [Spec][Ngram] 3/N: Fix synchronization issues in Ngram.cpp (#21186) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-23 00:37:07 -07:00
Lianmin Zheng
814202704b ci: unify PR test suite naming (#21187) 2026-03-23 00:18:45 -07:00
yudian0504
3d312643b9 [BUGFIX] Fix CP residual size mismatch crash when tp_size == attn_cp_size (#21170) 2026-03-23 00:12:58 -07:00
Lianmin Zheng
7757a9ddd0 ci: remove IS_BLACKWELL env var; auto-detect Blackwell (#21118) 2026-03-22 23:44:48 -07:00
kpham-sgl
bc4aaab6a1 [Spec][Ngram] 2/N: Rename branch length to max trie depth (#21181) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-22 23:35:25 -07:00
Cheng Wan
d6b12c401c Revert "[bugfix] Fix PPMissingLayer AttributeError when Using PP" (#21189) 2026-03-22 23:28:36 -07:00
Zhiqiang Xie
13f4f010d8 HiSparse for Sparse Attention (#20343) 2026-03-22 23:09:31 -07:00
Lianmin Zheng
7050011dee Enable JIT clamp_position and resolve_future_token_ids on ROCm (#21116) 2026-03-22 22:33:54 -07:00
Yuhao Yang
32a85ef128 [diffusion] CI: auto-skip diffusion tests when required pipeline class is missing from diffusers (#21139) 2026-03-23 12:15:21 +08:00
Mohammad Miadh Angkad
d8a5b1dbaf [Bugfix] Work around FlashInfer unified transport issue on GB (#20039) 2026-03-22 21:10:25 -07:00
Xiaoyu Zhang
a94d67d44b [SKILL] fix(bench): Support model-specific DenoisingStage variants in… (#21137) 2026-03-23 12:08:00 +08:00
fanghao
2b47bd3a34 [Bug Fix] Fix non-streaming request abort failure when --enable-metrics is enabled (#20625) 2026-03-22 19:58:49 -07:00
yuumn
889e8489e9 [diffusion] model: support FireRed-Image-Edit (#20862) 
Co-authored-by: yuumn <1010797597@qqã.com>
 2026-03-23 10:27:07 +08:00
Cishoon
999bad5aba Fix VRAM leak in overlap scheduling with structured output (#20640) (#20697) 2026-03-22 17:07:39 -07:00
Yilong Zhao
343998865a perf: pad max-num-requests in decode cuda graph for higher coverage (#20978) 2026-03-22 17:06:16 -07:00
Ziang Li
ce0541404f [FlashInfer v0.6.6][RL] Support fp8-last-n-bf16 RL for flashinfer_trtllm_routed moe backend (#20214) 2026-03-22 11:17:01 -07:00
Xiaoyu Zhang
c1fe5de69c [Diffusion] Clean up diffusion Triton kernels and modernize custom op registration (#21122) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-03-22 22:38:57 +08:00
Xiaoyu Zhang
766d225fcc Add SGLang CUDA crash API logging inspired by FlashInfer (#20910) 2026-03-22 16:39:40 +08:00
Shunkangz
bb737d7a82 Support Qwen3 MoE context parallel (#18233) 
Co-authored-by: Shunkang <182541032+Shunkangz@users.noreply.github.co>
Co-authored-by: Jiying Dong <87510204+dongjiyingdjy@users.noreply.github.com>
 2026-03-22 01:27:20 -07:00
kpham-sgl
6d160b42bb [Spec][Ngram] 1/N: Reference based Speculative Decoding refactor (#20393) 2026-03-22 00:55:10 -07:00
Xiaoyu Zhang
1b65c0d259 [Diffusion] Fix torch.compile RMSNorm fallback for Z-Image (#20962) 
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-03-22 15:38:22 +08:00
Bowen Li
3bc595acbc [FlashAttn] Add fused triton kernel for normal_decode_set_metadata (#20778) 
Co-authored-by: kinza99 <dh18324568312@163.com>
 2026-03-22 15:12:29 +08:00
Mick
f7fc2c8592 [diffusion] fix: fix accuracy for some image models (#20679) 2026-03-22 15:11:57 +08:00
shuwenn
2fba2bdad1 refactor: Remove dead code from utils/common.py (#20668) 2026-03-21 21:54:17 -07:00
Lianmin Zheng
76e4a8662c Replace clamp_position with JIT kernel + platform dispatch (#20999) 
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-03-21 21:26:26 -07:00
Changyi Yang
c1794e2944 [diffusion] fix: fix Sana corrupted output by removing spurious QK norm layers (#20656) 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
 2026-03-22 12:06:49 +08:00
Yuhao Yang
c32e35a2a5 [diffusion] CI: fix picklingerror for diffusion models using diffusers backend (#20854) 2026-03-22 11:51:03 +08:00