chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
7,024 Commits 1 Branch 0 Tags
af6bcadcf723b1924d8a6b44eeb920df32151cbf
Commit Graph

625 Commits

Author SHA1 Message Date
Yuan Luo
af6bcadcf7 [VLM] Support Piecewise CUDA Graph for Qwen2.5-VL (#13055) 
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
Co-authored-by: Yuhao Yang <yhyang201@gmail.com>
 2025-11-20 10:23:44 +08:00
Xinyuan Tong
a355794905 Expend compatibility check for all quantized MoE models (#13465) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-11-20 01:24:27 +08:00
Chen Haozhe
6c2e5fcd91 [feat][Ascend][Mindspore]: support model-impl of mindspore (#9234) 2025-11-19 09:17:47 +08:00
Zijian Zhang
aa8ecbda7a model: support JetVLM (#13289) 2025-11-18 12:02:03 +08:00
Carlo Mussolini
df56139226 Adding user defined hooks support (#13217) 2025-11-17 23:07:37 +08:00
lixiaolx
d368c7451a (1/n)support context parallel with deepseekv3.2-DSA (#12065) 2025-11-16 20:12:25 -08:00
Lifu Huang
254f62d879 Support spec decoding when LoRA is applied to target model (#12903) 2025-11-16 13:20:23 -08:00
Zilin Zhu
9509c4ccd3 [RL] enable offloading hybrid linear attn model (#13336) 2025-11-16 17:12:52 +08:00
kyleliang-nv
597d416070 [feature] Add layerwise NVTX support (#11870) 2025-11-15 19:20:56 -08:00
Even Zhou
2aec8b6e1b [Feature] Spec-Overlap supporting DP-ATTN; PD-Disaggregation; npugraph mode (#12443) 2025-11-15 21:51:07 +08:00
Yongfei Xu
d91b16eb16 Opt tp: tp attn support tp reduce scattered input (#10568) 2025-11-15 02:08:12 -08:00
Zilin Zhu
56fc483073 [RL] support only do cpu backup on draft model (#13318) 2025-11-15 16:31:57 +08:00
Ho-Ren (Jack) Chuang
6d5e16fb1c feat: Add FP4 (E2M1) KV Cache Support for MHA (#12612) 
Signed-off-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
Co-authored-by: Yichen Wang <yichen.wang@bytedance.com>
 2025-11-14 22:31:35 -08:00
Yuwei An
eae59b337e Piecewise Cuda Graph Support for gpt-oss model (#13045) 
Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>
 2025-11-15 09:28:00 +08:00
Lianmin Zheng
7a2254b260 [Auto Sync] Update backend.py, forward_batch_info.py, piece... (20251113) (#13221) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-11-13 18:54:11 -08:00
ykcombat
dd192a55f4 [Feature] Enable CUDA graph for PD-Multiplexing. (#11595) 2025-11-14 03:39:40 +08:00
ronnie_zheng
5324f37ab3 [Ascend]adapt enable-profile-cuda-graph for NPU (#12617) 
Co-authored-by: Leo920320 <1530593013@qq.com>
Co-authored-by: liupengcheng <6531487+leo920320@user.noreply.gitee.com>
 2025-11-12 19:57:24 +08:00
Xiaoyu Zhang
9caca6a45c [PieceWise CUDA Graph] Support awq/gptq model in piecewise cudagraph (#12518) 2025-11-11 11:56:15 +08:00
Xuchun Shang
ac19ce7efb [PP] put pp assert in model runner (#12934) 
Signed-off-by: Xuchun Shang <xuchun.shang@gmail.com>
 2025-11-10 14:58:19 +08:00
Atream
ddd1440d0f Refactor KTransformers heterogeneous compute with unified GPU-quantization backend (#12834) 
Co-authored-by: Chen Hongtao <56470055+chenht2022@users.noreply.github.com>
Co-authored-by: chenht2022 <cht22@mails.tsinghua.edu.cn>
Co-authored-by: skqliao <skqliao@gmail.com>
Co-authored-by: ovowei <1913953267@qq.com>
 2025-11-09 21:06:32 -08:00
Liangsheng Yin
61c7fe7aed Minor code cleanup / improvement for PREBUILT_EXTEND mode (#12948) 2025-11-10 11:59:54 +08:00
Ke Bao
db24d34603 Support piecewise cuda graph for MLA (#11812) 2025-11-10 09:13:48 +08:00
Liangsheng Yin
4f65a64666 Refactor / Unify event loop across PD-Disagg, Overlap, DP-Attn cases (#12839) 
Co-authored-by: cctry <17473714+cctry@users.noreply.github.com>
 2025-11-10 00:42:50 +08:00
Zijian Zhang
3633f8b0cf Add Jet-Nemotron (#12448) 2025-11-09 01:32:47 -08:00
Chen1022
c67fce160e Fix missing dp_max_padding argument in set_dp_buffer_len (#12812) 2025-11-07 21:27:43 +08:00
hlu1
bef37d6de8 [Deepseek V3.2] Only skip Indexer logits computation when is_extend_without_speculative (#12816) 
Signed-off-by: Hao Lu <14827759+hlu1@users.noreply.github.com>
 2025-11-07 01:34:04 -08:00
Nicolas Castet
2340798353 Register allgather/reducescatter buffers with symm memory (#12572) 2025-11-04 17:11:36 -08:00
Baizhou Zhang
d22d044734 Revert "Enable memory saver for hybrid model" (#12648) 2025-11-04 16:22:06 -08:00
Junrong Lin
173e0f704f Enable memory saver for hybrid model (#11974) 2025-11-04 14:55:26 +08:00
Yueyang Pan
952fbe47cb fix: fix the bug which leads qwen2_5_vl to crash with mixed_chunk (#11330) 
Signed-off-by: PanJason <pyyjason@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Yuan Luo <yuan.luo@hotmail.com>
 2025-11-03 09:26:03 -08:00
harrisonlimh
ffb32a8548 Conditionally recapture cuda graph after model weight update from disk (#12060) 2025-11-03 05:51:27 -08:00
Yuan Luo
819fc59123 Add prefix for torch symm mem (#12506) 
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
 2025-11-02 11:23:05 -08:00
Ho-Ren (Jack) Chuang
76196b3cbf feat: Add FP4 (E2M1) KV Cache Support with Quantization Utilities for MLA (#10078) 
Signed-off-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
Co-authored-by: Yichen Wang <yichen.wang@bytedance.com>
 2025-11-01 22:24:58 -07:00
Xinyuan Tong
0bfe1d145c fa3 & trtllm_mha spec overlap (#11874) 2025-10-31 17:38:13 -07:00
Ke Bao
a4bf5c6ad2 Support Kimi Linear (#12469) 
Co-authored-by: yizhang2077 <1109276519@qq.com>
 2025-10-31 14:03:35 -07:00
ykcombat
41efcaeb45 [Feature] PD-Multiplexing Context and Scheduler, lazy import spatial. (#12275) 2025-11-01 00:40:01 +08:00
Yuhong Guo
ab95d35fcb feat: Add Non-intrusive Tensor Dumping for Model Inference (#10566) 2025-10-31 12:04:48 +08:00
Even Zhou
ce6b17c0f9 [Feature] Support DeepSeek MTP on NPU (#11897) 
Co-authored-by: liupeng374 <liupeng374@huawei.com>
 2025-10-30 21:56:56 +08:00
Makcum888e
8e2ac2e628 [NPU] fix pp_size>1 (#12195) 2025-10-30 11:18:36 +08:00
Yingchun Lai
ec92b0cefe EPLB: prefer to use physical experts in the same gpu or node (#10874) 2025-10-28 21:01:11 -07:00
Yineng Zhang
64cf868eba chore: cleanup quant deps (#12268) 2025-10-28 02:03:57 -07:00
Yineng Zhang
ea39952797 Revert "[Feature] PD-Multiplexing Context and Scheduler." (#12267) 2025-10-28 02:00:37 -07:00
Xuchun Shang
a1f2dc90e4 [Bug fix] [PP] fix wrong dtype for quantified model (#12247) 
Signed-off-by: Xuchun Shang <xuchun.shang@gmail.com>
 2025-10-28 01:27:24 -07:00
fzyzcjy
691c8534cf Support releasing CUDA graph memory when paused (#7873) 
Co-authored-by: ryang-max <y1cunhui.yang@gmail.com>
Co-authored-by: ryang <38470282+ryang-max@users.noreply.github.com>
 2025-10-28 14:40:50 +08:00
satyamk7054
9fc3e8aac7 Add support for Matryoshka embeddings (#126) (#11142) 
Co-authored-by: Satyam Kumar <satyamk@linkedin.com>
 2025-10-28 02:49:36 +08:00
ykcombat
05ad28f25e [Feature] PD-Multiplexing Context and Scheduler. (#11592) 2025-10-28 00:54:43 +08:00
ash-sigh
0b3b3e9a69 transfer mrope_position_delta to device when first running (#11047) 2025-10-26 13:06:09 -07:00
zyksir
96a5a949f6 [Fix] fix allreduce bug in Piecewise Graph (#12106) 2025-10-26 21:15:48 +08:00
Kai-Hsun Chen
6371f7af27 [quantization] AWQ Marlin doesn't work when dtype is bfloat16 (#11494) 
Signed-off-by: Kai-Hsun Chen <khchen@x.ai>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
 2025-10-26 15:49:45 +08:00
Lianmin Zheng
7b36c47b3b Clean up attention backend selection code & Other minor rename (#12136) 2025-10-25 23:50:12 -07:00