chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
5,548 Commits 1 Branch 0 Tags
05a352665479ef8ea308d96b9b614c76c80740c0
Commit Graph

159 Commits

Author SHA1 Message Date
Xiaoyu Zhang
05a3526654 Restruct gpu_memory_settings in a unify function and relax max_cuda_graph_bs (#10372) 
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: sglang-bot <sglangbot@gmail.com>
 2025-09-26 15:10:49 -07:00
feng397
944f1ea0ec fix capture_bs when speculative decoding enabled (#10730) 2025-09-22 00:43:12 -07:00
Zhihao Zhang
e7bc600304 [Feature] Speculative decoding support lookahead (#9873) 
Co-authored-by: a4zhangfei <a4zhangfei@qq.com>
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
 2025-09-18 16:42:41 -07:00
brayden-hai
67073dde85 Garbage collector regression in the online server (#10621) 2025-09-18 11:52:35 -07:00
Qiaolin Yu
8cda5a622c Standalone speculative decoding (#10090) 2025-09-07 20:55:09 -07:00
Lzhang-hub
37d83c6e6d Qwen2.5-VL eagle3 infer (#8801) 2025-09-07 20:44:34 -07:00
miter
a0b22f2f17 remove redundant rank0_log function. (#9560) 
Co-authored-by: linhuang <linhuang@ruijie.com.cn>
 2025-08-24 23:17:55 -07:00
VDV1985
2c4b4b786b [feature] Ascend NPU graph support (#9399) 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
Co-authored-by: yezhifeng (D) <y00897525@china.huawei.com>
Co-authored-by: anon189Ty <Stari_Falcon@outlook.com>
Co-authored-by: Maksim <makcum888e@mail.ru>
Co-authored-by: ssshinigami <44640852+ssshinigami@users.noreply.github.com>
 2025-08-20 21:13:27 -07:00
Even Zhou
de2dd73831 Revert "[feature] Rework Ascend NPU graph support" (#9385) 2025-08-20 00:35:10 -07:00
Even Zhou
3680d6f88b [feature] Rework Ascend NPU graph support (#9350) 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
Co-authored-by: yezhifeng (D) <y00897525@china.huawei.com>
Co-authored-by: anon189Ty <Stari_Falcon@outlook.com>
Co-authored-by: Maksim <makcum888e@mail.ru>
Co-authored-by: ssshinigami <44640852+ssshinigami@users.noreply.github.com>
 2025-08-19 20:32:27 -07:00
Even Zhou
f4fafacc5d Revert "[feature] Ascend NPU graph support (#8027)" (#9348) 2025-08-19 10:11:23 -07:00
VDV1985
94371dbbd6 [feature] Ascend NPU graph support (#8027) 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
Co-authored-by: yezhifeng (D) <y00897525@china.huawei.com>
Co-authored-by: anon189Ty <Stari_Falcon@outlook.com>
Co-authored-by: Maksim <makcum888e@mail.ru>
Co-authored-by: ssshinigami <44640852+ssshinigami@users.noreply.github.com>
 2025-08-16 17:25:17 -07:00
Cheng Wan
b87aacb5c5 [DP Attention] Refactor: adding some utility functions (#9136) 2025-08-13 21:08:06 -07:00
Cheng Wan
5f5b3b2449 [5/n] DP Enhancement: Correct num_token_non_padded (#9107) 2025-08-12 12:23:46 -07:00
DarkSharpness
7ba5ad5766 [Fix] Fix flashinfer cpu <-> gpu synchronization (#8340) 2025-08-10 03:11:40 +00:00
PGFLMG
b7cd743038 [Feat] QWen-1M context support[2/2]: Update block sparse attention backend (#5949) 2025-08-06 23:49:36 -07:00
Baizhou Zhang
f2d68ded6d Rename lora_path to lora_id in batches (#8437) 2025-08-03 21:08:28 -07:00
Cheng Wan
cb099d2095 [CUDA Graph] save cuda graph memory by using next_token_logits_buffer (#8579) 2025-08-03 03:06:47 -07:00
Nicolas Castet
82e6c3a65a Add support for NCCL symmetric memory for TP allreduces (#8238) 2025-08-01 23:30:55 +00:00
Brayden Zhong
4acf690206 [Optimization][Perf] Disable the GC during CUDA graph capture to speed up by up to 3x (#8577) 2025-07-31 11:31:21 -07:00
Cheng Wan
c0fb25e949 DP Enhancement (#8280) 2025-07-24 21:36:21 -07:00
Lifu Huang
4e3defe5a7 Support start up LoRA server without initial adapters (#8019) 2025-07-19 15:38:09 -07:00
Qiaolin Yu
41650b0d70 feat: support compatibility between MTP and two-batch-overlap (#7225) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2025-06-27 01:10:27 -07:00
fzyzcjy
3d7cdb2ebd Fix MTP error when enabling two-batch overlap (#7569) 2025-06-26 15:40:54 -07:00
valarLip
e984d5073b enable aiter_biased_grouped_topk kernel (#7423) 2025-06-24 02:09:42 -07:00
u4lr451
ed0a0b692c Perormance: Enable cuda graph for dp idle batch (#7269) 
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
 2025-06-23 17:34:13 -07:00
Cheng Wan
ac5010e0ba Fix CUDA Graph Check under Deepep with DP FFN (#7451) 2025-06-22 20:35:58 -07:00
Cheng Wan
e879d8b7a8 [Feature] Comprehensive Hybrid Parallelism Support (#6389) 2025-06-20 14:43:11 -07:00
u4lr451
10d60cd41b feat: mtp support dp-attention (#6081) 
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: tianqilin.99 <tianqilin.99@bytedance.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
 2025-06-17 00:33:28 -07:00
Lianmin Zheng
c64290dcb5 Use seq_len_fill_value in the cuda graph runners (#7233) 2025-06-16 15:57:07 -07:00
Lianmin Zheng
6b12d6a8d5 Simplify the heuristics for setting --mem-fraction-static (#7054) 2025-06-10 19:01:39 -07:00
kyle-pena-kuzco
b56de8f943 Open AI API hidden states (#6716) 2025-06-10 14:37:29 -07:00
Lianmin Zheng
4a102a2b02 Minor style fix in cuda_graph_runner.py (#7053) 2025-06-10 06:32:41 -07:00
Lianmin Zheng
6406408a70 Clean up server_args.py (#7037) 2025-06-10 05:34:29 -07:00
Lianmin Zheng
dc0705a504 Simplify prepare_extend_after_decode (#6987) 2025-06-09 16:39:21 -07:00
Baizhou Zhang
971a0dfa32 Extend cuda graph capture bs for B200 (#6937) 2025-06-08 05:13:22 -07:00
fzyzcjy
2fc1299562 Remove unnecessary kernels of num_token_non_padded (#6965) 2025-06-08 05:09:17 -07:00
Lianmin Zheng
60fdad7cf3 Sync the changes on cuda graph runners (#6932) 2025-06-06 18:23:52 -07:00
fzyzcjy
e05e29d178 Refactor CustomOp to avoid confusing bugs (#5382) 2025-06-02 10:27:36 -07:00
Lianmin Zheng
20fd53b8f6 Correctly abort the failed grammar requests & Improve the handling of abort (#6803) 2025-06-01 19:00:07 -07:00
Lianmin Zheng
2d72fc47cf Improve profiler and integrate profiler in bench_one_batch_server (#6787) 2025-05-31 15:53:55 -07:00
Baizhou Zhang
f2bd3515fb Tune memory arguments on B200 (#6718) 2025-05-29 00:03:22 -07:00
fzyzcjy
ae6a5b2950 Minor refactor two-batch overlap (#6682) 2025-05-28 15:54:17 -07:00
fzyzcjy
0d47788025 Support overlapping two batches (#4068) 2025-05-24 17:39:07 -07:00
Baizhou Zhang
d4c038daed [Fix]Fix capture fail bug for DeepSeek (#6275) 2025-05-21 11:11:20 -07:00
fzyzcjy
2716830802 Speed up when having padding tokens in DeepEP (#6175) 2025-05-17 16:44:05 -07:00
quinnrong94
2e4babdb0a [Feat] Support FlashMLA backend with MTP and FP8 KV cache (#6109) 
Co-authored-by: Yingyi <yingyihuang2000@outlook.com>
Co-authored-by: neiltian <neiltian@tencent.com>
Co-authored-by: lukec <118525388+sleepcoo@users.noreply.github.com>
Co-authored-by: kexueyu <kexueyu@tencent.com>
Co-authored-by: vincentmeng <vincentmeng@tencent.com>
Co-authored-by: pengmeng <pengmeng@tencent.com>
 2025-05-15 00:48:09 -07:00
Lianmin Zheng
e8e18dcdcc Revert "fix some typos" (#6244) 2025-05-12 12:53:26 -07:00
applesaucethebun
d738ab52f8 fix some typos (#6209) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-05-13 01:42:38 +08:00
Lianmin Zheng
fba8eccd7e Log if cuda graph is used & extend cuda graph capture to cuda-graph-max-bs (#6201) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-05-12 00:17:33 -07:00