chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
4,451 Commits 1 Branch 0 Tags
6ad6c8c9e6623fba797379566e4d1a3887bfee7f
Commit Graph

440 Commits

Author SHA1 Message Date
eigen
6ad6c8c9e6 feat: openai oss attention sink support with trtllm-gen backend #8825 (#8834) 
Co-authored-by: averyhuang <averyh@nvidia.com>
 2025-08-06 19:18:27 -07:00
Trevor Morris
c0e84297c2 Use reduce scatter for DP (#8539) 2025-08-06 16:21:26 -07:00
HouseWest
ca47e24f5d [Feature] improve TBO: two chunk overlap (#8144) 2025-08-05 21:11:01 -07:00
eigen
40e3b2beeb feat: add trtllm-gen mha from direct call (#8782) 
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2025-08-05 03:28:39 -07:00
Baizhou Zhang
f2d68ded6d Rename lora_path to lora_id in batches (#8437) 2025-08-03 21:08:28 -07:00
Cheng Wan
0e0eef00ce [DP] fix the compatibility issue between DP attention and --attention-backend triton (#8723) 2025-08-03 03:06:57 -07:00
Cheng Wan
cb099d2095 [CUDA Graph] save cuda graph memory by using next_token_logits_buffer (#8579) 2025-08-03 03:06:47 -07:00
Nicolas Castet
82e6c3a65a Add support for NCCL symmetric memory for TP allreduces (#8238) 2025-08-01 23:30:55 +00:00
Cheng Wan
6c88f6c8d9 [5/N] MoE Refactor: Update MoE parallelism arguments (#8658) 2025-08-01 01:20:03 -07:00
Faraz
4b04998d38 TRTLLM Gen MLA Decode Kernel Integration (same as #7938) (#8632) 
Signed-off-by: Faraz Khoubsirat <58580514+farazkh80@users.noreply.github.com>
 2025-07-31 16:03:40 -07:00
Chang Su
743638bc03 misc: Remove debug print to logger.info (#8633) 2025-07-31 12:56:52 -07:00
Brayden Zhong
4acf690206 [Optimization][Perf] Disable the GC during CUDA graph capture to speed up by up to 3x (#8577) 2025-07-31 11:31:21 -07:00
Cheng Wan
7a1f7fc504 [Feature] Hybrid EP and TP (#8590) 2025-07-31 02:53:25 -07:00
Lifu Huang
fb16fbaf52 Fix incorrect KV cache allocation for MTP models. (#8482) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-07-28 22:54:50 -07:00
Qiaolin Yu
2810338401 [feat] Support different attention backends for prefill and decode (#6338) 
Co-authored-by: tianqilin.99 <tianqilin.99@bytedance.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2025-07-28 11:42:29 +08:00
Lianmin Zheng
ed2e313eb6 Clean up server_args, triton cache manager (#8332) 2025-07-25 14:14:51 -07:00
Cheng Wan
c0fb25e949 DP Enhancement (#8280) 2025-07-24 21:36:21 -07:00
Xinyuan Tong
38000a5f44 Fix gemma3n with hybrid swa (#8240) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-07-23 13:29:18 -07:00
Zhiqiang Xie
f39037fffb HiCache Fix (#8288) 
Co-authored-by: pansicheng <sicheng.pan.chn@gmail.com>
 2025-07-23 16:51:32 +08:00
Lifu Huang
8abd3e77fe Introduce Stable LoRA ID System for Overlapped Updates and Prefix Caching (#8261) 2025-07-23 00:32:16 -07:00
Lifu Huang
4e3defe5a7 Support start up LoRA server without initial adapters (#8019) 2025-07-19 15:38:09 -07:00
Mick
3964b352c3 chore: tune mem fraction static for vlm (#6881) 2025-07-18 17:19:27 -07:00
Cheng Wan
9069884b51 [ci] disable memory imbalance check for draft worker (#8108) 2025-07-16 20:41:47 -07:00
Xiaoze Fan
570d33437b [Feature] Layer-wise Prefill (#7634) 
Signed-off-by: jason-fxz <jason341132@qq.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-07-17 01:57:46 +08:00
ykcombat
d4d0c7c367 [Feature]TP Group Switching for PD-Multiplexing (#7653) 2025-07-15 02:35:46 +08:00
Lifu Huang
e2ed9d049a Refactor dynamic LoRA update to fix incorrect handling of variant weight shapes (#7844) 2025-07-13 18:36:01 -07:00
Hanming Lu
9379da77de SWA Prefix Cache (#7367) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2025-07-13 12:31:07 -07:00
Mick
b5e3d6031c vlm: support video as an input modality (#5888) 2025-07-09 23:48:35 -07:00
Cheng Wan
8fc910db03 DP Attention with Auto DeepEP Dispatch (#7222) 2025-07-05 01:54:24 -07:00
Lianmin Zheng
14229ccf8f Move mem_fraction_static adjustment for multimodal models to server_args.py & Fix session control & Other cleanups (#7748) 2025-07-04 16:33:33 -07:00
Chunyuan WU
1dce6c480f [CPU] support the case where num_attention_heads or intermediate_size is not divisible by the TP size (#6771) 2025-07-03 09:51:38 -07:00
ronnie_zheng
1e0e549766 Ascend attention backend(PA&MLA) (#7722) 
Co-authored-by: Maksim <makcum888e@mail.ru>
Co-authored-by: VDV1985 <vladdv85@mail.ru>
 2025-07-03 09:23:19 -07:00
Chunyuan WU
8f844db699 [CPU] fix all_reduce and all_gather (#6770) 
Co-authored-by: blzheng <beilei.zheng@intel.com>
 2025-07-02 22:39:45 -07:00
Chunyuan WU
40e5cb7a9c [CPU] Bind threads and numa node for each TP rank (#6549) 
Co-authored-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>
 2025-07-02 19:57:59 -07:00
Zilin Zhu
0626f678de [RL] support update_weights_from_distributed with different group and multiple weights (#7292) 2025-07-02 19:29:11 -07:00
lukec
886d344964 support llama4 eagle3 (#6985) 
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
Co-authored-by: Shenggui Li <somerlee.9@gmail.com>
Co-authored-by: Yingyi Huang <yingyihuang2000@outlook.com>
Co-authored-by: yizhang2077 <1109276519@qq.com>
 2025-06-30 22:34:10 -07:00
Lianmin Zheng
22352d47a9 Improve streaming, log_level, memory report, weight loading, and benchmark script (#7632) 
Co-authored-by: Kan Wu <wukanustc@gmail.com>
 2025-06-29 23:16:19 -07:00
fzyzcjy
0c9c6c75a8 Move files related to EPLB (#7580) 2025-06-29 15:39:38 -07:00
Lifu Huang
49538d111b Support dynamic LoRA loading / unloading in engine/server API (#7446) 2025-06-27 21:00:27 -07:00
fzyzcjy
9efb2993da Tiny add logs for expert location updater (#7308) 2025-06-27 19:12:33 -07:00
tarinkk
eb6c2c1663 Hybrid kv cache for LLaMA4 (#6563) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: tarinkk <rt572@physics.rutger.edu>
Co-authored-by: tarinkk <rt572@rutgers.physics.edu>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-06-27 18:58:55 -07:00
Qiaolin Yu
41650b0d70 feat: support compatibility between MTP and two-batch-overlap (#7225) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2025-06-27 01:10:27 -07:00
fzyzcjy
3d7cdb2ebd Fix MTP error when enabling two-batch overlap (#7569) 2025-06-26 15:40:54 -07:00
Yuhong Guo
a1c1ebe935 Fix FP8 KV Cache Support in FA3 Backend (#7148) 2025-06-25 02:14:40 -07:00
xianzhiT
9f1787fa60 Support multi-thread model weight loading (#7277) 2025-06-24 10:39:10 -07:00
valarLip
e984d5073b enable aiter_biased_grouped_topk kernel (#7423) 2025-06-24 02:09:42 -07:00
u4lr451
ed0a0b692c Perormance: Enable cuda graph for dp idle batch (#7269) 
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
 2025-06-23 17:34:13 -07:00
Cheng Wan
ac5010e0ba Fix CUDA Graph Check under Deepep with DP FFN (#7451) 2025-06-22 20:35:58 -07:00
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
Lifu Huang
1998ce4046 Refactor LoRAManager and LoRAMemoryPool state management logic for dynamic LoRA loading support (#7412) 2025-06-21 16:09:19 -07:00