chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
4,268 Commits 1 Branch 0 Tags
fb16fbaf521c2439ac6e6664ff6669ce20e14a11
Commit Graph

303 Commits

Author SHA1 Message Date
Lifu Huang
fb16fbaf52 Fix incorrect KV cache allocation for MTP models. (#8482) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-07-28 22:54:50 -07:00
Qiaolin Yu
2810338401 [feat] Support different attention backends for prefill and decode (#6338) 
Co-authored-by: tianqilin.99 <tianqilin.99@bytedance.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
 2025-07-28 11:42:29 +08:00
Lianmin Zheng
ed2e313eb6 Clean up server_args, triton cache manager (#8332) 2025-07-25 14:14:51 -07:00
Cheng Wan
c0fb25e949 DP Enhancement (#8280) 2025-07-24 21:36:21 -07:00
Xinyuan Tong
38000a5f44 Fix gemma3n with hybrid swa (#8240) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-07-23 13:29:18 -07:00
Zhiqiang Xie
f39037fffb HiCache Fix (#8288) 
Co-authored-by: pansicheng <sicheng.pan.chn@gmail.com>
 2025-07-23 16:51:32 +08:00
Lifu Huang
8abd3e77fe Introduce Stable LoRA ID System for Overlapped Updates and Prefix Caching (#8261) 2025-07-23 00:32:16 -07:00
Lifu Huang
4e3defe5a7 Support start up LoRA server without initial adapters (#8019) 2025-07-19 15:38:09 -07:00
Mick
3964b352c3 chore: tune mem fraction static for vlm (#6881) 2025-07-18 17:19:27 -07:00
Cheng Wan
9069884b51 [ci] disable memory imbalance check for draft worker (#8108) 2025-07-16 20:41:47 -07:00
Xiaoze Fan
570d33437b [Feature] Layer-wise Prefill (#7634) 
Signed-off-by: jason-fxz <jason341132@qq.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-07-17 01:57:46 +08:00
ykcombat
d4d0c7c367 [Feature]TP Group Switching for PD-Multiplexing (#7653) 2025-07-15 02:35:46 +08:00
Lifu Huang
e2ed9d049a Refactor dynamic LoRA update to fix incorrect handling of variant weight shapes (#7844) 2025-07-13 18:36:01 -07:00
Hanming Lu
9379da77de SWA Prefix Cache (#7367) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2025-07-13 12:31:07 -07:00
Lianmin Zheng
14229ccf8f Move mem_fraction_static adjustment for multimodal models to server_args.py & Fix session control & Other cleanups (#7748) 2025-07-04 16:33:33 -07:00
Chunyuan WU
1dce6c480f [CPU] support the case where num_attention_heads or intermediate_size is not divisible by the TP size (#6771) 2025-07-03 09:51:38 -07:00
ronnie_zheng
1e0e549766 Ascend attention backend(PA&MLA) (#7722) 
Co-authored-by: Maksim <makcum888e@mail.ru>
Co-authored-by: VDV1985 <vladdv85@mail.ru>
 2025-07-03 09:23:19 -07:00
Chunyuan WU
8f844db699 [CPU] fix all_reduce and all_gather (#6770) 
Co-authored-by: blzheng <beilei.zheng@intel.com>
 2025-07-02 22:39:45 -07:00
Chunyuan WU
40e5cb7a9c [CPU] Bind threads and numa node for each TP rank (#6549) 
Co-authored-by: srinarayan-srikanthan <srinarayan.srikanthan@intel.com>
 2025-07-02 19:57:59 -07:00
Zilin Zhu
0626f678de [RL] support update_weights_from_distributed with different group and multiple weights (#7292) 2025-07-02 19:29:11 -07:00
lukec
886d344964 support llama4 eagle3 (#6985) 
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
Co-authored-by: Shenggui Li <somerlee.9@gmail.com>
Co-authored-by: Yingyi Huang <yingyihuang2000@outlook.com>
Co-authored-by: yizhang2077 <1109276519@qq.com>
 2025-06-30 22:34:10 -07:00
Lianmin Zheng
22352d47a9 Improve streaming, log_level, memory report, weight loading, and benchmark script (#7632) 
Co-authored-by: Kan Wu <wukanustc@gmail.com>
 2025-06-29 23:16:19 -07:00
fzyzcjy
0c9c6c75a8 Move files related to EPLB (#7580) 2025-06-29 15:39:38 -07:00
Lifu Huang
49538d111b Support dynamic LoRA loading / unloading in engine/server API (#7446) 2025-06-27 21:00:27 -07:00
tarinkk
eb6c2c1663 Hybrid kv cache for LLaMA4 (#6563) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: tarinkk <rt572@physics.rutger.edu>
Co-authored-by: tarinkk <rt572@rutgers.physics.edu>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-06-27 18:58:55 -07:00
Qiaolin Yu
41650b0d70 feat: support compatibility between MTP and two-batch-overlap (#7225) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
 2025-06-27 01:10:27 -07:00
Yuhong Guo
a1c1ebe935 Fix FP8 KV Cache Support in FA3 Backend (#7148) 2025-06-25 02:14:40 -07:00
xianzhiT
9f1787fa60 Support multi-thread model weight loading (#7277) 2025-06-24 10:39:10 -07:00
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
Lifu Huang
1998ce4046 Refactor LoRAManager and LoRAMemoryPool state management logic for dynamic LoRA loading support (#7412) 2025-06-21 16:09:19 -07:00
Stefan He
3774f07825 Multi-Stage Awake: Support Resume and Pause KV Cache and Weights separately (#7099) 2025-06-19 00:56:37 -07:00
YanbingJiang
094c116f7d Update python API of activation, topk, norm and rope and remove vllm dependency (#6614) 
Co-authored-by: Wu, Chunyuan <chunyuan.wu@intel.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
Co-authored-by: sdp <sdp@gnr799219.jf.intel.com>
 2025-06-17 22:11:50 -07:00
u4lr451
10d60cd41b feat: mtp support dp-attention (#6081) 
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: tianqilin.99 <tianqilin.99@bytedance.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
 2025-06-17 00:33:28 -07:00
KavioYu
873ae12cee support custom weight loader for model runner (#7122) 
Co-authored-by: kavioyu <kavioyu@tencent.com>
 2025-06-16 16:28:15 -07:00
fzyzcjy
b4c41f7276 Refactor DeepGEMM integration (#7150) 2025-06-13 20:41:03 -07:00
Byron Hsu
c2b16795b5 Add decode req pool (#6980) 2025-06-09 21:23:36 -07:00
Baizhou Zhang
6716b41786 Update default settings for blackwell (#7023) 2025-06-09 20:37:47 -07:00
fzyzcjy
f5599ef124 Refactor global_server_args_dict (#6866) 2025-06-07 03:10:35 -07:00
HAI
b819381fec AITER backend extension and workload optimizations (#6838) 
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: Hubert Lu <Hubert.Lu@amd.com>
 2025-06-05 23:00:18 -07:00
fzyzcjy
0de5e7d40f Support layerwise rebalancing experts (#6851) 2025-06-05 00:05:52 -07:00
zyksir
8e3797be1c support 1 shot allreduce in 1-node and 2-node using mscclpp (#6277) 2025-06-04 22:11:24 -07:00
Cheng Wan
81964328b7 Set num_fused_shared_experts as num_shared_experts when shared_experts fusion is not disabled (#6736) 2025-06-04 15:53:22 -07:00
Cheng Wan
8a5480528d [Refactor] Rename n_share_experts_fusion as num_fused_shared_experts (#6735) 2025-06-03 17:48:24 -07:00
Lianmin Zheng
20fd53b8f6 Correctly abort the failed grammar requests & Improve the handling of abort (#6803) 2025-06-01 19:00:07 -07:00
Lianmin Zheng
2d72fc47cf Improve profiler and integrate profiler in bench_one_batch_server (#6787) 2025-05-31 15:53:55 -07:00
YanbingJiang
888cb175a6 Add intel_amx backend for Radix Attention for CPU (#6408) 
Co-authored-by: Chunyuan WU <chunyuan.wu@intel.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2025-05-30 21:37:42 -07:00
Jianan Ji
22630ca242 Support sliding window in triton backend (#6509) 2025-05-30 01:11:53 -07:00
fzyzcjy
447be24228 Fix OOM when updating expert locations (#6660) 2025-05-27 09:59:53 -07:00
fzyzcjy
0d47788025 Support overlapping two batches (#4068) 2025-05-24 17:39:07 -07:00
kk
7a5e6ce1cb Fix GPU OOM (#6564) 
Co-authored-by: michael <michael.zhang@amd.com>
 2025-05-24 16:38:39 -07:00