chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
3,922 Commits 1 Branch 0 Tags
1e0e549766a0b13164283d759120200d73d71858
Commit Graph

83 Commits

Author SHA1 Message Date
ronnie_zheng
1e0e549766 Ascend attention backend(PA&MLA) (#7722) 
Co-authored-by: Maksim <makcum888e@mail.ru>
Co-authored-by: VDV1985 <vladdv85@mail.ru>
 2025-07-03 09:23:19 -07:00
Lianmin Zheng
22352d47a9 Improve streaming, log_level, memory report, weight loading, and benchmark script (#7632) 
Co-authored-by: Kan Wu <wukanustc@gmail.com>
 2025-06-29 23:16:19 -07:00
tarinkk
eb6c2c1663 Hybrid kv cache for LLaMA4 (#6563) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: tarinkk <rt572@physics.rutger.edu>
Co-authored-by: tarinkk <rt572@rutgers.physics.edu>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-06-27 18:58:55 -07:00
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
Liangsheng Yin
5ea5d22170 Fix CPU offloading for MLA memory pool (#7409) 2025-06-22 02:39:05 +08:00
Shangming Cai
187b85b7f3 [PD] Optimize custom mem pool usage and bump mooncake version (#7393) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-20 09:50:39 -07:00
Stefan He
3774f07825 Multi-Stage Awake: Support Resume and Pause KV Cache and Weights separately (#7099) 2025-06-19 00:56:37 -07:00
shangmingc
c26d7349d3 [PD] Add custom memory pool option to support Mooncake PD with NVLink (#7264) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-17 17:21:37 -07:00
Lianmin Zheng
b1286a116a [EAGLE] Refactor code for page size > 1 & more simplifications (#7213) 2025-06-16 03:04:29 -07:00
Lianmin Zheng
fff10809bf Revert "[EAGLE] Refactor code for page size > 1 & more simplifications" (#7210) 2025-06-15 02:48:00 -07:00
Lianmin Zheng
5f1ab32717 [EAGLE] Refactor code for page size > 1 & more simplifications (#7163) 2025-06-14 23:16:23 -07:00
Lianmin Zheng
38af4f68a9 Fix grammar abort & Minor style fixes (#7204) 2025-06-14 22:49:41 -07:00
Lianmin Zheng
a6305c7d50 Lianmin/simplify memory pool (#7202) 2025-06-14 22:25:37 -07:00
Lianmin Zheng
a023856b12 Move host memory pools into a separate file (#7200) 2025-06-14 21:31:42 -07:00
Byron Hsu
db0cc57e75 [PD] Support decode retract and update decode.py (#7196) 2025-06-14 19:48:05 -07:00
sogalin
02543b545c Fix misusing the "_is_cuda". (#7091) 2025-06-11 11:21:31 -07:00
Baizhou Zhang
d4c038daed [Fix]Fix capture fail bug for DeepSeek (#6275) 2025-05-21 11:11:20 -07:00
Lianmin Zheng
03886917bd Disable all two stream overlap on amd (#6475) 2025-05-20 19:06:59 -07:00
wangxiyu191
155214952b refactor: Extract repeated member variables in KVCache subclasses to base class. (#6323) 2025-05-18 15:28:15 -07:00
doujiang24
9d24c3ffb0 chore: tiny remove duplicated code (#6392) 
Signed-off-by: doujiang24 <doujiang24@gmail.com>
 2025-05-18 02:17:32 -07:00
Simon (Jiyou) Li
b29a026e14 KV‑Cache (MHA, MLA): add missing start_layer / end_layer fields to MHATokenToKVPoolHost and MLATokenToKVPoolHost (#6016) 
Co-authored-by: 继优 <jiyou.ljy@alibaba-inc.com>
Co-authored-by: chus-chus <chus-chus@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-05-09 15:50:06 -07:00
Ke Bao
6579cd7daf Fix set kv cache multi-stream (#5975) 2025-05-02 09:26:00 -07:00
Ying Sheng
11383cec3c [PP] Add pipeline parallelism (#5724) 2025-04-30 18:18:07 -07:00
Ke Bao
799c4bb502 Fuse MLA set kv cache kernel (#5748) 2025-04-26 18:42:22 -07:00
Liangsheng Yin
e69a219074 Enhance GPU memory settings (#5604) 2025-04-21 15:15:00 -07:00
Zhiqiang Xie
70645f4d7d upstream hicache fixes (#5570) 2025-04-20 23:08:30 -07:00
Byron Hsu
c951d312ed [PD] Fix large page size + chunk prefill (#5588) 2025-04-20 17:21:54 -07:00
fzyzcjy
80ac527d22 [PD] Fix DeepSeek cannot be run on latest master (#5568) 2025-04-20 02:19:48 -07:00
Byron Hsu
ab4b5606e4 [PD] Support page size > 1 (#5561) 2025-04-19 21:54:27 -07:00
ybyang
e9fc2ac7b6 [PD Bug] fix MLA get_contiguous_buf_infos error (#5384) 2025-04-14 22:56:39 +08:00
huangtingwei
5fbafbb8f8 fix MLATokenToKVPoolHost get_size_per_token bug (#5161) 
Co-authored-by: AniZpZ <zhuangsen.zp@antgroup.com>
 2025-04-13 12:37:26 -07:00
Teng Ma
7e4f72dd8c [PD] Add get_contiguous_buf_infos interface for MLATokenToKVPool (#5204) 2025-04-10 20:05:34 +08:00
Zhiqiang Xie
e119f04215 Large page size aligned hierarchical caching (#4581) 2025-04-01 22:38:15 -07:00
Lianmin Zheng
b26bc86b36 Support page size > 1 + eagle (#4908) 2025-03-30 00:46:23 -07:00
c1lovez1
93cf7fc5cd Unify variable naming: replace is_in_free_group with is_not_in_free_group (#4698) 2025-03-23 21:51:08 -07:00
Zhiqiang Xie
4d25305700 Move mem_state update into debug mode (#4525) 2025-03-23 00:52:27 -07:00
Byron Hsu
c7c7dbebbe [PD] Release initial code (#4654) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: makro
Co-authored-by: dhou-xai
 2025-03-21 14:47:47 -07:00
Zhiqiang Xie
a98290aea3 Unit test for Hierarchical Caching (#4486) 2025-03-17 17:45:00 -07:00
Zhiqiang Xie
f5bbf6037d Fix: Complete int32 to int64 conversion (#4465) 2025-03-16 18:14:27 -07:00
JieXin Liang
1a3fa75f2f [Fix] use torch.cat instead of torch.concat to prevent entering the Autograd backends. (#4466) 2025-03-16 00:02:47 -07:00
Lianmin Zheng
2c4f5ccac1 Fix minor style (#4460) 2025-03-15 21:51:12 -07:00
Wang Ran (汪然)
158430473e Fix typos (#4368) 2025-03-15 21:27:58 -07:00
Chen Shengzhi
86d9baedc2 [Fix] Fix errors when using the device except cuda. (#4455) 2025-03-15 16:33:00 -07:00
Lu Changqi
0e0ec70200 Hierarchical Caching supports MLA (#4009) 
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-03-13 20:42:14 -07:00
Lianmin Zheng
a5a892ffd3 Fix auto merge & add back get_flat_data_by_layer (#4393) 2025-03-13 08:46:25 -07:00
Lianmin Zheng
4fea040ca1 Fix a regression introduced by overlapping KV cache writing (#4375) 2025-03-13 03:49:05 -07:00
Lianmin Zheng
c76040e31b Support page size > 1 (#4356) 2025-03-12 22:22:39 -07:00
Zhiqiang Xie
10b544ae9b Hierarchical Caching Refactoring and Fixing TP issue (#4082) 2025-03-12 11:22:35 -07:00
Zhiqiang Xie
9376ac361d Memory pool fix for upstream change about eagle (#4170) 2025-03-07 00:58:20 -08:00
Zhiqiang Xie
aee30630d8 Add a pointer to the real KV cache pool (#4113) 2025-03-05 21:39:07 -08:00