chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
3,795 Commits 1 Branch 0 Tags
ac5010e0ba1491892ef5f5aeb603df98237ff7f9
Commit Graph

111 Commits

Author SHA1 Message Date
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
Liangsheng Yin
5ea5d22170 Fix CPU offloading for MLA memory pool (#7409) 2025-06-22 02:39:05 +08:00
Shangming Cai
187b85b7f3 [PD] Optimize custom mem pool usage and bump mooncake version (#7393) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-20 09:50:39 -07:00
DarkSharpness
47367b768d [Refactor] Clean up radix cache related API (#7303) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-06-20 00:58:48 +08:00
Stefan He
3774f07825 Multi-Stage Awake: Support Resume and Pause KV Cache and Weights separately (#7099) 2025-06-19 00:56:37 -07:00
Zhiqiang Xie
e56685ac1b Upstreaming hicache bug fixes (#7267) 2025-06-17 17:44:57 -07:00
shangmingc
c26d7349d3 [PD] Add custom memory pool option to support Mooncake PD with NVLink (#7264) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-17 17:21:37 -07:00
Lianmin Zheng
b1286a116a [EAGLE] Refactor code for page size > 1 & more simplifications (#7213) 2025-06-16 03:04:29 -07:00
Baizhou Zhang
d2679f5109 Fix ChunkCache object has no attribute 'disable' (#7217) 2025-06-15 20:55:15 -07:00
Lianmin Zheng
fff10809bf Revert "[EAGLE] Refactor code for page size > 1 & more simplifications" (#7210) 2025-06-15 02:48:00 -07:00
Lianmin Zheng
5f1ab32717 [EAGLE] Refactor code for page size > 1 & more simplifications (#7163) 2025-06-14 23:16:23 -07:00
Lianmin Zheng
38af4f68a9 Fix grammar abort & Minor style fixes (#7204) 2025-06-14 22:49:41 -07:00
Lianmin Zheng
a6305c7d50 Lianmin/simplify memory pool (#7202) 2025-06-14 22:25:37 -07:00
Lianmin Zheng
a023856b12 Move host memory pools into a separate file (#7200) 2025-06-14 21:31:42 -07:00
Byron Hsu
db0cc57e75 [PD] Support decode retract and update decode.py (#7196) 2025-06-14 19:48:05 -07:00
Faradawn Yang
777688b892 [feat]: Emit fixed-size KV blocks events (#6824) 2025-06-11 13:07:58 -07:00
sogalin
02543b545c Fix misusing the "_is_cuda". (#7091) 2025-06-11 11:21:31 -07:00
Chang Su
4685fbb888 [VLM] Support chunk prefill for VLM (#6355) 
Co-authored-by: yizhang2077 <1109276519@qq.com>
 2025-05-22 20:32:41 -07:00
Byron Hsu
0a4fc73b48 [PD] Fix failure abort (#6535) 2025-05-22 20:32:03 -07:00
Baizhou Zhang
d4c038daed [Fix]Fix capture fail bug for DeepSeek (#6275) 2025-05-21 11:11:20 -07:00
Lianmin Zheng
03886917bd Disable all two stream overlap on amd (#6475) 2025-05-20 19:06:59 -07:00
Trevor Morris
7adf245ba2 [Metrics] Add KV events publishing (#6098) 2025-05-19 14:19:54 -07:00
wangxiyu191
155214952b refactor: Extract repeated member variables in KVCache subclasses to base class. (#6323) 2025-05-18 15:28:15 -07:00
doujiang24
9d24c3ffb0 chore: tiny remove duplicated code (#6392) 
Signed-off-by: doujiang24 <doujiang24@gmail.com>
 2025-05-18 02:17:32 -07:00
Lifu Huang
3cf1473a09 Use monotonic clock for interval measurement (#6211) 
Signed-off-by: Lifu Huang <lifu.hlf@gmail.com>
 2025-05-17 16:49:18 -07:00
Simon (Jiyou) Li
b29a026e14 KV‑Cache (MHA, MLA): add missing start_layer / end_layer fields to MHATokenToKVPoolHost and MLATokenToKVPoolHost (#6016) 
Co-authored-by: 继优 <jiyou.ljy@alibaba-inc.com>
Co-authored-by: chus-chus <chus-chus@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-05-09 15:50:06 -07:00
Zhiqiang Xie
f8e460930a Fix prefill OOM error in the case of large page size (#5081) 2025-05-05 16:02:55 -07:00
Ke Bao
6579cd7daf Fix set kv cache multi-stream (#5975) 2025-05-02 09:26:00 -07:00
Ying Sheng
11383cec3c [PP] Add pipeline parallelism (#5724) 2025-04-30 18:18:07 -07:00
Ke Bao
799c4bb502 Fuse MLA set kv cache kernel (#5748) 2025-04-26 18:42:22 -07:00
Liangsheng Yin
e69a219074 Enhance GPU memory settings (#5604) 2025-04-21 15:15:00 -07:00
Zhiqiang Xie
70645f4d7d upstream hicache fixes (#5570) 2025-04-20 23:08:30 -07:00
Byron Hsu
c951d312ed [PD] Fix large page size + chunk prefill (#5588) 2025-04-20 17:21:54 -07:00
fzyzcjy
80ac527d22 [PD] Fix DeepSeek cannot be run on latest master (#5568) 2025-04-20 02:19:48 -07:00
Zhiqiang Xie
e2574ee986 fix hicache write back (#5543) 2025-04-19 21:56:22 -07:00
Byron Hsu
ab4b5606e4 [PD] Support page size > 1 (#5561) 2025-04-19 21:54:27 -07:00
ybyang
e9fc2ac7b6 [PD Bug] fix MLA get_contiguous_buf_infos error (#5384) 2025-04-14 22:56:39 +08:00
huangtingwei
5fbafbb8f8 fix MLATokenToKVPoolHost get_size_per_token bug (#5161) 
Co-authored-by: AniZpZ <zhuangsen.zp@antgroup.com>
 2025-04-13 12:37:26 -07:00
Teng Ma
7e4f72dd8c [PD] Add get_contiguous_buf_infos interface for MLATokenToKVPool (#5204) 2025-04-10 20:05:34 +08:00
Zhiqiang Xie
3fadc64793 bug fix for hicache host eviction (#4989) 2025-04-02 00:33:50 -07:00
Zhiqiang Xie
e119f04215 Large page size aligned hierarchical caching (#4581) 2025-04-01 22:38:15 -07:00
Lianmin Zheng
b26bc86b36 Support page size > 1 + eagle (#4908) 2025-03-30 00:46:23 -07:00
c1lovez1
93cf7fc5cd Unify variable naming: replace is_in_free_group with is_not_in_free_group (#4698) 2025-03-23 21:51:08 -07:00
Zhiqiang Xie
4d25305700 Move mem_state update into debug mode (#4525) 2025-03-23 00:52:27 -07:00
Byron Hsu
c7c7dbebbe [PD] Release initial code (#4654) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: makro
Co-authored-by: dhou-xai
 2025-03-21 14:47:47 -07:00
Zhiqiang Xie
a98290aea3 Unit test for Hierarchical Caching (#4486) 2025-03-17 17:45:00 -07:00
Zhiqiang Xie
f5bbf6037d Fix: Complete int32 to int64 conversion (#4465) 2025-03-16 18:14:27 -07:00
JieXin Liang
1a3fa75f2f [Fix] use torch.cat instead of torch.concat to prevent entering the Autograd backends. (#4466) 2025-03-16 00:02:47 -07:00
Lianmin Zheng
2c4f5ccac1 Fix minor style (#4460) 2025-03-15 21:51:12 -07:00
Wang Ran (汪然)
158430473e Fix typos (#4368) 2025-03-15 21:27:58 -07:00