chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
9,426 Commits 1 Branch 0 Tags
fd983b09b68c076d448a55266a29ffdfdff2d06e
Commit Graph

61 Commits

Author SHA1 Message Date
EkiRui
fd983b09b6 [Performance] Optimize radix cache eviction performance (#14339) 
Signed-off-by: Xingrui Yi <yixingrui@linux.alibaba.com>
Co-authored-by: Xuchun Shang <xuchun.shang@gmail.com>
 2026-02-03 09:44:20 +08:00
cctry
027f314050 [Fix] data race in req_to_token pool (#17850) 2026-02-02 14:38:15 -08:00
zhangheng
f33022d039 [RadixTree][3/N Refactor]：Support unified insert/evict params (#17401) 2026-01-22 17:36:31 +08:00
zhangheng
20b0523eca [RadixTree][1/N Refactor]: Support unified match_prefix params (#17142) 
Co-authored-by: yizhang2077 <1109276519@qq.com>
Co-authored-by: pansicheng <sicheng.pan.chn@gmail.com>
 2026-01-19 22:39:40 +08:00
Yi Zhang
737a1183d6 [BUGFIX] fix radix cache memory consumption to avoid OOM (#17191) 
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
 2026-01-17 16:47:37 +08:00
Neal Vaidya
a4c762811a Remove incorrect BlockRemoved event emission during node splits (#14934) 
Signed-off-by: Neal Vaidya <nealv@nvidia.com>
 2025-12-16 14:11:24 -08:00
PiteXChen
eac5b66485 [RadixTree] Optimize the Time Complexity of Node Retrieval Operation from O(n*m) to O(n) (#13334) 
Signed-off-by: CLFutureX <chenyongqyl@163.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-12-08 15:55:09 -08:00
Yan Ru Pei
f446b51c41 fix: malformed KV events for NVIDIA Dynamo (#13488) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-11-28 14:55:20 -08:00
Lianmin Zheng
e83bd1fadc [Auto Sync] Update schedule_batch.py, schedule_policy.py, b... (20251122) (#13763) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Hanming Lu <hanming@x.ai>
 2025-11-24 14:33:31 -08:00
Liangsheng Yin
b2f7b08c49 Refactor cache init logic (#13800) 2025-11-24 11:41:46 +08:00
Liangsheng Yin
ac43822634 Refactor eagle bigram key matching (#13714) 2025-11-22 20:40:42 +08:00
Lianmin Zheng
b41afa3754 [Auto Sync] Update evict_policy.py, radix_cache.py (20251120) (#13669) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: cctry <shiyang@x.ai>
 2025-11-21 22:07:36 -08:00
Liangsheng Yin
f09eee036d Tiny simplify evcition metrics collector (#12983) 2025-11-11 23:23:48 +08:00
Liangsheng Yin
665416f6dd Unify memory management across (overlap, non-overlap) x (page>=1) x (spec, non-spec, spec v2) x (retract, finished) (#12224) 2025-11-11 02:56:22 +08:00
ShawnKung
afee2843d5 feat(metrics): add scheduler and hiradix cache metrics (#10218) (#10225) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-11-10 18:14:47 +08:00
PiteXChen
e2a784ecda [RadixTree] Reduce Syscalls, Optimize Collection Filtering and Align with cpp (#12239) 
Signed-off-by: CLFutureX <chenyongqyl@163.com>
 2025-11-10 17:47:59 +08:00
cctry
3029d30189 Fix crash after flush cache (#12107) 
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
 2025-10-27 23:52:27 +08:00
Teng Ma
585e1223f0 [HiCache] feat: add more eviction policy (#11506) 2025-10-18 15:49:45 +00:00
Chang Su
627974405d [Lint] Add python/sglang to ruff F401 checks and remove unused imports in files (#11685) 2025-10-17 16:49:46 -07:00
Baizhou Zhang
b0d1d717e1 Revert "make radix cache deterministic" (#11728) 2025-10-16 14:36:15 -07:00
cctry
1d7f783501 Refactor kv cache free (#11351) 2025-10-14 17:45:19 -07:00
Alex Chi Z
dc965db0e0 make radix cache deterministic (#10721) 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
 2025-10-14 21:01:52 +08:00
hzh0425
ee3bd8a1c8 feat(hicache): Support passing prefix keys for l3 store. (#9045) 
Co-authored-by: pansicheng <sicheng.pan.chn@gmail.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-10-10 00:22:05 -07:00
Ke Bao
24bc3fb0f9 EAGLE cache fix for SWARadixCache (#11231) 
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-10-07 18:21:37 +08:00
Alex Chi Z
1a31229cd4 fix: radix cache memory accounting (#10637) 
Signed-off-by: Alex Chi Z <iskyzh@gmail.com>
 2025-10-02 22:47:33 -07:00
Ke Bao
91847e382a Fix eagle radix cache (#10846) 2025-09-30 22:59:20 +08:00
Xinyuan Tong
12d6cf18f0 Refactors radix cache for extra key support (#10317) 
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
 2025-09-22 02:16:16 +08:00
Xuchun Shang
1ccd59c715 [HICache] introduce evict policy (#10190) 
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
 2025-09-18 11:10:20 +08:00
DarkSharpness
948b01a04c [Refactor] Remove Hicache Load & Write threads (#10127) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-09-08 22:18:50 -07:00
Zhiqiang Xie
8b6966d020 [HiCache] Storage Refactoring (#9797) 
Co-authored-by: pansicheng <27603155+pansicheng@users.noreply.github.com>
 2025-08-31 22:58:21 +08:00
Zhiqiang Xie
54e872d343 [HiCache] resolve conflict between chunked-prefill and hicache hit count (#9776) 2025-08-30 01:30:54 +08:00
Zhiqiang Xie
43de1d7304 HiCache Storage fix host memory leak (#9648) 2025-08-26 10:49:40 -07:00
Faradawn Yang
ff1f68252c [fix] Set Radix tree root node hash to None - Nvidia Dynamo Integration (#9030) 
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-11 14:20:39 -07:00
Zhiqiang Xie
9f78f391ae HiCache Storage: generate hash when inserting new nodes (#9053) 2025-08-11 14:18:59 -07:00
Zhiqiang Xie
9d33fcfb8e Hicache Storage Layer Prototype (#7704) 2025-07-18 15:20:19 +08:00
Zhiqiang Xie
2fc824b84c Kernels for efficient KV cache IO (#7313) 2025-07-06 22:53:36 -07:00
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
DarkSharpness
47367b768d [Refactor] Clean up radix cache related API (#7303) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-06-20 00:58:48 +08:00
Faradawn Yang
777688b892 [feat]: Emit fixed-size KV blocks events (#6824) 2025-06-11 13:07:58 -07:00
Trevor Morris
7adf245ba2 [Metrics] Add KV events publishing (#6098) 2025-05-19 14:19:54 -07:00
Lifu Huang
3cf1473a09 Use monotonic clock for interval measurement (#6211) 
Signed-off-by: Lifu Huang <lifu.hlf@gmail.com>
 2025-05-17 16:49:18 -07:00
Zhiqiang Xie
f5bbf6037d Fix: Complete int32 to int64 conversion (#4465) 2025-03-16 18:14:27 -07:00
JieXin Liang
1a3fa75f2f [Fix] use torch.cat instead of torch.concat to prevent entering the Autograd backends. (#4466) 2025-03-16 00:02:47 -07:00
Lianmin Zheng
c76040e31b Support page size > 1 (#4356) 2025-03-12 22:22:39 -07:00
luzengxiangcn
62b362b1f1 Debug radixcache: refactor recursive helper methods (#3029) 
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
 2025-03-05 16:11:42 -08:00
Ying Sheng
d3d4d76758 [Eagle] Refactor eagle speculative decoding (#3986) 
Co-authored-by: Ke Bao <ISPObaoke@163.com>
 2025-03-05 08:06:07 -08:00
Zhiqiang Xie
08104b56de Sanity check to prevent performance regression (#3171) 
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
 2025-01-27 12:28:17 -08:00
SangBin Cho
9208618b3e [Core] in batch prefix caching by delay scheduling (#2442) 2024-12-11 12:51:50 -08:00
Lianmin Zheng
59cbf47626 Unify the memory pool api and tp worker API (#1724) 2024-10-19 23:19:26 -07:00
Lianmin Zheng
769bf11c05 Fix the race condition in overlap mode (#1712) 2024-10-19 06:50:56 -07:00