chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
6,077 Commits 1 Branch 0 Tags
a2ba0bc3df90ce4527997505eb2703686c4033d7
Commit Graph

57 Commits

Author SHA1 Message Date
Ziming Huang
6d2d0ce285 [PD] Improve eagle acceptance rate by transferring draft model hidden states (#10801) 
Co-authored-by: Shangming Cai <csmthu@gmail.com>
 2025-10-20 11:52:18 +08:00
Chang Su
627974405d [Lint] Add python/sglang to ruff F401 checks and remove unused imports in files (#11685) 2025-10-17 16:49:46 -07:00
Shangming Cai
868403f642 [PD] Add PD support for hybrid model (Qwen3-Next, DeepSeek V3.2 Exp) (#10912) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: hzh0425 <hzh0425@apache.org>
Co-authored-by: ZeldaHuang <hzm414167@alibaba-inc.com>
 2025-10-15 18:59:14 -07:00
Liangsheng Yin
bfadb5ea5f Adjust overlap event loop (#11507) 2025-10-14 00:33:19 +08:00
Liangsheng Yin
501dfa6b42 Remove sampling info events and overlap thread file (#11300) 2025-10-07 21:34:25 +08:00
Liangsheng Yin
1519a89cfd Remove overlap thread (#11210) 
Co-authored-by: Lianmin Zheng <15100009+merrymercy@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-10-07 20:12:12 +08:00
Liangsheng Yin
458611de77 Unify forward output datastructure (#11124) 2025-10-03 00:28:57 +08:00
Lianmin Zheng
2d62af6be5 Fix metrics and request tracing (TimeStats) (#11123) 2025-10-01 13:03:07 -07:00
Shangming Cai
e23e280e16 Add support for topk metadata transferring for PD (#10616) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-28 00:09:38 +08:00
Ziming Huang
b73ac629cd [BugFix] Fix incorrect hidden_states_tensor in pd disaggregation + eagle (#9976) 2025-09-17 10:37:14 -07:00
Yingchun Lai
b1721edbac [PD metrics] Add latency Histogram metrics of each stage for generate requests (#8710) 2025-09-16 01:52:49 +08:00
Jimmy
f40038fb09 [Vulnerability]feat(conn): set bootstrap server host (#9931) 2025-09-05 17:36:17 +08:00
Zhiqiang Xie
54e872d343 [HiCache] resolve conflict between chunked-prefill and hicache hit count (#9776) 2025-08-30 01:30:54 +08:00
SCDESPERTATE
b5c6529e17 [PD] Improve disaggregation metrics output: update the metrics to keep reflecting real stats (#7317) 2025-08-24 23:16:43 -07:00
datdo-msft
98b44e9e56 [PD] Propagate internal server errors from aborted requests to clients instead of blindly returning 200's (#8936) 2025-08-18 14:23:46 -07:00
Shangming Cai
384f8ab5ce [PD] Support PD disaggregation with Prefill PP (#8846) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: root <huzhiyuan@xiaohongshu.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Francis <38564764+ssssnow@users.noreply.github.com>
Co-authored-by: zitto <zhjc1124@gmail.com>
 2025-08-16 18:31:31 -07:00
Shangming Cai
d98a4913ea [PD] Refactor parallel sizes and add pp support for mooncake (#8571) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-08-04 20:18:11 -07:00
ybyang
6f9baf1002 [Improvements] Merge health check route (#8444) 
Signed-off-by: ybyang <ybyang7@iflytek.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Kan Wu <wukanustc@gmail.com>
 2025-08-03 01:59:06 -07:00
Lianmin Zheng
a4c3b121d8 Split the scheduler into multiple mixin classes to reduce the file size (#8483) 2025-07-29 12:46:50 -07:00
Shangming Cai
22e00eeb4a [Bugfix] Prevent PD server crash from invalid grammar (#8062) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-07-28 00:17:51 +08:00
Cheng Wan
8fc910db03 DP Attention with Auto DeepEP Dispatch (#7222) 2025-07-05 01:54:24 -07:00
Hongbo Xu
e21aa1df67 [PD] Add different TP sizes support for no-MLA models (#6793) 
Co-authored-by: shangmingc <csmthu@gmail.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-25 02:00:22 -07:00
linzhuo
afeed46530 clean duplicate code (#7512) 2025-06-25 01:22:20 -07:00
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
Cheng Wan
e879d8b7a8 [Feature] Comprehensive Hybrid Parallelism Support (#6389) 2025-06-20 14:43:11 -07:00
Atream
4f838c09cd [PD] Transfer hidden states for mtp when disaggregation (#7242) 2025-06-19 11:22:47 -07:00
Byron Hsu
88f9c347b2 [PD] use int32 for kv indices & get num_reserved_decode_tokens from server_args (#7214) 2025-06-15 11:51:03 -07:00
Byron Hsu
db0cc57e75 [PD] Support decode retract and update decode.py (#7196) 2025-06-14 19:48:05 -07:00
Byron Hsu
7d316991b2 [PD] Update prefill.py (#7190) 2025-06-14 15:59:54 -07:00
Povilas Kanapickas
bd7cfbd2f8 [Fix] Reduce busy polling when scheduler is idle (#6026) 2025-06-12 14:58:22 -07:00
shangmingc
3ce94f71f9 [PD] Handle P/D failure and reconnect without affecting other instances (#6263) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-05-26 19:21:01 -07:00
Byron Hsu
2d831c6ef9 [PD] Support structured output (#6560) 2025-05-23 21:49:00 -07:00
Byron Hsu
8233cc10fd [PD] Support logprob & Add failure test (#6558) 2025-05-23 14:29:20 -07:00
Byron Hsu
d2e0881a34 [PD] support spec decode (#6507) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-05-23 12:03:05 -07:00
shangmingc
58f10679e1 Fix missing http status import for PD failure handler (#6520) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-05-22 15:23:54 +08:00
Byron Hsu
3bde101099 [PD] Abort request if transfer fails (#6504) 2025-05-21 21:44:25 -07:00
Byron Hsu
7513558074 [PD] Add doc and simplify sender.send (#6019) 2025-05-21 21:22:21 -07:00
shangmingc
f1c896007a [PD] Add support for different TP sizes per DP rank (#5922) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-05-12 13:55:42 -07:00
Lianmin Zheng
fba8eccd7e Log if cuda graph is used & extend cuda graph capture to cuda-graph-max-bs (#6201) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-05-12 00:17:33 -07:00
ybyang
c6c6264073 [PD] support pd fake transfer for warmup (#5726) 2025-04-29 00:33:20 +08:00
Liangsheng Yin
40d9b8acce Improve overlap scheduling (#5788) 2025-04-28 11:19:16 +08:00
Liangsheng Yin
c55550cbf0 [PD] Better logs (#5715) 2025-04-25 17:25:45 +08:00
Cheng Wan
711efe7814 Integrating PD disaggregation with DP attention and DeepEP (#5435) 
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-04-23 01:46:01 -07:00
Byron Hsu
bf98d2e377 [PD] Support prefill overlap + Ensure no race condition (#5609) 2025-04-21 12:12:56 -07:00
Byron Hsu
deded17f38 [PD] Fix edge case and simplify large page size + chunked prefill (#5589) 2025-04-21 10:27:02 -07:00
Byron Hsu
c951d312ed [PD] Fix large page size + chunk prefill (#5588) 2025-04-20 17:21:54 -07:00
Byron Hsu
ab4b5606e4 [PD] Support page size > 1 (#5561) 2025-04-19 21:54:27 -07:00
shangmingc
dca90f1db8 [PD] Remove the requirement of config file for mooncake backend (#5460) 2025-04-19 19:31:00 +08:00
Cheng Wan
6aca583420 Fix several minor issues in PD disaggregation (#5444) 2025-04-15 23:04:41 -07:00
Liangsheng Yin
33b16ad178 Distinguish bootstrap key only in decode server (#5422) 2025-04-15 20:59:28 +08:00