chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
3,878 Commits 1 Branch 0 Tags
b2264076dc201212ea689c6fec957bb458c5e2b6
Commit Graph

40 Commits

Author SHA1 Message Date
Shangming Cai
eb429b88a4 [PD] Respect sampling_params.max_new_tokens when PD disaggregation is activated (#7598) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-27 22:22:01 -07:00
tarinkk
eb6c2c1663 Hybrid kv cache for LLaMA4 (#6563) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: tarinkk <rt572@physics.rutger.edu>
Co-authored-by: tarinkk <rt572@rutgers.physics.edu>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-06-27 18:58:55 -07:00
Shangming Cai
7b9a174a7a [PD][Spec] Fix hidden state transfer for spec decode (#7516) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-25 00:42:07 -07:00
Liangsheng Yin
05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) 2025-06-22 12:37:18 +08:00
Atream
02bf31ef29 [fix] PD disaggregation when enable mtp and tp!=dp (#7420) 2025-06-21 12:03:11 -07:00
Cheng Wan
e879d8b7a8 [Feature] Comprehensive Hybrid Parallelism Support (#6389) 2025-06-20 14:43:11 -07:00
Atream
4f838c09cd [PD] Transfer hidden states for mtp when disaggregation (#7242) 2025-06-19 11:22:47 -07:00
Stefan He
3774f07825 Multi-Stage Awake: Support Resume and Pause KV Cache and Weights separately (#7099) 2025-06-19 00:56:37 -07:00
Byron Hsu
96be97bfff Minor PD style fix (#7215) 2025-06-15 16:12:12 -07:00
Byron Hsu
88f9c347b2 [PD] use int32 for kv indices & get num_reserved_decode_tokens from server_args (#7214) 2025-06-15 11:51:03 -07:00
Byron Hsu
db0cc57e75 [PD] Support decode retract and update decode.py (#7196) 2025-06-14 19:48:05 -07:00
Byron Hsu
7d316991b2 [PD] Update prefill.py (#7190) 2025-06-14 15:59:54 -07:00
Povilas Kanapickas
bd7cfbd2f8 [Fix] Reduce busy polling when scheduler is idle (#6026) 2025-06-12 14:58:22 -07:00
Byron Hsu
c2b16795b5 Add decode req pool (#6980) 2025-06-09 21:23:36 -07:00
ishandhanani
f1569876d5 feat: add direct routing strategy to DP worker (#6884) 2025-06-09 11:44:05 -07:00
shangmingc
3ce94f71f9 [PD] Handle P/D failure and reconnect without affecting other instances (#6263) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-05-26 19:21:01 -07:00
Byron Hsu
2d831c6ef9 [PD] Support structured output (#6560) 2025-05-23 21:49:00 -07:00
Byron Hsu
8233cc10fd [PD] Support logprob & Add failure test (#6558) 2025-05-23 14:29:20 -07:00
Byron Hsu
d2e0881a34 [PD] support spec decode (#6507) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-05-23 12:03:05 -07:00
Byron Hsu
0a4fc73b48 [PD] Fix failure abort (#6535) 2025-05-22 20:32:03 -07:00
shangmingc
58f10679e1 Fix missing http status import for PD failure handler (#6520) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-05-22 15:23:54 +08:00
Byron Hsu
3bde101099 [PD] Abort request if transfer fails (#6504) 2025-05-21 21:44:25 -07:00
shangmingc
f1c896007a [PD] Add support for different TP sizes per DP rank (#5922) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-05-12 13:55:42 -07:00
applesaucethebun
2ce8793519 Add typo checker in pre-commit (#6179) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-05-11 12:55:00 +08:00
Liangsheng Yin
6d4d3bc81d Fix not "import os" (#6057) 2025-05-06 22:06:41 +08:00
fzyzcjy
3008db9c1a [PD] Allow customizing reserved tokens to avoid KV cache waste (#6002) 2025-05-05 11:23:15 +08:00
ybyang
c6c6264073 [PD] support pd fake transfer for warmup (#5726) 2025-04-29 00:33:20 +08:00
IAN
11e27d0926 [PD]: Support Muti Prefill in one node (#5704) 
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
 2025-04-26 00:30:47 +08:00
Liangsheng Yin
c55550cbf0 [PD] Better logs (#5715) 2025-04-25 17:25:45 +08:00
Cheng Wan
711efe7814 Integrating PD disaggregation with DP attention and DeepEP (#5435) 
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com>
 2025-04-23 01:46:01 -07:00
Byron Hsu
e65b9f21e3 [PD] Support decode overlap schedule (#5608) 2025-04-21 12:06:16 -07:00
Byron Hsu
ab4b5606e4 [PD] Support page size > 1 (#5561) 2025-04-19 21:54:27 -07:00
shangmingc
dca90f1db8 [PD] Remove the requirement of config file for mooncake backend (#5460) 2025-04-19 19:31:00 +08:00
Cheng Wan
6aca583420 Fix several minor issues in PD disaggregation (#5444) 2025-04-15 23:04:41 -07:00
Liangsheng Yin
33b16ad178 Distinguish bootstrap key only in decode server (#5422) 2025-04-15 20:59:28 +08:00
shangmingc
ffde65a094 [PD] Fix dynamic port support and MLA buffer for Mooncake (#5415) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: ybyang <ybyang7@iflytek.com>
 2025-04-15 19:29:31 +08:00
Byron Hsu
a9499885e9 [PD] Add transfer backend abstraction (#5328) 2025-04-14 01:39:39 +08:00
Teng Ma
4c31ae9f6d [PD] Support KV transfer with mooncake (#4880) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Shangming Cai <caishangming@linux.alibaba.com>
Co-authored-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
Co-authored-by: shangmingc <csmthu@gmail.com>
 2025-04-10 14:23:23 +08:00
Xuchun Shang
8154de5a32 [PD] Remove invalid parameter (#4721) 
Signed-off-by: Xuchun Shang <xuchun.shang@linux.alibaba.com>
 2025-03-24 13:14:16 -07:00
Byron Hsu
c7c7dbebbe [PD] Release initial code (#4654) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: makro
Co-authored-by: dhou-xai
 2025-03-21 14:47:47 -07:00