chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 17 Packages Projects Releases Wiki Activity
11,116 Commits 1 Branch 0 Tags
47cb9b9d116d1c5c112e894110ec94de80f2db6b
Commit Graph
94 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Shangming Cai a094eb9c21 Temporarily disable flaky qwen3 cp test in CI (#21178) 2026-03-22 21:13:52 -07:00
Shangming Cai 4c52b7fcc6 [CI] Improve PP consistency check success rate (#20838) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-19 19:12:50 +08:00
Shangming Cai fd05532da1 Add logging for BootstrapServer for CI diagnosis (#20844) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-19 14:42:12 +08:00
Shangming Cai 8b46f1f4ec [PD] Add retry interval in ensure_prefill_info (#20832) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-18 16:02:20 +08:00
Shangming Caiandhnyls2002 2acb20f53b [Disagg] Non-blocking try_ensure_parallel_info in pending queue, consolidate rank mapping into PrefillServerInfo (#20785) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
 2026-03-17 17:26:18 -07:00
Shangming Cai 17c81a3e07 Revert "[PD] Make pending reqs resolving more robust" (#20779) 2026-03-17 20:31:12 +08:00
Shangming Cai 5d5c31c6e4 [PP] Add CP pyobj broadcasting when enable dynamic CPP (#20738) 2026-03-17 12:20:11 +08:00
Shangming Cai a4528a5737 [CI] Bring back CI test for Mamba PD Disaggregation (#20675) 2026-03-17 01:04:21 +08:00
Shangming Cai 738cbde902 [PD] Make pending reqs resolving more robust (#20505) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-16 14:12:13 +08:00
Shangming Cai 4f91b3069e Skip flaky test in CI for disaggregation decode offload (#20622) 2026-03-15 16:30:34 +08:00
Shangming Cai 99a3b25c9b [PP] Fix recv tensor dict potential race condition (#20341) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-14 13:35:01 +08:00
Shangming Cai af4c28904d [PD] Fix the infinite loop in deocde resolve_pending_reqs (#20371) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-11 14:11:19 -07:00
Shangming Cai ea6ac1d545 Add CODEOWNERS for mooncake transfer engine (#20226) 2026-03-09 20:22:06 -07:00
Shangming Cai c2b66d320d [HiCache] Add an env var to control transfer engine reuse (#19867) 2026-03-04 20:36:32 +08:00
Shangming Cai facde4c6d3 [PD] Enable all CP ranks for KVCache transfer (#19765) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-03 19:35:21 +08:00
Shangming Cai 3f36f27eae [Bugfix] Fix nixl and mori backend for missing decode tp size in PD module (#19690) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-02 19:55:19 +08:00
Shangming Cai 0a6678bf3a [PD] Remove unused server args for disaggregation (#19618) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-03-02 11:38:50 +08:00
Shangming Cai 366574b2b8 [PD] Cleanup BootstrapServer init and ready check (#19551) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-28 16:41:42 +08:00
Shangming CaiandVladislav Nosivskoy b01f3590be [PD] Support PD with context parallel after refactor (#19504) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Vladislav Nosivskoy <vladnosiv@gmail.com>
 2026-02-28 13:11:15 +08:00
Shangming Cai 288300aafd [PD] Tiny code cleanup for prefill info registering (#19414) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-27 01:35:26 +08:00
Shangming Cai e55e65535e [Bugfix] Add rids to the batch filtering for two batch overlap (#19418) 2026-02-26 06:57:25 -08:00
Shangming Cai 97f1fa5e6b [NPU] Fix disaggregation metadata buffer bootstrap_room_dtype for npu backend (#19423) 2026-02-26 21:10:50 +08:00
Shangming Cai 27fd014726 [PD] Add kv_cache_dtype consistency check for PD Disaggregation (#19407) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-26 17:15:58 +08:00
Shangming Cai 8d9ee6669e Fix comment for tp_rank calculation in dp_attention (#19306) 2026-02-25 15:19:10 +08:00
Shangming Cai 0fac2796b6 [PD-Disagg] Improve KVManager init across all backends (#19240) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-25 10:37:09 +08:00
Shangming Cai eccee4c48e [PD] Change bootstrap_room metadata dtype from int64 to uint64 (#19141) 2026-02-22 14:20:16 +08:00
Shangming Cai 1f39bf6523 [Bugfix] Add warnings when NSA indexer cache indice mismatch in PD module (#18727) 2026-02-13 15:20:05 +08:00
Shangming Cai 2a8a48c0ca Reuse initialized transfer engine in mooncake store (#18460) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-13 01:21:35 +08:00
Shangming Cai bffd765417 Refactoring Mooncake TE as a shared distributed component (#17810) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-09 10:53:11 +08:00
Shangming Cai 52401bec1d chore: bump mooncake version to 0.3.9 (#18316) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-07 17:30:01 +08:00
Shangming Cai afae4c7178 [PD] Minor code cleanup for mooncake backend (#18279) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-02-05 17:38:09 +08:00
Shangming Cai 3ad3268e06 [CI] Skip PD hybrid attention test with different TP temporarily (#17791) 2026-01-27 10:59:33 +08:00
Shangming Cai 2262c5c9b5 Add ZhengWG to CI_Permission (#17572) 2026-01-22 17:24:06 +08:00
Shangming Cai 23d765d1f2 [Doc] Update pipeline parallelism documentation (#17414) 2026-01-20 20:10:56 +08:00
Shangming Cai 71cb9d0302 [PD] Fix PP dynamic chunking with DP attention (#17339) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-01-20 11:27:13 +08:00
Shangming Cai 0227db8926 [PD] Optimize MHA models pp util calculation logic (#17306) 2026-01-19 11:23:15 +08:00
Shangming Cai 4c59782e0f Fix hybrid attention PD Disaggregation test (#17099) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-01-15 23:38:58 +08:00
Shangming Cai 4b14f622e1 [CI] Add PD Disaggregation aarch64 test (#16572) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-01-10 14:44:54 +08:00
Shangming Cai 0c4e155a3c chore: bump mooncake version to 0.3.8.post1 (#16792) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-01-09 18:42:27 +08:00
Shangming Cai 973116e6bb [Doc] Optimize pipeline parallelism doc (#16630) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-01-07 14:52:42 +08:00
Shangming Cai 130f60ee78 Add CODEOWNERS for scheduler_pp_mixin.py (#16481) 2026-01-05 21:12:20 +08:00
Shangming Cai 9cac3c8628 Add liusy58 into CI_PERMISSION (#16373) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2026-01-03 21:36:35 -08:00
Shangming Cai db3821a9ef [PP] Add a minimum chunk value for PP dynamic chunking (#16140) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-30 16:35:39 +08:00
Shangming Cai 41addd2e08 chore: bump mooncake version to 0.3.8 (#15886) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-27 23:34:24 +08:00
Shangming Cai b3f83cc1c5 Fix pipeline parallelism doc typos (#15688) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-23 22:19:15 +08:00
Shangming Cai c15fa1c577 [2/N] Update doc of Pipeline Parallelism with case study (#15684) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-23 20:56:15 +08:00
Shangming Cai 17e81c750a [PP] Add dynamic chunking PP test (#15395) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-19 01:09:40 +08:00
Shangming Cai ee1ca51de8 [PP] Fix dynamic chunking strategy for PP (#15372) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-18 14:24:55 +08:00
Shangming Cai eeb2b9b259 [PP] Minor code cleanup for Pipeline Parallelism (#15329) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-17 23:31:25 +08:00
Shangming Cai 6682475124 [CI] Improve flaky 4 GPU test success rate (#15234) 2025-12-17 00:02:26 +08:00
Shangming Caiandybyang 36fcf71fff [Qwen3-next] Add PD disaggregation support for mamba with extra_buffer (#15180) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: ybyang <10629930+whybeyoung@users.noreply.github.com>
 2025-12-16 14:36:00 +08:00
Shangming Cai d277a86dea [CI] Add disaggregation decode PP test (#15114) 2025-12-14 23:54:25 +08:00
Shangming Caiandalpha-baby c8cf1cafdb [1/N] Update doc of Pipeline Parallelism (#14985) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: alpha-baby <fujianhao1997@qq.com>
 2025-12-12 19:32:52 +08:00
Shangming Cai a076d75e98 [Fix] Remove unused import from test_disaggregation_hicache.py (#14880) 2025-12-11 15:50:47 +08:00
Shangming Cai 93452a8252 [PD] Support decode pp for PD disaggregation (#14265) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-12-03 14:35:29 +08:00
Shangming Cai c6d34a0688 Move piecewise cuda graph test to manual dir to fix CI (#14121) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-11-29 00:52:01 -08:00
Shangming Cai b704b0a922 Optimize uneven PP layer distribution logic to improve PP performance (#13977) 2025-11-26 22:05:03 +08:00
Shangming Cai 4c3573e493 [PD] Clarify init method docstrings for kvsender and kvreceiver (#13476) 2025-11-18 12:20:19 +08:00
Shangming Cai c2d4716da0 chore: bump mooncake version to 0.3.7.post2 (#12599) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-11-04 01:09:31 -08:00
Shangming Cai ab8b83f71d chore: upgrade mooncake 0.3.7.post1 (#12541) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-11-03 00:34:32 -08:00
Shangming Cai 358ae3563d Tiny fix eos handling for PD disaggregation (#12334) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-31 17:57:10 -07:00
Shangming Cai 41a113356a Fix potential eos bug on decode instance when PD is enabled (#12206) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-28 01:29:02 -07:00
Shangming Cai 7fceeef599 Fix flaky hicache test with mooncake backend (#11953) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-22 21:00:47 +08:00
Shangming Cai f3cd5d2510 [CI] Fix b200 flashinfer installation (#11915) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-21 22:28:50 +08:00
Shangming Cai 05d3667ab9 [CI] disable glm4.1v and fix the flashinfer installation (#11902) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-21 18:38:35 +08:00
Shangming Cai a2ba0bc3df Tiny clean up for PD module and doc (#11747) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-20 11:52:42 +08:00
Shangming Cai 1de3924b18 [CI] Add GLM4MoE model test (#11706) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-16 16:25:58 +08:00
Shangming Cai 476c67d7fc Fix missing a2a backend init of GLM4.5 MoE Block (#11692) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-15 19:13:08 -07:00
Shangming Caihzh0425ZeldaHuang
 		868403f642 [PD] Add PD support for hybrid model (Qwen3-Next, DeepSeek V3.2 Exp) (#10912) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: hzh0425 <hzh0425@apache.org>
Co-authored-by: ZeldaHuang <hzm414167@alibaba-inc.com>
 2025-10-15 18:59:14 -07:00
Shangming CaiandLiangsheng Yin 6d0364681c Fix 1-step draft model forward (#11653) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
 2025-10-15 19:11:33 +08:00
Shangming Cai c5fe3c0b75 Tiny fix test run estimated time (#11544) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-13 02:23:13 -07:00
Shangming Cai 70fbb3adf6 [CI] Refactor PD disaggregation test suite (#11363) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-09 18:50:39 -07:00
Shangming Cai 0a7c4bded7 [Doc] Update mooncake nvlink transport doc for PD disaggregation (#11321) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-08 00:59:29 -07:00
Shangming Cai c560410da7 Refactor and optimize mooncake CI (#11162) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-05 14:08:52 -07:00
Shangming Cai 2c7f4ca2f2 Optimize debug log position of PD abort request (#11090) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-10-03 23:07:02 +08:00
Shangming Cai 9c339d6b47 [PD] Extract the PP transfer layer calculate logic from Mooncake to Common backend (#10565) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-28 00:10:41 +08:00
Shangming Cai e23e280e16 Add support for topk metadata transferring for PD (#10616) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-28 00:09:38 +08:00
Shangming Cai 23632d350c Fix latest main ci (#10799) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-23 12:46:13 -07:00
Shangming Cai d21c35224d Fix hicache mooncake backend CI (#10792) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-23 02:04:44 -07:00
Shangming Cai 70e4b21853 Fix flaky logprobs test (#10728) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-22 00:46:26 -07:00
Shangming Cai 74cd6e3902 chore: upgrade mooncake 0.3.6.post1 to fix gb200 dockerfile (#10681) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-20 00:12:26 -07:00
Shangming Cai 60fc5b51f6 chore: upgrade mooncake 0.3.6 (#10596) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-18 00:19:30 -07:00
Shangming Cai a13dd1e492 [PD] Improve disaggregation common backend and refactor mooncake backend (#10273) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-17 22:58:07 -07:00
Shangming Cai edd6a07bc0 Minor fix lint introduced by #10466 (#10507) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-16 01:38:25 -07:00
Shangming Cai f5f6b3b4b5 Refactor fused_add_rmsnorm import logic (#10207) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-09 00:23:58 -07:00
Shangming Cai 00974e4f6e [CI] Refactor disaggregation tests (#10068) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-06 22:14:46 +08:00
Shangming Cai a25e8e42eb Move multi-tokenizer event loop to better place (#9902) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-09-01 23:12:21 -07:00
Shangming Cai 25ef53f05f [PD] Fix nvlink transport accuracy through transferring metadata with tcp (#9261) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-08-20 19:29:10 -07:00
Shangming Cai d8ed60f254 [CI] Fix disaggregation failure tolerance CI (#9378) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-08-19 23:31:08 -07:00
Shangming CairootYing ShengFranciszitto
 		384f8ab5ce [PD] Support PD disaggregation with Prefill PP (#8846) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: root <huzhiyuan@xiaohongshu.com>
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Francis <38564764+ssssnow@users.noreply.github.com>
Co-authored-by: zitto <zhjc1124@gmail.com>
 2025-08-16 18:31:31 -07:00
Shangming Cai 8ca07bd948 [CI] Fix sgl-router disaggregation test (#9222) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-08-15 02:24:44 -07:00
Shangming Cai 4caca4f6b4 Fix typo in REVIEWERS (#9113) 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
 2025-08-12 11:55:49 -07:00
Shangming Cai eb429b88a4 [PD] Respect sampling_params.max_new_tokens when PD disaggregation is activated (#7598) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-06-27 22:22:01 -07:00
shangmingc 89a554181f [PD] Fix unclosed prefill connection warning of mini_lb (#5155) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-04-08 09:15:06 -07:00