Commit Graph
381 Commits
Author SHA1 Message Date
MOHENOOandyangjia1 534ac384db [HiCacheStorage & PD] fix prefill bootstrap request host memory leaks (#15439)
Co-authored-by: yangjia1 <yangjia1@kingsoft.com>
2026-01-07 11:03:57 +08:00
fba785c459 use aync load for encoder_server (#15456)
Signed-off-by: liuanqi <liuanqi6@xiaomi.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: liuanqi <liuanqi6@xiaomi.com>
2026-01-06 14:31:52 +08:00
Raayan DharandShangming Cai 4397cda7dc feat: raise error in PD when page sizes are mismatched (#14474)
Signed-off-by: Raayan Dhar raayan.dhar@gmail.com <raayan.dhar@gmail.com>
Signed-off-by: raayandhar <raayan.dhar@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-01-05 19:54:01 +08:00
Netanel Haber bebd625ba1 EVS Framework: Support NemotronH_Nano_VL_V2 (#14051) 2026-01-05 16:18:07 +08:00
Ke Bao 76bc07a335 Move swa memory pool to a seperate file (#16347) 2026-01-04 22:39:30 +08:00
Hudson Xing 8a9ca41fda Move log_prefill_stats_late to correct location in PP mode (#15946) 2025-12-31 12:00:24 +08:00
Jumiarandliusy58 664f611e83 Add profiling capture support to the encoder server (#15730)
Signed-off-by: liuanqi <liuanqi6@xiaomi.com>
Co-authored-by: liusy58 <liusy58@linux.alibaba.com>
2025-12-30 19:56:25 +08:00
Cheng Wan 60f1ca6925 Refactor: Moving extend_logprob_start_len calculation out of prepare_for_extend (#16105) 2025-12-30 12:38:33 +08:00
Cheng Wan c2e0913e17 Fix extend_input_len calculation in decode.py (#16103) 2025-12-29 13:13:24 -08:00
Hudson Xing 5c393e8153 Fix temp_prefill_info assertion error in PP disaggregation mode (#15943) 2025-12-27 22:46:55 +08:00
Huaixin ChangandJunjie Mao cf34d0ab32 [Fix] assert error in log_prefill_stats (#15881)
Signed-off-by: Chang Huaixin (OpenAnolis) <changhuaixin@linux.alibaba.com>
Co-authored-by: Junjie Mao (Alibaba) <banxing.mjj@alibaba-inc.com>
2025-12-26 19:54:35 +08:00
Tianyu Guo fa2966983a Support PP for zmq_to_scheduler (#15312) 2025-12-23 17:07:55 +08:00
Baidu-AIAKandsunhailiang bc3ca30023 [PD] Support fake decode for PD disaggregation without prefill node (#14628)
Co-authored-by: sunhailiang <sunhailiang@baidu.com>
2025-12-23 12:43:33 +08:00
Liangsheng Yinandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 3c882db3ad Adjust wrong mtp meaning introduce by mimo (#15632)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-23 02:06:46 +08:00
+6 160a06cab2 [Feature] Xiaomi MiMo-V2-Flash day0 support (#15207)
Co-authored-by: 谢学扬 <xiexueyang@xiaomi.com>
Co-authored-by: tz <tangzhen3@xiaomi.com>
Co-authored-by: 李家乐 <lijiale10@xiaomi.com>
Co-authored-by: 张晨 <zhangchen50@xiaomi.com>
Co-authored-by: Shaohui Liu <liushaohui3@xiaomi.com>
Co-authored-by: 王晨 <wangchen77@xiaomi.com>
Co-authored-by: jiangzihan <jiangzihan@xiaomi.com>
Co-authored-by: xiexueyang <xyxie_wangyi@163.com>
Co-authored-by: Linghao Zhang <zhanglinghao@xiaomi.com>
Co-authored-by: ispobock <ispobaoke@gmail.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: JoyFuture <35593546+JoyFuture@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: root <root@bj9-ml-g8h20e-k8s-slave106-20251106.alicn.idc.xiaomi.com>
2025-12-19 11:40:07 +08:00
Lianmin Zheng d1f0063262 Clean up __init__ function of the scheduler and event loop for PD (#15298) 2025-12-18 01:35:14 -08:00
Scott Lee 169a75dfb8 Add request-level timestamp for when prefill finishes (#14860) 2025-12-17 13:34:46 -08:00
Shangming Caiandybyang 36fcf71fff [Qwen3-next] Add PD disaggregation support for mamba with extra_buffer (#15180)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: ybyang <10629930+whybeyoung@users.noreply.github.com>
2025-12-16 14:36:00 +08:00
Hanming Lu e61dabf5e4 [Qwen3-next] support mamba radix cache for overlap scheduler (#14792) 2025-12-14 18:54:16 -08:00
9acb21ae27 feat: support EPD disaggregation (#12263)
Co-authored-by: liusy58 <liusy58@linux.alibaba.com>
Co-authored-by: ZhengWG <zwg0606@gmail.com>
Co-authored-by: Nicholas <45984215+liusy58@users.noreply.github.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com>
2025-12-14 22:30:08 +08:00
YAMY 0e7d7969d5 [PP Prefill][NIXL] Fix PP mode transfer completion tracking to wait for all ranks (#15027) 2025-12-13 00:55:28 -08:00
Kevin LiandShangming Cai 8fa8d9d7e8 [PD] Add decode PP event loop for PD disaggregation (#14945)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2025-12-12 20:15:00 +08:00
c01b2ee094 [PP] Refactor PP to async mode (#11852)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Signed-off-by: Xuchun Shang <xuchun.shang@gmail.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: bluecoffee8 <jasperli2002@gmail.com>
Co-authored-by: zhangxiaolei123456 <zhangxiaolei.666@bytedance.com>
Co-authored-by: ybyang <10629930+whybeyoung@users.noreply.github.com>
2025-12-12 12:54:16 +08:00
Rain H 32a32cf7d2 Enhance prefill PP node robustness (#14494) 2025-12-06 18:00:54 -08:00
Kevin Li 04df80a9a1 Support PP x PD decode with nixl backend (#14392) 2025-12-04 12:22:17 +08:00
Shangming Cai 93452a8252 [PD] Support decode pp for PD disaggregation (#14265)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2025-12-03 14:35:29 +08:00
Vikramandjinke15 42271376d1 [bug fix] use npu phy id in container env (#14266)
Co-authored-by: jinke15 <jinke15@jd.com>
2025-12-03 11:33:43 +08:00
Lianmin Zhenggithub-actions[bot] <github-actions[bot]@users.noreply.github.com>Hanming LuHanming Lu
64092c8b55 [Auto Sync] Rename is_hybrid to is_hybrid_swa (#14252)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Hanming Lu <hanming@x.ai>
2025-12-01 23:24:24 -08:00
Kartik Ramesh 11ce05163d Fix NIXL exception message (#14172) 2025-12-02 10:39:45 +08:00
Teng MaandShangming Cai f5f3a5d98c [PD] Support json file configuration for Transfer Engine (#14059)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2025-12-01 10:47:33 +08:00
Liangsheng Yin eff6a07c8f Fix get_load API (#13991) 2025-11-26 22:07:51 +08:00
vipwangerxiaoandPeng Wang 15729dbc8e Use dynamically maintained num_waiting_tokens in get_load() (#13203)
Signed-off-by: Peng Wang <peng_wang@linux.alibaba.com>
Co-authored-by: Peng Wang <peng_wang@linux.alibaba.com>
2025-11-26 20:34:45 +08:00
luchangli 64225a8ae9 fix nixl prefill crash make decode health check failed (#13657)
Signed-off-by: liluchang <liluchang@kingsoft.com>
2025-11-25 21:40:11 +08:00
Liangsheng Yin 981ca8313f [misc] Rename minilb install env & remove files & fix lint (#13831) 2025-11-24 13:50:00 +08:00
Liangsheng Yin 863124684c [Spec v2] Remove allocate_lens and enable over-allocation (#13478) 2025-11-22 22:49:10 +08:00
Mike QiuandMike_Qiu 6be65ae462 Fix target MLA with eagle3 support for PD disaggregation (#13555)
Signed-off-by: Michael Qiu <qiudayu.qdy@antgroup.com>
Co-authored-by: Mike_Qiu <qiudayu.qdy@antgroup.com>
2025-11-21 12:16:05 +08:00
Shangming Cai 4c3573e493 [PD] Clarify init method docstrings for kvsender and kvreceiver (#13476) 2025-11-18 12:20:19 +08:00
Mike QiuandMike_Qiu 9ba3597d2a Fix cache_tokens calculate issue when retracted (#11900)
Signed-off-by: Michael Qiu <qiudayu.qdy@antgroup.com>
Co-authored-by: Mike_Qiu <qiudayu.qdy@antgroup.com>
2025-11-17 21:25:38 +08:00
Xuchun ShangandShangming Cai 80797c2a15 [PD][bug fix] fix memleak when last_batch is none (#13144)
Signed-off-by: Xuchun Shang <xuchun.shang@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2025-11-17 21:22:20 +08:00
Even Zhou 2aec8b6e1b [Feature] Spec-Overlap supporting DP-ATTN; PD-Disaggregation; npugraph mode (#12443) 2025-11-15 21:51:07 +08:00
Yingchun Lai 875a25ddef refactor: remove duplicate function _get_bootstrap_info_from_server (#13277) 2025-11-15 02:48:17 +08:00
huangtingwei 4ac65e3c7d [PD / HiCache]fix deocde kvcache offload manager memory leak (#12774) 2025-11-14 03:31:53 +08:00
Jimmyandwangchao b40f605fde fix(tcp-port): replace bind_server_socket to get_zmq_socket(Port conflict) (#11961)
Co-authored-by: wangchao <wcsjtu@163.com>
2025-11-12 18:05:16 +08:00
Teng MaandShangming Cai 44e391b6e9 [PD] Add custom gpu id to device topo support (#12817)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2025-11-12 15:00:45 +08:00
Ziming Huang 5de25f786e [BugFix] Fix prefill memory leak in PD + GDN (#12994) 2025-11-12 00:48:07 +08:00
Feng Su fc8cda14cf Sglang Tracing: optimize trace_event_batch() (#13036)
Signed-off-by: Feng Su <sufeng@linux.alibaba.com>
2025-11-11 15:02:20 +08:00
Liangsheng Yin 665416f6dd Unify memory management across (overlap, non-overlap) x (page>=1) x (spec, non-spec, spec v2) x (retract, finished) (#12224) 2025-11-11 02:56:22 +08:00
Liangsheng Yin f1f4c451ab Add process_prefill_chunk back to fix PP event loop (#13009) 2025-11-11 00:51:36 +08:00
Liangsheng Yin dc8a5a1ce7 [Refactor / Style] Unify all event loops (except for PP) (#12959) 2025-11-10 14:49:12 +08:00
Liangsheng Yin 61c7fe7aed Minor code cleanup / improvement for PREBUILT_EXTEND mode (#12948) 2025-11-10 11:59:54 +08:00