Commit Graph
711 Commits
Author SHA1 Message Date
Yuwei An 4ac8e09df0 Piecewise CUDA Graph Support & Torch Compile Backend (#10062)
Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>
2025-10-12 11:55:57 +08:00
20a6c0a63d Beta spec-overlap for EAGLE (#11398)
Co-authored-by: Lianmin Zheng <15100009+merrymercy@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
2025-10-12 11:02:22 +08:00
Lorenzo LuandChang Su b5dcfd4154 Add option to disable any_whitespace for xgrammar and llguidance backends. (#8919)
Co-authored-by: Chang Su <chang.s.su@oracle.com>
2025-10-11 22:24:58 +08:00
ykcombat f5754d1256 [Documentation][Configuration] Server args and documentation of PD-Multiplexing. (#11427) 2025-10-11 21:36:07 +08:00
Zhiyu 129d299278 Enable native ModelOpt quantization support (2/3) (#9991)
Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>
2025-10-11 07:48:14 +00:00
Lianmin Zheng 9b8ebb2798 move more files under srt/utils (#11285) 2025-10-09 16:46:15 -07:00
Trevor Morris a4b424c632 [DeepSeek-V3.2] Include indexer kv cache when estimating kv cache size (#11309) 2025-10-08 23:59:46 -07:00
Sundara Raman Ramachandran 53bd00d975 [Generative Score API] Multi-Item scoring with custom attention mask. (#10979) 2025-10-08 18:47:32 -07:00
Lifu HuangandHieu Pham edefab0c64 [2/2] Support MHA prefill with FlashAttention 4. (#10937)
Co-authored-by: Hieu Pham <hyhieu@gmail.com>
2025-10-08 00:54:20 -07:00
Cheng Wan 3c06b673af [8/N] MoE Refactor: deprecate EPMoE (#11211) 2025-10-07 21:51:41 -07:00
Chang Su 7ba3de0e92 [oai serving chat] Add argument --sampling-defaults and fix ChatCompletionRequest defaults (#11304) 2025-10-08 00:36:05 +00:00
Xinyuan Tong e3c7f09146 Update tool parser and related documentation (#11223) 2025-10-07 11:03:40 -07:00
Lianmin Zheng 708f4ff490 Rename max_micro_batch_size -> pp_max_micro_batch_size (#11279) 2025-10-06 15:50:56 -07:00
Zhiyu 155cbb51f0 Enable native ModelOpt quantization support (1/3) (#7149)
Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>
2025-10-06 13:24:15 -07:00
+2 efbc687c28 Support DeepSeek V3.2 Exp (#11061)
Co-authored-by: Stefan He <11166516+hebiao064@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <95566987+hnyls2002@users.noreply.github.com>
Co-authored-by: Baizhou Zhang <56809903+fridge003@users.noreply.github.com>
Co-authored-by: DarkSharpness <76582120+darksharpness@users.noreply.github.com>
Co-authored-by: ZhengdQin <46387172+zhengdqin@users.noreply.github.com>
Co-authored-by: DarkSharpness <2040703891@qq.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Zhengda Qin <zhengdqin@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
2025-10-06 00:24:15 -07:00
Yuan Luoandluoyuan.luo 590f2da052 [Feat] Support Torch Symm Mem AllReduce (#10571)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2025-10-05 13:55:19 -07:00
fzyzcjy fdc4e1e570 Tiny move files to utils folder (#11166) 2025-10-03 22:40:06 +08:00
Matt Nappoandmolocule 8c57490210 [Feature] Option to save model weights to CPU when memory saver mode is enabled (#10873)
Co-authored-by: molocule <34072934+molocule@users.noreply.github.com>
2025-10-03 16:48:19 +08:00
b8zhong a2faf8940c [1/n] Enable DCA CUDA graph capture (#9537) 2025-10-03 11:30:00 +08:00
Lianmin Zheng 6a261aaca5 Minor fixes for server_args, parallel_state, and test_deterministic.py (#11159) 2025-10-02 12:12:49 -07:00
b8zhong 948278f173 fix cpp JIT compilation issue of ngram speculative decoding (#10837) 2025-10-02 21:05:01 +08:00
fzyzcjy 5e786cca3a Support single batch overlap (#10422) 2025-10-02 18:04:36 +08:00
Liangsheng Yin 25e7dbe8af Fix ngram spec with page size > 1 (#11135) 2025-10-02 12:34:23 +08:00
Shisong Maandmashisong 86cb4db058 [Feature] Add EIC as sglang HiCache Storage backend (#10271)
Co-authored-by: mashisong <mashisong@bytedance.com>
2025-10-01 08:43:34 -05:00
Trevor Morris a6cc86df9d Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization (#11081) 2025-09-30 10:33:12 -07:00
Zaili Wang 229d2b95f1 [CPU] Adding Memory Capacity Acquisition Functionality (#11102) 2025-09-30 08:41:20 -07:00
narutolhy d17986f8c6 Enable optional FP32 compute for LM Head (#10729)
Thanks to MiniMax Team and Chenyang Zhao's support.
2025-09-29 20:45:17 -07:00
li-kesenandYi Zhang 2bc61dd194 Remove hybrid_linear_attn attention backend and refactor attention registry (#10816)
Co-authored-by: Yi Zhang <1109276519@qq.com>
2025-09-30 10:16:16 +08:00
Lianmin Zheng dda34c2f93 Fix mem fraction static for nightly tests (#11076) 2025-09-29 12:57:41 -07:00
Lianmin Zhengandsglang-bot a17e70f5cc Use more general heuristics to set the default value of --mem-fraction-static (#10975)
Co-authored-by: sglang-bot <sglangbot@gmail.com>
2025-09-29 10:11:03 -07:00
Zhihao Zhanganda4zhangfei 24f7cb1ece [speculative decoding] rename lookahead to ngram (#11010)
Co-authored-by: a4zhangfei <a4zhangfei@qq.com>
2025-09-28 21:06:59 -07:00
huangtingwei e05555fad8 [HiCacheStorage] mooncake store support page_first_direct layout (#10591) 2025-09-28 20:45:48 -07:00
Mick 2e7633982c fix: show failed models in nightly ci (#10986) 2025-09-28 12:38:29 -07:00
Xinyuan Tongyhyang201Yangmin LiYuan Luogemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
62e2e99db6 fix: make inference deterministic for large TP (#10930)
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Yuan Luo <yuan.luo@hotmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-09-27 02:46:45 -07:00
hzh0425andTeng Ma c8a5d12abe [HiCache]: Support dynamic loading backends for hicache (#10551)
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
2025-09-26 18:34:11 -07:00
05a3526654 Restruct gpu_memory_settings in a unify function and relax max_cuda_graph_bs (#10372)
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: sglang-bot <sglangbot@gmail.com>
2025-09-26 15:10:49 -07:00
hzh0425andShangming Cai 7ec5b4e89c [PD-HiCache]: Support Async Offloading KVCache In Decode Side (#10192)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2025-09-25 23:20:49 -07:00
Lianmin ZhengYingchun Laigemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
f68dd998b9 Rename customer label -> custom label (#10899)
Co-authored-by: Yingchun Lai <laiyingchun@apache.org>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-09-25 16:19:53 -07:00
yi wang fce170480a integrate AIBrix KVcache (#10376) 2025-09-25 14:47:09 +08:00
d7b20dd65d chore: Initial support for input config files (#10534)
Co-authored-by: root <root@umbriel-b200-017.ipp4a1.colossus.nvidia.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
2025-09-24 14:45:52 -07:00
Lianmin Zheng ea338676b5 Clean up server args (#10770) 2025-09-23 00:22:32 -07:00
ronnie_zhengandwangqihui01 095093ee5a [Ascend] optimize Qwen-vl on Ascend (#10556)
Co-authored-by: wangqihui01 <wangqh10@163.com>
2025-09-22 17:18:16 -07:00
Baizhou ZhangandYineng Zhang aa1c5cf5bd Add warnings and remove dependency for deterministic inference (#10724)
Co-authored-by: Yineng Zhang <me@zhyncs.com>
2025-09-22 10:56:02 -07:00
e2ac7888b8 [2/2] Support deterministic inference for temperature > 0 (#10678)
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: hebiao064 <hebiaobuaa@gmail.com>
2025-09-21 19:36:08 -07:00
Ethan (Yusheng) Su 134b4f7ec2 Support deterministic inference with triton backend (#10694) 2025-09-22 09:20:40 +08:00
Lifu Huang 08ecd0aa2a [3/4] Speed up CSGMV backend perf by 10% through dynamic chunking + kernel optimization (#10592) 2025-09-20 22:47:48 -07:00
fzyzcjy 720c1c8ca3 Super tiny fix extra logs (#10697) 2025-09-20 21:30:54 -07:00
Yineng Zhangandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> d403c143e3 feat: update server args (#10696)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-09-20 18:52:19 -07:00
Stefan He cba0d8c309 [Feature] Support deterministic inference with FA3 backend (#10651) 2025-09-20 17:50:21 -07:00
8ecef73f12 [1/2] Support deterministic inference with flashinfer attention backend (#10645)
Co-authored-by: hebiao064 <hebiaobuaa@gmail.com>
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2025-09-19 23:34:29 -07:00