chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
4,796 Commits 1 Branch 0 Tags
9c8e4f69c363c5063d5570f54ff6b9e8b1906746
Commit Graph

264 Commits

Author SHA1 Message Date
Hongbo Xu
9c8e4f69c3 [5/n]decouple quantization implementation from vLLM dependency (#9454) 2025-08-21 12:52:07 -07:00
fzyzcjy
e85cb1ce9d Fix quant kernel test errors and benchmark wrong output speeds (#7604) 2025-08-21 03:48:41 -07:00
Martin Vit
7cd2ee06d7 feat: Add Triton fallback option and SM120 MoE configs for FP8 models (#9251) 2025-08-20 19:33:15 -07:00
Trevor Morris
a91e90d9a3 [2/2] Fuse routed scaling factor into select_experts (#8690) 2025-08-20 15:10:16 -07:00
Enrique Shockwave
e483ab6d20 enable marlin fp8 blockwise (#8990) 2025-08-18 18:53:15 -07:00
Jiaqi Gu
3c2c9f6c9e [Bug] Fix input arguments of flashinfer_trtllm_moe (#9317) 2025-08-18 18:03:19 -07:00
Zhiyu
2256d62d36 Modelopt quant config adaptation (#8829) 2025-08-18 11:27:30 -07:00
b8zhong
716e682721 [Fix] Add undefined update_tensor_inplace function (#6307) 2025-08-18 11:11:00 +08:00
blzheng
ebbb75e917 [CPU] Fix TP padding issue on Phi-4 (#8289) 2025-08-17 16:25:26 -07:00
fzyzcjy
b498cd21d7 Tiny make fp4 moe method parameters more static (#8520) 2025-08-17 13:26:02 -07:00
kk
1c1f8a118e Combine fp4.py and mxfp4.py into one file and support dynamic mxfp4 quantization in mxfp4.py (#9049) 
Co-authored-by: wunhuang <wunhuang@amd.com>
 2025-08-16 19:01:54 -07:00
Trevor Morris
eff4eb3fdd Add fp4 quantize before all-gather for Flashinfer cutlass MoE DP (max throughput) (#7667) 2025-08-15 22:08:11 -07:00
Cheng Wan
84b006b278 Cleanup MoE Refactor (#9223) 2025-08-15 02:28:33 -07:00
Cheng Wan
295895120d [6/N] MoE Refactor: Cleanup MoE-related configs (#8849) 2025-08-14 21:14:53 -07:00
Hongbo Xu
2cc9eeab01 [4/n]decouple quantization implementation from vLLM dependency (#9191) 
Co-authored-by: AniZpZ <aniz1905@gmail.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2025-08-14 12:05:46 -07:00
Xiaoyu Zhang
63d82a776a refine mxfp4 shuffling log (#9194) 2025-08-14 10:57:29 -07:00
Peng Zhang
5aa1ebd242 [2/n]decouple quantization implementation from vLLM dependency (#8112) 
Co-authored-by: walker-ai <yiyun.wyt@antgroup.com>
Co-authored-by: leoneo <1320612015@qq.com>
 2025-08-14 03:19:03 -07:00
Alex Yang
1bc183c6de Faster weight processing (trtllm-gen moe nvfp4) (#9162) 2025-08-13 21:09:34 -07:00
Hongbo Xu
a669bc2f74 Replace sglang.srt.layers.quantization.scalar_types with sgl_kernel.scalar_type (#8951) 2025-08-13 19:41:41 -07:00
Stefan He
930fe467bd Support Triton FP8 Gemm can handle hidden_dim not divisible by 16 (#9093) 
Co-authored-by: Xiaoyu Zhang <35585791+BBuf@users.noreply.github.com>
 2025-08-12 21:21:55 -07:00
Elfie Guo
8723b4f146 Use FlashInfer's TRTLLM FP8 Blockscale GEMM (#8588) 2025-08-12 20:08:40 -07:00
ichernob
83123f481e [Quantization] Supported w8a8 int8 quantized Gemma3 and Qwen-VL models (#8619) 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
 2025-08-12 13:31:18 -07:00
fzyzcjy
5190ba7f42 Fuse two kernels of hidden states padding into quantization kernel (#9005) 
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
 2025-08-12 01:20:13 -07:00
Jianwei Dong
83262dcb29 Fix mismatch between padded_scales shape and reshape dimensions in modelopt quantization (#8766) 2025-08-11 23:44:40 -07:00
SijiaYang
90f44b74e6 fix: w4afp8 accuracy problem and rebase (#8752) 
Signed-off-by: yangsijia.614 <yangsijia.614@bytedance.com>
Co-authored-by: Jinwu <ayrnb@users.noreply.github.com>
 2025-08-11 13:41:19 -07:00
Lianmin Zheng
9a44b643c6 Fix CI (#9012) 2025-08-09 13:33:42 -07:00
Even Zhou
137e75daa1 [Feature] Optimize DeepSeek's DeepEP on Ascend NPU (#8355) 
Co-authored-by: ronnie_zheng <zl19940307@163.com>
Co-authored-by: Hexq0210 <hexq0809521@gmail.com>
 2025-08-09 01:35:00 -07:00
tql.99
3f2e315f6e optimize: reduce shulffle and quantization overhead in cutlass_moe sm90 (#8962) 
Co-authored-by: 戚余航 <qiyuhang@bytedance.com>
 2025-08-09 00:29:12 -07:00
Lianmin Zheng
706bd69cc5 Clean up server_args.py to have a dedicated function for model specific adjustments (#8983) 2025-08-08 19:56:50 -07:00
Cheng Wan
1d24db8348 Expert Parallelism for GPT-OSS (#8944) 2025-08-08 00:46:42 -07:00
Xiaoyu Zhang
0d1e27a0c5 Better optimization log for gpt-oss model (#8953) 2025-08-08 00:11:48 -07:00
Stefan He
aaf0ad8cdf remove vllm fp8quant from fp8.py (#8937) 2025-08-07 15:50:52 -07:00
Xiaoyu Zhang
3ae33fcd0a Fix hopper launch gpt-oss model illegal memory (#8908) 2025-08-07 10:02:40 -07:00
Xiaoyu Zhang
47824c1488 [Perf] Auto enable best flashinfer mxfp4 kernel in b200 (#8898) 2025-08-07 01:08:41 -07:00
Xiaoyu Zhang
4373df5525 add flashinfer mxfp4 (#8847) 2025-08-06 16:23:41 -07:00
Shu Wang
288ae41f7a [NVIDIA] Fix num_experts in modelopt_quant (#8811) 2025-08-06 14:35:07 -07:00
Ying Sheng
168033d5fb Support mxfp4 for GPT-OSS (#8843) 
Co-authored-by: Co-author fzyzcjy <ch271828n@outlook.com>
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
Co-authored-by: zhuofan1123 <zhuofanl@nvidia.com>
Co-authored-by: liz-badada <jinyanc@nvidia.com>
Co-authored-by: xutizhou <xutingz@nvidia.com>
Co-authored-by: linhu-nv <linhu@nvidia.com>
 2025-08-06 00:05:25 -07:00
Ying Sheng
c1d2061f97 Add initial support for gpt-oss (#8824) 2025-08-05 13:42:01 -07:00
Yineng Zhang
5e91fed1c5 Revert "[NVIDIA]Fix local_num_experts for EP (#8779)" (#8797) 2025-08-04 23:30:43 -07:00
Shu Wang
b01eeb80f8 [NVIDIA]Fix local_num_experts for EP (#8779) 2025-08-04 22:01:14 -07:00
kk
d4bf5a8524 Support OCP MXFP4 quantization on AMD GPUs (#8255) 
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: Hubert Lu <Hubert.Lu@amd.com>
 2025-08-04 18:14:52 -07:00
Trevor Morris
9bd4872a34 [bugfix] Fix typo in modelopt quant: 'FusedMoE' object has no attribute 'local_num_experts' (#8768) 2025-08-04 11:08:08 -07:00
azhurkevich
915140fd18 [NVIDIA] Add Low Latency NVFP4 decode kernels from Flashinfer (#8552) 
Co-authored-by: Cheng Wan <cwan@x.ai>
 2025-08-04 03:10:02 -07:00
Yuan Luo
3b87a9e8ae Fix bug of refactoring TopKOutput in w4afp8 (#8745) 
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
 2025-08-03 20:05:02 -07:00
tql.99
e67276ecb3 feat: support cutlass_moe_fp8 kernel for fusedmoe in sm90 (#8678) 2025-08-03 10:47:15 -07:00
Cheng Wan
a437aa9987 [hotfix] fix mixtral with tensor-level compressed-tensor quantization (#8721) 2025-08-02 22:59:25 -07:00
fzyzcjy
403566bcca Remove assertions about per group quant fp8 (#8717) 2025-08-02 17:08:40 -07:00
Trevor Morris
89caf7a3c6 [bugfix] Apply routed scaling factor to cutlass_fused_experts_fp8 (#8688) 2025-08-01 19:00:24 -07:00
Ke Bao
e252192679 Fix deepgemm masked grouped gemm jit compile (#8679) 2025-08-01 15:37:59 -07:00
Kaixi Hou
aa4c66b564 [NVIDIA] Enable Flashinfer MoE blockscale fp8 backend for TP MoE (#8450) 
Co-authored-by: kushanam <42385577+kushanam@users.noreply.github.com>
 2025-07-31 19:56:34 -07:00