chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
1,651 Commits 1 Branch 0 Tags
287427e2e66aef4e4d857cfd666fe849e9f73617
Commit Graph
249 Commits
Author SHA1 Message Date
Zhiyu 287427e2e6 Enable Nvidia's ModelOpt fp8 quantized models (#2535) 2025-01-06 14:54:52 -08:00
Lianmin Zheng b8574f6953 Clean up eagle code (#2756) 2025-01-06 14:54:18 -08:00
yigexrootHAIBruce XueYineng Zhang
 		c7ae474a49 [Feature, Hardware] Enable DeepseekV3 on AMD GPUs (#2601) 
Co-authored-by: root <root@banff-cyxtera-s83-5.amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
Co-authored-by: Bruce Xue <yigex@xilinx.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2025-01-02 16:23:19 -08:00
Lianmin Zheng bdf946bf81 Support loading pre-sharded moe weights (#2716) 2025-01-02 15:07:37 -08:00
Yineng ZhangandWANDY666 ba5112ff69 feat: support moe_align_block_size_triton (#2712) 
Co-authored-by: WANDY666 <1060304770@qq.com>
 2025-01-02 21:47:44 +08:00
Lianmin Zhengandkavioyu ad20b7957e Eagle speculative decoding part 3: small modifications to the general scheduler (#2709) 
Co-authored-by: kavioyu <kavioyu@tencent.com>
 2025-01-02 02:09:08 -08:00
kkandwunhuang 148254d4db Improve moe reduce sum kernel performance (#2705) 
Co-authored-by: wunhuang <wunhuang@amd.com>
 2025-01-02 01:11:06 -08:00
Xiaoyu Zhang 286cad3ee3 h200 tuning fused_moe_triton config for Mixtral 8x7B/8x22B and Qwen2 57BA14B (#2689) 2024-12-31 23:17:36 +08:00
Lianmin Zhengandyukavio b0524c3789 Eagle speculative decoding part 2: Fix cuda graph + DP attention hanging (#2684) 
Co-authored-by: yukavio <kavioyu@gmail.com>
 2024-12-31 02:25:05 -08:00
Lianmin Zhengandyukavio f44d143949 Support target model verification in the attention backend (#2678) 
Co-authored-by: yukavio <kavioyu@gmail.com>
 2024-12-30 22:58:55 -08:00
Lianmin Zheng 21ec66e59e Minor follow-up fixes for the logprob refactor (#2670) 2024-12-30 05:42:08 -08:00
mobicham a29dd9501d Add GemLite caching after each capture (#2669) 2024-12-30 05:27:29 -08:00
Lianmin Zheng 9c6ba2484f Refactor logprob computation to return the real logprob used in sampling (#2664) 2024-12-30 04:51:38 -08:00
HAI e6f523b5f2 fix typo in python/sglang/srt/layers/quantization/fp8.py (#2655) 2024-12-29 23:45:02 -08:00
HandH1998 afa0341e57 Update Triton configs for block fp8 kernels (#2641) 2024-12-29 22:53:47 +08:00
HAI 30828e7192 AMD: set weights and scaling numbers properly for block FP8 (#2637) 2024-12-29 03:23:39 -08:00
Lianmin Zheng 3815b23ccb Clean up wrapper in flashinfer backend (#2638) 2024-12-29 00:45:57 -08:00
Yineng ZhangandHandH1998 7863e4368a add configs for block fp8 related kernels (#2628) 
Co-authored-by: HandH1998 <1335248067@qq.com>
 2024-12-28 23:12:04 +08:00
Xiaoyu Zhang 9254a33ad4 avoid fused_moe_triton padding circular import (#2624) 2024-12-28 14:01:35 +08:00
HandH1998 6e5305158c update sgl_moe_align_block_size usage (#2617) 2024-12-28 00:01:13 +08:00
kkwunhuangHAI
 		70dc2fbe2d Change extend attention kernel launch parameter for ROCm platform to … (#2610) 
Co-authored-by: wunhuang <wunhuang@amd.com>
Co-authored-by: HAI <hixiao@gmail.com>
 2024-12-27 00:32:17 -08:00
kkandwunhuang 7ca751ff7d Fused moe triton cfg opt for rocm (#2612) 
Co-authored-by: wunhuang <wunhuang@amd.com>
 2024-12-26 23:38:22 -08:00
HAI 7722c11c1d Regression fix to AMD/ROCm from recent change (#2606) 2024-12-26 20:22:14 -08:00
Lianmin Zheng dc3bee4815 Fix test and benchmark scripts (#2598) 2024-12-26 07:56:26 -08:00
Xiaoyu Zhang 9a23c48456 h100 tuning fused_moe_triton for qwen2 moe (#2560) 2024-12-26 03:13:31 -08:00
Yineng Zhang 635a042623 docs: update deepseek v3 example (#2592) 2024-12-26 17:43:37 +08:00
Yineng ZhangispobockHandH1998
 		60e2fdcf4f use sgl-kernel moe_align_block_size (#2581) 
Co-authored-by: ispobock <ispobaoke@163.com>
Co-authored-by: HandH1998 <1335248067@qq.com>
 2024-12-26 06:29:08 +08:00
HandH1998andzhyncs 53aed988cb Refactor MoE (#2575) 
Co-authored-by: zhyncs <me@zhyncs.com>
 2024-12-26 00:02:14 +08:00
Ke Bao e835a50021 Reorg moe code (#2563) 2024-12-24 01:10:22 +08:00
Lianmin Zheng 23e5e50fd5 Fix gemlite import (#2553) 2024-12-22 20:21:17 -08:00
Lianmin Zheng 8496701934 [Misc] Fix metrics, weight update lock, request logging (#2543) 2024-12-22 06:27:22 -08:00
Jerry Zhang feb2b768ba Add integration with gemlite weight only quant (#2528) 2024-12-21 00:25:25 +08:00
Lianmin Zheng ba36b5520a Revert "Small fixes for torchao quant" (#2493) 2024-12-16 15:04:16 -08:00
Lianmin Zheng 7a1aecb938 Simplify pytorch sampling kernel and logit processor (#2491) 2024-12-16 14:11:09 -08:00
Jerry Zhang 82699474fd Small fixes for torchao quant (#2476) 2024-12-16 14:08:12 -08:00
xiaobochen b532a5fd16 fix moe-ep accuracy issue for fp8 (#2489) 2024-12-16 20:54:02 +08:00
Yineng Zhang 5f2595be43 hotfix: checking for HIP (#2485) 2024-12-15 02:47:26 +08:00
Ke Bao 2f9bd0fafd Fix correctness issue for triton decoding kernel (#2479) 2024-12-14 16:50:54 +08:00
Fred Reiss 993956c6b1 Add support for IBM Granite 3.x models (#2437) 2024-12-11 06:30:23 -08:00
Lianmin Zheng 641b7d0ae0 [Minor] Improve code style (#2422) 2024-12-09 06:30:35 -08:00
Xiaoyu Zhang 3844feb9bb Add a unittest for fused_moe (#2416) 2024-12-08 22:46:10 -08:00
Lianmin Zheng a6ca736c8e Simplify stream_output (#2398) 2024-12-08 12:27:13 -08:00
Yineng Zhang 6128f7cff5 fix: specify dtype with begin_forward aka plan (#2404) 2024-12-08 20:07:30 +08:00
Ke Bao 61dec545b0 Remove unused vars in the triton backend (#2401) 2024-12-08 03:37:03 -08:00
Ke Bao 7dc66fcb40 Optimize Triton decoding kernel for long context (#2394) 2024-12-08 01:17:37 -08:00
HAI 95f93f493a Fp8 MoE optimizations on AMD (#2388) 2024-12-07 21:18:26 +08:00
Yineng Zhang d332aa3b0c fix: resolve fp8 moe issue (#2387) 2024-12-07 19:28:53 +08:00
Qun Yang 37ee906f61 Add more support for intel Gaudi accelerators (#2357) 2024-12-06 01:16:33 -08:00
Yineng ZhangandHaiShaw 84d96b3ae5 Move FP8 to SGLang (#2370) 
Co-authored-by: HaiShaw <hixiao@gmail.com>
 2024-12-06 15:42:10 +08:00
xiaobochen 3d32e4a32c Resubmit MoE-EP (#2371) 2024-12-06 15:05:21 +08:00