chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
3,265 Commits 1 Branch 0 Tags
cfc9f9ab8d7eb2bfd91511fc34b159886b7edbf7
Commit Graph

50 Commits

Author SHA1 Message Date
quinnrong94
2e4babdb0a [Feat] Support FlashMLA backend with MTP and FP8 KV cache (#6109) 
Co-authored-by: Yingyi <yingyihuang2000@outlook.com>
Co-authored-by: neiltian <neiltian@tencent.com>
Co-authored-by: lukec <118525388+sleepcoo@users.noreply.github.com>
Co-authored-by: kexueyu <kexueyu@tencent.com>
Co-authored-by: vincentmeng <vincentmeng@tencent.com>
Co-authored-by: pengmeng <pengmeng@tencent.com>
 2025-05-15 00:48:09 -07:00
Lianmin Zheng
e8e18dcdcc Revert "fix some typos" (#6244) 2025-05-12 12:53:26 -07:00
applesaucethebun
d738ab52f8 fix some typos (#6209) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-05-13 01:42:38 +08:00
Lianmin Zheng
fba8eccd7e Log if cuda graph is used & extend cuda graph capture to cuda-graph-max-bs (#6201) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-05-12 00:17:33 -07:00
applesaucethebun
2ce8793519 Add typo checker in pre-commit (#6179) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-05-11 12:55:00 +08:00
Ying Sheng
11383cec3c [PP] Add pipeline parallelism (#5724) 2025-04-30 18:18:07 -07:00
Lianmin Zheng
21514ff5bd Disable flaky eagle tests (#5753) 2025-04-25 15:54:39 -07:00
JieXin Liang
97cb762bb6 [misc] remove is_cuda_available (#5319) 2025-04-20 18:16:51 -07:00
u4lr451
211c7b31b8 Fix: Incorrect parameters passed to forward_batch_generation (#5506) (#5511) 2025-04-17 18:49:59 -07:00
Yineng Zhang
4cb53ecd0c fix: log warning when disable cuda graph (#5209) 2025-04-09 14:16:13 -07:00
fzyzcjy
86a876d883 Optimize topk operation in llama4 (#5128) 2025-04-09 02:50:22 -07:00
Baizhou Zhang
efbae697b3 [Revision] Replace enable_flashinfer_mla argument with attention_backend (#5052) 2025-04-05 01:23:02 -07:00
Qingquan Song
e983e43248 Add Eagle Speculative Decoding to FA3 Backend (#4951) 
Co-authored-by: hebiao064 <hebiaobuaa@gmail.com>
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com>
Co-authored-by: zcnrex <zcnrex@gmail.com>
 2025-04-02 13:09:02 -07:00
Baizhou Zhang
e62d60fe6d [Fix] avoid stream sync and torch compile in prefill for fa3 backend (#4932) 2025-03-30 13:53:44 -07:00
Lianmin Zheng
b26bc86b36 Support page size > 1 + eagle (#4908) 2025-03-30 00:46:23 -07:00
Brayden Zhong
e84f4ba0ab [Misc] Fix issues reported by torchfix (#4837) 2025-03-27 20:10:32 -07:00
Alex Sun
af6535e7aa [ROCm] Enable MTP (NextN) on AMD GPU (#4631) 2025-03-23 22:58:05 -07:00
Baizhou Zhang
90532b7627 [Fix] Fix raw_bs bug when using flashinfer mla and eagle (#4557) 2025-03-18 21:26:53 -07:00
James Liu
9e0186f352 [Feature] Support EAGLE 3 (#4247) 2025-03-18 07:35:23 -07:00
Ying Sheng
1b859295f4 [Eagle] Remove the greedy branch and some redundant code (#4363) 
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-16 02:48:55 -07:00
JieXin Liang
1a3fa75f2f [Fix] use torch.cat instead of torch.concat to prevent entering the Autograd backends. (#4466) 2025-03-16 00:02:47 -07:00
Baizhou Zhang
9fb48f951f Support nextn for flashinfer mla attention backend (#4218) 2025-03-09 00:01:54 -08:00
Lianmin Zheng
d4017a6b63 [EAGLE] many fixes for eagle (#4195) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-07 22:12:13 -08:00
Lianmin Zheng
bc1534ff32 Fix a draft model accuracy bug in eagle; support step=1; return logprob in eagle (#4134) 
Co-authored-by: Sehoon Kim <kssteven418@gmail.com>
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-06 06:13:59 -08:00
Ying Sheng
d3d4d76758 [Eagle] Refactor eagle speculative decoding (#3986) 
Co-authored-by: Ke Bao <ISPObaoke@163.com>
 2025-03-05 08:06:07 -08:00
Lianmin Zheng
77a3954bf7 Simplify eagle tests and TP sync in grammar backend (#4066) 2025-03-04 13:40:40 -08:00
William
0d4e3228cf [Feature] Add test for speculative_token_map (#4016) 2025-03-04 04:26:24 -08:00
Ke Bao
9fafa62db7 Share target model embed and head weights for nextn (#4033) 2025-03-03 13:30:04 -08:00
Zhousx
7fbab730bd [feat] add small vocab table for eagle's draft model[1]. (#3822) 
Co-authored-by: Achazwl <323163497@qq.com>
Co-authored-by: Chayenne <zhaochen20@outlook.com>
 2025-03-02 18:58:45 -08:00
Ke Bao
862dd76c76 Support NextN (MTP) speculative decoding for DeepSeek-V3/R1 (#3582) 2025-02-15 05:28:34 +08:00
Ying Sheng
d23cb9a01e [Eagle] reduce one draft forward (#3468) 2025-02-10 20:21:49 +08:00
Ke Bao
2d61132374 Support Eagle2 for Triton backend (#3466) 2025-02-10 20:00:42 +08:00
Yineng Zhang
1646149a83 fix draft cuda graph capture failure (#3431) 2025-02-09 23:16:20 +08:00
Yineng Zhang
fad315cb8e fix EAGLE 2 non greedy case (#3407) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2025-02-09 07:28:34 +08:00
Yineng Zhang
fa82dfccdd fix EagleVerifyInput (#3378) 2025-02-07 22:30:43 +08:00
Yineng Zhang
f9905d59a8 support speculative decoding kernel in sgl-kernel (#3373) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2025-02-07 20:29:51 +08:00
Yineng Zhang
d39899e85c upgrade flashinfer v0.2.0.post2 (#3288) 
Co-authored-by: pankajroark <pankajroark@users.noreply.github.com>
 2025-02-04 21:41:40 +08:00
Yineng Zhang
013021b6a1 refactor EAGLE 2 (#3269) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
 2025-02-03 20:52:30 +08:00
Lianmin Zheng
1dda8c5e4c Return more infos for computing average acceptance length (#3152) 2025-01-26 04:51:54 -08:00
Lianmin Zheng
287d07a669 Misc fixes for eagle (flush_cache, CPU overhead) (#3014) 2025-01-20 20:27:38 -08:00
996_icu
b730aa6b9e [EAGLE] Fix some boundary situation when retract reqs and req's max token = 1 (#2939) 
Co-authored-by: josephyou <josephyou@tencent.com>
 2025-01-20 17:46:43 -08:00
justdoit
4093aa4660 [Fix]eagle2 health_generate is first request,apiserver will core (#2853) 2025-01-13 01:01:21 -08:00
justdoit
a47bf39123 [Eagle2] Fix multiple concurrent request crashes (#2730) 2025-01-10 14:00:43 -08:00
Lianmin Zheng
8a6906127a Improve linear.py to load sharded weights & remove the dependency of Parameters from vllm (#2784) 
Co-authored-by: SangBin Cho rkooo567@gmail.com
 2025-01-07 23:29:10 -08:00
JJJJOHNSON
694e41925e [eagle2] fix end check when target model verify (#2723) 2025-01-07 21:46:02 -08:00
Lianmin Zheng
b8574f6953 Clean up eagle code (#2756) 2025-01-06 14:54:18 -08:00
yukavio
8c8779cd05 [Fix] fix retract error in eagle speculative decoding (#2711) 
Co-authored-by: kavioyu <kavioyu@tencent.com>
 2025-01-02 10:28:39 -08:00
yukavio
815dce0554 Eagle speculative decoding part 4: Add EAGLE2 worker (#2150) 
Co-authored-by: kavioyu <kavioyu@tencent.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
 2025-01-02 03:22:34 -08:00
Lianmin Zheng
ad20b7957e Eagle speculative decoding part 3: small modifications to the general scheduler (#2709) 
Co-authored-by: kavioyu <kavioyu@tencent.com>
 2025-01-02 02:09:08 -08:00
Lianmin Zheng
3815b23ccb Clean up wrapper in flashinfer backend (#2638) 2024-12-29 00:45:57 -08:00