chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
2,488 Commits 1 Branch 0 Tags
c0e9a36c5f5e8e018663a3b84c401780c5999381
Commit Graph

92 Commits

Author SHA1 Message Date
JieXin Liang
c0e9a36c5f Optimize Triton decoding kernel for dynamic workload (#4553) 2025-03-18 21:25:38 -07:00
James Liu
9e0186f352 [Feature] Support EAGLE 3 (#4247) 2025-03-18 07:35:23 -07:00
Lianmin Zheng
5493c3343e Fix data parallel + tensor parallel (#4499) 2025-03-17 05:13:16 -07:00
Ying Sheng
1b859295f4 [Eagle] Remove the greedy branch and some redundant code (#4363) 
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-16 02:48:55 -07:00
Lianmin Zheng
8e66fbecee Improve DP attention (#4390) 
Co-authored-by: dhou-xai <dhou@x.ai>
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
 2025-03-13 08:23:56 -07:00
Lianmin Zheng
c76040e31b Support page size > 1 (#4356) 2025-03-12 22:22:39 -07:00
Yineng Zhang
d1da58e275 unify is_cuda and is_hip (#4321) 2025-03-11 18:12:56 -07:00
Lianmin Zheng
08c4d764a5 lazy import attn backends (#4200) 2025-03-08 00:41:35 -08:00
Lianmin Zheng
d4017a6b63 [EAGLE] many fixes for eagle (#4195) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-07 22:12:13 -08:00
Lianmin Zheng
bc1534ff32 Fix a draft model accuracy bug in eagle; support step=1; return logprob in eagle (#4134) 
Co-authored-by: Sehoon Kim <kssteven418@gmail.com>
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: Sehoon Kim <sehoon@x.ai>
 2025-03-06 06:13:59 -08:00
Lianmin Zheng
98c73d71cb [Minor] make the __init__ function of model_runner.py shorter (#4132) 2025-03-06 01:51:12 -08:00
Baizhou Zhang
fc91d08a8f [Revision] Add fast decode plan for flashinfer mla (#4012) 2025-03-05 11:20:41 -08:00
Lianmin Zheng
e074d84e5b [Minor] more code cleanup (#4077) 2025-03-04 21:23:47 -08:00
Lianmin Zheng
ac2387279e Support penalty in overlap mode; return logprob with chunked prefill; improve benchmark scripts (#3988) 
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: dhou-xai <dhou@x.ai>
Co-authored-by: Hanming Lu <hanming_lu@berkeley.edu>
 2025-03-03 00:12:04 -08:00
Lianmin Zheng
9e1014cf99 Revert "Add fast decode plan for flashinfer mla" (#4008) 2025-03-02 19:29:10 -08:00
Baizhou Zhang
fa56106731 Add fast decode plan for flashinfer mla (#3987) 2025-03-02 19:16:37 -08:00
Qiaolin Yu
40782f05d7 Refactor: Move return_hidden_states to the generate input (#3985) 
Co-authored-by: Beichen-Ma <mabeichen12@gmail.com>
 2025-03-01 17:51:29 -08:00
Qiaolin Yu
d6898dd253 Add return hidden state in the native API (#3897) 
Co-authored-by: Beichen-Ma <mabeichen12@gmail.com>
Co-authored-by: Chayenne <zhaochen20@outlook.com>
 2025-02-26 22:06:54 -08:00
who who who
4606e2a3fe Bug: fix capture_bs (#3857) 2025-02-25 08:40:35 -08:00
HAI
d81ac4434e MI30x: More graph captures for larger batch sizes and concurrencies (#3420) 2025-02-12 03:04:38 +08:00
Jackmin801
5f0e7de339 [Feat] Return hidden states (experimental) (#3364) 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
 2025-02-10 15:54:37 -08:00
Yineng Zhang
bc72e5bd32 add cuda graph capture failure possible solution (#3430) 2025-02-09 22:57:11 +08:00
Yineng Zhang
fad315cb8e fix EAGLE 2 non greedy case (#3407) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2025-02-09 07:28:34 +08:00
Yineng Zhang
013021b6a1 refactor EAGLE 2 (#3269) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: merrymercy <lianminzheng@gmail.com>
Co-authored-by: Ying1123 <sqy1415@gmail.com>
 2025-02-03 20:52:30 +08:00
Yineng Zhang
4eb4b401cc update and simplify CustomOp (#3249) 2025-02-01 18:56:44 +08:00
Lianmin Zheng
1dda8c5e4c Return more infos for computing average acceptance length (#3152) 2025-01-26 04:51:54 -08:00
Lianmin Zheng
3d8f1c9bcf Use int64 as indices for set_kv_buffer (#3039) 2025-01-21 19:46:09 -08:00
Lianmin Zheng
89cd923581 Roll back to use vllm custom allreduce (#3006) 2025-01-20 04:03:15 -08:00
Yineng Zhang
5dc54f1a62 feat: remove vllm distributed (#2907) 
Co-authored-by: Zhangyi <1109276519@qq.com>
 2025-01-17 22:31:51 +08:00
Lianmin Zheng
8b6ce52e92 Support multi-node DP attention (#2925) 
Co-authored-by: dhou-xai <dhou@x.ai>
 2025-01-16 11:15:00 -08:00
Lianmin Zheng
679c3bcacf Fix typo in cuda_graph_bs (#2813) 2025-01-09 03:03:24 -08:00
Lianmin Zheng
bdc1acf6cd Misc fix for min_p_sampling, --cuda-graph-bs (#2761) 2025-01-07 02:52:53 -08:00
Lianmin Zheng
b8574f6953 Clean up eagle code (#2756) 2025-01-06 14:54:18 -08:00
Lianmin Zheng
ad20b7957e Eagle speculative decoding part 3: small modifications to the general scheduler (#2709) 
Co-authored-by: kavioyu <kavioyu@tencent.com>
 2025-01-02 02:09:08 -08:00
Lianmin Zheng
b0524c3789 Eagle speculative decoding part 2: Fix cuda graph + DP attention hanging (#2684) 
Co-authored-by: yukavio <kavioyu@gmail.com>
 2024-12-31 02:25:05 -08:00
Lianmin Zheng
f44d143949 Support target model verification in the attention backend (#2678) 
Co-authored-by: yukavio <kavioyu@gmail.com>
 2024-12-30 22:58:55 -08:00
mobicham
a29dd9501d Add GemLite caching after each capture (#2669) 2024-12-30 05:27:29 -08:00
Lianmin Zheng
9c6ba2484f Refactor logprob computation to return the real logprob used in sampling (#2664) 2024-12-30 04:51:38 -08:00
Ke Bao
e835a50021 Reorg moe code (#2563) 2024-12-24 01:10:22 +08:00
Lianmin Zheng
21e9e63ad5 Print progress bar during cuda graph capture (#2502) 2024-12-17 06:33:46 -08:00
Lianmin Zheng
7a1aecb938 Simplify pytorch sampling kernel and logit processor (#2491) 2024-12-16 14:11:09 -08:00
Lianmin Zheng
641b7d0ae0 [Minor] Improve code style (#2422) 2024-12-09 06:30:35 -08:00
Lianmin Zheng
a6ca736c8e Simplify stream_output (#2398) 2024-12-08 12:27:13 -08:00
Lianmin Zheng
71e2a27753 Fix the cuda graph capture range for small #max-running-requests (#2359) 2024-12-06 14:13:57 +08:00
Lianmin Zheng
2b0fc5941d [Minor] Code style improvements (#2355) 2024-12-04 19:02:08 -08:00
Lianmin Zheng
07ec07ad1f Improve torch compile for fused moe (#2327) 2024-12-03 01:58:25 -08:00
Lianmin Zheng
3c5538f781 Update CI threshold (#2186) 2024-11-25 15:24:17 -08:00
Lianmin Zheng
c4336b2b60 Use custom allreduce w/ torch.compile (#2185) 2024-11-25 14:55:01 -08:00
Lianmin Zheng
5652c56535 Update CI threshold & Improve code style (#2159) 2024-11-24 06:29:38 -08:00
Xuehai Pan
62a4a339eb docs: fix module docstrings and copyright headers (#2077) 2024-11-22 22:16:53 +08:00