chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
891 Commits 1 Branch 0 Tags
f42e9bfb52b4e1af282806697ba29de1164df480
Commit Graph

118 Commits

Author SHA1 Message Date
Jerry Zhang
63e845d0bb Add float8 dynamic quant to torchao_utils (#1528) 2024-09-28 12:27:54 -07:00
Ying Sheng
f39a0197fd Revert "kernel: use tensor cores for flashinfer gqa kernels" (#1511) 2024-09-24 22:50:31 -07:00
Ke Bao
8d4ed42ad5 MoE torch compile (#1497) 2024-09-24 01:46:59 -07:00
Ke Bao
a68cb201dd Fix triton head num (#1482) 2024-09-21 10:25:20 +08:00
Yineng Zhang
b4408b0d16 feat: update linear deps 1/N (#1305) 2024-09-19 20:53:11 +08:00
Lianmin Zheng
7f24ea95c3 Fuse top_k and top_k in the sampler (#1457) 2024-09-18 04:35:35 -07:00
HAI
aa2750beb3 [Bugfix] Enable SGLang on AMD GPUs via PyTorch for ROCm (#1419) (#1453) 2024-09-18 02:01:35 -07:00
HAI
3a6e04185b [Feature, Hardware] Enable SGLang on AMD GPUs via PyTorch for ROCm (#1420) 2024-09-17 07:43:52 +00:00
Lianmin Zheng
2fa5cec775 Simplify sampler and its error handling (#1441) 2024-09-16 21:23:31 -07:00
Lianmin Zheng
899cf5c438 Remove deprecated configs (#1431) 2024-09-15 08:52:18 -07:00
Lianmin Zheng
9ba1f09760 [Fix] Fix logprob and normalized_logprob (#1428) 2024-09-15 06:36:06 -07:00
Lianmin Zheng
9463bc1385 Enable torch.compile for triton backend (#1422) 2024-09-14 15:38:37 -07:00
Jerry Zhang
30b404ce72 Add torchao quant for mixtral and qwen_moe (#1418) 2024-09-14 06:46:55 +00:00
Liangsheng Yin
70b6802982 Optimize conflicts between CUDA graph and vocab mask tensors (#1392) 2024-09-13 20:27:53 -07:00
Zihao Ye
debbdb5178 kernel: use tensor cores for flashinfer gqa kernels (#1403) 2024-09-12 00:38:18 -07:00
Lianmin Zheng
3efa798116 Support cuda graph in the triton attention backend (#1401) 2024-09-12 00:36:55 -07:00
Lianmin Zheng
fec185ce0c Refactor attention backend (#1381) 2024-09-11 11:44:26 -07:00
Byron Hsu
8c0efa514d remove assertion in triton attention and add an unit test (#1385) 2024-09-11 03:22:07 -07:00
Liangsheng Yin
144bc70fcc Organize flashinfer indices update (#1378) 2024-09-10 17:38:59 -07:00
Lianmin Zheng
46094e0c1b Deprecate --disable-flashinfer and introduce --attention-backend (#1380) 2024-09-10 17:11:16 -07:00
Lianmin Zheng
3a6e8b6d78 [Minor] move triton attention kernels into a separate folder (#1379) 2024-09-10 15:15:08 -07:00
Lianmin Zheng
6c7cb90365 [Minor] improve kill scripts and torchao import (#1375) 2024-09-11 04:27:03 +10:00
William
e72275cf7f Support MiniCPM3 (#1371) 2024-09-10 19:57:52 +10:00
Liangsheng Yin
69b3bb9ae1 Unify forward mode (#1360) 2024-09-09 13:49:29 -07:00
Jerry Zhang
a7c47e0f02 Add torchao quant (int4/int8/fp8) to llama models (#1341) 
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
 2024-09-09 05:32:41 -07:00
Byron Hsu
8e6bdf851c [triton] Support head_dim not 2^n in triton extend and decode attention (#1281) 2024-09-09 01:30:24 -07:00
Liangsheng Yin
a5a134f39f Fix bugs in sampler with CUDA graph / torch.compile (#1306) 2024-09-02 23:18:48 +00:00
Liangsheng Yin
47f20da223 Fix regex mask (#1296) 2024-09-01 21:50:58 -07:00
Byron Hsu
00b19f198f [triton] Remove the zero initialization of qk_acc by directly writing the result (#1288) 2024-09-01 03:12:06 -07:00
Ke Bao
6cb32ef92c Support Triton fp8 e5m2 kv cache (#1286) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-09-01 19:46:40 +10:00
Liangsheng Yin
381dd57bd6 Sampler cudagraph (#1253) 2024-08-28 18:58:52 -07:00
Yineng Zhang
c411f32e1c feat: replace GeluAndMul (#1234) 2024-08-28 14:07:02 +00:00
Yineng Zhang
b1a540ec42 feat: update GemmaRMSNorm (#1232) 2024-08-28 22:47:34 +10:00
Yineng Zhang
f25f4dfde5 hotfix: revert sampler CUDA Graph (#1242) 2024-08-28 21:16:47 +10:00
Yineng Zhang
198974cd1a feat: support sm75 with FlashInfer v0.1.6 (#1233) 2024-08-28 18:39:12 +10:00
Yineng Zhang
3602692c7c feat: replace get_act_fn for gpt_bigcode (#1231) 2024-08-27 21:15:31 +10:00
Liangsheng Yin
75ce37f401 Move sampler into CUDA graph (#1201) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-26 07:02:50 -07:00
Ke Bao
2c615d120f [Feature] Support fp8 e5m2 kv cache with flashinfer (#1204) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-25 17:38:11 -07:00
Lianmin Zheng
f6af3a6561 Cleanup readme, llava examples, usage examples and nccl init (#1194) 2024-08-24 08:02:23 -07:00
Yineng Zhang
c9064e6fd9 feat: use gelu_tanh_and_mul (#1193) 2024-08-24 01:58:16 -07:00
Liangsheng Yin
83e23c69b3 Improve code style of sampler (#1168) 2024-08-21 16:48:24 -07:00
Yineng Zhang
1fb9459908 fix: custom op fallback forward native when lower sm80 (#1177) 2024-08-21 14:26:35 -07:00
Ke Bao
df191254ab Optimize MLA/GQA/MQA Triton decoding (#1138) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-19 20:23:07 +10:00
yichuan~
b997a18d74 [Feat]Add support for optional start len of logprobs (#1035) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
 2024-08-18 23:45:41 -07:00
Lianmin Zheng
5a261bd055 Fix the deadlock in multi-node tp (#1122) 2024-08-16 01:39:24 -07:00
Ying Sheng
93d4e354d8 [Fix] Window attention compatible with RadixAttention and chunked prefill (#1112) 2024-08-15 10:33:20 -07:00
Ying Sheng
14cb544d56 [Fix] fix flashinfer usage for window attention (#1107) 2024-08-15 00:53:24 -07:00
Ying Sheng
96a2093ef0 [Fix] Compatibility of window attention and cuda graph (#1090) 2024-08-14 10:37:01 -07:00
Lianmin Zheng
a59636bb5e Update grok 1 model (#1095) 2024-08-14 04:40:44 -07:00
Ying Sheng
0909bb0d2f [Feat] Add window attention for gemma-2 (#1056) 2024-08-13 17:01:26 -07:00