Lianmin Zheng
|
3c5538f781
|
Update CI threshold (#2186)
|
2024-11-25 15:24:17 -08:00 |
|
Lianmin Zheng
|
c4336b2b60
|
Use custom allreduce w/ torch.compile (#2185)
|
2024-11-25 14:55:01 -08:00 |
|
Lianmin Zheng
|
5652c56535
|
Update CI threshold & Improve code style (#2159)
|
2024-11-24 06:29:38 -08:00 |
|
Xuehai Pan
|
62a4a339eb
|
docs: fix module docstrings and copyright headers (#2077)
|
2024-11-22 22:16:53 +08:00 |
|
Lianmin Zheng
|
722530fa01
|
Enable overlap scheduler by default for the triton attention backend (#2105)
|
2024-11-20 02:58:35 -08:00 |
|
Yineng Zhang
|
766192610e
|
feat: update torch 2.5.1 (#2069)
|
2024-11-18 21:29:13 +08:00 |
|
Ke Bao
|
62832bb272
|
Support cuda graph for DP attention (#2061)
|
2024-11-17 16:29:20 -08:00 |
|
Lianmin Zheng
|
a509552087
|
[minor] Improve code style and compatibility (#1961)
|
2024-11-08 02:19:41 -08:00 |
|
Lianmin Zheng
|
86e0dde555
|
Improve the user control of new_token_ratio (#1811)
|
2024-10-26 16:39:41 -07:00 |
|
Lianmin Zheng
|
2b80978859
|
Provide an argument to set the maximum batch size for cuda graph (#1809)
|
2024-10-26 15:09:33 -07:00 |
|
Lianmin Zheng
|
e646c5901e
|
Fix logprob in the overlapped mode (#1795)
|
2024-10-25 11:06:57 -07:00 |
|
yizhang2077
|
def55bc876
|
Qwen2vl support cuda graph and disable radix cache (#1780)
|
2024-10-25 10:45:17 -04:00 |
|
Lianmin Zheng
|
86a2c473b7
|
[Fix] Fix seq_lens_sum for cuda graph runner in padded cases (#1789)
|
2024-10-24 21:26:05 -07:00 |
|
Lianmin Zheng
|
384d85ba35
|
Re-introduce get_cuda_graph_seq_len_fill_value (#1783)
|
2024-10-24 13:30:11 -07:00 |
|
Lianmin Zheng
|
fc82f5a743
|
[Fix] Fix cuda graph padding for triton attention backend (#1782)
|
2024-10-24 12:33:15 -07:00 |
|
Lianmin Zheng
|
0089c4bc96
|
[Fix] Fix NaN issues by fixing the cuda graph padding values for flashinfer (#1779)
|
2024-10-24 04:16:59 -07:00 |
|
Lianmin Zheng
|
05b3bf5e8e
|
Crash the server on warnings in CI (#1772)
|
2024-10-23 16:27:13 -07:00 |
|
Lianmin Zheng
|
ad4125d1a9
|
Fuse more ops & Simplify token mapping (#1758)
|
2024-10-22 23:20:43 -07:00 |
|
Liangsheng Yin
|
94cde10920
|
Llama3.2 vision model support (#1551)
|
2024-10-21 15:01:21 -07:00 |
|
Lianmin Zheng
|
09603c6dc9
|
Maintain seq_lens_sum to make more FlashInfer operations non-blocking (#1741)
|
2024-10-21 01:43:16 -07:00 |
|
Lianmin Zheng
|
f0f8a7699b
|
Simplify the nan detection and greedy check in sampler (#1709)
|
2024-10-18 20:21:24 -07:00 |
|
Lianmin Zheng
|
36d5acfca5
|
Rename InputMetadata -> ForwardBatch (#1543)
|
2024-09-30 02:41:11 -07:00 |
|
Lianmin Zheng
|
3f0fe08d37
|
Let ModelRunner take InputMetadata as input, instead of ScheduleBatch (#1541)
|
2024-09-29 20:28:45 -07:00 |
|
Ke Bao
|
8d4ed42ad5
|
MoE torch compile (#1497)
|
2024-09-24 01:46:59 -07:00 |
|
Lianmin Zheng
|
39bb49d156
|
Update dockerfile to include datamodel_code_generator (#1492)
|
2024-09-22 04:49:16 -07:00 |
|
Lianmin Zheng
|
2d346a57c2
|
Fix padding in the cuda graph (#1469)
|
2024-09-19 01:52:15 -07:00 |
|
Ke Bao
|
76524b70d1
|
Fix torch compile for deepseek-v2 (#1442)
|
2024-09-17 00:52:08 -07:00 |
|
Lianmin Zheng
|
9ba1f09760
|
[Fix] Fix logprob and normalized_logprob (#1428)
|
2024-09-15 06:36:06 -07:00 |
|
Liangsheng Yin
|
70b6802982
|
Optimize conflicts between CUDA graph and vocab mask tensors (#1392)
|
2024-09-13 20:27:53 -07:00 |
|
Lianmin Zheng
|
3efa798116
|
Support cuda graph in the triton attention backend (#1401)
|
2024-09-12 00:36:55 -07:00 |
|
Lianmin Zheng
|
fec185ce0c
|
Refactor attention backend (#1381)
|
2024-09-11 11:44:26 -07:00 |
|
Liangsheng Yin
|
144bc70fcc
|
Organize flashinfer indices update (#1378)
|
2024-09-10 17:38:59 -07:00 |
|
Liangsheng Yin
|
a5a134f39f
|
Fix bugs in sampler with CUDA graph / torch.compile (#1306)
|
2024-09-02 23:18:48 +00:00 |
|
Liangsheng Yin
|
381dd57bd6
|
Sampler cudagraph (#1253)
|
2024-08-28 18:58:52 -07:00 |
|
Yineng Zhang
|
f25f4dfde5
|
hotfix: revert sampler CUDA Graph (#1242)
|
2024-08-28 21:16:47 +10:00 |
|
Lianmin Zheng
|
6cc38b2bf3
|
[Minor] Add more type annotations (#1237)
|
2024-08-28 00:54:26 -07:00 |
|
Liangsheng Yin
|
75ce37f401
|
Move sampler into CUDA graph (#1201)
Co-authored-by: Yineng Zhang <me@zhyncs.com>
|
2024-08-26 07:02:50 -07:00 |
|
Lianmin Zheng
|
bea2bb9eea
|
Improve multi-node stability (#1171)
|
2024-08-20 22:35:05 -07:00 |
|
Lianmin Zheng
|
5a261bd055
|
Fix the deadlock in multi-node tp (#1122)
|
2024-08-16 01:39:24 -07:00 |
|
Lianmin Zheng
|
326df4bab2
|
Use a single workspace for flashinfer (#1077)
|
2024-08-14 19:25:37 -07:00 |
|
Ying Sheng
|
96a2093ef0
|
[Fix] Compatibility of window attention and cuda graph (#1090)
|
2024-08-14 10:37:01 -07:00 |
|
Lianmin Zheng
|
8f790ac100
|
Fix a bug in cuda graph runner (#1094)
|
2024-08-14 03:25:38 -07:00 |
|
Ying Sheng
|
9f662501a3
|
Move torch.compile configs into cuda_graph_runner.py (#993)
|
2024-08-08 13:20:30 -07:00 |
|
Liangsheng Yin
|
1ac304eeb4
|
Adjust InputeMetadata and ScheduleBatch (#981)
|
2024-08-08 01:11:22 -07:00 |
|
Liangsheng Yin
|
87e8c090e9
|
Organize code (rename, movement) (#953)
|
2024-08-06 20:50:32 -07:00 |
|
Liangsheng Yin
|
cdcbde5fc3
|
Code structure refactor (#807)
|
2024-07-29 23:04:48 -07:00 |
|