Baizhou Zhang
|
f2d68ded6d
|
Rename lora_path to lora_id in batches (#8437)
|
2025-08-03 21:08:28 -07:00 |
|
Cheng Wan
|
cb099d2095
|
[CUDA Graph] save cuda graph memory by using next_token_logits_buffer (#8579)
|
2025-08-03 03:06:47 -07:00 |
|
Nicolas Castet
|
82e6c3a65a
|
Add support for NCCL symmetric memory for TP allreduces (#8238)
|
2025-08-01 23:30:55 +00:00 |
|
Brayden Zhong
|
4acf690206
|
[Optimization][Perf] Disable the GC during CUDA graph capture to speed up by up to 3x (#8577)
|
2025-07-31 11:31:21 -07:00 |
|
Cheng Wan
|
c0fb25e949
|
DP Enhancement (#8280)
|
2025-07-24 21:36:21 -07:00 |
|
Lifu Huang
|
4e3defe5a7
|
Support start up LoRA server without initial adapters (#8019)
|
2025-07-19 15:38:09 -07:00 |
|
Qiaolin Yu
|
41650b0d70
|
feat: support compatibility between MTP and two-batch-overlap (#7225)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
|
2025-06-27 01:10:27 -07:00 |
|
fzyzcjy
|
3d7cdb2ebd
|
Fix MTP error when enabling two-batch overlap (#7569)
|
2025-06-26 15:40:54 -07:00 |
|
valarLip
|
e984d5073b
|
enable aiter_biased_grouped_topk kernel (#7423)
|
2025-06-24 02:09:42 -07:00 |
|
u4lr451
|
ed0a0b692c
|
Perormance: Enable cuda graph for dp idle batch (#7269)
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
|
2025-06-23 17:34:13 -07:00 |
|
Cheng Wan
|
ac5010e0ba
|
Fix CUDA Graph Check under Deepep with DP FFN (#7451)
|
2025-06-22 20:35:58 -07:00 |
|
Cheng Wan
|
e879d8b7a8
|
[Feature] Comprehensive Hybrid Parallelism Support (#6389)
|
2025-06-20 14:43:11 -07:00 |
|
u4lr451
|
10d60cd41b
|
feat: mtp support dp-attention (#6081)
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: tianqilin.99 <tianqilin.99@bytedance.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
|
2025-06-17 00:33:28 -07:00 |
|
Lianmin Zheng
|
c64290dcb5
|
Use seq_len_fill_value in the cuda graph runners (#7233)
|
2025-06-16 15:57:07 -07:00 |
|
Lianmin Zheng
|
6b12d6a8d5
|
Simplify the heuristics for setting --mem-fraction-static (#7054)
|
2025-06-10 19:01:39 -07:00 |
|
kyle-pena-kuzco
|
b56de8f943
|
Open AI API hidden states (#6716)
|
2025-06-10 14:37:29 -07:00 |
|
Lianmin Zheng
|
4a102a2b02
|
Minor style fix in cuda_graph_runner.py (#7053)
|
2025-06-10 06:32:41 -07:00 |
|
Lianmin Zheng
|
6406408a70
|
Clean up server_args.py (#7037)
|
2025-06-10 05:34:29 -07:00 |
|
Lianmin Zheng
|
dc0705a504
|
Simplify prepare_extend_after_decode (#6987)
|
2025-06-09 16:39:21 -07:00 |
|
Baizhou Zhang
|
971a0dfa32
|
Extend cuda graph capture bs for B200 (#6937)
|
2025-06-08 05:13:22 -07:00 |
|
fzyzcjy
|
2fc1299562
|
Remove unnecessary kernels of num_token_non_padded (#6965)
|
2025-06-08 05:09:17 -07:00 |
|
Lianmin Zheng
|
60fdad7cf3
|
Sync the changes on cuda graph runners (#6932)
|
2025-06-06 18:23:52 -07:00 |
|
fzyzcjy
|
e05e29d178
|
Refactor CustomOp to avoid confusing bugs (#5382)
|
2025-06-02 10:27:36 -07:00 |
|
Lianmin Zheng
|
20fd53b8f6
|
Correctly abort the failed grammar requests & Improve the handling of abort (#6803)
|
2025-06-01 19:00:07 -07:00 |
|
Lianmin Zheng
|
2d72fc47cf
|
Improve profiler and integrate profiler in bench_one_batch_server (#6787)
|
2025-05-31 15:53:55 -07:00 |
|
Baizhou Zhang
|
f2bd3515fb
|
Tune memory arguments on B200 (#6718)
|
2025-05-29 00:03:22 -07:00 |
|
fzyzcjy
|
ae6a5b2950
|
Minor refactor two-batch overlap (#6682)
|
2025-05-28 15:54:17 -07:00 |
|
fzyzcjy
|
0d47788025
|
Support overlapping two batches (#4068)
|
2025-05-24 17:39:07 -07:00 |
|
Baizhou Zhang
|
d4c038daed
|
[Fix]Fix capture fail bug for DeepSeek (#6275)
|
2025-05-21 11:11:20 -07:00 |
|
fzyzcjy
|
2716830802
|
Speed up when having padding tokens in DeepEP (#6175)
|
2025-05-17 16:44:05 -07:00 |
|
quinnrong94
|
2e4babdb0a
|
[Feat] Support FlashMLA backend with MTP and FP8 KV cache (#6109)
Co-authored-by: Yingyi <yingyihuang2000@outlook.com>
Co-authored-by: neiltian <neiltian@tencent.com>
Co-authored-by: lukec <118525388+sleepcoo@users.noreply.github.com>
Co-authored-by: kexueyu <kexueyu@tencent.com>
Co-authored-by: vincentmeng <vincentmeng@tencent.com>
Co-authored-by: pengmeng <pengmeng@tencent.com>
|
2025-05-15 00:48:09 -07:00 |
|
Lianmin Zheng
|
e8e18dcdcc
|
Revert "fix some typos" (#6244)
|
2025-05-12 12:53:26 -07:00 |
|
applesaucethebun
|
d738ab52f8
|
fix some typos (#6209)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
|
2025-05-13 01:42:38 +08:00 |
|
Lianmin Zheng
|
fba8eccd7e
|
Log if cuda graph is used & extend cuda graph capture to cuda-graph-max-bs (#6201)
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
|
2025-05-12 00:17:33 -07:00 |
|
applesaucethebun
|
2ce8793519
|
Add typo checker in pre-commit (#6179)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
|
2025-05-11 12:55:00 +08:00 |
|
Ying Sheng
|
11383cec3c
|
[PP] Add pipeline parallelism (#5724)
|
2025-04-30 18:18:07 -07:00 |
|
JieXin Liang
|
e4b6133b78
|
[fix] relax mem_fraction_static for h200 (#5893)
Co-authored-by: alcanerian <alcanerian@gmail.com>
|
2025-04-29 17:01:12 -07:00 |
|
Qiaolin Yu
|
8c0cfca87d
|
Feat: support cuda graph for LoRA (#4115)
Co-authored-by: Beichen Ma <mabeichen12@gmail.com>
|
2025-04-28 23:30:44 -07:00 |
|
Lianmin Zheng
|
21514ff5bd
|
Disable flaky eagle tests (#5753)
|
2025-04-25 15:54:39 -07:00 |
|
JieXin Liang
|
917324862e
|
[fix] reduce dp capture bs (#5634)
Co-authored-by: alcanerian <alcanerian@gmail.com>
|
2025-04-22 11:08:45 -07:00 |
|
Lianmin Zheng
|
1343200299
|
Clean up mem settings (#5610)
|
2025-04-21 17:19:00 -07:00 |
|
Liangsheng Yin
|
e69a219074
|
Enhance GPU memory settings (#5604)
|
2025-04-21 15:15:00 -07:00 |
|
Lianmin Zheng
|
177320a582
|
Clean up imports (#5467)
|
2025-04-16 15:26:49 -07:00 |
|
Lianmin Zheng
|
0769b14bf9
|
[Minor] Move torch.compile patch to a better place (#5397)
|
2025-04-15 18:37:07 -07:00 |
|
Zhaoyang Hao
|
5d13440162
|
[FIX] Fix concatenation error in capture_bs when open --disable-cuda-graph-padding and without MTP (#5412)
|
2025-04-15 01:42:27 -07:00 |
|
Yineng Zhang
|
4cb53ecd0c
|
fix: log warning when disable cuda graph (#5209)
|
2025-04-09 14:16:13 -07:00 |
|
Baizhou Zhang
|
e62d60fe6d
|
[Fix] avoid stream sync and torch compile in prefill for fa3 backend (#4932)
|
2025-03-30 13:53:44 -07:00 |
|
Lianmin Zheng
|
b26bc86b36
|
Support page size > 1 + eagle (#4908)
|
2025-03-30 00:46:23 -07:00 |
|
tarinkk
|
7f19e083c1
|
Support (1 <= dp < tp) in the dp attention in DeepEP (#4770)
Co-authored-by: Cheng Wan <cwan39@gatech.edu>
|
2025-03-27 17:09:35 -07:00 |
|
AinL
|
17000d2b3a
|
Remove Unintended Capture Batch Sizes in AMD HIP Graph Runner (#4638)
|
2025-03-27 08:41:33 -07:00 |
|