Commit Graph
126 Commits
Author SHA1 Message Date
Yineng Zhang f24fc5b86d fix typo (#6248) 2025-05-12 15:45:12 -07:00
Lianmin ZhengandSangBin Cho fba8eccd7e Log if cuda graph is used & extend cuda graph capture to cuda-graph-max-bs (#6201)
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
2025-05-12 00:17:33 -07:00
applesaucethebunandBrayden Zhong 2ce8793519 Add typo checker in pre-commit (#6179)
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
2025-05-11 12:55:00 +08:00
XinyuanTong e88dd482ed [CI]Add performance CI for VLM (#6038)
Signed-off-by: Xinyuan Tong <justinning0323@outlook.com>
2025-05-07 19:20:03 -07:00
Lianmin Zheng 35ca04d2fa [CI] fix port conflicts (#5789) 2025-04-27 05:17:44 -07:00
vzed 094891c01a fix: Use is not None instead of != None for None checks. (#5687) 2025-04-26 19:26:57 -07:00
Yineng Zhang b1f6d89b5f fix: update truss bench_serving (#5683) 2025-04-23 13:28:35 -07:00
Yineng Zhang 7282ab741a fix: update bench_speculative (#5649) 2025-04-22 16:08:15 -07:00
fzyzcjy 9924bbe153 Fix bench_serving fail when zero warmup requests (#5574) 2025-04-20 14:16:03 -07:00
Lianmin Zheng 177320a582 Clean up imports (#5467) 2025-04-16 15:26:49 -07:00
Yuhong Guo 3dfc6023ce Fix bench_serving with random-ids (#5214) 2025-04-15 01:34:35 -07:00
Xiaoyu Zhang 924ca7c92c Add DeepSeek V3/R1 shared experts fusion (#4918) 2025-04-04 01:59:29 -07:00
Yineng Zhang fda6bb78da update bench_serving (#4958) 2025-04-01 15:10:56 -07:00
chaobo jiaandrudy152 ef9a378a20 [Feature] add multi-rank support for Lora (#4492)
Co-authored-by: rudy152 <czh1137892874@gmail.com>
2025-03-28 09:38:44 -07:00
5d7edc8e55 Support FA3 as Attention backend by using --attention-backend fa3 (#4680)
Co-authored-by: qsong <qsong@linkedin.com>
Co-authored-by: qingquansong <ustcsqq@gmail.com>
2025-03-23 23:28:11 -07:00
Xu Song 470b474075 Update bench_serving.py (#4454) 2025-03-15 16:33:58 -07:00
Mingshan 0fe7c13be1 Fix bench_serving flush cache not recognizing OPENAI_API_KEY (#4181)
Signed-off-by: Mingshan <git@brighill.com>
2025-03-08 01:03:38 -08:00
Lzhang-hub 3a3918121f fix bench serving bug (#4135) 2025-03-06 05:34:02 -08:00
Lianmin Zheng e074d84e5b [Minor] more code cleanup (#4077) 2025-03-04 21:23:47 -08:00
Lianmin Zheng 66301e124f Improve code styles (#4021) 2025-03-03 03:20:23 -08:00
ac2387279e Support penalty in overlap mode; return logprob with chunked prefill; improve benchmark scripts (#3988)
Co-authored-by: SangBin Cho <rkooo567@gmail.com>
Co-authored-by: dhou-xai <dhou@x.ai>
Co-authored-by: Hanming Lu <hanming_lu@berkeley.edu>
2025-03-03 00:12:04 -08:00
Kebe ec0a72c2d9 Fix bench_serving not recognizing OPENAI_API_KEY (#3870)
Signed-off-by: Kebe <mail@kebe7jun.com>
2025-02-27 20:18:53 -08:00
Lianmin Zheng 287d07a669 Misc fixes for eagle (flush_cache, CPU overhead) (#3014) 2025-01-20 20:27:38 -08:00
Lianmin Zheng 8f2c522aba Improve benchmark scripts and error message printing (#2922) 2025-01-16 06:24:31 -08:00
Pratyush Patel 8f15789314 Add more metrics to serving benchmark. (#2819) 2025-01-10 23:30:44 +08:00
Lianmin Zheng bdc1acf6cd Misc fix for min_p_sampling, --cuda-graph-bs (#2761) 2025-01-07 02:52:53 -08:00
Lianmin Zheng dc3bee4815 Fix test and benchmark scripts (#2598) 2024-12-26 07:56:26 -08:00
Ying Sheng 8a56b43175 [Bench] Flush cache before benchmarking (#2566) 2024-12-24 11:21:21 +08:00
Lianmin Zheng a2486eb58f Fix a bug with logprob streaming + chunked prefill (#2403) 2024-12-08 03:55:27 -08:00
Lianmin Zheng 3ddb1c4679 [Minor] Fix logger and style (#2325) 2024-12-02 20:45:53 -08:00
bjmsongandroot 01017d4c20 Support LoRA in Completion API (#2243)
Co-authored-by: root <bjmsong@126.com>
2024-11-29 16:13:38 -08:00
Byron Hsu 4b0a1c9365 Replace prob based with threshold based load balancing (#2170) 2024-11-24 23:17:11 -08:00
Lianmin Zheng 8e1adb8441 Allow overwrite flashinfer use_tensorcore (#2169) 2024-11-24 20:58:17 -08:00
Byron Hsu cbedd1db1d [router] cache-aware load-balancing router v1 (#2114) 2024-11-23 08:34:48 -08:00
Yineng Zhang ad47749b82 fix: resolve bench_serving args (#2139) 2024-11-23 17:45:42 +08:00
Yunmeng 60769be14d Add concurrency option for benchmark (#2136) 2024-11-23 17:07:07 +08:00
bjmsongandroot ad30d5cf9a Benchmark with Pytorch Profiler easily (#2110)
Co-authored-by: root <bjmsong@126.com>
2024-11-21 23:29:50 -08:00
Yineng Zhang 55bd97f3e5 minor: add dataset dump and questions shuffle (#2093) 2024-11-19 14:07:27 -08:00
Lianmin Zheng 2558d6a675 Fix the default arguments of bench_offline_throughput.py & simplify detokenizer manager (#2042) 2024-11-15 05:02:44 -08:00
zolinthecowandByronHsu f6dd648620 Offline LLM Engine Benchmark Throughput (#1968)
Co-authored-by: ByronHsu <byronhsu1230@gmail.com>
2024-11-14 21:59:33 -08:00
Ke Bao b808a38365 Filter empty prompt in random bench serving (#2011) 2024-11-12 14:53:41 +08:00
Byron Hsu 8169c6f4ef Add gen-shared-prefix dataset in bench_serving (#1990) 2024-11-11 08:39:56 +08:00
Yineng Zhang 793b79dbe9 feat: support truss endpoint for benchmark serving (#1906) 2024-11-03 12:56:10 -08:00
Lianmin Zheng 175afed370 Improve benchmark scripts (#1672) 2024-10-14 21:53:01 -07:00
Lianmin Zheng 4a292f670d [Minor] Add some utility functions (#1671) 2024-10-14 20:08:03 -07:00
Ying Shengandhnyls2002 04b262cd91 [Fix] Fix major performance bug in certain cases (#1563)
Co-authored-by: hnyls2002 <hnyls2002@gmail.com>
2024-10-04 08:51:11 +00:00
Lianmin Zheng 5e62a6b706 Add bench_server_latency.py (#1452) 2024-09-18 00:56:06 -07:00
Lianmin Zheng e4d68afcf0 [Minor] Many cleanup (#1357) 2024-09-09 04:14:11 -07:00
min-xu-et fa13b95d6b fixed a typo (#1143) 2024-08-18 14:29:09 -07:00
Lianmin Zheng 3c1f5a9220 Fix duplicated imports in hf_transformers_utils.py (#1141) 2024-08-17 18:03:00 -07:00