4 Commits

Author	SHA1	Message	Date
min-xu-et	7dd8a7e6d9	fixed an error handling in bench_latency.py (#904)	2024-08-03 17:42:17 -07:00
Liangsheng Yin	c020f9ceda	Support chunked prefill when radix cache is disabled (#811)	2024-08-01 00:29:01 -07:00
Ying Sheng	e7487b08bc	Adjust default mem fraction to avoid OOM (#823)	2024-07-30 01:58:31 -07:00
Liangsheng Yin	cdcbde5fc3	Code structure refactor (#807)	2024-07-29 23:04:48 -07:00