chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
3,178 Commits 1 Branch 0 Tags
b6cf3532b504081ea1d5b52244c0e889b7544856
Commit Graph
2115 Commits
Author SHA1 Message Date
Kaichen Zhang - NTU 3579162ab1 [Fix] Multi-images loading error (#1218) 2024-08-26 03:58:51 +00:00
Mingyi 7514b9f8d3 [CI] Fix CI (#1217) 2024-08-26 02:56:42 +00:00
Mingyi 158e8f1e2d improve the threshold and ports in tests (#1215) 2024-08-25 19:02:08 -07:00
Ke BaoandYineng Zhang 2c615d120f [Feature] Support fp8 e5m2 kv cache with flashinfer (#1204) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-25 17:38:11 -07:00
Lianmin Zheng 15f1a49d2d Update CI workflows (#1210) 2024-08-25 16:43:07 -07:00
Ying Sheng 308d024092 [CI] Fix the issue of unit test hanging (#1211) 2024-08-25 16:21:37 -07:00
Lianmin Zheng 902278008a [Minor] Improve the function organization in TokenizerManager & improve loggers (#1208) 2024-08-25 14:46:34 -07:00
ChayenneandYing Sheng 30b4f771b0 Support Alibaba-NLP/gte-Qwen2-7B-instruct embedding Model (#1186) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
 2024-08-25 10:29:12 -07:00
Kaichen Zhang - NTU 66e7dcaf70 [Fix] Fixing the multi-images error for llava-onevision (#1205) 2024-08-25 10:28:23 -07:00
Ying Sheng 1cb4da5c5f [Fix] the issue of random order when input is a list (#1199) 2024-08-24 21:43:03 -07:00
Ying Sheng e61d13acdf [CI] Fix the problem of hf runner too slow (#1202) 2024-08-24 18:35:55 -07:00
Lianmin Zheng f6af3a6561 Cleanup readme, llava examples, usage examples and nccl init (#1194) 2024-08-24 08:02:23 -07:00
Yineng Zhang c9064e6fd9 feat: use gelu_tanh_and_mul (#1193) 2024-08-24 01:58:16 -07:00
Kaichen Zhang - NTUandBo Li a5b14ad043 [Feat/WIP] add llava-onevision, with support for (1) siglip encoder, (2) qwen2 decoder (3) openai api compatible server. (#1123) 
Co-authored-by: Bo Li <drluodian@gmail.com>
 2024-08-23 14:11:16 -07:00
Ying Sheng 5fafcac008 Fix benchmark script (#1185) 2024-08-22 09:03:25 +00:00
Liangsheng Yin 364d3d72a7 Fix broken penalty (#1184) 2024-08-22 08:16:35 +00:00
Lianmin Zheng 5623826f73 [Minor] Improve logging and rename the health check endpoint name (#1180) 2024-08-21 19:24:36 -07:00
Liangsheng Yin 83e23c69b3 Improve code style of sampler (#1168) 2024-08-21 16:48:24 -07:00
intervitens 068e9eae55 Support min-p sampling (#1167) 2024-08-21 22:49:32 +00:00
rainredandlzhang d6aeb9fa15 [Feature] Add a function to convert sampling_params to kwargs (#1170) 
Co-authored-by: lzhang <zhanglei@modelbest.cn>
 2024-08-21 14:28:35 -07:00
Yineng Zhang 1fb9459908 fix: custom op fallback forward native when lower sm80 (#1177) 2024-08-21 14:26:35 -07:00
Lianmin Zheng bea2bb9eea Improve multi-node stability (#1171) 2024-08-20 22:35:05 -07:00
Shan Yu cd10654e7e [Feat] Support update weights without restart server (#1157) 2024-08-20 13:48:24 -07:00
LucienandYineng Zhang 6242c399ab Generate 1 token to verify the health of the inference service in /health (#1154) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-21 03:14:34 +10:00
Yineng Zhang 04707b09b7 misc: add hypervisor vendor (#1165) 2024-08-21 02:14:51 +10:00
Xu-Chenchenxu02Yineng Zhang
 		ff2cfdb1a2 [Feature] add disable-custom-all-reduce (#1148) 
Co-authored-by: chenxu02 <chenxu02@zhihu.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-20 08:44:12 -07:00
Lianmin Zheng a8ae640328 Improve docs and warnings (#1164) 2024-08-20 08:31:29 -07:00
Juwan Yoo d8476818ef feat: allow streaming for multi-prompt and/or parallel sampling (#1134) 2024-08-20 08:06:55 -07:00
Ke BaoandYineng Zhang df191254ab Optimize MLA/GQA/MQA Triton decoding (#1138) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-19 20:23:07 +10:00
yichuan~Ying ShengYineng ZhangLianmin ZhengLiangsheng Yin
 		b997a18d74 [Feat]Add support for optional start len of logprobs (#1035) 
Co-authored-by: Ying Sheng <sqy1415@gmail.com>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
 2024-08-18 23:45:41 -07:00
min-xu-et fa13b95d6b fixed a typo (#1143) 2024-08-18 14:29:09 -07:00
Lianmin Zheng 3c1f5a9220 Fix duplicated imports in hf_transformers_utils.py (#1141) 2024-08-17 18:03:00 -07:00
Lianmin Zheng 57d0bd91ec Improve benchmark (#1140) 2024-08-17 17:43:23 -07:00
Lianmin Zheng cdc8d60752 Improve the code style: more comments and remove useless packages (#1139) 2024-08-17 14:37:52 -07:00
Yineng Zhang 9208591f05 fix: use fp16 dtype for sm75 (#1136) 2024-08-18 00:45:42 +10:00
Liangsheng Yin f624f6a6cc Fix port conflicts between local CI and runner CI. (#1131) 2024-08-16 15:12:38 -07:00
Liangsheng Yin 3694f8f996 Mixed style of chunked prefill (#1013) 2024-08-16 09:13:00 +00:00
Lianmin Zheng 5a261bd055 Fix the deadlock in multi-node tp (#1122) 2024-08-16 01:39:24 -07:00
Yineng Zhang 5bd953749b chore: bump v0.2.13 (#1111) 2024-08-16 03:50:43 +10:00
Lianmin Zheng 0cb099e20a set CUDA_DEVICE_MAX_CONNECTIONS=1 (#1113) 2024-08-16 03:47:39 +10:00
Ying Sheng 93d4e354d8 [Fix] Window attention compatible with RadixAttention and chunked prefill (#1112) 2024-08-15 10:33:20 -07:00
Yineng Zhang 9195d1362a misc: rm unused model_loader (#1110) 2024-08-15 08:29:35 -07:00
Ying Sheng 14cb544d56 [Fix] fix flashinfer usage for window attention (#1107) 2024-08-15 00:53:24 -07:00
Lianmin Zheng e86b1ccbf0 Enable chunked prefill by default (#1040) 2024-08-14 21:56:20 -07:00
Ying Sheng 8d2d876fc8 [Fix] fix the typo bug for window attention (#1106) 2024-08-14 21:56:01 -07:00
Lianmin Zheng 326df4bab2 Use a single workspace for flashinfer (#1077) 2024-08-14 19:25:37 -07:00
Ying Sheng 6767e2229f Support jinja as chat template file (#1104) 2024-08-14 17:43:14 -07:00
Liangsheng Yin 73cf6834f2 Support stop_token_ids in sglang API (#1092) 2024-08-15 00:31:39 +00:00
Ying Sheng 96a2093ef0 [Fix] Compatibility of window attention and cuda graph (#1090) 2024-08-14 10:37:01 -07:00
Liangsheng Yinandzhyncs a34dd86a7d Use dtype to control generate (#1082) 
Co-authored-by: zhyncs <me@zhyncs.com>
 2024-08-14 15:58:07 +00:00