chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Commit Graph

  • 52cefdbf57 fix: resolve the fp8 bug introduced by vLLM 0.5.5 (#1276) Yineng Zhang 2024-09-01 00:44:29 +10:00
  • 51c554d812 Allow more flexible assistant and system response (#1256) Christopher Chou 2024-08-30 11:51:44 -07:00
  • 79ece2c51f Report median instead of mean in bench_latency.py (#1269) Lianmin Zheng 2024-08-30 06:05:01 -07:00
  • 55f5976b42 Update README.md - Supported Models add Exaone 3.0 (#1267) 김종곤 2024-08-30 17:49:07 +09:00
  • b7f8341014 EXAONE 3.0 Model Support (#1258) 김종곤 2024-08-30 17:08:28 +09:00
  • f414352ae6 Transpose mla weight offline (#1261) Ke Bao 2024-08-30 14:45:40 +08:00
  • a362340b33 fix: multimodal_config in monkey_patch_vllm_dummy_weight_loader (#1260) lxww302 2024-08-29 23:43:41 -07:00
  • 381dd57bd6 Sampler cudagraph (#1253) Liangsheng Yin 2024-08-28 18:58:52 -07:00
  • 8153168c96 fix data racing due to mutable reference using deepcopy (#1255) Zhiqiang Xie 2024-08-28 18:57:54 -07:00
  • 6c34d6339c make json_schema usable from gen (#1254) Enrique Shockwave 2024-08-29 02:57:10 +01:00
  • 13ac95b894 chore: bump v0.2.14.post2 (#1250) Yineng Zhang 2024-08-29 04:46:33 +10:00
  • 492143bf32 fix: resolve qwen2 moe weight loader (#1252) Yineng Zhang 2024-08-29 04:25:46 +10:00
  • 0a97d7962d [Fix] Fix OOM in llava base class (#1249) Lianmin Zheng 2024-08-28 08:38:50 -07:00
  • c411f32e1c feat: replace GeluAndMul (#1234) Yineng Zhang 2024-08-29 00:07:02 +10:00
  • bf53bf5142 [Fix] Fix llava on multi images (#1247) Lianmin Zheng 2024-08-28 06:33:05 -07:00
  • b1a540ec42 feat: update GemmaRMSNorm (#1232) Yineng Zhang 2024-08-28 22:47:34 +10:00
  • 66975360e7 fix: increase max_new_tokens when testing generation models (#1244) Yineng Zhang 2024-08-28 22:12:36 +10:00
  • 6c49831394 Add sglang.bench_latency to CI (#1243) Lianmin Zheng 2024-08-28 04:20:54 -07:00
  • f25f4dfde5 hotfix: revert sampler CUDA Graph (#1242) Yineng Zhang 2024-08-28 21:16:47 +10:00
  • 184ae1c683 Update README.md (#1239) Lianmin Zheng 2024-08-28 02:15:52 -07:00
  • 198974cd1a feat: support sm75 with FlashInfer v0.1.6 (#1233) Yineng Zhang 2024-08-28 18:39:12 +10:00
  • 6cc38b2bf3 [Minor] Add more type annotations (#1237) Lianmin Zheng 2024-08-28 00:54:26 -07:00
  • 1ece2cda3d Fix bench latency benchmark (#1225) Liangsheng Yin 2024-08-28 00:37:32 -07:00
  • c8a9e79186 Fix readme (#1236) Dr. Artificial曾小健 2024-08-28 14:51:41 +08:00
  • 3602692c7c feat: replace get_act_fn for gpt_bigcode (#1231) Yineng Zhang 2024-08-27 21:15:31 +10:00
  • 909f34363b [FIX] Wrong logger (#1230) havetc 2024-08-27 12:10:46 +02:00
  • 5ff25cdf5b [Minor] add delete test and delete tmp file on ci server (#1227) yichuan~ 2024-08-26 22:04:52 -07:00
  • 2f1d92834f [FEAT] Support batches cancel (#1222) caiyueliang 2024-08-27 07:28:26 +08:00
  • c61a1b6f97 Torch compile CI throughput test (#1223) Liangsheng Yin 2024-08-26 13:52:58 -07:00
  • 9935f97b3e [FEAT] JSON constrained support (#1125) havetc 2024-08-26 18:37:26 +02:00
  • c5fe11a8e1 chore: bump v0.2.14 (#1155) Yineng Zhang 2024-08-27 00:28:24 +10:00
  • 75ce37f401 Move sampler into CUDA graph (#1201) Liangsheng Yin 2024-08-26 07:02:50 -07:00
  • 97589a60a2 [CI] Parallelize unit tests in CI (#1219) Mingyi 2024-08-25 21:54:02 -07:00
  • 632d506d0b minor: improve CI and dependencies (#1212) Liangsheng Yin 2024-08-25 21:26:31 -07:00
  • 3579162ab1 [Fix] Multi-images loading error (#1218) Kaichen Zhang - NTU 2024-08-26 11:58:51 +08:00
  • 7514b9f8d3 [CI] Fix CI (#1217) Mingyi 2024-08-25 19:56:42 -07:00
  • 158e8f1e2d improve the threshold and ports in tests (#1215) Mingyi 2024-08-25 19:02:08 -07:00
  • d3efcb3930 Update workflow files (#1214) Lianmin Zheng 2024-08-25 17:45:35 -07:00
  • 2c615d120f [Feature] Support fp8 e5m2 kv cache with flashinfer (#1204) Ke Bao 2024-08-26 08:38:11 +08:00
  • 61bb223e0f Update CI runner docs (#1213) Lianmin Zheng 2024-08-25 17:31:52 -07:00
  • 15f1a49d2d Update CI workflows (#1210) Lianmin Zheng 2024-08-25 16:43:07 -07:00
  • 308d024092 [CI] Fix the issue of unit test hanging (#1211) Ying Sheng 2024-08-25 16:21:37 -07:00
  • ab4990e4bf [Minor] Temporarily skip flaky test (#1209) Ying Sheng 2024-08-25 14:49:23 -07:00
  • 902278008a [Minor] Improve the function organization in TokenizerManager & improve loggers (#1208) Lianmin Zheng 2024-08-25 14:46:34 -07:00
  • 30b4f771b0 Support Alibaba-NLP/gte-Qwen2-7B-instruct embedding Model (#1186) Chayenne 2024-08-26 01:29:12 +08:00
  • 66e7dcaf70 [Fix] Fixing the multi-images error for llava-onevision (#1205) Kaichen Zhang - NTU 2024-08-26 01:28:23 +08:00
  • bc4c7a3545 Relax the assert in moe throughput test to fix the flaky CI (#1207) Lianmin Zheng 2024-08-25 10:27:02 -07:00
  • 1cb4da5c5f [Fix] the issue of random order when input is a list (#1199) Ying Sheng 2024-08-24 21:43:03 -07:00
  • e61d13acdf [CI] Fix the problem of hf runner too slow (#1202) Ying Sheng 2024-08-24 18:35:55 -07:00
  • b20daf982a Update README.md (#1198) Lianmin Zheng 2024-08-24 14:50:05 -07:00
  • f6af3a6561 Cleanup readme, llava examples, usage examples and nccl init (#1194) Lianmin Zheng 2024-08-24 08:02:23 -07:00
  • c9064e6fd9 feat: use gelu_tanh_and_mul (#1193) Yineng Zhang 2024-08-24 18:58:16 +10:00
  • a5b14ad043 [Feat/WIP] add llava-onevision, with support for (1) siglip encoder, (2) qwen2 decoder (3) openai api compatible server. (#1123) Kaichen Zhang - NTU 2024-08-24 05:11:16 +08:00
  • 5fafcac008 Fix benchmark script (#1185) Ying Sheng 2024-08-22 02:03:25 -07:00
  • 364d3d72a7 Fix broken penalty (#1184) Liangsheng Yin 2024-08-22 01:16:35 -07:00
  • 5623826f73 [Minor] Improve logging and rename the health check endpoint name (#1180) Lianmin Zheng 2024-08-21 19:24:36 -07:00
  • 83e23c69b3 Improve code style of sampler (#1168) Liangsheng Yin 2024-08-21 16:48:24 -07:00
  • ac1b74fa85 [Docs] Fix rendering of details in README (#1179) Zhanghao Wu 2024-08-21 16:05:33 -07:00
  • 068e9eae55 Support min-p sampling (#1167) intervitens 2024-08-22 01:49:32 +03:00
  • d6aeb9fa15 [Feature] Add a function to convert sampling_params to kwargs (#1170) rainred 2024-08-22 05:28:35 +08:00
  • 1fb9459908 fix: custom op fallback forward native when lower sm80 (#1177) Yineng Zhang 2024-08-22 07:26:35 +10:00
  • bea2bb9eea Improve multi-node stability (#1171) Lianmin Zheng 2024-08-20 22:35:05 -07:00
  • cd10654e7e [Feat] Support update weights without restart server (#1157) Shan Yu 2024-08-20 13:48:24 -07:00
  • 350a81609b fix: resolve README render (#1166) Yineng Zhang 2024-08-21 03:23:52 +10:00
  • 6242c399ab Generate 1 token to verify the health of the inference service in /health (#1154) Lucien 2024-08-21 01:14:34 +08:00
  • 04707b09b7 misc: add hypervisor vendor (#1165) Yineng Zhang 2024-08-21 02:14:51 +10:00
  • ff2cfdb1a2 [Feature] add disable-custom-all-reduce (#1148) Xu-Chen 2024-08-20 23:44:12 +08:00
  • a8ae640328 Improve docs and warnings (#1164) Lianmin Zheng 2024-08-20 08:31:29 -07:00
  • d8476818ef feat: allow streaming for multi-prompt and/or parallel sampling (#1134) Juwan Yoo 2024-08-20 08:06:55 -07:00
  • df191254ab Optimize MLA/GQA/MQA Triton decoding (#1138) Ke Bao 2024-08-19 18:23:07 +08:00
  • b997a18d74 [Feat]Add support for optional start len of logprobs (#1035) yichuan~ 2024-08-18 23:45:41 -07:00
  • d8627ed16d [Docs] Add instruction for running on clouds and kubernetes with SkyPilot (#1144) Zhanghao Wu 2024-08-18 23:01:55 -07:00
  • fa13b95d6b fixed a typo (#1143) min-xu-et 2024-08-18 14:29:09 -07:00
  • 3c1f5a9220 Fix duplicated imports in hf_transformers_utils.py (#1141) Lianmin Zheng 2024-08-17 18:03:00 -07:00
  • 57d0bd91ec Improve benchmark (#1140) Lianmin Zheng 2024-08-17 17:43:23 -07:00
  • cdc8d60752 Improve the code style: more comments and remove useless packages (#1139) Lianmin Zheng 2024-08-17 14:37:52 -07:00
  • 9208591f05 fix: use fp16 dtype for sm75 (#1136) Yineng Zhang 2024-08-17 22:45:42 +08:00
  • 5d0d40d0eb Fix CI accuracy && time out limit (#1133) Liangsheng Yin 2024-08-16 21:41:11 -07:00
  • f624f6a6cc Fix port conflicts between local CI and runner CI. (#1131) Liangsheng Yin 2024-08-16 15:12:38 -07:00
  • 3694f8f996 Mixed style of chunked prefill (#1013) Liangsheng Yin 2024-08-16 02:13:00 -07:00
  • 5a261bd055 Fix the deadlock in multi-node tp (#1122) Lianmin Zheng 2024-08-16 01:39:24 -07:00
  • 6aa8ad14f8 fix: resolve Python.h header missing (#1119) Yineng Zhang 2024-08-16 13:46:43 +08:00
  • 26e9c12c15 ci: compatible with fork repo (#1115) Yineng Zhang 2024-08-16 02:26:44 +08:00
  • 87a0db82b8 update hyperparameter guide (#1114) Lianmin Zheng 2024-08-15 10:54:24 -07:00
  • 5bd953749b chore: bump v0.2.13 (#1111) Yineng Zhang 2024-08-16 01:50:43 +08:00
  • 0cb099e20a set CUDA_DEVICE_MAX_CONNECTIONS=1 (#1113) Lianmin Zheng 2024-08-15 10:47:39 -07:00
  • 93d4e354d8 [Fix] Window attention compatible with RadixAttention and chunked prefill (#1112) Ying Sheng 2024-08-15 10:33:20 -07:00
  • 9195d1362a misc: rm unused model_loader (#1110) Yineng Zhang 2024-08-15 23:29:35 +08:00
  • 14cb544d56 [Fix] fix flashinfer usage for window attention (#1107) Ying Sheng 2024-08-15 00:53:24 -07:00
  • e86b1ccbf0 Enable chunked prefill by default (#1040) Lianmin Zheng 2024-08-14 21:56:20 -07:00
  • 8d2d876fc8 [Fix] fix the typo bug for window attention (#1106) Ying Sheng 2024-08-14 21:56:01 -07:00
  • 326df4bab2 Use a single workspace for flashinfer (#1077) Lianmin Zheng 2024-08-14 19:25:37 -07:00
  • 6767e2229f Support jinja as chat template file (#1104) Ying Sheng 2024-08-14 17:43:14 -07:00
  • 73cf6834f2 Support stop_token_ids in sglang API (#1092) Liangsheng Yin 2024-08-14 17:31:39 -07:00
  • 1c2b5f5240 docs: update nsys usage (#1103) Yineng Zhang 2024-08-15 01:39:15 +08:00
  • 96a2093ef0 [Fix] Compatibility of window attention and cuda graph (#1090) Ying Sheng 2024-08-14 10:37:01 -07:00
  • a34dd86a7d Use dtype to control generate (#1082) Liangsheng Yin 2024-08-14 08:58:07 -07:00
  • 67c0d832a6 docs: update pr template (#1099) Yineng Zhang 2024-08-14 20:25:39 +08:00
  • a59636bb5e Update grok 1 model (#1095) Lianmin Zheng 2024-08-14 04:40:44 -07:00
  • fe5024325b docs: update README (#1098) Yineng Zhang 2024-08-14 19:40:05 +08:00