chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
402 Commits 1 Branch 0 Tags
df7c4c19b4b93ec5b7ba9f4e227f802f8d82a246
Commit Graph
69 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Liangsheng Yin eedc12e12e Support Deepseek MoE Model (#689) 2024-07-21 03:09:29 -07:00
Liangsheng Yin caaad53b52 Support gpt-bigcode model class (#681) 2024-07-20 18:34:37 -07:00
Liangsheng Yin 69d19188fc Decouple kv (#679) 2024-07-20 14:16:45 -07:00
Liangsheng Yin f424e76d96 Fix illegal tokens during sampling (#676) 2024-07-20 03:11:15 -07:00
Liangsheng Yin 39c57317e1 Revert "Temporary fix invalid sample results" (#673) 2024-07-20 02:06:31 -07:00
Liangsheng Yin 8f4b1559e7 Temporary fix invalid sample results (#668) 2024-07-20 00:51:05 -07:00
Liangsheng Yin 7620cd37dd Fix jump forward when streaming (#665) 2024-07-19 16:42:06 -07:00
Liangsheng Yin a9ef49c12c Detokenize incrementally when streaming (#653) 2024-07-18 17:57:40 -07:00
Liangsheng Yin abd5385ac5 Move global_server_args_dict (#642) 2024-07-17 13:49:15 -07:00
Liangsheng Yin 3de2f30a27 Flashinfer sample kernel (#617) 2024-07-17 13:24:43 -07:00
Liangsheng Yin 8832ecb1e4 Reduce docker size (#632) 2024-07-16 16:12:12 -07:00
Liangsheng Yin 5ff60eda78 Fix vertexai (#633) 2024-07-16 16:07:19 -07:00
Liangsheng Yin c9ee3d3559 Fix model forward grad (#628) 2024-07-15 22:09:09 -07:00
Liangsheng Yin a56858ba67 Unify index operations (#620) 2024-07-14 12:55:55 -07:00
Liangsheng Yin 564a898ad9 Optimize mem indices mangement (#619) 2024-07-13 23:39:37 -07:00
Liangsheng Yin 10143e1a5f Memorypool chunked prefetch (#614) 2024-07-13 15:24:03 -07:00
Liangsheng Yin f25b76c02a add LogitsMetadata (#604) 2024-07-08 17:46:55 -07:00
Liangsheng Yin 0877f1e75b Fix streaming (#600) 2024-07-07 01:55:58 -07:00
Liangsheng Yin 5304b4ef58 Add --enable-p2p-check option (#599) 2024-07-06 23:34:10 -07:00
Liangsheng Yin 96c503eb60 fix the broken server args (#585) 2024-07-03 16:01:19 -07:00
Liangsheng Yin 05471f2103 Update test_flashinfer (#560) 2024-06-24 15:23:57 +08:00
Liangsheng Yin 92cb93f390 Fix latency benchmark (#557) 2024-06-22 15:11:04 +08:00
Liangsheng Yin ad5f04d6ce Fix the Jump-Forward with Chinese (#551) 2024-06-16 21:45:04 +08:00
Liangsheng Yin 40e53d65cb Add disk cache for loading ShareGPT dataset. (#542) 2024-06-13 16:37:12 +08:00
Liangsheng Yin 9c902b1954 Decode Incrementally (#517) 2024-06-11 23:39:12 -07:00
Liangsheng Yin f06e90c2cf Optimize retract (#440) 2024-05-26 00:07:26 +08:00
Liangsheng Yin 5b647543c1 Fix the broken --disable-radix-cache (#451) 2024-05-19 13:00:12 +08:00
Liangsheng Yin 690d162d97 Format code (#441) 2024-05-14 22:40:46 +08:00
Liangsheng Yin 39191c8515 Cache optimizations (#418) 2024-05-13 12:47:13 +08:00
Liangsheng Yin 6ec65f4555 Make public APIs more standard. (#416) 2024-05-09 15:39:22 +08:00
Liangsheng Yin d5de20a3ee Fix sync() when fork(1) (#412) 2024-05-08 15:15:18 +08:00
Liangsheng Yin 14522e6a26 Organize Benchmark (#381) 2024-05-05 16:14:17 +08:00
Liangsheng Yin 95c4e0dfac Format Benchmark Code (#399) 2024-04-28 21:06:22 +08:00
Liangsheng Yin 19818b9c2f Minor: style improvement of radix_cache and memory_pool (#395) 2024-04-26 01:01:36 +08:00
Liangsheng Yin 9216b10678 Improve performance when running with full parallel (#394) 2024-04-25 17:29:07 +08:00
Liangsheng Yin da19434c2f Benchmark Updates (#382) 2024-04-24 02:23:01 +08:00
Liangsheng Yin 150d7020ed Revert removing the unused imports (#385) 2024-04-23 22:36:33 +08:00
Liangsheng Yin 9acc6e3504 add .isort.cfg (#378) 2024-04-22 22:38:09 +08:00
Liangsheng Yin 1bf1cf1953 Reduce overhead when fork(1) (#375) 2024-04-21 17:25:14 +08:00
Liangsheng Yin 62b3812b69 Time cost utils (#355) 2024-04-09 23:27:31 +08:00
Liangsheng Yin ed27a6b992 Revert "Eliminate 2 gpu ops during sampling when logit_bias is zero" (#345) 2024-04-03 12:45:01 +08:00
Liangsheng YinandQubitium 463c6632a8 Eliminate 2 gpu ops during sampling when logit_bias is zero (#343) 
Co-authored-by: Qubitium <417764+Qubitium@users.noreply.github.com>
 2024-04-02 19:14:55 +08:00
Liangsheng Yin 2af565b3bb [model] DBRX-instruct support (#337) 2024-03-28 10:05:19 -07:00
Liangsheng Yin 3842eba5fa Logprobs Refractor (#331) 2024-03-28 14:34:49 +08:00
Liangsheng Yin 24e59f5350 model_runner simplify (#329) 2024-03-24 19:48:37 +08:00
Liangsheng Yin 7523541962 model_rpc style improvement (#293) 2024-03-24 15:41:24 +08:00
Liangsheng Yin 1b35547927 Organize server_args (#277) 2024-03-11 20:06:52 +08:00
Liangsheng Yin 89885b31ef Gemma Support (#256) 2024-03-11 12:14:27 +08:00
Liangsheng Yin a7ace9c88d Fix qwen config (#261) 2024-03-10 18:54:18 -07:00
Liangsheng Yin dfb13ac455 Fix addr reuse in check_port (#253) 2024-03-03 17:09:16 +08:00
Liangsheng Yin ec90b9c054 Upload agent_calls.jsonl download link (#226) 2024-02-24 19:03:46 +08:00
Liangsheng YinandEnrique Shockwave c4e9ebe3a4 Fix stop str merging (#225) 
Co-authored-by: Enrique Shockwave <33002121+qeternity@users.noreply.github.com>
 2024-02-24 16:05:21 +08:00
Liangsheng Yinandcomaniac 91e036334f Adjust outlines version. (#200) 
Co-authored-by: comaniac <hao.yu.cody@gmail.com>
 2024-02-17 13:40:39 +08:00
Liangsheng Yin 37b42297f8 import outlines (#168) 2024-02-09 10:13:02 +08:00
Liangsheng YinandJa1Zhou b1a3a454ee add --disable-disk-cache (#160) 
Co-authored-by: Ja1Zhou <50169346+Ja1Zhou@users.noreply.github.com>
 2024-02-08 00:50:12 +08:00
Liangsheng Yin 26f0bedc8f jump-forward rename (#144) 2024-02-05 16:50:37 +08:00
Liangsheng Yin bb3a3b6675 Support Faster JSON decoding for llava (#137) 
When sending fast-forwarded reqs to model_rpc, re-calculate `pad_input_ids`
 2024-02-03 23:32:05 +08:00
Liangsheng Yin cd8c3ccd95 Fix is_multimodal_model judge (#132) 2024-02-03 11:48:01 +08:00
hnyls2002 9c121f2a45 minor fix: result dump format 2024-02-02 09:58:24 +00:00
Liangsheng Yin 79cb018e4b Add city doc benchmark mode (#129) 2024-02-01 13:38:47 +08:00
Liangsheng Yin 81561f8e2d Flush Cache API (#103) 2024-01-25 21:32:59 -08:00
Liangsheng Yin 01ee0fbc05 fast regex decode 
Auto-detect constant str path in regex FSM, then extend instead.
 2024-01-25 01:16:25 +08:00
Liangsheng Yin ca13f3b8c5 Disk FSM cache and adjust code. (#63) 2024-01-20 21:26:11 -08:00
Liangsheng Yin 40ab1f0129 Fix the possible bug of decode out of memory (#36) 2024-01-19 11:01:15 -08:00
Liangsheng Yin 01ca82d765 fix radix cache match (#7) 2024-01-15 09:42:46 -08:00
Liangsheng Yin 08ab2a1655 Json Decode && Mutl-Turns (#4) 2024-01-15 00:49:29 -08:00
hnyls2002 f652494df1 fix typo 2024-01-10 04:21:17 +00:00
Liangsheng Yin 331848de9d Add SRT json decode example (#2) 2024-01-09 12:35:44 -08:00
Liangsheng Yin ead5b39f82 Add flashinfer && Oultines (#1) 2024-01-08 08:26:18 -08:00