chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
695 Commits 1 Branch 0 Tags
bea2bb9eeae6cf6f1bdfbb6aaaae2d91adea7bac
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Liangsheng Yin 5d0d40d0eb Fix CI accuracy && time out limit (#1133) 2024-08-16 21:41:11 -07:00
Liangsheng Yin f624f6a6cc Fix port conflicts between local CI and runner CI. (#1131) 2024-08-16 15:12:38 -07:00
Liangsheng Yin 3694f8f996 Mixed style of chunked prefill (#1013) 2024-08-16 09:13:00 +00:00
Liangsheng Yin 73cf6834f2 Support stop_token_ids in sglang API (#1092) 2024-08-15 00:31:39 +00:00
Liangsheng Yinandzhyncs a34dd86a7d Use dtype to control generate (#1082) 
Co-authored-by: zhyncs <me@zhyncs.com>
 2024-08-14 15:58:07 +00:00
Liangsheng Yin e205527cb1 Fix jump forward final state circular path bug. (#1084) 2024-08-13 21:14:05 -07:00
Liangsheng Yin fb7421db0d minor: some potential bugs (#1044) 2024-08-12 05:35:44 +00:00
Liangsheng Yin 7de6034534 Fix the prefix indices (#1037) 2024-08-11 17:57:02 -07:00
Liangsheng Yin 7b6a5332ca Fix triton args init (#1034) 2024-08-11 12:11:26 -07:00
Liangsheng Yin fcc0f5ed99 Fix wrong assert (#1028) 2024-08-11 09:22:16 +00:00
Liangsheng Yin 43fbb6d919 Fix input_ids && rename to fill_ids (#1021) 2024-08-10 16:24:12 -07:00
Liangsheng Yin 62757db6f0 Reduce the overhead when cache is disabled (#1010) 2024-08-09 16:36:57 -07:00
Liangsheng Yin 73fa2d49d5 Some warnings to crash when CI (#1009) 2024-08-09 15:16:23 -07:00
Liangsheng Yin 1ac304eeb4 Adjust InputeMetadata and ScheduleBatch (#981) 2024-08-08 01:11:22 -07:00
Liangsheng Yin 6ed4e3b8fb Fix chunked prefill (#984) 2024-08-07 22:28:42 -07:00
Liangsheng Yin 2b8257f325 Adjust max prefix len (#980) 2024-08-08 00:41:26 +00:00
Liangsheng Yin 7623091d97 RadixCache method adjust (#977) 2024-08-07 15:52:24 -07:00
Liangsheng Yin f724f1f1e9 PrefillAdder abstraction (#968) 2024-08-07 13:47:28 -07:00
Liangsheng Yin 4d929107ae Run purge-cache only in sgl-project (#976) 2024-08-07 13:16:36 -07:00
Liangsheng Yin fbe0c818c2 Purge self-runner's pip cache weekly (#975) 2024-08-07 12:43:12 -07:00
Liangsheng Yin a01ddd9605 misc: fix the req_to_token member change (#967) 2024-08-07 01:52:10 -07:00
Liangsheng Yin 7fa54a1ab3 Make req_pool_indices on CPU (#960) 2024-08-07 01:41:25 -07:00
Liangsheng Yin 87e8c090e9 Organize code (rename, movement) (#953) 2024-08-06 20:50:32 -07:00
Liangsheng Yin ad56e68495 Fix stuck in get_new_prefill_batch (#948) 2024-08-06 01:05:58 -07:00
Liangsheng Yin bb66cc4c52 Fix CI && python3.8 compatible (#920) 2024-08-04 16:02:05 -07:00
Liangsheng Yin 947402c829 Reorder CI unit tests. (#908) 2024-08-03 16:18:50 -07:00
Liangsheng Yin 12ce3befb6 Update runner docs (#879) 2024-08-01 17:37:47 -07:00
Liangsheng Yin 70c78cfb03 Update runner docs (#876) 2024-08-01 15:32:33 -07:00
Liangsheng Yin c020f9ceda Support chunked prefill when radix cache is disabled (#811) 2024-08-01 00:29:01 -07:00
Liangsheng Yin a6c7ebbbcb Add req slots leaking check (#842) 2024-07-30 18:29:01 -07:00
Liangsheng Yin 6b0f2e9088 Add --max-total-tokens (#840) 2024-07-30 13:33:55 -07:00
Liangsheng Yin cdcbde5fc3 Code structure refactor (#807) 2024-07-29 23:04:48 -07:00
Liangsheng Yin 3520f75fb1 Remove inf value for chunked prefill size (#812) 2024-07-29 18:34:25 -07:00
Liangsheng Yin c8e9fed87a Organize public APIs (#809) 2024-07-29 15:34:16 -07:00
Liangsheng Yin 7cd4f244a4 Chunked prefill (#800) 2024-07-29 03:32:58 -07:00
Liangsheng Yin 2ec39ab712 Chunked prefill support (#797) 2024-07-29 02:21:50 -07:00
Liangsheng Yin ba29504b21 Update supported models (#763) 2024-07-27 15:53:53 +10:00
Liangsheng Yin d9fccfefe2 Fix context length (#757) 2024-07-26 18:13:13 -07:00
Liangsheng Yin 679ebcbbdc Deepseek v2 support (#693) 2024-07-26 17:10:07 -07:00
Liangsheng Yin 04ec6ba2ac Fix dockerfile and triton cache manager (#720) 2024-07-25 03:04:21 -07:00
Liangsheng Yin 4cd64b8ee6 Auto adjust new ratio (#708) 2024-07-23 22:06:02 -07:00
Liangsheng Yin 268684439b Use min new token ratio at start (#701) 2024-07-23 11:52:50 -07:00
Liangsheng Yin eedc12e12e Support Deepseek MoE Model (#689) 2024-07-21 03:09:29 -07:00
Liangsheng Yin caaad53b52 Support gpt-bigcode model class (#681) 2024-07-20 18:34:37 -07:00
Liangsheng Yin 69d19188fc Decouple kv (#679) 2024-07-20 14:16:45 -07:00
Liangsheng Yin f424e76d96 Fix illegal tokens during sampling (#676) 2024-07-20 03:11:15 -07:00
Liangsheng Yin 39c57317e1 Revert "Temporary fix invalid sample results" (#673) 2024-07-20 02:06:31 -07:00
Liangsheng Yin 8f4b1559e7 Temporary fix invalid sample results (#668) 2024-07-20 00:51:05 -07:00
Liangsheng Yin 7620cd37dd Fix jump forward when streaming (#665) 2024-07-19 16:42:06 -07:00
Liangsheng Yin a9ef49c12c Detokenize incrementally when streaming (#653) 2024-07-18 17:57:40 -07:00
Liangsheng Yin abd5385ac5 Move global_server_args_dict (#642) 2024-07-17 13:49:15 -07:00
Liangsheng Yin 3de2f30a27 Flashinfer sample kernel (#617) 2024-07-17 13:24:43 -07:00
Liangsheng Yin 8832ecb1e4 Reduce docker size (#632) 2024-07-16 16:12:12 -07:00
Liangsheng Yin 5ff60eda78 Fix vertexai (#633) 2024-07-16 16:07:19 -07:00
Liangsheng Yin c9ee3d3559 Fix model forward grad (#628) 2024-07-15 22:09:09 -07:00
Liangsheng Yin a56858ba67 Unify index operations (#620) 2024-07-14 12:55:55 -07:00
Liangsheng Yin 564a898ad9 Optimize mem indices mangement (#619) 2024-07-13 23:39:37 -07:00
Liangsheng Yin 10143e1a5f Memorypool chunked prefetch (#614) 2024-07-13 15:24:03 -07:00
Liangsheng Yin f25b76c02a add LogitsMetadata (#604) 2024-07-08 17:46:55 -07:00
Liangsheng Yin 0877f1e75b Fix streaming (#600) 2024-07-07 01:55:58 -07:00
Liangsheng Yin 5304b4ef58 Add --enable-p2p-check option (#599) 2024-07-06 23:34:10 -07:00
Liangsheng Yin 96c503eb60 fix the broken server args (#585) 2024-07-03 16:01:19 -07:00
Liangsheng Yin 05471f2103 Update test_flashinfer (#560) 2024-06-24 15:23:57 +08:00
Liangsheng Yin 92cb93f390 Fix latency benchmark (#557) 2024-06-22 15:11:04 +08:00
Liangsheng Yin ad5f04d6ce Fix the Jump-Forward with Chinese (#551) 2024-06-16 21:45:04 +08:00
Liangsheng Yin 40e53d65cb Add disk cache for loading ShareGPT dataset. (#542) 2024-06-13 16:37:12 +08:00
Liangsheng Yin 9c902b1954 Decode Incrementally (#517) 2024-06-11 23:39:12 -07:00
Liangsheng Yin f06e90c2cf Optimize retract (#440) 2024-05-26 00:07:26 +08:00
Liangsheng Yin 5b647543c1 Fix the broken --disable-radix-cache (#451) 2024-05-19 13:00:12 +08:00
Liangsheng Yin 690d162d97 Format code (#441) 2024-05-14 22:40:46 +08:00
Liangsheng Yin 39191c8515 Cache optimizations (#418) 2024-05-13 12:47:13 +08:00
Liangsheng Yin 6ec65f4555 Make public APIs more standard. (#416) 2024-05-09 15:39:22 +08:00
Liangsheng Yin d5de20a3ee Fix sync() when fork(1) (#412) 2024-05-08 15:15:18 +08:00
Liangsheng Yin 14522e6a26 Organize Benchmark (#381) 2024-05-05 16:14:17 +08:00
Liangsheng Yin 95c4e0dfac Format Benchmark Code (#399) 2024-04-28 21:06:22 +08:00
Liangsheng Yin 19818b9c2f Minor: style improvement of radix_cache and memory_pool (#395) 2024-04-26 01:01:36 +08:00
Liangsheng Yin 9216b10678 Improve performance when running with full parallel (#394) 2024-04-25 17:29:07 +08:00
Liangsheng Yin da19434c2f Benchmark Updates (#382) 2024-04-24 02:23:01 +08:00
Liangsheng Yin 150d7020ed Revert removing the unused imports (#385) 2024-04-23 22:36:33 +08:00
Liangsheng Yin 9acc6e3504 add .isort.cfg (#378) 2024-04-22 22:38:09 +08:00
Liangsheng Yin 1bf1cf1953 Reduce overhead when fork(1) (#375) 2024-04-21 17:25:14 +08:00
Liangsheng Yin 62b3812b69 Time cost utils (#355) 2024-04-09 23:27:31 +08:00
Liangsheng Yin ed27a6b992 Revert "Eliminate 2 gpu ops during sampling when logit_bias is zero" (#345) 2024-04-03 12:45:01 +08:00
Liangsheng YinandQubitium 463c6632a8 Eliminate 2 gpu ops during sampling when logit_bias is zero (#343) 
Co-authored-by: Qubitium <417764+Qubitium@users.noreply.github.com>
 2024-04-02 19:14:55 +08:00
Liangsheng Yin 2af565b3bb [model] DBRX-instruct support (#337) 2024-03-28 10:05:19 -07:00
Liangsheng Yin 3842eba5fa Logprobs Refractor (#331) 2024-03-28 14:34:49 +08:00
Liangsheng Yin 24e59f5350 model_runner simplify (#329) 2024-03-24 19:48:37 +08:00
Liangsheng Yin 7523541962 model_rpc style improvement (#293) 2024-03-24 15:41:24 +08:00
Liangsheng Yin 1b35547927 Organize server_args (#277) 2024-03-11 20:06:52 +08:00
Liangsheng Yin 89885b31ef Gemma Support (#256) 2024-03-11 12:14:27 +08:00
Liangsheng Yin a7ace9c88d Fix qwen config (#261) 2024-03-10 18:54:18 -07:00
Liangsheng Yin dfb13ac455 Fix addr reuse in check_port (#253) 2024-03-03 17:09:16 +08:00
Liangsheng Yin ec90b9c054 Upload agent_calls.jsonl download link (#226) 2024-02-24 19:03:46 +08:00
Liangsheng YinandEnrique Shockwave c4e9ebe3a4 Fix stop str merging (#225) 
Co-authored-by: Enrique Shockwave <33002121+qeternity@users.noreply.github.com>
 2024-02-24 16:05:21 +08:00
Liangsheng Yinandcomaniac 91e036334f Adjust outlines version. (#200) 
Co-authored-by: comaniac <hao.yu.cody@gmail.com>
 2024-02-17 13:40:39 +08:00
Liangsheng Yin 37b42297f8 import outlines (#168) 2024-02-09 10:13:02 +08:00
Liangsheng YinandJa1Zhou b1a3a454ee add --disable-disk-cache (#160) 
Co-authored-by: Ja1Zhou <50169346+Ja1Zhou@users.noreply.github.com>
 2024-02-08 00:50:12 +08:00
Liangsheng Yin 26f0bedc8f jump-forward rename (#144) 2024-02-05 16:50:37 +08:00
Liangsheng Yin bb3a3b6675 Support Faster JSON decoding for llava (#137) 
When sending fast-forwarded reqs to model_rpc, re-calculate `pad_input_ids`
 2024-02-03 23:32:05 +08:00
Liangsheng Yin cd8c3ccd95 Fix is_multimodal_model judge (#132) 2024-02-03 11:48:01 +08:00