chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
3,455 Commits 1 Branch 0 Tags
41ba767f0c69559eec3ef82874b5e7670ed66bc2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Liangsheng Yin 17d080b7ae Remove Cargo.lock, add it into .gitignore (#6438) 2025-05-20 12:01:32 +08:00
Liangsheng Yin a3e4e9bf9e Better PD initialization (#5751) 2025-05-07 01:12:57 +08:00
Liangsheng Yin 6d4d3bc81d Fix not "import os" (#6057) 2025-05-06 22:06:41 +08:00
Liangsheng Yin 40d9b8acce Improve overlap scheduling (#5788) 2025-04-28 11:19:16 +08:00
Liangsheng Yin beb65c7433 [PD]Reduce kv transfer threads (#5791) 2025-04-27 23:03:30 +08:00
Liangsheng Yin c55550cbf0 [PD] Better logs (#5715) 2025-04-25 17:25:45 +08:00
Liangsheng Yin e69a219074 Enhance GPU memory settings (#5604) 2025-04-21 15:15:00 -07:00
Liangsheng Yin 33b16ad178 Distinguish bootstrap key only in decode server (#5422) 2025-04-15 20:59:28 +08:00
Liangsheng Yin 44afde82d7 Fix PD disaggregation bugs (#5326) 2025-04-14 19:27:30 +08:00
Liangsheng Yin f765579046 Fix typo: infight -> inflight (#5357) 2025-04-14 01:25:30 +08:00
Liangsheng Yin 8616357a97 Fix deepseek awq v3 (#3450) 2025-02-12 22:09:52 +08:00
Liangsheng Yin e7ebecf82e Fix cache hit rate when chunked prefill (#2555) 2024-12-26 03:14:28 -08:00
Liangsheng Yin 5f12f0e7af Fix chunked prefill when ignore eos (#2290) 2024-12-01 00:37:53 -08:00
Liangsheng YinandLianmin Zheng b9fd178f1b Fix retraction + overlap (#1860) 
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
 2024-10-31 18:27:42 -07:00
Liangsheng Yin 07bf2e846a Allow consecutive ports when launching multiple sglang servers. (#1802) 2024-10-26 06:43:24 +00:00
Liangsheng Yin a628dd8e31 Set ZMQ buffer size heuristic (#1801) 2024-10-25 23:15:56 -07:00
Liangsheng Yin 1e8903414a Fix possible ZMQ hanging (#1800) 2024-10-25 23:07:07 -07:00
Liangsheng Yin a2f5e7555f Fix memory leak when doing chunked prefill (#1787) 2024-10-25 08:01:17 -07:00
Liangsheng Yin 3f5ac88d02 Fix out of memory message. (#1771) 2024-10-23 15:20:39 -07:00
Liangsheng Yin 5e1558f1f2 Update max_req_len and max_req_input_len (#1748) 2024-10-21 16:12:04 -07:00
Liangsheng Yin 94cde10920 Llama3.2 vision model support (#1551) 2024-10-21 15:01:21 -07:00
Liangsheng Yin efb099cdee Fix prefill oom (#1743) 2024-10-21 03:54:35 -07:00
Liangsheng Yin 9e0dac1ad7 Fix regex and logprob conflicts when chunked prefilling (#1703) 2024-10-17 18:33:21 -07:00
Liangsheng Yin b6b4094621 Fix filter_batch function call (#1681) 2024-10-15 22:59:26 -07:00
Liangsheng Yin 5d0ba4038f Refine the add request reasons to avoid corner cases. (#1574) 2024-10-04 18:00:18 -07:00
Liangsheng Yin 100f5b8bc9 Simplify flashinfer dispatch (#1552) 2024-10-01 00:28:42 -07:00
Liangsheng Yin 619bb6ddda Dispatch flashinfer wrappers (#1550) 2024-09-30 23:12:36 -07:00
Liangsheng Yin b88ea90d4a Fix bugs of logprobs_nums (#1548) 2024-09-30 17:09:54 -07:00
Liangsheng Yin 99ec439da4 Organize Attention Backends (#1547) 2024-09-30 15:54:18 -07:00
Liangsheng Yin 55b974f96f Process image in parallel (#1539) 2024-09-29 18:52:43 -07:00
Liangsheng Yin fd9ad817ec Organize image inputs (#1531) 2024-09-29 06:28:55 +00:00
Liangsheng Yin 4353acb469 minor: fix config (#1524) 2024-09-27 01:49:16 -07:00
Liangsheng Yin 7c162fa9c5 Fix schedule bug (#1451) 2024-09-17 22:59:32 -07:00
Liangsheng Yin 36078fb247 fix schedule bug (#1450) 2024-09-17 16:33:53 -07:00
Liangsheng Yin 70b6802982 Optimize conflicts between CUDA graph and vocab mask tensors (#1392) 2024-09-13 20:27:53 -07:00
Liangsheng Yin 4462137777 Add no commit to main rule (#1393) 2024-09-12 05:40:45 +08:00
Liangsheng Yin 144bc70fcc Organize flashinfer indices update (#1378) 2024-09-10 17:38:59 -07:00
Liangsheng Yin fbb4754cb8 Fix vocab mask update bug (#1376) 2024-09-10 13:10:36 -07:00
Liangsheng Yin 69b3bb9ae1 Unify forward mode (#1360) 2024-09-09 13:49:29 -07:00
Liangsheng Yin 05bea6883c Fix some online scheduling delay (#1345) 2024-09-07 20:46:27 -07:00
Liangsheng Yin ab4a83b259 Optimize schedule (#1339) 2024-09-05 14:30:26 -07:00
Liangsheng Yin a5a134f39f Fix bugs in sampler with CUDA graph / torch.compile (#1306) 2024-09-02 23:18:48 +00:00
Liangsheng Yin 47f20da223 Fix regex mask (#1296) 2024-09-01 21:50:58 -07:00
Liangsheng Yin 381dd57bd6 Sampler cudagraph (#1253) 2024-08-28 18:58:52 -07:00
Liangsheng Yin 1ece2cda3d Fix bench latency benchmark (#1225) 2024-08-28 00:37:32 -07:00
Liangsheng Yin c61a1b6f97 Torch compile CI throughput test (#1223) 2024-08-26 13:52:58 -07:00
Liangsheng YinandYineng Zhang 75ce37f401 Move sampler into CUDA graph (#1201) 
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2024-08-26 07:02:50 -07:00
Liangsheng Yin 632d506d0b minor: improve CI and dependencies (#1212) 2024-08-26 04:26:31 +00:00
Liangsheng Yin 364d3d72a7 Fix broken penalty (#1184) 2024-08-22 08:16:35 +00:00
Liangsheng Yin 83e23c69b3 Improve code style of sampler (#1168) 2024-08-21 16:48:24 -07:00
Liangsheng Yin 5d0d40d0eb Fix CI accuracy && time out limit (#1133) 2024-08-16 21:41:11 -07:00
Liangsheng Yin f624f6a6cc Fix port conflicts between local CI and runner CI. (#1131) 2024-08-16 15:12:38 -07:00
Liangsheng Yin 3694f8f996 Mixed style of chunked prefill (#1013) 2024-08-16 09:13:00 +00:00
Liangsheng Yin 73cf6834f2 Support stop_token_ids in sglang API (#1092) 2024-08-15 00:31:39 +00:00
Liangsheng Yinandzhyncs a34dd86a7d Use dtype to control generate (#1082) 
Co-authored-by: zhyncs <me@zhyncs.com>
 2024-08-14 15:58:07 +00:00
Liangsheng Yin e205527cb1 Fix jump forward final state circular path bug. (#1084) 2024-08-13 21:14:05 -07:00
Liangsheng Yin fb7421db0d minor: some potential bugs (#1044) 2024-08-12 05:35:44 +00:00
Liangsheng Yin 7de6034534 Fix the prefix indices (#1037) 2024-08-11 17:57:02 -07:00
Liangsheng Yin 7b6a5332ca Fix triton args init (#1034) 2024-08-11 12:11:26 -07:00
Liangsheng Yin fcc0f5ed99 Fix wrong assert (#1028) 2024-08-11 09:22:16 +00:00
Liangsheng Yin 43fbb6d919 Fix input_ids && rename to fill_ids (#1021) 2024-08-10 16:24:12 -07:00
Liangsheng Yin 62757db6f0 Reduce the overhead when cache is disabled (#1010) 2024-08-09 16:36:57 -07:00
Liangsheng Yin 73fa2d49d5 Some warnings to crash when CI (#1009) 2024-08-09 15:16:23 -07:00
Liangsheng Yin 1ac304eeb4 Adjust InputeMetadata and ScheduleBatch (#981) 2024-08-08 01:11:22 -07:00
Liangsheng Yin 6ed4e3b8fb Fix chunked prefill (#984) 2024-08-07 22:28:42 -07:00
Liangsheng Yin 2b8257f325 Adjust max prefix len (#980) 2024-08-08 00:41:26 +00:00
Liangsheng Yin 7623091d97 RadixCache method adjust (#977) 2024-08-07 15:52:24 -07:00
Liangsheng Yin f724f1f1e9 PrefillAdder abstraction (#968) 2024-08-07 13:47:28 -07:00
Liangsheng Yin 4d929107ae Run purge-cache only in sgl-project (#976) 2024-08-07 13:16:36 -07:00
Liangsheng Yin fbe0c818c2 Purge self-runner's pip cache weekly (#975) 2024-08-07 12:43:12 -07:00
Liangsheng Yin a01ddd9605 misc: fix the req_to_token member change (#967) 2024-08-07 01:52:10 -07:00
Liangsheng Yin 7fa54a1ab3 Make req_pool_indices on CPU (#960) 2024-08-07 01:41:25 -07:00
Liangsheng Yin 87e8c090e9 Organize code (rename, movement) (#953) 2024-08-06 20:50:32 -07:00
Liangsheng Yin ad56e68495 Fix stuck in get_new_prefill_batch (#948) 2024-08-06 01:05:58 -07:00
Liangsheng Yin bb66cc4c52 Fix CI && python3.8 compatible (#920) 2024-08-04 16:02:05 -07:00
Liangsheng Yin 947402c829 Reorder CI unit tests. (#908) 2024-08-03 16:18:50 -07:00
Liangsheng Yin 12ce3befb6 Update runner docs (#879) 2024-08-01 17:37:47 -07:00
Liangsheng Yin 70c78cfb03 Update runner docs (#876) 2024-08-01 15:32:33 -07:00
Liangsheng Yin c020f9ceda Support chunked prefill when radix cache is disabled (#811) 2024-08-01 00:29:01 -07:00
Liangsheng Yin a6c7ebbbcb Add req slots leaking check (#842) 2024-07-30 18:29:01 -07:00
Liangsheng Yin 6b0f2e9088 Add --max-total-tokens (#840) 2024-07-30 13:33:55 -07:00
Liangsheng Yin cdcbde5fc3 Code structure refactor (#807) 2024-07-29 23:04:48 -07:00
Liangsheng Yin 3520f75fb1 Remove inf value for chunked prefill size (#812) 2024-07-29 18:34:25 -07:00
Liangsheng Yin c8e9fed87a Organize public APIs (#809) 2024-07-29 15:34:16 -07:00
Liangsheng Yin 7cd4f244a4 Chunked prefill (#800) 2024-07-29 03:32:58 -07:00
Liangsheng Yin 2ec39ab712 Chunked prefill support (#797) 2024-07-29 02:21:50 -07:00
Liangsheng Yin ba29504b21 Update supported models (#763) 2024-07-27 15:53:53 +10:00
Liangsheng Yin d9fccfefe2 Fix context length (#757) 2024-07-26 18:13:13 -07:00
Liangsheng Yin 679ebcbbdc Deepseek v2 support (#693) 2024-07-26 17:10:07 -07:00
Liangsheng Yin 04ec6ba2ac Fix dockerfile and triton cache manager (#720) 2024-07-25 03:04:21 -07:00
Liangsheng Yin 4cd64b8ee6 Auto adjust new ratio (#708) 2024-07-23 22:06:02 -07:00
Liangsheng Yin 268684439b Use min new token ratio at start (#701) 2024-07-23 11:52:50 -07:00
Liangsheng Yin eedc12e12e Support Deepseek MoE Model (#689) 2024-07-21 03:09:29 -07:00
Liangsheng Yin caaad53b52 Support gpt-bigcode model class (#681) 2024-07-20 18:34:37 -07:00
Liangsheng Yin 69d19188fc Decouple kv (#679) 2024-07-20 14:16:45 -07:00
Liangsheng Yin f424e76d96 Fix illegal tokens during sampling (#676) 2024-07-20 03:11:15 -07:00
Liangsheng Yin 39c57317e1 Revert "Temporary fix invalid sample results" (#673) 2024-07-20 02:06:31 -07:00
Liangsheng Yin 8f4b1559e7 Temporary fix invalid sample results (#668) 2024-07-20 00:51:05 -07:00
Liangsheng Yin 7620cd37dd Fix jump forward when streaming (#665) 2024-07-19 16:42:06 -07:00
Liangsheng Yin a9ef49c12c Detokenize incrementally when streaming (#653) 2024-07-18 17:57:40 -07:00