chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Commit Graph

  • fc82f5a743 [Fix] Fix cuda graph padding for triton attention backend (#1782) Lianmin Zheng 2024-10-24 12:33:15 -07:00
  • 0089c4bc96 [Fix] Fix NaN issues by fixing the cuda graph padding values for flashinfer (#1779) Lianmin Zheng 2024-10-24 04:16:59 -07:00
  • 72e7b57a75 [Bug] Catch any errors caused by parsing json schema (#1776) zolinthecow 2024-10-24 01:54:53 -07:00
  • 87a7cfa080 Fix MockTokenizer in the unit tests (#1774) Lianmin Zheng 2024-10-23 17:47:05 -07:00
  • 8f8f96a621 Fix the perf regression due to additional_stop_token_ids (#1773) Lianmin Zheng 2024-10-23 16:45:21 -07:00
  • 05b3bf5e8e Crash the server on warnings in CI (#1772) Lianmin Zheng 2024-10-23 16:27:13 -07:00
  • 3f5ac88d02 Fix out of memory message. (#1771) Liangsheng Yin 2024-10-23 15:20:39 -07:00
  • 0d800090b4 Fix missing additional_stop_token_ids (#1769) Lianmin Zheng 2024-10-23 12:18:59 -07:00
  • b7d0559496 Update docs (#1768) Lianmin Zheng 2024-10-23 11:28:48 -07:00
  • 80a905475d Fix stop condition for <|eom_id|> (#1766) Lianmin Zheng 2024-10-23 10:47:12 -07:00
  • 9af7b88e3c [Fix] Fix abort in dp (#1767) Lianmin Zheng 2024-10-23 10:46:29 -07:00
  • fbcbb26327 Fix perf regression for set_kv_buffer (#1765) Lianmin Zheng 2024-10-23 09:57:08 -07:00
  • 2fce449b1c [API] add get memory pool size (#1760) Ying Sheng 2024-10-23 00:02:29 -07:00
  • ad4125d1a9 Fuse more ops & Simplify token mapping (#1758) Lianmin Zheng 2024-10-22 23:20:43 -07:00
  • 17536e7e3d Fix edge case for truncated (#1747) Byron Hsu 2024-10-22 21:00:25 -07:00
  • 1f26e8b8e4 Release v0.3.4.post1 (#1749) Lianmin Zheng 2024-10-21 21:16:43 -07:00
  • 5e1558f1f2 Update max_req_len and max_req_input_len (#1748) Liangsheng Yin 2024-10-21 16:12:04 -07:00
  • 94cde10920 Llama3.2 vision model support (#1551) Liangsheng Yin 2024-10-21 15:01:21 -07:00
  • 00611286a1 Fix sliding window attention and gemma-2 unit tests in CI (#1746) Lianmin Zheng 2024-10-21 13:47:12 -07:00
  • e68b9e7667 misc: add CODEOWNERS (#1737) Yineng Zhang 2024-10-21 06:28:32 -07:00
  • 7ce3606891 Faster overlap mode scheduler (#1738) Lianmin Zheng 2024-10-21 04:30:52 -07:00
  • efb099cdee Fix prefill oom (#1743) Liangsheng Yin 2024-10-21 03:54:35 -07:00
  • 09603c6dc9 Maintain seq_lens_sum to make more FlashInfer operations non-blocking (#1741) Lianmin Zheng 2024-10-21 01:43:16 -07:00
  • cf470fea32 Make token mapping non-blocking in the overlapped mode (#1740) Lianmin Zheng 2024-10-20 23:25:14 -07:00
  • 45d5af2416 Add GLM-4 TextGeneration Model support for SGLang (#1736) sixgod 2024-10-21 12:08:30 +08:00
  • b121bc03a3 Simplify batch result resolution (#1735) Lianmin Zheng 2024-10-20 19:47:14 -07:00
  • e12358dc91 Simplify the usage of device (#1734) Lianmin Zheng 2024-10-20 18:17:41 -07:00
  • 554fbf93cd [Bugfix] qwen2vl forward_extend (#1727) yizhang2077 2024-10-20 17:38:35 +08:00
  • b48edff67f Split the overlapped version of TpModelWorkerClient into a separate file (#1726) Lianmin Zheng 2024-10-20 00:29:29 -07:00
  • 593b19f29d Temporarily skip this test_mixed_batch for QWen2VL (#1725) Lianmin Zheng 2024-10-20 00:05:45 -07:00
  • 59cbf47626 Unify the memory pool api and tp worker API (#1724) Lianmin Zheng 2024-10-19 23:19:26 -07:00
  • 95946271af Update README.md Ying Sheng 2024-10-19 22:29:12 -07:00
  • 5c4ce65631 Update README.md (#1722) Ying Sheng 2024-10-19 22:27:38 -07:00
  • cbbc82b7b8 Support qwen2 vl model (#1721) Yineng Zhang 2024-10-19 21:44:38 -07:00
  • 8bee20f80b Update vllm to 0.6.3 (#1711) (#1720) Yineng Zhang 2024-10-19 20:45:41 -07:00
  • 12cad0feae Simplify the interface of tp_worker (#1718) Lianmin Zheng 2024-10-19 17:39:38 -07:00
  • b6cd903604 Update readme and workflow (#1716) Lianmin Zheng 2024-10-19 12:58:55 -07:00
  • 087257ea03 Release v0.3.4 (#1714) Lianmin Zheng 2024-10-19 08:17:41 -07:00
  • 736f04025d Update README.md (#1713) Lianmin Zheng 2024-10-19 07:11:02 -07:00
  • 769bf11c05 Fix the race condition in overlap mode (#1712) Lianmin Zheng 2024-10-19 06:50:56 -07:00
  • 3db43d1b08 Fix is_all_ready for overlap copy (#1710) Lianmin Zheng 2024-10-18 21:01:52 -07:00
  • f0f8a7699b Simplify the nan detection and greedy check in sampler (#1709) Lianmin Zheng 2024-10-18 20:21:24 -07:00
  • 2bcfba1b08 Skip unnecessary penalizer (#1707) Lianmin Zheng 2024-10-18 17:54:03 -07:00
  • bc12d4033f Add grouped free operations (#1706) Lianmin Zheng 2024-10-18 13:21:05 -07:00
  • 392f2863c8 Add dtype for more operations (#1705) Lianmin Zheng 2024-10-18 12:18:15 -07:00
  • 6d0fa73ece Simplify flashinfer utilities (#1704) Lianmin Zheng 2024-10-17 22:54:14 -07:00
  • 9e0dac1ad7 Fix regex and logprob conflicts when chunked prefilling (#1703) Liangsheng Yin 2024-10-17 18:33:21 -07:00
  • a95d5589c3 Add matched_stop token or str to distinguish between eos or stop str finish_reason generation (#1684) Gleb Drozdov 2024-10-17 22:06:52 +04:00
  • d17d19e5b8 Fix mixed batch for multi modal models (#1702) Lianmin Zheng 2024-10-17 10:27:26 -07:00
  • dd3809fad8 Fix engine unit test (#1701) Lianmin Zheng 2024-10-17 09:53:32 -07:00
  • 7feba41584 Fix failed ci tests on long prompts; Better error messages for embedding models (#1700) Lianmin Zheng 2024-10-17 09:23:29 -07:00
  • 30ee36305e Fix the failed unit tests (#1699) Lianmin Zheng 2024-10-17 08:13:29 -07:00
  • e5db40dcbc ORJson. Faster Json serialization (#1694) Michael Feil 2024-10-17 08:03:08 -07:00
  • b170930534 feat: radix tree code optimize (#1697) wxsm 2024-10-17 23:01:27 +08:00
  • 5ab20cceba Use SGLang imports for linear layer (#1696) Jani Monoses 2024-10-17 17:50:01 +03:00
  • 02f7f3e488 Update the transformers version in CI (#1690) Lianmin Zheng 2024-10-16 19:03:55 -07:00
  • 2782132be8 Add date to logging messages (#1623) (#1679) Zeng Zhongchao 2024-10-17 09:54:55 +08:00
  • d19cc0b9c9 Update README.md (#1689) Lianmin Zheng 2024-10-16 18:36:24 -07:00
  • b0facb3316 add orjson for jsonresponse (#1688) Michael Feil 2024-10-16 18:14:30 -07:00
  • ecb8bad276 Returning a per request metric for number of cached_tokens read (#1599) havetc 2024-10-16 20:49:22 +02:00
  • dbec2f1847 Launch a thread to overlap CPU and GPU (#1687) Lianmin Zheng 2024-10-16 11:20:17 -07:00
  • e4b367baa8 [Event] Add online meetup meeting link (#1686) Ying Sheng 2024-10-16 10:58:14 -07:00
  • d10b933a36 Fix srt dependency (#1685) Ke Bao 2024-10-16 23:21:20 +08:00
  • 9116b2896f Add a new event loop (#1677) Lianmin Zheng 2024-10-16 01:33:20 -07:00
  • a5114b6f91 Add OLMo model (#1676) Jani Monoses 2024-10-16 10:11:18 +03:00
  • b6b4094621 Fix filter_batch function call (#1681) Liangsheng Yin 2024-10-15 22:59:26 -07:00
  • f1088e0fc8 Fix memory leak during abort (#1674) Lianmin Zheng 2024-10-15 08:15:08 -07:00
  • 175afed370 Improve benchmark scripts (#1672) Lianmin Zheng 2024-10-14 21:53:01 -07:00
  • 4a292f670d [Minor] Add some utility functions (#1671) Lianmin Zheng 2024-10-14 20:08:03 -07:00
  • cd0be7489f [doc] improve engine doc and add to readme (#1670) Byron Hsu 2024-10-14 19:56:21 -07:00
  • 56503d9bc9 [1/N] Remove CacheConfig import in all model files (#1658) Byron Hsu 2024-10-14 09:06:34 -07:00
  • 02bc95796d Simplify chunked prefill (#1667) Lianmin Zheng 2024-10-14 06:47:50 -07:00
  • 24f3e1511c [Minor] Improve style (#1666) Lianmin Zheng 2024-10-14 05:25:00 -07:00
  • 6790240cc3 Fix unit test order to balance the tasks in CI (#1665) Lianmin Zheng 2024-10-14 02:01:44 -07:00
  • 061e546313 Support double sparsity (#1459) Shuo Yang 2024-10-14 02:00:41 -07:00
  • 0c1e87964b Move filter_batch out of stream_output (#1663) Lianmin Zheng 2024-10-14 01:15:34 -07:00
  • 869f1c02c4 Add a test case to test retract (#1662) Lianmin Zheng 2024-10-13 20:32:37 -07:00
  • 2725f8da61 [Minor] Rename no_eos_trim to no_stop_trim (#1661) Ying Sheng 2024-10-13 20:30:03 -07:00
  • da1ffed689 Add output_ids into ScheduleBatch (#1659) Lianmin Zheng 2024-10-13 19:54:02 -07:00
  • 4876117171 [Fix] fix eos trim inconsistency (#1650) Ying Sheng 2024-10-13 01:07:09 -07:00
  • c3f2fc5a7a [doc] Add engine section in backend.md (#1656) Byron Hsu 2024-10-13 00:33:58 -07:00
  • 7ee6c259ff Simplify the event loop and expose --num-continuous-decode-steps as an argument (#1652) Lianmin Zheng 2024-10-12 21:35:30 -07:00
  • 9610fcd469 Fix the batch_is_full check for jump-forward decoding (#1654) Lianmin Zheng 2024-10-12 19:47:24 -07:00
  • 31fad29ab0 Add get_tokenizer function for Engine class (#1653) Patrick Yi 2024-10-12 22:39:35 -04:00
  • 9da5a60b18 Add an option to disable penalizer (#1651) Lianmin Zheng 2024-10-12 17:53:23 -07:00
  • 69aa937aa5 Fix unit tests and type annotations (#1648) Lianmin Zheng 2024-10-12 14:49:24 -07:00
  • 5d638c92f5 [Feature, Hardware] Enable SGLang on XPU GPUs via PyTorch (#1480) Zhang, Liangang 2024-10-13 02:10:32 +08:00
  • e37cdab0c6 Fix ignore_eos (#1645) Lianmin Zheng 2024-10-12 00:36:28 -07:00
  • 1d9deeacdb fix missing ignore_eos in v1/chat/completions (#1642) LI MOU 2024-10-12 12:37:20 +08:00
  • dafb6a5266 [Fix] Fix the style of test_large_max_new_tokens.py (#1638) Lianmin Zheng 2024-10-11 16:05:58 -07:00
  • 862cd265e5 [engine] support async and streaming (#1614) Byron Hsu 2024-10-11 15:26:25 -07:00
  • 00c7e6368b Release v0.3.3.post1 (#1636) Lianmin Zheng 2024-10-11 07:56:16 -07:00
  • 23cc66f7b6 Add back data parallelism (#1635) Lianmin Zheng 2024-10-11 07:22:48 -07:00
  • 5d09ca5735 Fix constrained decoding (#1634) Lianmin Zheng 2024-10-11 06:26:20 -07:00
  • 81c3327402 Added a "Back To Top" Button (#1633) Janumala Akhilendra 2024-10-11 18:55:30 +05:30
  • f13d86f920 Add image_token in conversation.py (#1632) Lianmin Zheng 2024-10-11 05:07:51 -07:00
  • aba9eae4c6 Fix the correctness test in bench_latency.py when tp > 1 and test_generation_models.py (#1631) Lianmin Zheng 2024-10-11 05:03:20 -07:00
  • bbd72bfc86 Add the ability to enable and disable the Profiler via HTTP API. (#1626) 科英 2024-10-11 17:34:25 +08:00
  • b503881bd2 [Bug] Fix the Image Input of Batch Generation (#1579) Yiding-Lu 2024-10-11 17:25:04 +08:00
  • 58093b868f Nit about the decorator of PortArgs.init_new (#1611) glen-amd 2024-10-11 02:17:47 -07:00