chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Commit Graph

  • 8275049ce3 Add device support (#1607) Zhang, Liangang 2024-10-11 17:05:58 +08:00
  • 5476ccad8f Update README.md Lianmin Zheng 2024-10-11 01:59:49 -07:00
  • b040ed71f7 Update README.md (#1629) Lianmin Zheng 2024-10-11 01:58:25 -07:00
  • c9e6658699 Update README.md (#1625) Kushal Agrawal 2024-10-11 14:27:42 +05:30
  • e11ab79e68 [Performance, hardware] MoE tuning update to AMD MI300x GPUs (#1619) HAI 2024-10-10 22:48:15 -07:00
  • 01fdb2f377 Fix test_vision_openai_server on CI (#1620) Byron Hsu 2024-10-10 16:34:13 -07:00
  • c996e8ccd4 [Minor] Fix logging typo (#1615) Amos You 2024-10-08 21:11:19 -07:00
  • 7b69d91b4f Release v0.3.3 (#1605) Lianmin Zheng 2024-10-08 12:58:41 -07:00
  • e8613df071 [Engine] Fix generate hanging issue after the first call (#1606) Byron Hsu 2024-10-07 21:26:56 -07:00
  • c5325aba75 [Profile] Add pytorch profiler (#1604) Ying Sheng 2024-10-07 14:37:16 -07:00
  • ebbc42d989 Optimize broadcast & Reorg code (#1598) Lianmin Zheng 2024-10-07 13:05:53 -07:00
  • 3ff641132e Remove references to squeezellm (#1603) Jani Monoses 2024-10-07 21:30:41 +03:00
  • 2b302b9393 Fix the port_args in bench_latency (#1597) Lianmin Zheng 2024-10-07 00:44:38 -07:00
  • 68f8b60d22 Fix chunked prefill condition (#1594) Ke Bao 2024-10-07 14:34:14 +08:00
  • 6a5b352aaf Use is_flashinfer_available to replace is_hip for flashinfer check (#1596) Lianmin Zheng 2024-10-06 22:54:05 -07:00
  • 565b05f02f Use atexit hook to implicitly shutdown Runtime (#1595) Byron Hsu 2024-10-06 22:18:45 -07:00
  • b6aad70ab1 [Fix] Fix the case where prompt_len = 0 (#1593) Lianmin Zheng 2024-10-06 20:30:02 -07:00
  • 551a3a9d38 Provide an offline engine API (#1567) Byron Hsu 2024-10-06 20:27:03 -07:00
  • 91877a9f9c Fix modality for image inputs (#1592) Lianmin Zheng 2024-10-06 15:43:32 -07:00
  • f7cce751f9 Update README.md (#1591) Lianmin Zheng 2024-10-06 15:14:29 -07:00
  • 17e998f1a8 Test consistency for single and batch seperately (#1590) Byron Hsu 2024-10-06 15:02:27 -07:00
  • c98e84c21e [Minor, Performance] Use torch.argmax for greedy sampling (#1589) Ying Sheng 2024-10-06 13:15:05 -07:00
  • 9c064bf78a [LoRA, Performance] Speedup multi-LoRA serving - Step 1 (#1587) Ying Sheng 2024-10-06 10:33:44 -07:00
  • 58d1082e39 Clean up event loop (#1586) Lianmin Zheng 2024-10-06 03:24:04 -07:00
  • 4d086719e5 [Bug] Fix decode stats error on output_len 1 (#1585) HAI 2024-10-06 01:09:09 -07:00
  • 9244f27f0a [Minor] Improve the style and fix flaky tests (#1584) Lianmin Zheng 2024-10-06 00:10:48 -07:00
  • 2422de5193 Support min_tokens in sgl.gen (#1573) Byron Hsu 2024-10-05 21:51:12 -07:00
  • 521f862d90 Fix runtime.generate when sampling param is not passed (#1582) Byron Hsu 2024-10-05 17:59:05 -07:00
  • 34c32d2820 Fix styling (#1583) Byron Hsu 2024-10-05 17:52:14 -07:00
  • dde8bb16fe default sampling param should be deepcopied (#1581) Byron Hsu 2024-10-05 17:27:43 -07:00
  • 8ac3ccc060 Backend method not found when SRT Runtime is used (#1576) Byron Hsu 2024-10-05 11:47:35 -07:00
  • 9b0926ceeb Add llama implementation with no tensor parallel linears (#1561) Jerry Zhang 2024-10-05 11:22:27 -07:00
  • 1c1bdc7699 [Event] Update README.md (#1572) Ying Sheng 2024-10-05 11:16:47 -07:00
  • 6bfdb4031d [Easy] use .text() instead of .text (#1577) Byron Hsu 2024-10-05 11:07:41 -07:00
  • f8fb4ce9b0 chore: update README.md (#1580) Ikko Eltociear Ashimine 2024-10-06 03:05:57 +09:00
  • 5d0ba4038f Refine the add request reasons to avoid corner cases. (#1574) Liangsheng Yin 2024-10-04 18:00:18 -07:00
  • 04b262cd91 [Fix] Fix major performance bug in certain cases (#1563) Ying Sheng 2024-10-04 01:51:11 -07:00
  • 2432ad40c6 [Minifix] Remove extra space in cot example (#1569) FredericOdermatt 2024-10-04 10:16:53 +02:00
  • 45473d4b2b Make input_ids a torch.Tensor (#1568) Lianmin Zheng 2024-10-04 01:09:59 -07:00
  • 114bbc8651 Use ipc instead of tcp in zmq (#1566) Lianmin Zheng 2024-10-04 00:45:52 -07:00
  • 32eb6e96f2 Organize sampling batch info better (#1562) Lianmin Zheng 2024-10-03 18:29:49 -07:00
  • e0b5dbcec1 [FP8 KV Cache] Avoid KeyError at loading pre-quantized FP8 model with kv_scale (#1559) HAI 2024-10-03 01:52:26 -07:00
  • e6852b0dd2 [Fix] Fix AttributeError in Qwen2.5 LoRA: 'Qwen2ForCausalLM' object has no attribute 'get_hidden_dim' (#1536) Minsang Song 2024-10-03 12:41:15 +09:00
  • 4ae0969c0a Move status check in the memory pool to CPU (#1557) Lianmin Zheng 2024-10-02 18:23:35 -07:00
  • 317631cada [Fix] Move ScheduleBatch out of SamplingInfo (#1556) Lianmin Zheng 2024-10-02 17:18:04 -07:00
  • b564835364 [Fix] do not maintain regex_fsm in SamplingBatchInfo (#1555) Lianmin Zheng 2024-10-02 13:19:44 -07:00
  • 2c7d0a5b8b [Fix] Fix all the Huggingface paths (#1553) Theresa Barton 2024-10-02 10:12:07 -07:00
  • 8cdc76f6d4 [Performance, Hardware] MoE tuning on AMD MI300x GPUs (#1554) kk 2024-10-03 00:52:46 +08:00
  • f202ed9712 [Refactor] Simplify io_struct and tokenizer_manager (#1549) Ying Sheng 2024-10-01 10:25:32 -07:00
  • 100f5b8bc9 Simplify flashinfer dispatch (#1552) Liangsheng Yin 2024-10-01 00:28:42 -07:00
  • 619bb6ddda Dispatch flashinfer wrappers (#1550) Liangsheng Yin 2024-09-30 23:12:36 -07:00
  • b88ea90d4a Fix bugs of logprobs_nums (#1548) Liangsheng Yin 2024-09-30 17:09:54 -07:00
  • 99ec439da4 Organize Attention Backends (#1547) Liangsheng Yin 2024-09-30 15:54:18 -07:00
  • 0f4fb19bc8 [Fix, LoRA] fix LoRA with updates in main (#1545) Ying Sheng 2024-09-30 10:06:08 -07:00
  • 63ba2f8d7b Clean up batch data structures: Introducing ModelWorkerBatch (#1544) Lianmin Zheng 2024-09-30 06:41:49 -07:00
  • 36d5acfca5 Rename InputMetadata -> ForwardBatch (#1543) Lianmin Zheng 2024-09-30 02:41:11 -07:00
  • 3f0fe08d37 Let ModelRunner take InputMetadata as input, instead of ScheduleBatch (#1541) Lianmin Zheng 2024-09-29 20:28:45 -07:00
  • 55b974f96f Process image in parallel (#1539) Liangsheng Yin 2024-09-29 18:52:43 -07:00
  • f86c1e611f Move scheduler code from tp_worker.py to scheduler.py (#1538) Lianmin Zheng 2024-09-29 17:42:45 -07:00
  • acaffd233f [Fix] fix ipv6 url when warm up model (#1537) Xinyu Yang 2024-09-30 02:02:40 +08:00
  • 048685430d Improve process creation (#1534) Lianmin Zheng 2024-09-29 02:36:12 -07:00
  • fd9ad817ec Organize image inputs (#1531) Liangsheng Yin 2024-09-28 23:28:55 -07:00
  • e165a9fc1b Make detokenizer_manager.py not asyncio (#1532) Lianmin Zheng 2024-09-28 19:33:09 -07:00
  • 4e4459b91f Multiple minor fixes (#1530) Lianmin Zheng 2024-09-28 14:43:35 -07:00
  • 065bb94753 Fix RuntimeEndpoint.select method (#1495) Jeffrey Fong 2024-09-29 05:04:06 +08:00
  • f42e9bfb52 [bugfix] Add modelscope package to avoid docker image without modelscope (#1520) Kylin 2024-09-29 03:43:22 +08:00
  • 840c5dbcb3 [FIX] Catch syntax error of Regex Guide to avoid crash (#1521) Ninglin Du 2024-09-29 03:42:06 +08:00
  • 63e845d0bb Add float8 dynamic quant to torchao_utils (#1528) Jerry Zhang 2024-09-28 12:27:54 -07:00
  • 9aa6553d2a [Feature] Support reward model LxzGordon/URM-LLaMa-3.1-8B (#1525) Ying Sheng 2024-09-27 23:32:11 -07:00
  • b1e330bcb0 [Event] Update meeting link (#1529) Ying Sheng 2024-09-27 13:30:04 -07:00
  • 4353acb469 minor: fix config (#1524) Liangsheng Yin 2024-09-27 01:49:16 -07:00
  • 9ae1db0bdc [Fix] Ignore import error (#1513) Lianmin Zheng 2024-09-25 11:32:21 -07:00
  • 37c5899fc2 Release v0.3.2 (#1512) Ying Sheng 2024-09-24 23:17:09 -07:00
  • f39a0197fd Revert "kernel: use tensor cores for flashinfer gqa kernels" (#1511) Ying Sheng 2024-09-24 22:50:31 -07:00
  • 3c93187caf Add support for tie_word_embeddings when loading weights + support for SmolLM (#1508) TianyiQ 2024-09-24 21:50:20 -07:00
  • fb2d0680e0 [Fix] Fix clean_up_tokenization_spaces in tokenizer (#1510) Lianmin Zheng 2024-09-24 21:37:33 -07:00
  • 067d8e16fc Simplify bench_latency.py (#1503) Lianmin Zheng 2024-09-24 17:42:07 -07:00
  • e6692bf4a5 debug radixcache stack_overflow (#1499) luzengxiangcn 2024-09-24 19:58:01 +08:00
  • 28b4d8e144 Update test_srt_backend.py (#1502) Lianmin Zheng 2024-09-24 03:17:10 -07:00
  • bc068e9618 [CI] Move AMD test to a separate file (#1500) Lianmin Zheng 2024-09-24 02:06:28 -07:00
  • 8d4ed42ad5 MoE torch compile (#1497) Ke Bao 2024-09-24 16:46:59 +08:00
  • 2854a5ea9f Fix the overhead due to penalizer in bench_latency (#1496) Lianmin Zheng 2024-09-23 07:38:14 -07:00
  • 42a2d82ba7 minor: add mla fp8 test (#1494) Yineng Zhang 2024-09-23 20:40:17 +08:00
  • e4780cf839 [API, Feature] Support response prefill for openai API (#1490) Ying Sheng 2024-09-22 06:46:17 -07:00
  • 39bb49d156 Update dockerfile to include datamodel_code_generator (#1492) Lianmin Zheng 2024-09-22 04:49:16 -07:00
  • 6f3cf1297e [CI, AMD] Add AMD tests to CI (#1491) Ying Sheng 2024-09-22 04:45:10 -07:00
  • 13f1357ef0 Add a unit test for data parallelism (#1489) Lianmin Zheng 2024-09-22 02:21:05 -07:00
  • 2a99993cd9 Pr fix max workers (#1456) wellhowtosay 2024-09-22 17:20:26 +08:00
  • 167591e864 Better unit tests for adding a new model (#1488) Lianmin Zheng 2024-09-22 01:50:37 -07:00
  • 441c22db8c doc: update backend (#1486) Yineng Zhang 2024-09-21 22:05:12 +08:00
  • ce636ac441 fix incorrect links in documentation (#1481) Ran Chen 2024-09-21 05:36:23 -07:00
  • 82136eb0b5 chore: bump v0.3.1.post3 (#1483) Yineng Zhang 2024-09-21 11:17:45 +08:00
  • b8ccaf4d73 Add MLA gsm8k eval (#1484) Ke Bao 2024-09-21 11:16:13 +08:00
  • a68cb201dd Fix triton head num (#1482) Ke Bao 2024-09-21 10:25:20 +08:00
  • 014982b5e0 Add OLMoE (#1476) Niklas Muennighoff 2024-09-19 19:32:49 -07:00
  • a6db88626e minor: add quant eval compared with base (#1475) Yineng Zhang 2024-09-20 01:57:19 +08:00
  • b4408b0d16 feat: update linear deps 1/N (#1305) Yineng Zhang 2024-09-19 20:53:11 +08:00
  • 2cd7e181dd Fix env vars in bench_latency (#1472) Lianmin Zheng 2024-09-19 03:19:26 -07:00
  • 5ce55aee15 Release v0.3.1.post2 (#1470) Lianmin Zheng 2024-09-19 02:03:38 -07:00
  • 2d346a57c2 Fix padding in the cuda graph (#1469) Lianmin Zheng 2024-09-19 01:52:15 -07:00