chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Commit Graph

  • b050d9283f fix: set ulimit -n 65535 (#647) zhyncs 2024-07-18 19:35:45 +10:00
  • 6a4dc99697 misc: rm rpyc from PACKAGE_LIST (#649) zhyncs 2024-07-18 19:35:38 +10:00
  • d774acad5c Remove the dependency of rpyc (#646) Mingyi 2024-07-18 02:13:54 -07:00
  • d93388da3e feat: add check_env (#645) zhyncs 2024-07-18 14:39:28 +10:00
  • 476584cb6e Increase the capacity of the memory pool (#643) Ying Sheng 2024-07-17 15:44:41 -07:00
  • abd5385ac5 Move global_server_args_dict (#642) Liangsheng Yin 2024-07-17 13:49:15 -07:00
  • 3de2f30a27 Flashinfer sample kernel (#617) Liangsheng Yin 2024-07-17 13:24:43 -07:00
  • 4efcc59d4f misc: add issue and pr template (#638) zhyncs 2024-07-18 04:58:11 +10:00
  • 2e341cd493 misc: add pre-commit config (#637) zhyncs 2024-07-18 04:55:39 +10:00
  • a8552cb18b feat: support internlm2 (#636) zhyncs 2024-07-17 15:40:03 +10:00
  • a470e60c97 clean up step function (#635) Ying Sheng 2024-07-16 20:15:24 -07:00
  • 5f90e0769c Update README.md Ying Sheng 2024-07-16 19:18:54 -07:00
  • 8832ecb1e4 Reduce docker size (#632) Liangsheng Yin 2024-07-16 16:12:12 -07:00
  • 5ff60eda78 Fix vertexai (#633) Liangsheng Yin 2024-07-16 16:07:19 -07:00
  • c193002297 Add support for VertexAI safety settings (#624) Aidan Cooper 2024-07-16 19:54:42 +01:00
  • fe3be1595d Add qwen2 tie word embedding (#630) ylying 2024-07-17 02:48:49 +08:00
  • 0aa189f150 Disable NCCL_NVLS by default (#631) Ying Sheng 2024-07-16 09:05:10 -07:00
  • f6b29f6920 Update docker file (#629) Ying Sheng 2024-07-16 01:12:37 -07:00
  • c9ee3d3559 Fix model forward grad (#628) Liangsheng Yin 2024-07-15 22:09:09 -07:00
  • 41d1f67704 Fix flush cache (#627) Lianmin Zheng 2024-07-15 19:56:55 -07:00
  • 56f5fc4ab5 Bump version to 0.1.21 (#626) Ying Sheng 2024-07-15 13:10:53 -07:00
  • 6a2941f4d0 Improve tensor parallel performance (#625) Ying Sheng 2024-07-15 07:10:51 -07:00
  • 5ac8b80677 Simplify mem state (#623) Mingyi 2024-07-15 02:01:09 -07:00
  • bae9541e4c Update benchmark script (#621) Ying Sheng 2024-07-14 14:38:13 -07:00
  • a56858ba67 Unify index operations (#620) Liangsheng Yin 2024-07-14 12:55:55 -07:00
  • 564a898ad9 Optimize mem indices mangement (#619) Liangsheng Yin 2024-07-13 23:39:37 -07:00
  • 5d264a90ac Bump version to 0.1.20 (#618) Lianmin Zheng 2024-07-13 17:27:55 -07:00
  • 5949b1ca0e Fix memory pool index error (#616) Ying Sheng 2024-07-13 16:45:11 -07:00
  • 0feca02dd9 Improve benchmark scripts (#615) Lianmin Zheng 2024-07-13 15:59:04 -07:00
  • 10143e1a5f Memorypool chunked prefetch (#614) Liangsheng Yin 2024-07-13 15:24:03 -07:00
  • 65c6577696 Improve benchmark scripts & fix llava (#613) Lianmin Zheng 2024-07-13 15:00:26 -07:00
  • 665815969a Enable cuda graph by default (#612) Lianmin Zheng 2024-07-13 05:29:46 -07:00
  • 396a69240f Cleanup attention backend: flashinfer and triton (#611) Lianmin Zheng 2024-07-12 18:21:11 -07:00
  • af4e7910e7 Clean up the usage of flashinfer (#610) Lianmin Zheng 2024-07-12 13:00:03 -07:00
  • 519e20cfda Code clean up: Remove deprecated prefill move InputMetadata to infer_batch.py (#609) Lianmin Zheng 2024-07-12 12:28:09 -07:00
  • d9a6902986 Fix bench latency (#607) Lianmin Zheng 2024-07-11 14:37:01 -07:00
  • ad872feb14 bump version to 0.1.19 Lianmin Zheng 2024-07-09 02:23:14 -07:00
  • da2e5d6546 Fix the default argument of OpenAI Chat completion (#605) Lianmin Zheng 2024-07-09 01:52:55 -07:00
  • ce62dc73f0 Update model_support.md Lianmin Zheng 2024-07-09 01:32:46 -07:00
  • 02b7258658 [Feat] Expose logprob options to sgl.gen API (#503) 胡译文 2024-07-09 15:35:39 +08:00
  • d557e9f3b7 Update chat template for qwen and yi-1.5. (#530) prophe 2024-07-09 14:55:44 +08:00
  • 740c46a152 Add Qwen2 MoE support (#603) Tommy Yang 2024-07-09 14:44:59 +08:00
  • b38687226a Make sglang compat with vllm 0.5.1 (#598) Tommy Yang 2024-07-09 14:44:22 +08:00
  • 710f614ebe add minicpm support (#602) Pan Lyu 2024-07-09 14:27:04 +08:00
  • f25b76c02a add LogitsMetadata (#604) Liangsheng Yin 2024-07-08 17:46:55 -07:00
  • f4e885b7c3 Reduce number of workspaces (#601) Mingyi 2024-07-07 19:35:22 -07:00
  • 0877f1e75b Fix streaming (#600) Liangsheng Yin 2024-07-07 01:55:58 -07:00
  • 5304b4ef58 Add --enable-p2p-check option (#599) Liangsheng Yin 2024-07-06 23:34:10 -07:00
  • 26908d9568 * fix(detokenizer_manager.py): fix truncated decoded output (#586) Pan Lyu 2024-07-07 05:53:22 +08:00
  • c0982ac553 Fix Llava model (#594) Mingyi 2024-07-06 00:58:46 -07:00
  • dc1b8bcfaa Format (#593) Ying Sheng 2024-07-05 10:06:17 -07:00
  • 5a57b8addd Add Gemma2 (#592) Ying Sheng 2024-07-05 09:48:54 -07:00
  • d737da5f17 Update README.md Lianmin Zheng 2024-07-04 00:55:40 -07:00
  • ac11388756 Add docker file (#588) Ying Sheng 2024-07-04 00:53:49 -07:00
  • dc8cef1d0c Update README.md Lianmin Zheng 2024-07-04 00:05:40 -07:00
  • 2f11936f95 bump version to 0.1.18 Ying Sheng 2024-07-04 06:27:29 +00:00
  • 63fbef9876 fix flashinfer & http log level Lianmin Zheng 2024-07-03 23:19:33 -07:00
  • 2a754e57b0 2x performance improvement for large prefill & Fix workspace conflicts (#579) Ying Sheng 2024-07-03 16:14:57 -07:00
  • 96c503eb60 fix the broken server args (#585) Liangsheng Yin 2024-07-04 07:01:19 +08:00
  • 441cca773d support gptj style rope in llama Chen Xuechen Li 2024-07-03 12:23:30 -07:00
  • c7709d3abe Update install commands (#583) Lianmin Zheng 2024-07-03 02:07:34 -07:00
  • 9380f50ff9 Turn on flashinfer by default (#578) Ying Sheng 2024-07-02 02:25:07 -07:00
  • 95dc093b19 [BugFix] gemma loading weights "lm_head.weight" key error (#577) Daniel Hernandez Garcia 2024-07-02 06:10:07 +01:00
  • d9ac639202 Fix flashinfer version (#576) Yueyang Pan 2024-07-02 07:08:39 +02:00
  • 26294b2f3d Update README.md Lianmin Zheng 2024-07-01 09:54:08 -07:00
  • 75b31a2a88 Update run_batch interface and max_prefill_tokens (#574) Ying Sheng 2024-06-30 18:26:04 -07:00
  • 11616fc6bd Minor fix in compiler & format (#545) sglang 2024-06-29 23:42:14 -07:00
  • 9ce89bc14b Update benchmark script (#571) Ying Sheng 2024-06-28 00:44:22 -07:00
  • badf3fa020 Expose dtype argument (#569) Lianmin Zheng 2024-06-27 23:30:39 -07:00
  • 945aa9beb2 Update readme (#568) Lianmin Zheng 2024-06-27 11:37:49 -07:00
  • 2e6e62e156 Increase the number of thread limitation for tp worker managers. (#567) Lianmin Zheng 2024-06-26 09:33:45 -07:00
  • a385ee27bd Warmup cublas (#566) Lianmin Zheng 2024-06-25 12:46:00 -07:00
  • eb1ae6ae0c Add sglang.bench_latency for offline benchmark (#564) Lianmin Zheng 2024-06-25 03:38:04 -07:00
  • 2187f36237 Add a new arguments log_level_http to control the HTTP logging (#563) Lianmin Zheng 2024-06-25 01:16:20 -07:00
  • 9465b668b9 Allow running with vllm==0.4.3 (#561) Lianmin Zheng 2024-06-24 15:24:21 -07:00
  • 05471f2103 Update test_flashinfer (#560) Liangsheng Yin 2024-06-24 15:23:57 +08:00
  • 1fa15099d8 Add LlamaForClassification (#559) Lianmin Zheng 2024-06-22 00:45:33 -07:00
  • 303ef8883e Clean up logits processor (#558) Lianmin Zheng 2024-06-22 00:25:24 -07:00
  • 92cb93f390 Fix latency benchmark (#557) Liangsheng Yin 2024-06-22 15:11:04 +08:00
  • e94e60d6fb make flashinfer workspace larger Lianmin Zheng 2024-06-21 17:32:36 -07:00
  • d2f8bfb2e1 Follow-up fixes for flashinfer 0.0.5 (#556) Lianmin Zheng 2024-06-20 23:19:52 -07:00
  • b7e2f800ac Update flashinfer to 0.0.5 (#554) Lianmin Zheng 2024-06-20 20:29:06 -07:00
  • 09593e9bc9 Multi-node Tensor Parallelism (#550) Ying Sheng 2024-06-17 20:41:24 -07:00
  • 53a7ebd89a Update fused_moe (#553) Lianmin Zheng 2024-06-17 09:47:58 -07:00
  • ad5f04d6ce Fix the Jump-Forward with Chinese (#551) Liangsheng Yin 2024-06-16 21:45:04 +08:00
  • bbec01c9aa Fix tp worker only checking req[0] for stream (#546) Qubitium-modelcloud 2024-06-15 13:56:10 +08:00
  • 40e53d65cb Add disk cache for loading ShareGPT dataset. (#542) Liangsheng Yin 2024-06-13 16:37:12 +08:00
  • fb9296f0ed Higher priority for user input of max_prefill_tokens & format (#540) Ying Sheng 2024-06-12 21:48:40 -07:00
  • 1374334d38 Fix dependency & crash issues (#539) Ying Sheng 2024-06-12 21:23:19 -07:00
  • 94aead9e8d Fix dependency (#538) Lianmin Zheng 2024-06-12 13:17:35 -07:00
  • 9c902b1954 Decode Incrementally (#517) Liangsheng Yin 2024-06-12 14:39:12 +08:00
  • 111991fe23 Fix Regression: Disable p2p for 4090 (#531) ZhouXingg 2024-06-12 14:27:17 +08:00
  • a8c787d2b3 Add ChatGLM Model Support (#516) Qubitium 2024-06-12 07:39:52 +08:00
  • 5f283991e9 [Minor] Correct Optional type hints in api (#526) Fabian Preiß 2024-06-12 01:37:27 +02:00
  • b6667a53b9 Fix RAG nb, parea setup (parea -> parea-ai) (#525) Fabian Preiß 2024-06-12 01:36:43 +02:00
  • 542bc733d6 Fix missing numpy dependency in pyproject.toml (#524) Fabian Preiß 2024-06-10 21:13:50 +02:00
  • f6dbd24043 Improve doc strings (#518) Lianmin Zheng 2024-06-08 02:06:52 -07:00
  • e8a2327d52 Update version to 0.1.17 (#515) Lianmin Zheng 2024-06-07 19:49:18 -07:00
  • 91f93f141f Crash the server when error or OOM happens (#514) Lianmin Zheng 2024-06-07 19:22:34 -07:00
  • f70f72586a Fix rid state map leak + Refractor .finished (#505) Qubitium 2024-06-08 04:20:40 +08:00