chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Commit Graph

  • 44f011d224 Super tiny typo fix (#2564) fzyzcjy 2024-12-27 00:28:01 +08:00
  • ed91e003bb [UTILS] improve makefile a bit by adding help info (#2570) kzhou003 2024-12-26 08:24:18 -08:00
  • 531d6ea968 fix: package data missing (#2521) yudian0504 2024-12-27 00:16:48 +08:00
  • dc3bee4815 Fix test and benchmark scripts (#2598) Lianmin Zheng 2024-12-26 07:56:26 -08:00
  • a74d194146 [unittest] add unit test to test quant args of srt engine (#2574) Zhizhou Sha 2024-12-26 06:54:43 -08:00
  • 3169e66c23 Fix duplicated handling of GetWeightsByNameReqInput (#2565) fzyzcjy 2024-12-26 22:49:32 +08:00
  • 773951548d Fix logprob_start_len for multi modal models (#2597) Lianmin Zheng 2024-12-26 06:27:45 -08:00
  • 637de9e8ce update readme of DeepSeek V3 (#2596) fsygd 2024-12-26 21:31:56 +08:00
  • acb340728c [Feature] Support new parameter - EBNF in xgrammar (#2526) Adarsh Shirawalmath 2024-12-26 18:42:41 +05:30
  • 08effbff35 Error occurs when loading the gemma model in bitsandbytes format. (#2557) Sangchun Ha (Patrick) 2024-12-26 22:10:37 +09:00
  • 60bd32723a Update README.md (#2594) Lianmin Zheng 2024-12-26 03:31:50 -08:00
  • e7ebecf82e Fix cache hit rate when chunked prefill (#2555) Liangsheng Yin 2024-12-26 03:14:28 -08:00
  • 9a23c48456 h100 tuning fused_moe_triton for qwen2 moe (#2560) Xiaoyu Zhang 2024-12-26 19:13:31 +08:00
  • 635a042623 docs: update deepseek v3 example (#2592) Yineng Zhang 2024-12-26 17:43:37 +08:00
  • 2dccecf432 fix: only enable moe_align_block_size for now (#2590) Yineng Zhang 2024-12-26 16:56:59 +08:00
  • 75ad0a143f docs: add deepseek v3 launch instructions (#2589) Yineng Zhang 2024-12-26 15:26:54 +08:00
  • efc52f85e2 chore: bump v0.4.1 (#2582) Yineng Zhang 2024-12-26 07:14:51 +08:00
  • 60e2fdcf4f use sgl-kernel moe_align_block_size (#2581) Yineng Zhang 2024-12-26 06:29:08 +08:00
  • d7c0e872b0 chore: bump 0.0.2.post8 for sgl-kernel (#2580) Yineng Zhang 2024-12-26 06:11:39 +08:00
  • 31548116a8 fix moe_align_block_size_kernel for shared memory issue (#2579) Yineng Zhang 2024-12-26 05:31:04 +08:00
  • 53aed988cb Refactor MoE (#2575) HandH1998 2024-12-26 00:02:14 +08:00
  • 8a56b43175 [Bench] Flush cache before benchmarking (#2566) Ying Sheng 2024-12-23 19:21:21 -08:00
  • e835a50021 Reorg moe code (#2563) Ke Bao 2024-12-24 01:10:22 +08:00
  • 23e5e50fd5 Fix gemlite import (#2553) Lianmin Zheng 2024-12-22 20:21:17 -08:00
  • 25e5d589e3 Doc: Update Grammar Backend (#2545) Shi Shuai 2024-12-23 01:14:40 +00:00
  • 41b1db69b8 A better aio rwlock that guarantees the order (#2547) Lianmin Zheng 2024-12-22 15:44:32 -08:00
  • 8496701934 [Misc] Fix metrics, weight update lock, request logging (#2543) Lianmin Zheng 2024-12-22 06:25:57 -08:00
  • 7d672d277b [kernel optimize] benchmark write_req_to_token_pool_triton and optimize kernel (#2509) Xiaoyu Zhang 2024-12-22 18:31:02 +08:00
  • d4b174817d docs: update sponsorship (DataCrunch) (#2523) Yineng Zhang 2024-12-22 18:29:04 +08:00
  • 19ba2b0ea9 Add lora_paths to v1_chat_generate_request (#2529) Lei 2024-12-22 02:23:33 -08:00
  • 4e1e3cff20 fix #2528 (#2541) Yineng Zhang 2024-12-22 00:14:41 +08:00
  • 8f4d04e540 chore: bump v0.4.0.post2 (#2525) Yineng Zhang 2024-12-21 21:16:34 +08:00
  • feb2b768ba Add integration with gemlite weight only quant (#2528) Jerry Zhang 2024-12-20 08:25:25 -08:00
  • d95a5f5bf5 fix followup #2517 (#2524) Yineng Zhang 2024-12-19 23:24:30 +08:00
  • 4b83db24f1 fix: continue to use flashinfer 0.1.6 temporarily (#2517) Yineng Zhang 2024-12-19 14:03:24 +08:00
  • 64456cf023 docs: update README (#2516) Yineng Zhang 2024-12-19 13:44:02 +08:00
  • bb4a922023 feat: add llama3 eval (#2515) Yineng Zhang 2024-12-19 13:37:09 +08:00
  • 21e9e63ad5 Print progress bar during cuda graph capture (#2502) Lianmin Zheng 2024-12-17 06:20:44 -08:00
  • 1fc84cf60b Update readme (#2500) Lianmin Zheng 2024-12-17 04:33:36 -08:00
  • 361ea8d912 Fix openai protocols and pass top_k, min_p (#2499) Lianmin Zheng 2024-12-17 04:14:14 -08:00
  • 33c5ff2845 Add lora_path to chat completion (#2438) Lei 2024-12-17 03:47:49 -08:00
  • 5ce9daea59 ROCm support for sglang.check_env (#2426) Hui Liu 2024-12-17 03:45:14 -08:00
  • ce094a5d79 Clean up GPU memory after killing sglang processes (#2457) Ata Fatahi 2024-12-17 06:42:40 -05:00
  • e21026690d benchmark decoding attention kernel with cudnn (#2467) bjmsong 2024-12-17 19:31:57 +08:00
  • bd6196163e Small fix for the order of apply_torchao_config (#2495) Lianmin Zheng 2024-12-16 19:21:11 -08:00
  • 56198b45d9 Add a benchmark script for in-batch prefix caching (#2494) Lianmin Zheng 2024-12-16 18:49:02 -08:00
  • ba36b5520a Revert "Small fixes for torchao quant" (#2493) Lianmin Zheng 2024-12-16 15:04:16 -08:00
  • 9cd9dc83b3 Temporarily disable unit test of torch native attention backend (#2492) Lianmin Zheng 2024-12-16 14:17:27 -08:00
  • 7a1aecb938 Simplify pytorch sampling kernel and logit processor (#2491) Lianmin Zheng 2024-12-16 14:11:09 -08:00
  • 82699474fd Small fixes for torchao quant (#2476) Jerry Zhang 2024-12-16 14:08:12 -08:00
  • 7154b4b1df minor: update flashinfer nightly (#2490) Yineng Zhang 2024-12-16 23:02:49 +08:00
  • b532a5fd16 fix moe-ep accuracy issue for fp8 (#2489) xiaobochen 2024-12-16 20:54:02 +08:00
  • a0592c059f [Benchmark] add a benchmark for hf/vllm/sglang rmsnorm (#2486) Xiaoyu Zhang 2024-12-15 13:52:08 +08:00
  • e8dbdf75bc fix typo (#2487) Yineng Zhang 2024-12-15 13:44:55 +08:00
  • e04d3f2897 adapt tensorrt llm custom all reduce to sgl-kernel (#2481) yizhang2077 2024-12-15 13:15:59 +08:00
  • 5f2595be43 hotfix: checking for HIP (#2485) Yineng Zhang 2024-12-15 02:47:26 +08:00
  • 0ba2c58947 Remove cuda graph batch size adjustment for dp attention (#2484) Ke Bao 2024-12-14 23:53:54 +08:00
  • fccbfa3752 format: add clang-format for sgl-kernel (#2483) Yineng Zhang 2024-12-14 22:36:04 +08:00
  • 2f9bd0fafd Fix correctness issue for triton decoding kernel (#2479) Ke Bao 2024-12-14 16:50:54 +08:00
  • 5282a4735f [Minor] Fix grok model loader (#2473) Lianmin Zheng 2024-12-12 14:34:47 -08:00
  • f0ed9c353e feat: support dev image (#2469) Yineng Zhang 2024-12-13 02:23:52 +08:00
  • e3b3acfa6f Rename rust folder to sgl-router (#2464) Ata Fatahi 2024-12-12 12:40:41 -05:00
  • 2673fa29d4 fix: set runtime path (#2466) Yineng Zhang 2024-12-12 18:05:48 +08:00
  • dedaf8cd48 minor: update pypi tag (#2463) Yineng Zhang 2024-12-12 15:21:45 +08:00
  • 32ed016041 chore: bump v0.0.2 for sgl-kernel (#2462) Yineng Zhang 2024-12-12 14:58:05 +08:00
  • 6efa9e4a6d Bump sglang-router to 0.1.1 (#2459) Ata Fatahi 2024-12-11 20:40:03 -05:00
  • 7791fd9948 Include version info into the router package (#2456) Ata Fatahi 2024-12-11 20:31:20 -05:00
  • 2ac36b9a7b Make request payload size configurable (#2444) Ata Fatahi 2024-12-11 19:55:21 -05:00
  • 2d60a5ee75 Update v0.1.0.md Byron Hsu 2024-12-11 13:48:18 -08:00
  • 2e4a5907c9 [router] Release router 0.1.0 with dynamic scaling and fault tolerance (#2455) Byron Hsu 2024-12-11 13:42:35 -08:00
  • c0ee46fe10 [router] Update doc for dynamic scaling and fault tolerance (#2454) Byron Hsu 2024-12-11 13:11:42 -08:00
  • 9208618b3e [Core] in batch prefix caching by delay scheduling (#2442) SangBin Cho 2024-12-11 12:51:50 -08:00
  • 864bf2ba00 [router] remove main.rs because only lib.rs is used for py binding (#2453) Byron Hsu 2024-12-11 12:13:19 -08:00
  • a4cca7fc53 [router] Add retries based fault tolerance (#2452) Byron Hsu 2024-12-11 12:13:08 -08:00
  • 993956c6b1 Add support for IBM Granite 3.x models (#2437) Fred Reiss 2024-12-11 06:30:23 -08:00
  • f8548295d6 Fix warmup in bench_offline_throughput.py (#2449) Lianmin Zheng 2024-12-11 06:16:01 -08:00
  • 959735fc9e Fix model loader for more quantization formats (#2448) Lianmin Zheng 2024-12-11 05:21:23 -08:00
  • f67723940d decoding attention kernel benchmark (#2425) bjmsong 2024-12-11 20:46:59 +08:00
  • 626a99ac13 chore: update ao v0.7.0 (#2447) Yineng Zhang 2024-12-11 20:44:28 +08:00
  • ece724910a Make torch TP composable with torchao (#2436) Ke Wen 2024-12-11 04:21:42 -08:00
  • 0fb88aaa77 [router] Use borrow if possible to save cost (#2441) Byron Hsu 2024-12-11 01:38:50 -08:00
  • d4de9a6235 [router] Refactor: decouple select and send stage (#2440) Byron Hsu 2024-12-11 00:51:21 -08:00
  • 7310aede97 fix: compatible with PEP 440 (#2435) Yineng Zhang 2024-12-11 06:48:45 +08:00
  • 5de9a58eca fix: use manylinux2014_x86_64 tag (#2434) Yineng Zhang 2024-12-11 06:17:41 +08:00
  • 56fcd8e8a5 feat: support sgl-kernel PyPI (#2433) Yineng Zhang 2024-12-11 06:06:19 +08:00
  • 2b340adfb1 Typo fix in router.md (#2424) Adarsh Shirawalmath 2024-12-10 11:19:40 +05:30
  • 8586b72da0 [feat] Enable chunked prefill for llava-onevision (#2412) Ying Sheng 2024-12-09 09:52:38 -08:00
  • 641b7d0ae0 [Minor] Improve code style (#2422) Lianmin Zheng 2024-12-09 06:30:35 -08:00
  • 0ce091a82d [Minor] Improve code style (#2419) Lianmin Zheng 2024-12-09 03:05:59 -08:00
  • 835f8afc77 Migrate llama_classification to use the /classify interface (#2417) Lianmin Zheng 2024-12-08 23:30:51 -08:00
  • 3844feb9bb Add a unittest for fused_moe (#2416) Xiaoyu Zhang 2024-12-09 14:46:10 +08:00
  • 27f7bed7a7 reduce watchdog interval to 5s (#2410) Byron Hsu 2024-12-08 21:17:31 -08:00
  • 6387098f5f [router] add health checking in router init (#2393) Byron Hsu 2024-12-08 17:17:37 -08:00
  • 2a717c5078 [Router] fix interrupt from terminal (#2413) Byron Hsu 2024-12-08 16:58:41 -08:00
  • a1e697b25b [router] Improve cleanup logic (#2411) Byron Hsu 2024-12-08 15:24:02 -08:00
  • a6ca736c8e Simplify stream_output (#2398) Lianmin Zheng 2024-12-08 12:27:13 -08:00
  • f62055b528 minor: add random flashinfer vs triton use case (#2409) Yineng Zhang 2024-12-09 04:15:21 +08:00
  • 74bc9184c3 minor: add random use case (#2408) Yineng Zhang 2024-12-09 03:21:35 +08:00
  • 0f8eb15323 feat: support custom task runner (#2407) Yineng Zhang 2024-12-09 02:29:55 +08:00
  • 67470bbb28 minor: update correct measurement unit (#2406) Yineng Zhang 2024-12-08 20:55:04 +08:00