chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Commit Graph

  • fa3c9e0668 Fix popen_launch_server wait for 20 minutes when child process exits (#4777) fzyzcjy 2025-03-26 15:32:19 +08:00
  • 0d658ac3df Support recording experts workload in QWen2-MoE (#4775) Cheng Wan 2025-03-26 03:29:57 -04:00
  • ced35a0649 fix(typo): fix reply to replay in base_attn_backend.py (#4784) Thysrael 2025-03-26 15:19:12 +08:00
  • 26f07294f1 Warn users when release_memory_occupation is called without memory saver enabled (#4566) fzyzcjy 2025-03-26 15:18:14 +08:00
  • 34e07a65f1 [Fix] Fix unexpected idx bug of Phi-3-small (#4728) Baizhou Zhang 2025-03-25 22:33:48 -06:00
  • 15ddd84322 Add retry for flaky tests in CI (#4755) fzyzcjy 2025-03-26 07:53:12 +08:00
  • 52029bd1e3 Fix warmup error when dp=1 (#4753) fzyzcjy 2025-03-25 17:01:21 +08:00
  • eb934bdf3b Fix test_expert_distribution failure (#4752) fzyzcjy 2025-03-25 16:17:03 +08:00
  • e45ae444db Revert "Add DeepEP tests into CI (#4737)" (#4751) fzyzcjy 2025-03-25 15:44:01 +08:00
  • ac3fae8445 [Feature] Support "strict" in function calling (#4310) DarkSharpness 2025-03-25 14:15:25 +09:00
  • 2d1b83e57a add dsv3 int8 test (#4705) HandH1998 2025-03-25 12:57:58 +08:00
  • 199bb01d00 Add endpoints to dump selected expert ids (#4435) yuhsaun-t 2025-03-24 21:34:19 -07:00
  • 6b7038babd Speedup warmup when DP > 1 (#4695) fzyzcjy 2025-03-25 12:08:05 +08:00
  • 57eec0bfbc fix FlashMLA cudagraph config (#4691) lukec 2025-03-25 12:06:58 +08:00
  • f01b092519 Super tiny fix typo (#4738) fzyzcjy 2025-03-25 12:05:45 +08:00
  • 14269198e3 [Benchmark] tilelang vs deepgemm vs w8a8_block_fp8_matmul (#4735) Chunan Zeng 2025-03-24 20:56:31 -07:00
  • 9b7cf9ee6c support cu128 sgl-kernel (#4744) Yineng Zhang 2025-03-24 20:53:23 -07:00
  • 1e86457c90 model: Minicpmo (#3023) Mick 2025-03-25 11:08:40 +08:00
  • 64129fa632 Add DeepEP tests into CI (#4737) fzyzcjy 2025-03-25 10:54:31 +08:00
  • e9f8e42318 Support FP4 gemm (1/2) (#3899) Trevor Morris 2025-03-24 19:50:23 -07:00
  • 22c3702e1e [Model] Support Qwen2ForSequenceClassification (#4609) Ximingwang-09 2025-03-25 10:13:44 +08:00
  • 4c584fc632 Fix circular imports in gptq.py and unblock test explorer (#4736) Stefan He 2025-03-24 18:07:08 -07:00
  • 77cf771ebe Fix EAGLE3 for llama3.3 70b (#4716) Ke Bao 2025-03-25 08:31:19 +08:00
  • 8154de5a32 [PD] Remove invalid parameter (#4721) Xuchun Shang 2025-03-25 04:14:16 +08:00
  • c11cfda07b update pyproject (#4731) Yineng Zhang 2025-03-24 09:50:28 -07:00
  • 64edeb798f Support dynamic version name in sglang's pyproject.toml (#4720) Yuhong Guo 2025-03-24 23:56:31 +08:00
  • 65c24c28f9 [Quant Kernel] refactored per token group quant fp8 to support int8 up-to 2x faster (#4396) Chunan Zeng 2025-03-23 23:44:17 -07:00
  • 3980ff1be6 rename benchmark_deepgemm_fp8_group_gemm.py (#4605) Tongbao Zhang 2025-03-24 14:35:20 +08:00
  • 5d7edc8e55 Support FA3 as Attention backend by using --attention-backend fa3 (#4680) Stefan He 2025-03-23 23:28:11 -07:00
  • af6535e7aa [ROCm] Enable MTP (NextN) on AMD GPU (#4631) Alex Sun 2025-03-24 13:58:05 +08:00
  • 93cf7fc5cd Unify variable naming: replace is_in_free_group with is_not_in_free_group (#4698) c1lovez1 2025-03-24 12:51:08 +08:00
  • 2a206b22ed Fix RotaryEmbedding when using Triton backend for EXAONE-3.5-2.4B (#4064) Kyungmin Lee 2025-03-24 09:58:12 +09:00
  • 4d25305700 Move mem_state update into debug mode (#4525) Zhiqiang Xie 2025-03-23 00:52:27 -07:00
  • 11577cedb7 refactor: bug fixes and refactor for vlm (#4661) Mick 2025-03-23 13:48:49 +08:00
  • ca75741e86 Support async in DeepEP (#4610) fzyzcjy 2025-03-23 13:39:56 +08:00
  • c6d549e773 Multiple tiny code cleanups (#4608) fzyzcjy 2025-03-23 13:39:11 +08:00
  • 3c09548d1f close gemma2 in test_verl_engine.py temporarily (#4685) Yi Zhang 2025-03-23 07:36:46 +08:00
  • 8796cebb2c fix typo SGLang supports three grammar backends (#4679) BroadbentJim 2025-03-22 21:33:48 +00:00
  • c2bd094d6e Optimize Permute Kernel in DeepEP (#4643) xutizhou 2025-03-23 05:30:34 +08:00
  • f8f9244a61 [Bug Fix] Add partial rotary factor support for Phi-4 and upgrade to transformers v4.50.0 (#3984) Adarsh Shirawalmath 2025-03-23 02:57:39 +05:30
  • ecbfe58bb0 Bug fix for metrics counter (#4660) Zhiqiang Xie 2025-03-22 13:39:21 -07:00
  • 8f163b1653 Add EAGLE mtbench benchmark script (#4676) Ke Bao 2025-03-23 04:34:01 +08:00
  • e7a8610d51 fix flaky ut (#4670) Yineng Zhang 2025-03-22 12:36:50 -07:00
  • a2cc62a6db [CI fix] test skipping modelopt on AMD (#4677) Adarsh Shirawalmath 2025-03-23 01:06:02 +05:30
  • fb8886037c [Docs] Update docs for gemma3 and VLM chat templates (#4674) Adarsh Shirawalmath 2025-03-22 20:32:19 +05:30
  • 321ab756bc [1/3] fix dsv3 awq issue (#4556) AniZpZ 2025-03-22 16:07:17 +08:00
  • 38f25e87fc Correcting default configuration when benchmarking fused_moe (#4665) penguin_wwy 2025-03-22 15:52:34 +08:00
  • 8cd4250401 [quantization] fix channelwise conversion with scalar weight scale (#4596) Yun Dai 2025-03-22 00:47:52 -07:00
  • 6a384d5c01 Speed up per token and per tensor quant by 15% (#4639) Chunan Zeng 2025-03-22 00:37:57 -07:00
  • f69e0696bc update news for README (#4664) Yineng Zhang 2025-03-21 23:25:59 -07:00
  • f6ab4ca6bc fix: fix ipython running error for Engine due to outlines nest_asyncio (#4582) mlmz 2025-03-22 10:11:15 +08:00
  • c7c7dbebbe [PD] Release initial code (#4654) Byron Hsu 2025-03-21 14:47:47 -07:00
  • 417fc72f6f Align completion and chat_completion response to OpenAI API (#4637) Yuhong Guo 2025-03-21 13:59:04 +08:00
  • c6ec70290f [docs] Add links and fix grammars in deploy_on_k8s.md (#4641) Michael Yao 2025-03-21 13:55:23 +08:00
  • 4c56e5dbee Set deepgemm to the default value in the hopper architecture. (#4613) lukec 2025-03-21 13:03:00 +08:00
  • 7b5fc71972 fix SUPPORT_CUTLASS_BLOCK_FP8 flag (#4640) Cheng Wan 2025-03-21 00:45:07 -04:00
  • ad4e58bf67 Support fp8 gemm for blackwell (#4558) Shu Wang 2025-03-20 14:40:28 -05:00
  • bfb03c6182 Update doc for MTP and DP attention (#4622) Ke Bao 2025-03-21 02:31:48 +08:00
  • b36ab493b3 Enable setting sglang logger from Env Variable SGLANG_LOGGING_CONFIG_PATH (#4592) Yuhong Guo 2025-03-20 17:10:32 +08:00
  • 9e93ef3f8e [fix] fix illegal mem access and clean up triton attention backend (#4571) JieXin Liang 2025-03-20 17:01:52 +08:00
  • fad86a6863 Support n in OpenAI API completions (#3446) Chuyue Sun 2025-03-19 22:46:46 -07:00
  • df7014a8d2 avoid cudaStreamSynchronize in DeepSeekV2AttentionMLA (#4577) strgrb 2025-03-20 01:02:26 +08:00
  • 4942074174 [fix] fix initialization of _ENABLE_TORCH_INFERENCE_MODE (#4549) JieXin Liang 2025-03-20 00:57:59 +08:00
  • ba52fd1868 Add clang-format to pre-commit config (#4583) Hongbo Xu 2025-03-20 00:50:19 +08:00
  • b6944f97a6 Support FlashMLA backend cuda graph (#4514) lukec 2025-03-19 23:25:34 +08:00
  • f44db16c8e [Feature] Integrate DeepEP into SGLang (#4232) Jinyan Chen 2025-03-19 23:16:31 +08:00
  • f9c53cbb42 Create col-major and tma-aligned x_scale for deep_gemm.gemm_fp8_fp8_bf16_nt (#4515) strgrb 2025-03-19 15:02:43 +08:00
  • 90532b7627 [Fix] Fix raw_bs bug when using flashinfer mla and eagle (#4557) Baizhou Zhang 2025-03-18 21:26:53 -07:00
  • c0e9a36c5f Optimize Triton decoding kernel for dynamic workload (#4553) JieXin Liang 2025-03-19 12:25:38 +08:00
  • 588865f0e0 [Feature] Support Tensor Parallelism and Weight Slicing for Lora (#4274) aoshen524 2025-03-18 23:33:07 -04:00
  • 3196999f63 Reduce computation and communication in DP attention (#4521) Cheng Wan 2025-03-18 16:41:36 -04:00
  • 9e0186f352 [Feature] Support EAGLE 3 (#4247) James Liu 2025-03-18 10:35:23 -04:00
  • 8baf9a0c18 [Fix] Type annotation correction for UpdateWeightsFromTensorReqInput (#4532) Wei Wu 2025-03-18 15:52:47 +08:00
  • c787298547 use sgl custom all reduce (#4441) Yineng Zhang 2025-03-18 00:46:41 -07:00
  • 45212ce18b Add deepseek v2 torch compile pr test (#4538) Ke Bao 2025-03-18 15:29:24 +08:00
  • c16b33ccac cleanup deps 3/n (#4541) Yineng Zhang 2025-03-18 00:11:36 -07:00
  • 2d0045125f Fix the incorrect args in benchmark_and_profiling.md (#4542) Albert 2025-03-18 15:07:06 +08:00
  • 804d250a0d remove useless backend forward in rotary_embedding (#4500) Xiaoyu Zhang 2025-03-18 14:54:00 +08:00
  • dd865befde [Hotfix] solve fp8 w8a8 ci test fail (#4531) Xiaoyu Zhang 2025-03-18 14:17:04 +08:00
  • d373a48c98 fix: second_per_grid_ts should be used to get mrope position (#3682) Mick 2025-03-18 09:12:38 +08:00
  • 98be3bd306 refactor: rewrite bench-mmmu-sglang (#4458) Mick 2025-03-18 09:11:47 +08:00
  • a98290aea3 Unit test for Hierarchical Caching (#4486) Zhiqiang Xie 2025-03-17 17:45:00 -07:00
  • 9b81f9bd34 sglang quant module remove vllm dependency (#4507) Xiaoyu Zhang 2025-03-18 06:51:59 +08:00
  • f81a27f65e upgrade sgl-kernel 0.0.5.post3 (#4522) Yineng Zhang 2025-03-17 14:49:56 -07:00
  • 988ab646ec bump v0.0.5.post3 (#4520) Yineng Zhang 2025-03-17 13:05:59 -07:00
  • 3ded4b215d Revert "feat: update grouped_topk to support softmax and sigmoid" (#4505) Ke Bao 2025-03-18 02:30:26 +08:00
  • f4d7ab7a63 [sgl-router] improvement to avoid hang (#4482) Yinghai Lu 2025-03-17 10:37:50 -07:00
  • c38ca4fc8e Update readme (#4517) Lianmin Zheng 2025-03-17 08:22:42 -07:00
  • 82dec1f70b Remove redundant type conversion (#4513) Lianmin Zheng 2025-03-17 05:57:35 -07:00
  • 5f9b2c62ff [ROCm] fix dtype (#4510) yiakwy-xpu-ml-framework-team 2025-03-17 20:20:50 +08:00
  • 5493c3343e Fix data parallel + tensor parallel (#4499) Lianmin Zheng 2025-03-17 05:13:16 -07:00
  • f2ab37e500 [Doc] add doc for quantization w8a8_fp8 or w8a8_int8 (#4495) HandH1998 2025-03-17 17:25:00 +08:00
  • 91ba98fe50 [Fix] Resolve GPU Memory Leak in update_weights_from_tensor (#4446) Wei Wu 2025-03-17 16:54:30 +08:00
  • c614dbdf95 Nicer standalone engine inferface (#4480) Yinghai Lu 2025-03-17 01:42:04 -07:00
  • 927ca935a7 Constraint Decoding: Tool call with text (#4067) Xihuai Wang 2025-03-17 16:06:46 +08:00
  • ef3c2dd08e Support Online Quantization for W8A8 (#4485) Stefan He 2025-03-17 00:28:56 -07:00
  • 75b656488a Support serving DeepSeek-R1-Channel-INT8 with 32 L40S. (#4418) Wenbo Yang 2025-03-17 15:03:43 +08:00
  • 0f52fb55ec config: Update fused moe config (#4493) Mick 2025-03-17 14:51:58 +08:00
  • d6d21640d3 [Feature] Support Deepseek-VL2 (#2798) 萝卜菜 2025-03-17 14:07:59 +08:00
  • 0212d2e288 [Fix] use torch.inference_mode() instead of torch.no_grad() (#4372) JieXin Liang 2025-03-17 13:54:16 +08:00