chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Commit Graph

  • 29bd4c8135 [CI] Add CI Testing for Prefill-Decode Disaggregation with Router (#7540) Keyang Ru 2025-06-27 00:18:56 -07:00
  • 031f64aa1b Add e2e test for multi instance multi stage memory release/resume occupuation (#7208) Ata Fatahi 2025-06-26 17:40:38 -07:00
  • 3d7cdb2ebd Fix MTP error when enabling two-batch overlap (#7569) fzyzcjy 2025-06-27 06:40:54 +08:00
  • 604efe07e1 Updates Gemma3n MLP layer to adapt latest transformers version (#7573) Xinyuan Tong 2025-06-26 15:07:22 -07:00
  • 1b8cf77b01 [Fix] incorrect assert in EPLB (#7575) Cheng Wan 2025-06-26 14:59:20 -07:00
  • bb9b608c86 [PD][NIXL] Set is_sorted=False to fix NIXL_ERR_NOT_FOUND (#7330) Trevor Morris 2025-06-26 10:39:39 -07:00
  • 69183f8808 chore: bump v0.4.8.post1 (#7559) Yineng Zhang 2025-06-26 02:21:12 -07:00
  • 9b00990bea chore: remove vlm unnecessary import (#7541) Xinyuan Tong 2025-06-26 01:38:15 -07:00
  • 4d67025a1d chore: improve ci bug reporting (#7542) Mick 2025-06-26 16:32:44 +08:00
  • 0e05fe8cf4 Update seed in CPU UTs to avoid flaky failure with single test (#7544) YanbingJiang 2025-06-26 12:25:50 +08:00
  • 2390a2bc8d Add Tencent HunYuanMoEV1 model support (#7549) Meng, Peng 2025-06-26 11:59:53 +08:00
  • 16d76b9f23 [CMake] Fix sgl-kernel CMakeLists for Blackwell (#7543) Ruihang Lai 2025-06-25 22:00:46 -04:00
  • 5c2142579a [PD] Raise error for incompatible mooncake version and some minor fixes (#7527) Shangming Cai 2025-06-26 09:55:24 +08:00
  • b8df43ab9c Fix gathered_buffer issues in tbo (#7531) Qiaolin Yu 2025-06-25 14:42:21 -07:00
  • a1c1ebe935 Fix FP8 KV Cache Support in FA3 Backend (#7148) Yuhong Guo 2025-06-25 17:14:40 +08:00
  • fe2a0f962f minor: 'role' must be system/assistant/tool, but case insensitive for now (#7499) mlmz 2025-06-25 17:11:03 +08:00
  • 20beb3702b feat: add return hidden_states at async generation (#7507) eigen 2025-06-25 05:10:09 -04:00
  • 00fbd8a484 Fix typo of flash_cache (#7513) Stefan He 2025-06-25 02:04:41 -07:00
  • 802815e40b take aiter get_rope back (#7521) valarLip 2025-06-25 17:03:33 +08:00
  • 4c6675c4fc enable aiter fp8 blockscale quant (#7520) valarLip 2025-06-25 17:02:31 +08:00
  • e21aa1df67 [PD] Add different TP sizes support for no-MLA models (#6793) Hongbo Xu 2025-06-25 17:00:22 +08:00
  • f3cbd24541 feat: send kvmetrics from sglang scheduler (#6721) zixuanzhang226 2025-06-25 01:57:49 -07:00
  • 506a2d5934 npu fused op (#7386) ll819214 2025-06-25 16:54:20 +08:00
  • a07f8ae4b7 [CI] Upgrade mooncake to v0.3.4.post2 to fix potential slice failed bug (#7522) Shangming Cai 2025-06-25 16:49:22 +08:00
  • 7eb47b0f3d [CPU] [BF16] Call fused_experts_cpu, weight_packed_linear and bmm_cpu kernel in DeepSeek model (#6641) Chunyuan WU 2025-06-25 16:43:33 +08:00
  • bc2e5645c4 fix: force synchronization between TP workers when update_weights (#6626) DangKai 2025-06-25 16:35:59 +08:00
  • 3abc30364d [ci] add router benchmark script and CI (#7498) Simo Lin 2025-06-25 01:28:25 -07:00
  • afeed46530 clean duplicate code (#7512) linzhuo 2025-06-25 16:22:20 +08:00
  • 587b4c6e92 EPLB support for MTP (#7510) yilian49 2025-06-25 01:16:56 -07:00
  • 7b9a174a7a [PD][Spec] Fix hidden state transfer for spec decode (#7516) Shangming Cai 2025-06-25 15:42:07 +08:00
  • 03c039c48e [OAI] patch origin request_id logic (#7508) ybyang 2025-06-25 11:09:38 +08:00
  • 57ab776910 Fuse sorted_token_ids padding to moe_align_block_size kernel (#7437) Ke Bao 2025-06-25 08:44:27 +08:00
  • 112b496a6c misc: Improvement to serving_chat.py and add more ut (#7489) Chang Su 2025-06-24 17:19:51 -07:00
  • 3562256bb2 fix: Add --model as an alias for --model-path in server_args (#7505) Chang Su 2025-06-24 12:08:08 -07:00
  • 5f527834a8 [PD] NIXL: Register kv args in advance and cleanup finished requests (#6717) Trevor Morris 2025-06-24 11:26:09 -07:00
  • 9f1787fa60 Support multi-thread model weight loading (#7277) xianzhiT 2025-06-25 01:39:10 +08:00
  • 8ecad0b16f [benchmark] fbgemm benchmark support bandwidth report and support fbgemm_cutlass_gmm (#7422) Xiaoyu Zhang 2025-06-25 00:44:55 +08:00
  • 7151194bb2 Remove cumsum_buffer initilization (#7439) Ke Bao 2025-06-24 17:22:16 +08:00
  • 2ed68d7a6c [PD Disaggregation] replace transfer with batch transfer for better performance (#7236) Francis 2025-06-24 17:12:04 +08:00
  • e984d5073b enable aiter_biased_grouped_topk kernel (#7423) valarLip 2025-06-24 17:09:42 +08:00
  • 755f314785 [AMD] add aiter fused moe in DeepEP path (#7268) Alex Sun 2025-06-24 17:05:47 +08:00
  • 7c3a12c000 chore: bump v0.4.8 (#7493) Yineng Zhang 2025-06-23 23:14:22 -07:00
  • e846d95ef6 chore: bump sgl-kernel v0.2.0 (#7490) Yineng Zhang 2025-06-23 22:29:50 -07:00
  • 15f3401343 Fix MTP with Deepseek R1 Fp4 (#7376) Charles Chen 2025-06-23 21:38:07 -07:00
  • d04163b3fa Fix RequestValidationError response format (#7487) Chang Su 2025-06-23 20:35:11 -07:00
  • 7732bbe458 bugfix: Prevent global mutation of conv.stop_str across requests (#7347) huangtingwei 2025-06-24 10:36:23 +08:00
  • ed0a0b692c Perormance: Enable cuda graph for dp idle batch (#7269) u4lr451 2025-06-24 08:34:13 +08:00
  • fa42e41962 ci: Revert openai_server related tests in AMD suites (#7449) Chang Su 2025-06-23 15:28:22 -07:00
  • e5afb88b1c Support weight loading without mmap (#7469) Yuhong Guo 2025-06-24 06:13:59 +08:00
  • e5ddeb04d5 Quick fix for DeepGemm requant to also cover MTP. (#7378) Charles Chen 2025-06-23 12:08:54 -07:00
  • bdbb8d009a [perf] slightly imporve DeepSeek-R1-FP4 TP8 (#7481) JieXin Liang 2025-06-24 03:05:30 +08:00
  • 34c3f9b2d3 kvcache io kernels and test case (#7382) Zhiqiang Xie 2025-06-23 11:58:59 -07:00
  • 76139bfba0 update mooncake in dockerfile (#7480) Liangsheng Yin 2025-06-24 02:29:30 +08:00
  • f8d48fd311 Fix dtype for idle input in spec decoding (#7456) Cheng Wan 2025-06-23 11:23:25 -07:00
  • 34b6b8426f feat(func_call): Add more check in BaseFormatDetector.parse_streaming_increment (#7479) Chang Su 2025-06-23 11:15:47 -07:00
  • 25549433e8 Fix prefill OOM due to wrong token calculation when page > 1 (#7397) Liangsheng Yin 2025-06-24 02:12:29 +08:00
  • d6dddc19ff [CI] Upgrade mooncake to 0.3.4.post1 to fix 8 gpu tests (#7472) Shangming Cai 2025-06-24 02:10:50 +08:00
  • 55e03b10c4 Fix a bug in BatchTokenIDOut & Misc style and dependency updates (#7457) Lianmin Zheng 2025-06-23 06:20:39 -07:00
  • 8aa68ed5c4 Solve docker build failed in the virtual machine (#7290) kk 2025-06-23 17:10:30 +08:00
  • 506c4928f5 feat: integrate deepgemm into EPMoE (#6821) xutizhou 2025-06-23 16:38:58 +08:00
  • 30ceccc74a Update hyperparameter_tuning.md (#7454) Lianmin Zheng 2025-06-22 22:42:55 -07:00
  • ac5010e0ba Fix CUDA Graph Check under Deepep with DP FFN (#7451) Cheng Wan 2025-06-22 20:35:58 -07:00
  • 3cee035e99 add fused moe config for qwen3 in triton3.3.1 (#7445) Yi Zhang 2025-06-23 09:29:25 +08:00
  • 30f2a44a96 [misc] Add PD service discovery support in router (#7361) Simo Lin 2025-06-22 17:54:14 -07:00
  • bd4f581896 Fix torch compile run (#7391) kk 2025-06-23 06:33:09 +08:00
  • 50f1b6d6b1 Remove copy after bmm (#7441) Ke Bao 2025-06-23 06:07:44 +08:00
  • 5962e70d8d FlashInfer NVFP4 MoE with EP & 2-stream shared expert (#7327) Trevor Morris 2025-06-22 13:38:47 -07:00
  • edc21cc8ae Tiny add logging for GC (#7406) fzyzcjy 2025-06-22 12:40:02 +08:00
  • 05c9bc8956 [minor] simplify the TokenToKVPoolAllocator (#7414) Liangsheng Yin 2025-06-22 12:37:18 +08:00
  • b7a2df0a44 refactor(test): reorganize OpenAI test file structure (#7408) Chang Su 2025-06-21 19:37:48 -07:00
  • 1998ce4046 Refactor LoRAManager and LoRAMemoryPool state management logic for dynamic LoRA loading support (#7412) Lifu Huang 2025-06-21 16:09:19 -07:00
  • 72676cd6c0 feat(oai refactor): Replace openai_api with entrypoints/openai (#7351) Chang Su 2025-06-21 13:21:06 -07:00
  • 02bf31ef29 [fix] PD disaggregation when enable mtp and tp!=dp (#7420) Atream 2025-06-22 03:03:11 +08:00
  • 5ea5d22170 Fix CPU offloading for MLA memory pool (#7409) Liangsheng Yin 2025-06-22 02:39:05 +08:00
  • fdfd5224bf fix: Fix CI test_function_call_parser.py (#7425) Chang Su 2025-06-21 09:25:08 -07:00
  • 7f3ee861ae fix overlap pagecount (#6984) pansicheng 2025-06-21 15:34:45 +08:00
  • bec5891083 [BugFix]fix qwen25 invoke function call streaming responses with curly braces as the starting indicator (#7394) ehuaa 2025-06-21 13:50:08 +08:00
  • 9edf6608c9 [BugFix]: fix EmbeddingReqInput single input error (#7396) woodx 2025-06-21 11:35:21 +08:00
  • ab74f8f09d Remove batches api in docs & example (#7400) Jinn 2025-06-20 21:46:31 -05:00
  • 5e7fdc79fa [OAI Server Refactor] [ChatCompletions & Completions] Support Return Hidden State (#7329) Keyang Ru 2025-06-20 19:18:53 -07:00
  • 4d8d9b8efd chore: upgrade mooncake-transfer-engine 0.3.4 (#7401) Yineng Zhang 2025-06-20 16:38:54 -07:00
  • 5041df2d01 Fix 7285 Merge Conflicts (#7403) Cheng Wan 2025-06-20 16:02:50 -07:00
  • 256801e973 Update usage_processor.py (#7402) Cheng Wan 2025-06-20 15:55:38 -07:00
  • 73b13e69b4 Optimize DP attn scheduling for speculative decoding (#7285) Cheng Wan 2025-06-20 15:06:41 -07:00
  • 8609e637a9 Fix All-Gather under world size one (#7219) Cheng Wan 2025-06-20 14:57:34 -07:00
  • dea2b84bc3 [OAI Server Refactor] [ChatCompletions & Completions] Implement UsageInfo Processor (#7360) yhyang201 2025-06-21 05:51:21 +08:00
  • cfb2fb5afc [OAI refactor] Add rerank and score serving (#7399) woodx 2025-06-21 05:51:10 +08:00
  • 22bfed7509 [DeepSeekNextN] fix: residual of head norm can be None (#7398) Cheng Wan 2025-06-20 14:45:16 -07:00
  • e879d8b7a8 [Feature] Comprehensive Hybrid Parallelism Support (#6389) Cheng Wan 2025-06-20 14:43:11 -07:00
  • 0998808009 Refine OpenAI serving entrypoint to remove batch requests (#7372) Xinyuan Tong 2025-06-20 14:33:43 -07:00
  • 794be55af2 Support THUDM/GLM-4-0414 (GLM-Z1) Glm4ForCausalLM architecture. (#5485) Wenbo Yang 2025-06-21 01:07:43 +08:00
  • 187b85b7f3 [PD] Optimize custom mem pool usage and bump mooncake version (#7393) Shangming Cai 2025-06-21 00:50:39 +08:00
  • ceba0ce4f6 support return logprobs for pipeline (#7356) strgrb 2025-06-20 14:50:45 +08:00
  • 1ab6be1b26 Purge VerlEngine (#7326) Ata Fatahi 2025-06-19 23:47:21 -07:00
  • 4df5fc2156 Feat/refactor embedding server (#7322) woodx 2025-06-20 14:46:01 +08:00
  • a06912ad8b Fix judgment condition for enabling Deepseek V3/R1 shared expert fusion optimization (#7371) Li Hui 2025-06-20 12:58:00 +08:00
  • 97011abc8a [Doc] add embedding rerank doc (#7364) woodx 2025-06-20 12:53:54 +08:00
  • 1d6515ef2a [Bugfix]Fix hang bug using dp attention with HiRadixCache (#7159) Huang Long 2025-06-20 11:34:36 +08:00
  • dea8aa7ab8 Clean unused import for mimo mtp model (#7370) Li Hui 2025-06-20 10:16:14 +08:00
  • 906dbc34f1 [Docker] optimize dockerfile remove deepep and blackwell merge it to… (#7343) ybyang 2025-06-20 08:42:40 +08:00