chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Commit Graph

  • 7ff740a6ce Remove dp balance metadata and minimul token balance. (#11170) Liangsheng Yin 2025-10-03 01:48:15 +08:00
  • bfcd9b2433 [grpc] style fix for grpc compilation. (#11175) Liangsheng Yin 2025-10-03 01:44:29 +08:00
  • 458611de77 Unify forward output datastructure (#11124) Liangsheng Yin 2025-10-03 00:28:57 +08:00
  • 3511b37099 [proto] Add script to compile python protos (#11171) Chang Su 2025-10-02 08:45:51 -07:00
  • afcd3e1089 Tiny remove duplicated code (#11164) fzyzcjy 2025-10-02 21:56:31 +08:00
  • 12d6818380 Tiny fix ep_gather behavior different in CI (#11130) fzyzcjy 2025-10-02 21:55:53 +08:00
  • b65db0287b Tiny cleanup deepseek_v2.py (#11163) fzyzcjy 2025-10-02 21:54:52 +08:00
  • 948278f173 fix cpp JIT compilation issue of ngram speculative decoding (#10837) b8zhong 2025-10-02 06:05:01 -07:00
  • 7d00479950 Clean up ascend allocator (#11152) Liangsheng Yin 2025-10-02 20:34:26 +08:00
  • 083629c235 [model] Add mamba2 and Falcon-H1 support. (#10988) ilyasch2 2025-10-02 15:15:36 +04:00
  • b658be6f6a [router][grpc] Support tool call parser in streaming (#11160) Chang Su 2025-10-02 03:18:50 -07:00
  • 5e786cca3a Support single batch overlap (#10422) fzyzcjy 2025-10-02 18:04:36 +08:00
  • 0b9dfba787 Support dispatch low latency (#10263) fzyzcjy 2025-10-02 18:02:19 +08:00
  • 6a29003410 Remove unused pack .item() in paged allocator. (#11156) Liangsheng Yin 2025-10-02 18:01:21 +08:00
  • 2ac453b07f Tiny detect slow ranks (#10508) fzyzcjy 2025-10-02 18:00:33 +08:00
  • f35def8652 Fuse quantize and rope in trtllm_mla MTP (#10779) fzyzcjy 2025-10-02 17:59:37 +08:00
  • d61615fe93 Tiny fix missing alt stream in nextn layer (#10768) fzyzcjy 2025-10-02 17:58:23 +08:00
  • b1ccaf01cd Tiny improve dumper (#11132) fzyzcjy 2025-10-02 17:55:01 +08:00
  • 097725bb66 Clean up parallel_state.py (#11148) Lianmin Zheng 2025-10-02 01:09:13 -07:00
  • 44b1fbe258 Fix DeepSeek chunked prefill memory issue (#11149) fzyzcjy 2025-10-02 14:56:59 +08:00
  • c0dbbdd12b [ROCm] To reduce the compiling time when using torch compile. (#10559) sogalin 2025-10-02 14:53:14 +08:00
  • 25e7dbe8af Fix ngram spec with page size > 1 (#11135) Liangsheng Yin 2025-10-02 12:34:23 +08:00
  • 0b2aa8a70c Intoduce cpu tensor as metadata to avoid blocking gpu kernel launch (#10720) Zhang Junda 2025-10-02 10:51:25 +08:00
  • 609f65ba23 Remove debug print statement from scheduler output (#11145) Lianmin Zheng 2025-10-01 13:37:05 -07:00
  • 2d62af6be5 Fix metrics and request tracing (TimeStats) (#11123) Lianmin Zheng 2025-10-01 13:03:07 -07:00
  • a28b394fba [router] Add multi-turn tool calling loop support for MCP integration (#11143) Keyang Ru 2025-10-01 12:50:21 -07:00
  • 96fe2d0f15 [router] add pd service in grpc router for pd (#11120) Simo Lin 2025-10-01 11:09:21 -04:00
  • bfa274380b [HiCache] Configurable and Dynamic Prefetch Timeout (#10512) ykwd 2025-10-01 21:44:10 +08:00
  • 86cb4db058 [Feature] Add EIC as sglang HiCache Storage backend (#10271) Shisong Ma 2025-10-01 21:43:34 +08:00
  • 2e130b7618 [HiCache]bug fix: fixed blank item in host_mem_release_queue (#11005) zhangzuo21 2025-10-01 21:42:37 +08:00
  • ac1f2928ae feat: add fast_decode_plan from flashinfer, flashinfer to 0.4.0rc3 (#10760) eigen 2025-10-01 05:56:13 -04:00
  • 195a59fe23 Refactor AMD CI. (#11128) Sai Enduri 2025-10-01 01:12:28 -07:00
  • 47488cc353 docker: x86 dev builds for hopper and blackwell (#11075) ishandhanani 2025-10-01 00:06:38 -07:00
  • 6130529143 Quick Fix: fix Qwen3-VL launch failure caused by MRotaryEmbedding arg (#10985) yhyang201 2025-10-01 13:17:05 +08:00
  • a9ce2bcb3c [Doc] Update multimodal language models documentation (#11111) Xinyuan Tong 2025-09-30 22:10:31 -07:00
  • 5dddb331c4 [Auto Sync] Update base_grammar_backend.py, xgrammar_backen... (20250930) (#11115) Lianmin Zheng 2025-09-30 21:50:43 -07:00
  • 01a26544a3 [AMD] Add Tilelang and Fast Hadamard Transform builds to Dockerfile.rocm (#11114) Hubert Lu 2025-09-30 20:00:37 -07:00
  • 73d4a5f879 Organize spec-related data structures (#10735) Liangsheng Yin 2025-10-01 09:45:30 +08:00
  • 7fb551a75d [router] add mcp list and mcp call in output array (#11112) Keyang Ru 2025-09-30 18:41:54 -07:00
  • 1193f13181 fix: KimiK2Detector Improve tool call ID parsing with regex (#10972) Xinyuan Tong 2025-09-30 17:44:44 -07:00
  • 84a9f5d660 Feature/make PEFT adapter module format compatibile (#11080) Chenxi Li 2025-09-30 16:32:02 -07:00
  • 8ce830a8b0 [router][bugfix] Fix input_logprobs handling with None value and logprob_start_len = -1 (#11113) Chang Su 2025-09-30 16:09:40 -07:00
  • fb367acfcb Support Dots.ocr model (#11071) qrskannbara 2025-10-01 03:18:39 +08:00
  • a6cc86df9d Fix DSR1 accuracy for flashinfer_trtllm MoE with FP8 quantization (#11081) Trevor Morris 2025-09-30 10:33:12 -07:00
  • 229d2b95f1 [CPU] Adding Memory Capacity Acquisition Functionality (#11102) Zaili Wang 2025-09-30 23:41:20 +08:00
  • 9710f718fb [Eval] Add --repeat in run_eval (#11101) Liangsheng Yin 2025-09-30 23:35:54 +08:00
  • 91847e382a Fix eagle radix cache (#10846) Ke Bao 2025-09-30 22:59:20 +08:00
  • 5a290a5644 [router][grpc-server] Fix gRPC server shutdown (#11094) Simo Lin 2025-09-30 07:12:12 -04:00
  • 580051c5a8 chore: bump sgl-kernel v0.3.14 (#11067) PGFLMG 2025-09-30 17:53:24 +08:00
  • 1237aa19ce [Auto Sync] Update fused_moe_triton_config.py (20250930) (#11099) Lianmin Zheng 2025-09-30 00:57:16 -07:00
  • 5991119541 [Fix] Resolve performance drop in speculative decoding aiter backend (#11087) jacky.cheng 2025-09-30 14:51:30 +08:00
  • 424591d53d Fix spec filter batch when target extend (#10991) Ke Bao 2025-09-30 14:44:02 +08:00
  • d1676cd483 [router][tool call] Full support for ToolChoice (#11085) Chang Su 2025-09-29 22:36:03 -07:00
  • 33b3c0f85f [router] grpc router generate endpoint support (#11070) Simo Lin 2025-09-30 01:07:53 -04:00
  • e5281f84d5 Update CODEOWNERS for attention/ascend_backend.py (#11092) Lianmin Zheng 2025-09-29 21:28:49 -07:00
  • d17986f8c6 Enable optional FP32 compute for LM Head (#10729) narutolhy 2025-09-29 20:45:17 -07:00
  • 8831c55c3d [model] added support for w8a8int8 used by neuralmagic/Qwen2-0.5B-Ins… (#9642) DevashishLal-CB 2025-09-29 20:26:17 -07:00
  • 2bc61dd194 Remove hybrid_linear_attn attention backend and refactor attention registry (#10816) li-kesen 2025-09-30 10:16:16 +08:00
  • 6535fda127 [Profile] dump memory trace when cuda graph profile is enabled (#11083) Cheng Wan 2025-09-29 17:36:48 -07:00
  • 3713eb6135 feat(reasoning): improve enable thinking from request (#10875) Jimmy 2025-09-30 07:59:08 +08:00
  • 5937a56d47 [router][grpc] Add logprobs support to router (#11082) Chang Su 2025-09-29 15:55:06 -07:00
  • f065e5bea5 [router] Use get_pooled in process_single_choice (#11079) Chang Su 2025-09-29 15:48:00 -07:00
  • 9de1320b63 fix: fp8 mllama4 without vision modules being quantized (#10611) Mick 2025-09-30 05:17:12 +08:00
  • dda34c2f93 Fix mem fraction static for nightly tests (#11076) Lianmin Zheng 2025-09-29 12:57:41 -07:00
  • 4eeaff74a0 [router][tool call] Separate JsonParser and LlamaParser (#11073) Chang Su 2025-09-29 10:26:37 -07:00
  • a17e70f5cc Use more general heuristics to set the default value of --mem-fraction-static (#10975) Lianmin Zheng 2025-09-29 10:11:03 -07:00
  • 816b3a433a [router] add n to generate sampling params (#11069) Simo Lin 2025-09-29 10:37:43 -04:00
  • 3a641d9085 chore: upgrade sgl-kernel 0.3.13 (#11056) Yineng Zhang 2025-09-29 02:22:25 -07:00
  • 6f16bf9d9d [Ci Monitor] Auto uploaded performance data to sglang_ci_data repo (#10976) Xiaoyu Zhang 2025-09-29 16:17:27 +08:00
  • 5942fdb480 chore: upgrade cutedsl 4.2.1 (#11054) Yineng Zhang 2025-09-29 00:24:17 -07:00
  • af4ab65606 [router][tool call] Improve normal content extraction and error handling (non-stream) (#11050) Chang Su 2025-09-29 00:19:30 -07:00
  • 11965b0daf Fix sgl-kernel benchmark dead code (#11022) Xiaoyu Zhang 2025-09-29 15:06:40 +08:00
  • 71959545df Fix gemma 3 launch with transformers: the error: AttributeError: 'TransformersForCausalLM' object has no attribute 'tp_size' (#9614) Vincent Zhong 2025-09-29 02:18:29 -04:00
  • 24f7cb1ece [speculative decoding] rename lookahead to ngram (#11010) Zhihao Zhang 2025-09-29 12:06:59 +08:00
  • e05555fad8 [HiCacheStorage] mooncake store support page_first_direct layout (#10591) huangtingwei 2025-09-29 11:45:48 +08:00
  • 43fa9f22bd fix: check if weights are already local before downloading (#11015) Mick 2025-09-29 11:11:33 +08:00
  • e98d9346c7 [1/2] Support FA4 for MHA Prefill in sgl-kernel (#10940) Lifu Huang 2025-09-28 19:59:14 -07:00
  • 0c9174108a Unify SGL Kernel Releases (#10701) Kangyan-Zhou 2025-09-28 19:48:28 -07:00
  • 2572886367 [router] add harmony tool parser base structure and interface (#11036) Simo Lin 2025-09-28 22:46:38 -04:00
  • dba751a896 [router][tool call] Support normal content extraction before tool call (streaming) (#11038) Chang Su 2025-09-28 19:46:06 -07:00
  • 2e7633982c fix: show failed models in nightly ci (#10986) Mick 2025-09-29 03:38:29 +08:00
  • 336e9a6058 [router] migrate to rust python module for pythonic parser (#11033) Simo Lin 2025-09-28 14:48:59 -04:00
  • abb6781573 Update GLM-4.5 Model Doc (#11017) Yuxuan Zhang 2025-09-29 02:21:27 +08:00
  • 07440f5f34 Fix FusedSetKVBufferArg in RotaryEmbedding (#11003) Lianmin Zheng 2025-09-28 11:17:27 -07:00
  • 9816989bff [HiCache] bug: fix mooncake store batch set v1 (#11013) Teng Ma 2025-09-28 23:18:48 +08:00
  • 42245551ef [sgl-kernel] Optimize concat_mla_k kernel (#10543) Yuan Luo 2025-09-28 23:04:22 +08:00
  • 2a9d995c09 prepare for sglang+verl (#10555) lbk-sys 2025-09-28 16:39:17 +08:00
  • a9050b5c39 [bugfix]Add empty_context import to two_batch_overlap.py (#10964) wejoncy 2025-09-28 16:26:20 +08:00
  • 66face3598 Fix CI failure of TypeError: RotaryEmbedding.forward_cpu() got an unexpected keyword argument 'fused_set_kv_buffer_arg' (#11009) YanbingJiang 2025-09-28 15:31:08 +08:00
  • 5519766a4d [router] fix chat template loading and tokenizer path (#10999) Simo Lin 2025-09-27 23:54:12 -04:00
  • 72392f2908 [router] basic mcp support for openai router response api (#10978) Keyang Ru 2025-09-27 18:49:33 -07:00
  • c1c8dd1dd0 [router][tool parser] Modify tool parser to return both normal text and tool calls (non-stream) (#10995) Chang Su 2025-09-27 15:10:17 -07:00
  • f6bc3f529b Fix profiler (#10997) Lianmin Zheng 2025-09-27 14:56:18 -07:00
  • 8cc27fdc46 Use jsonschema to constrain required or specific tool choice (#10550) Tejesh Anand 2025-09-27 10:18:50 -07:00
  • 9c339d6b47 [PD] Extract the PP transfer layer calculate logic from Mooncake to Common backend (#10565) Shangming Cai 2025-09-28 00:10:41 +08:00
  • e23e280e16 Add support for topk metadata transferring for PD (#10616) Shangming Cai 2025-09-28 00:09:38 +08:00
  • 51f7c6bd3c Add auth to get server info (#10751) Muqi Li 2025-09-27 17:54:39 +08:00
  • 62e2e99db6 fix: make inference deterministic for large TP (#10930) Xinyuan Tong 2025-09-27 02:46:45 -07:00
  • 8ebf72fef3 [Fix] RuntimeError: get_cfg Unsupported input_type:Float4_e2m1fn_x2 in using aiter-mxfp4-moe (#10981) kk 2025-09-27 13:12:22 +08:00
  • 8260574729 fix: fp8 quantization failure of qwen 2.5 VL 7B model (#10112) Yueyang Pan 2025-09-27 07:05:23 +02:00