chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity
2,136 Commits 1 Branch 0 Tags
62bbd3439374003aa42ad17151ecb804e232c5a0
Commit Graph

572 Commits

Author SHA1 Message Date
Lianmin Zheng
62bbd34393 Revert "Extract generation_manager from tokenizer_manager" (#3829) 2025-02-24 14:49:16 -08:00
Lianmin Zheng
f2388f6b95 Revert "Rename TokenizerManager to StdOrchestrator" (#3828) 2025-02-24 14:47:59 -08:00
Zhiqiang Xie
6c7a152c5a Hierarchical Caching for SGLang (#2693) 
Co-authored-by: Wenxuan Tan <wenxuan.tan@wisc.edu>
Co-authored-by: Yineng Zhang <me@zhyncs.com>
 2025-02-23 21:56:30 -08:00
fzyzcjy
45360b2fa9 Improve: Rename TokenizerManager to StdOrchestrator (#3116) 2025-02-23 00:30:58 -08:00
fzyzcjy
3f41b18455 Improve: Extract generation_manager from tokenizer_manager (#3115) 2025-02-22 23:25:45 -08:00
Mick
45205d88a0 bench: Add MMMU benchmark for vLM (#3562) 2025-02-22 08:10:59 -08:00
fzyzcjy
9087694006 Improve: Use TypeBasedDispatcher in DetokenizerManager (#3117) 2025-02-21 19:50:46 -08:00
Shenggui Li
9af0e21ef5 [bug] fixed batch api for DeepSeek V3/R1 (#3754) 2025-02-21 10:28:16 -08:00
Yineng Zhang
714f3e6362 feat: support flashinfer mla with prefix cache (#3643) 2025-02-18 02:06:43 +08:00
Mick
bcc213df61 Model: Support Qwen 2.5 vl (#3258) 2025-02-16 00:58:53 -08:00
Yineng Zhang
70f894b810 feat: support flashinfer mla attention for deepseek v3 (#3550) 2025-02-14 08:50:14 +08:00
Jackmin801
5f0e7de339 [Feat] Return hidden states (experimental) (#3364) 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
 2025-02-10 15:54:37 -08:00
Mick
9f635ea50d [Fix] Address remaining issues of supporting MiniCPMV (#2977) 2025-01-28 00:22:13 -08:00
Zhiqiang Xie
08104b56de Sanity check to prevent performance regression (#3171) 
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
 2025-01-27 12:28:17 -08:00
Lianmin Zheng
53cef81587 Improve weight loading and code style (#3174) 2025-01-27 03:00:41 -08:00
YAMY
b045841bae Feature/function calling update (#2700) 
Co-authored-by: Mingyuan Ma <mamingyuan2001@berkeley.edu>
Co-authored-by: Chayenne <zhaochen20@outlook.com>
Co-authored-by: shuaills <shishuaiuoe@gmail.com>
 2025-01-26 09:57:51 -08:00
Lianmin Zheng
1dda8c5e4c Return more infos for computing average acceptance length (#3152) 2025-01-26 04:51:54 -08:00
Lianmin Zheng
d1a0863251 Add a test case for cached_tokens (#3145) 2025-01-26 01:39:28 -08:00
Lianmin Zheng
3d8f1c9bcf Use int64 as indices for set_kv_buffer (#3039) 2025-01-21 19:46:09 -08:00
Lianmin Zheng
287d07a669 Misc fixes for eagle (flush_cache, CPU overhead) (#3014) 2025-01-20 20:27:38 -08:00
996_icu
b730aa6b9e [EAGLE] Fix some boundary situation when retract reqs and req's max token = 1 (#2939) 
Co-authored-by: josephyou <josephyou@tencent.com>
 2025-01-20 17:46:43 -08:00
Hongpeng Guo
949b3fbfce [Doc] Update doc of custom logit processor (#3021) 
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
 2025-01-20 16:50:25 -08:00
Hongpeng Guo
583697cd71 [Enhancement] Custom Logit Processor Improvement (#2998) 
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
 2025-01-20 02:00:35 -08:00
Lianmin Zheng
09bcbe0123 Update TypeBasedDispatcher and balance CI tests (#3001) 2025-01-19 23:37:27 -08:00
Lianmin Zheng
03464890e0 Separate two entry points: Engine and HTTP server (#2996) 
Co-authored-by: fzyzcjy <5236035+fzyzcjy@users.noreply.github.com>
 2025-01-19 22:09:24 -08:00
Lianmin Zheng
cd493b5afc Improve metrics, logging, and importing orders (#2992) 2025-01-19 18:36:59 -08:00
Lianmin Zheng
61f42b5732 Move sgl.Runtime under sglang/lang (#2990) 2025-01-19 17:10:29 -08:00
Hongpeng Guo
e403d23757 [Feature] Add sampler custom logits processor (#2396) 
Signed-off-by: Hongpeng Guo <hpguo@anyscale.com>
 2025-01-19 14:46:53 -08:00
Seungduk Kim
d77caa2b75 [#2812] Make the decode status dict capcity adjustable by a CLI param (#2839) 2025-01-19 11:36:53 -08:00
giorgiopiatti-dfinity
8b6a4486ec fix missing revision arg when loading tokenizer (#2982) 2025-01-19 11:36:07 -08:00
Lianmin Zheng
93b77c8e8a Fix the request loggings to make it fully able to be easily replayed (#2973) 2025-01-18 21:45:00 -08:00
Lianmin Zheng
7906d1d298 Remove the unused write_with_records (#2972) 2025-01-18 20:20:23 -08:00
fzyzcjy
81d27c8e31 Refactor to add TypeBasedDispatcher to simplify dispatching (#2958) 2025-01-18 20:13:27 -08:00
Chang Su
4d4cdb3fe7 Frontend: better error message handling for FINISH_ABORT in scheduler.py (#2956) 2025-01-18 19:37:30 -08:00
Yang Zheng
2bd18e2d76 Memory pool: Minor optimize to avoid to (#2901) 2025-01-18 19:35:12 -08:00
Mick
3d93f84a00 [Feature] Support minicpmv v2.6 (#2785) 
Co-authored-by: Chayenne <zhaochen20@outlook.com>
Co-authored-by: yizhang2077 <1109276519@qq.com>
 2025-01-18 14:14:19 -08:00
Zhiqiang Xie
8af7048dcf Query remaining memory dynamically for PrefillAdder (#2941) 2025-01-17 20:20:26 -08:00
Chunyuan WU
63051738a9 Enable CPU device on SGLang (#2806) 2025-01-16 21:22:53 -08:00
Chang Su
a8ccacc8b8 [Frontend] Fix request length check and add option to disallow auto truncation in scheduler (#2876) 2025-01-16 14:51:19 -08:00
Lianmin Zheng
0427416b59 Fix zmq binding (#2930) 
Co-authored-by: Chunyuan WU <chunyuan.wu@intel.com>
 2025-01-16 14:36:07 -08:00
Lianmin Zheng
bc6915e3b9 Improve type annotation and styles (#2926) 2025-01-16 12:51:11 -08:00
Lianmin Zheng
8b6ce52e92 Support multi-node DP attention (#2925) 
Co-authored-by: dhou-xai <dhou@x.ai>
 2025-01-16 11:15:00 -08:00
Lianmin Zheng
8f2c522aba Improve benchmark scripts and error message printing (#2922) 2025-01-16 06:24:31 -08:00
Lianmin Zheng
f65c13b559 Remove normalized_prompt_logprobs from the engine to make code easier to maintain (#2902) 2025-01-15 04:54:14 -08:00
Cody Yu
b803b395b7 Disable graceful shutdown of tokenizer manager when not in the main thread (#2872) 2025-01-15 03:29:33 -08:00
Lianmin Zheng
46d4431889 Add a new api configure_logging to allow dumping the requests (#2875) 2025-01-13 14:24:00 -08:00
fzyzcjy
923f518337 CUDA-graph-compatible releasing and resuming KV cache and model weight memory (#2630) 2025-01-13 11:38:51 -08:00
Lianmin Zheng
c1e097ca66 Revert "Dump requests to a folder" (#2869) 2025-01-13 06:21:25 -08:00
Lianmin Zheng
336ff5b9f5 Fix typos in io_struct.py (#2867) 2025-01-13 05:13:02 -08:00
Lianmin Zheng
3b141e1509 Dump requests (#2862) 2025-01-13 04:51:56 -08:00