chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity
8,338 Commits 1 Branch 0 Tags
3de23274ee5261f9513976c7d445ad0ca72e4e9f
Commit Graph

71 Commits

Author SHA1 Message Date
Ke Bao
3de23274ee Clean up swa handling in fa3 backend (#15877) 2025-12-30 08:51:40 +08:00
Yubo Wang
762846531f Fix Illegal Memory Access when fa3 + spec + topk + page_size > 1 (#15469) 
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
 2025-12-24 00:13:57 +08:00
Yingchun Lai
160a06cab2 [Feature] Xiaomi MiMo-V2-Flash day0 support (#15207) 
Co-authored-by: 谢学扬 <xiexueyang@xiaomi.com>
Co-authored-by: tz <tangzhen3@xiaomi.com>
Co-authored-by: 李家乐 <lijiale10@xiaomi.com>
Co-authored-by: 张晨 <zhangchen50@xiaomi.com>
Co-authored-by: Shaohui Liu <liushaohui3@xiaomi.com>
Co-authored-by: 王晨 <wangchen77@xiaomi.com>
Co-authored-by: jiangzihan <jiangzihan@xiaomi.com>
Co-authored-by: xiexueyang <xyxie_wangyi@163.com>
Co-authored-by: Linghao Zhang <zhanglinghao@xiaomi.com>
Co-authored-by: ispobock <ispobaoke@gmail.com>
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com>
Co-authored-by: JoyFuture <35593546+JoyFuture@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: root <root@bj9-ml-g8h20e-k8s-slave106-20251106.alicn.idc.xiaomi.com>
 2025-12-19 11:40:07 +08:00
Ziming Huang
997ea57eaf Fix tensor mismatch error in sepc + topk > 1 + page_size > 1 (#14874) 2025-12-15 01:10:58 +08:00
Shu Wang
665cb02003 Fix regression caused by fa3 block_table (#15009) 2025-12-12 19:11:09 -08:00
Lianmin Zheng
64092c8b55 [Auto Sync] Rename is_hybrid to is_hybrid_swa (#14252) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
Co-authored-by: Hanming Lu <hanming@x.ai>
 2025-12-01 23:24:24 -08:00
Yubo Wang
18fb51583f Support FlashAttention3 page_size > 1 and topk > 1 case with paged attn and spec decode (#7725) 2025-11-26 11:44:41 +08:00
Xinyuan Tong
0bfe1d145c fa3 & trtllm_mha spec overlap (#11874) 2025-10-31 17:38:13 -07:00
Yongfei Xu
4793ec7d1a Opt MHA chunked prefix: merge prefix and extend kv cache to run mha once (#10953) 2025-10-23 20:58:10 -07:00
Liangsheng Yin
5ea96ac7cc Reduce one step decode for draft model. (#11561) 2025-10-14 23:52:04 +08:00
Liangsheng Yin
516738b096 Depreate global_server_args_dict (#11528) 2025-10-13 19:34:43 +08:00
Cheng Wan
1bdd010291 Revert "Deprecate global_server_args_dict" (#11520) 2025-10-12 17:40:40 -07:00
Liangsheng Yin
1083e7e3df Deprecate global_server_args_dict (#11331) 2025-10-13 01:20:47 +08:00
Liangsheng Yin
01e59e8247 Fix CI break by express-laned PRs. (#11499) 2025-10-12 21:06:06 +08:00
Mahmoud Ashraf
7b064f04f8 [bugfix]: use correct causality condition for flashattention, flashinfer, and triton backends (#10172) 2025-10-12 20:28:16 +08:00
Lifu Huang
edefab0c64 [2/2] Support MHA prefill with FlashAttention 4. (#10937) 
Co-authored-by: Hieu Pham <hyhieu@gmail.com>
 2025-10-08 00:54:20 -07:00
Liangsheng Yin
73d4a5f879 Organize spec-related data structures (#10735) 2025-10-01 09:45:30 +08:00
Yineng Zhang
0753ef831e [Auto Sync] Update flashattention_backend.py (20250922) (#10762) 
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gordon Gustafson <ggustafson@together.ai>
 2025-09-22 16:41:42 -07:00
Stefan He
cba0d8c309 [Feature] Support deterministic inference with FA3 backend (#10651) 2025-09-20 17:50:21 -07:00
cicirori
a2f7218a2e support using fa4 on deepseek on blackwell (#9928) 2025-09-16 16:16:06 -07:00
Ke Bao
603b3446dc Fix FA3 swa spec verify topk>1 (#9658) 2025-08-26 15:03:14 -07:00
Yongfei Xu
9708d353b7 Support MHA with chunked prefix cache for flashinfer/flashmla backend, support page size > 1 for MHA chunked prefix (#8616) 
Co-authored-by: xuyongfei.xyf <xuyongfei.xyf@antgroup.com>
 2025-08-21 18:19:44 -07:00
Stefan He
cded039b57 [FA3] Init Spec Page Table only when Spec is enabled to save ~40MB (#9455) 2025-08-21 15:11:38 -07:00
Ke Bao
0ff6d1fce1 Support FA3 backend for gpt-oss (#9028) 2025-08-13 10:41:50 -07:00
Cheng Wan
7a91330149 Save cuda graph memory for fa3 (#8567) 2025-08-03 03:06:31 -07:00
fzyzcjy
519ff5c8e6 Super tiny fix typo (#8046) 2025-07-17 21:15:51 +08:00
tarinkk
eb6c2c1663 Hybrid kv cache for LLaMA4 (#6563) 
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: tarinkk <rt572@physics.rutger.edu>
Co-authored-by: tarinkk <rt572@rutgers.physics.edu>
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com>
 2025-06-27 18:58:55 -07:00
Yuhong Guo
a1c1ebe935 Fix FP8 KV Cache Support in FA3 Backend (#7148) 2025-06-25 02:14:40 -07:00
u4lr451
ed0a0b692c Perormance: Enable cuda graph for dp idle batch (#7269) 
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
 2025-06-23 17:34:13 -07:00
u4lr451
10d60cd41b feat: mtp support dp-attention (#6081) 
Co-authored-by: austindeng <austindeng@tencent.com>
Co-authored-by: tianqilin.99 <tianqilin.99@bytedance.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
Co-authored-by: ch-wan <cwan39@gatech.edu>
 2025-06-17 00:33:28 -07:00
Lianmin Zheng
c64290dcb5 Use seq_len_fill_value in the cuda graph runners (#7233) 2025-06-16 15:57:07 -07:00
Lianmin Zheng
7ddf8e83d2 [EAGLE] Fix draft kv cache layout for fa3 and topk > 1 (#7239) 2025-06-16 05:47:51 -07:00
Lianmin Zheng
21615cc3fe Minor style and doc fix (#7228) 2025-06-16 01:03:13 -07:00
Binyao Jiang
22a6b9fc05 Remove unnecessary metadata_expand.max_seq_len_k operations in fa3 to… (#7140) 2025-06-12 23:25:52 -07:00
Lianmin Zheng
2dae104dca Minor cleanup of fa3 backend (#6999) 2025-06-10 03:58:44 -07:00
Lianmin Zheng
9d5fa68b90 Use torch.compile to fuse flash attention decode metadata preparation (#6973) 2025-06-08 23:05:40 -07:00
Ke Bao
631950280a Support EAGLE draft extend CUDA graph (#6606) 
Co-authored-by: Sehoon Kim <sehoonkim@berkeley.edu>
 2025-05-27 02:35:17 -07:00
Chang Su
066cf44546 [OAI] Add rid tracing for v1/embeddings and fix rid type in Chat (#6397) 2025-05-18 13:05:38 -07:00
JieXin Liang
1f30c05d4a [fix] fix fa3 forward_decode with spec_decode (#6395) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
 2025-05-18 12:50:15 -07:00
Chang Su
205d5cb407 perf: Optimize local attention memory allocation in FlashAttentionBackend (#6356) 2025-05-17 01:45:46 -07:00
Chang Su
912788c095 perf: optimize local_block_table memory allocation (#6273) 2025-05-13 17:18:38 -07:00
applesaucethebun
2ce8793519 Add typo checker in pre-commit (#6179) 
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca>
 2025-05-11 12:55:00 +08:00
Chang Su
1940cdec61 [Bugfix] Fix Llama4 gibberish output with long context and CUDA graph (#6162) 2025-05-09 15:33:02 -07:00
Minglei Zhu
79961afa82 optimize pad operations in fa3 to accelarate 100+us (#6077) 2025-05-07 23:40:08 -07:00
Lifu Huang
1acca3a2c6 FA3 speed up: skip len operation and get batch size directly from forward batch (#5969) 
Signed-off-by: Lifu Huang <lifu.hlf@gmail.com>
 2025-05-02 00:26:12 -07:00
Stefan He
6fc175968c Optimize a pad operation to accelerate 25us (#5945) 2025-05-01 10:48:55 -07:00
Ke Bao
799c4bb502 Fuse MLA set kv cache kernel (#5748) 2025-04-26 18:42:22 -07:00
Ke Bao
6b6e748775 Remove q concat in FA3 backend for DeepSeek decode (#5638) 2025-04-22 11:43:12 -07:00
Qingquan Song
188f0955fa Add Speculative Decoding Eagle3 topk > 1 (#5318) 
Co-authored-by: Stefan He <hebiaobuaa@gmail.com>
Co-authored-by: Yubo Wang <yubowang2019@gmail.com>
 2025-04-20 22:58:28 -07:00
Chang Su
c776234b45 Enable local attention during decode (#5479) 2025-04-17 02:07:43 -07:00