chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Add Llama4 attention backend auto-selection (#13421)

Browse Source 
Signed-off-by: jbernloehr <jbernloehr@nvidia.com>
This commit is contained in:
Jan Bernlöhr
2025-11-25 20:54:21 +01:00
committed by GitHub
parent 215a97fa6c
commit fcccaf9001
2 changed files with 24 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/basic_usage/llama4.md
View File

@@ -21,6 +21,15 @@ python3 -m sglang.launch_server \
- **OOM Mitigation**: Adjust `--context-length` to avoid a GPU out-of-memory issue. For the Scout model, we recommend setting this value up to 1M on 8\*H100 and up to 2.5M on 8\*H200. For the Maverick model, we don't need to set context length on 8\*H200. When hybrid kv cache is enabled, `--context-length` can be set up to 5M on 8\*H100 and up to 10M on 8\*H200 for the Scout model.
- **Attention Backend Auto-Selection**: SGLang automatically selects the optimal attention backend for Llama 4 based on your hardware. You typically don't need to specify `--attention-backend` manually:
- **Blackwell GPUs (B200/GB200)**: `trtllm_mha`
- **Hopper GPUs (H100/H200)**: `fa3`
- **AMD GPUs**: `aiter`
- **Intel XPU**: `intel_xpu`
- **Other platforms**: `triton` (fallback)
To override the auto-selection, explicitly specify `--attention-backend` with one of the supported backends: `fa3`, `aiter`, `triton`, `trtllm_mha`, or `intel_xpu`.
- **Chat Template**: Add `--chat-template llama-4` for chat completion tasks.
- **Enable Multi-Modal**: Add `--enable-multimodal` for multi-modal capabilities.
- **Enable Hybrid-KVCache**: Add `--hybrid-kvcache-ratio` for hybrid kv cache. Details can be seen in [this PR](https://github.com/sgl-project/sglang/pull/6563)