From eb38d6441375322878a428761e1d298cbe98a73b Mon Sep 17 00:00:00 2001 From: Ruihuan He Date: Tue, 20 Jan 2026 14:10:36 -0500 Subject: [PATCH] [Docs] Explain CUDA attention backend choices and aiter FP8 KV cache support (#17428) --- docs/advanced_features/attention_backend.md | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/advanced_features/attention_backend.md b/docs/advanced_features/attention_backend.md index 6fc385f8c..54b2dc989 100644 --- a/docs/advanced_features/attention_backend.md +++ b/docs/advanced_features/attention_backend.md @@ -25,7 +25,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu | **FlexAttention (PyTorch)** | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | | **TRTLLM MHA** | 16, 32 or 64 | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | | **Dual Chunk FlashAttention** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| **AITER (ROCm)** | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | +| **AITER (ROCm)** | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | | **Wave (ROCm)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **Ascend (NPU)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | **Intel XPU** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | @@ -111,6 +111,23 @@ If you set only one of `--prefill-attention-backend` or `--decode-attention-back If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase. ``` +## Attention Backend Selection Guide (CUDA) + +If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture. + +### Automatic Selection Logic + +**1. MHA Models (e.g., Llama, Qwen)** +- **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported. +- **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`. +- **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`. + +**2. MLA Models (e.g., DeepSeek V3)** +- **Hopper**: Defaults to `fa3` (requires CUDA 12.3+). +- **Blackwell**: Defaults to `trtllm_mla`. +- **Other Architectures**: Defaults to `triton`. + + ## User Guide ### Launch Command for Different Attention Backends