[Docs] Explain CUDA attention backend choices and aiter FP8 KV cache support (#17428)

This commit is contained in:
Ruihuan He
2026-01-20 14:10:36 -05:00
committed by GitHub
parent 6ea491e439
commit eb38d64413

View File

@@ -25,7 +25,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
| **FlexAttention (PyTorch)** | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| **TRTLLM MHA** | 16, 32 or 64 | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| **Dual Chunk FlashAttention** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **AITER (ROCm)** | ✅ | | ❌ | ✅ | ✅ | ❌ | ✅ |
| **AITER (ROCm)** | ✅ | | ❌ | ✅ | ✅ | ❌ | ✅ |
| **Wave (ROCm)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Ascend (NPU)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **Intel XPU** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
@@ -111,6 +111,23 @@ If you set only one of `--prefill-attention-backend` or `--decode-attention-back
If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase.
```
## Attention Backend Selection Guide (CUDA)
If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.
### Automatic Selection Logic
**1. MHA Models (e.g., Llama, Qwen)**
- **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported.
- **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`.
- **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`.
**2. MLA Models (e.g., DeepSeek V3)**
- **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
- **Blackwell**: Defaults to `trtllm_mla`.
- **Other Architectures**: Defaults to `triton`.
## User Guide
### Launch Command for Different Attention Backends