[Docs] Explain CUDA attention backend choices and aiter FP8 KV cache support (#17428)
This commit is contained in:
@@ -25,7 +25,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
|
||||
| **FlexAttention (PyTorch)** | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **TRTLLM MHA** | 16, 32 or 64 | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
|
||||
| **Dual Chunk FlashAttention** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **AITER (ROCm)** | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| **AITER (ROCm)** | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
|
||||
| **Wave (ROCm)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Ascend (NPU)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
|
||||
| **Intel XPU** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
|
||||
@@ -111,6 +111,23 @@ If you set only one of `--prefill-attention-backend` or `--decode-attention-back
|
||||
If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase.
|
||||
```
|
||||
|
||||
## Attention Backend Selection Guide (CUDA)
|
||||
|
||||
If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.
|
||||
|
||||
### Automatic Selection Logic
|
||||
|
||||
**1. MHA Models (e.g., Llama, Qwen)**
|
||||
- **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported.
|
||||
- **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`.
|
||||
- **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`.
|
||||
|
||||
**2. MLA Models (e.g., DeepSeek V3)**
|
||||
- **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
|
||||
- **Blackwell**: Defaults to `trtllm_mla`.
|
||||
- **Other Architectures**: Defaults to `triton`.
|
||||
|
||||
|
||||
## User Guide
|
||||
|
||||
### Launch Command for Different Attention Backends
|
||||
|
||||
Reference in New Issue
Block a user