[Docs] Add GDN attention backends matrix documentation (#19755)
Co-authored-by: Zeyu Wang <zeyu.wang@yahooinc.com>
This commit is contained in:
@@ -77,6 +77,28 @@ MLA page-size constraints:
|
||||
- Cutlass MLA: page_size = 128.
|
||||
- TRTLLM MLA: page_size ∈ {32, 64}.
|
||||
|
||||
### GDN Attention Backends
|
||||
|
||||
GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM).
|
||||
|
||||
The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`.
|
||||
|
||||
| **Backend** | **Decode** | **Prefill / Extend** | **Spec Decoding (Target Verify)** |
|
||||
|--------------------------|------------|----------------------|-----------------------------------|
|
||||
| **Triton (CUDA)** | ✅ | ✅ | ✅ |
|
||||
| **Triton (AMD/ROCm)** | ✅ | ✅ | ✅ |
|
||||
| **Triton (NPU)** | ✅ | ✅ | ❌ |
|
||||
| **Triton (CPU)** | ✅ | ✅ | ❌ |
|
||||
| **CuTe DSL (CUDA only)**| ✅ | ❌ | ❌ |
|
||||
|
||||
```{important}
|
||||
GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models:
|
||||
- **Blackwell (e.g., B200)**: `triton`, `trtllm_mha`, or `fa4` only.
|
||||
- **NPU (Ascend)**: `ascend` only.
|
||||
- **AMD (ROCm)**: `triton` recommended.
|
||||
- **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints.
|
||||
```
|
||||
|
||||
### Hybrid attention (different backends for prefill vs decode) (Experimental)
|
||||
|
||||
```{warning}
|
||||
@@ -278,6 +300,10 @@ To add a new attention backend, you can learn from the existing backends
|
||||
(`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`)
|
||||
and follow the steps below.
|
||||
|
||||
```{note}
|
||||
Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`.
|
||||
```
|
||||
|
||||
1. Run without cuda graph. Support the two forward functions
|
||||
- forward_extend
|
||||
- Will be used for prefill, prefill with KV cache, and target verification
|
||||
|
||||
Reference in New Issue
Block a user