diff --git a/docs/advanced_features/attention_backend.md b/docs/advanced_features/attention_backend.md index e4e1d683d..7d7fbe31d 100644 --- a/docs/advanced_features/attention_backend.md +++ b/docs/advanced_features/attention_backend.md @@ -77,6 +77,28 @@ MLA page-size constraints: - Cutlass MLA: page_size = 128. - TRTLLM MLA: page_size ∈ {32, 64}. +### GDN Attention Backends + +GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM). + +The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`. + +| **Backend** | **Decode** | **Prefill / Extend** | **Spec Decoding (Target Verify)** | +|--------------------------|------------|----------------------|-----------------------------------| +| **Triton (CUDA)** | ✅ | ✅ | ✅ | +| **Triton (AMD/ROCm)** | ✅ | ✅ | ✅ | +| **Triton (NPU)** | ✅ | ✅ | ❌ | +| **Triton (CPU)** | ✅ | ✅ | ❌ | +| **CuTe DSL (CUDA only)**| ✅ | ❌ | ❌ | + +```{important} +GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models: +- **Blackwell (e.g., B200)**: `triton`, `trtllm_mha`, or `fa4` only. +- **NPU (Ascend)**: `ascend` only. +- **AMD (ROCm)**: `triton` recommended. +- **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints. +``` + ### Hybrid attention (different backends for prefill vs decode) (Experimental) ```{warning} @@ -278,6 +300,10 @@ To add a new attention backend, you can learn from the existing backends (`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`) and follow the steps below. +```{note} +Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`. +``` + 1. Run without cuda graph. Support the two forward functions - forward_extend - Will be used for prefill, prefill with KV cache, and target verification