Add KV4-capable backend flashmla and update server args (#14989)

Signed-off-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
This commit is contained in:
Ho-Ren (Jack) Chuang
2025-12-12 11:50:27 -08:00
committed by GitHub
parent 4b7b5af36a
commit 171b442ad3
2 changed files with 2 additions and 1 deletions
+1 -1
View File
@@ -36,7 +36,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
| **Backend** | **Native Page Sizes** | **FP8 KV Cache** | **FP4 KV Cache** | **Chunked Prefix Cache** | **Spec topk=1** | **Spec topk>1** |
|----------------------------|---------------------------|------------------|------------------|--------------------------|-----------------|-----------------|
| **FlashInfer MLA** | 1 | ❌ | ✅ | ✅ | ✅ | ❌ |
| **FlashMLA** | 64 | ✅ | | ✅ | ✅ | ❌ |
| **FlashMLA** | 64 | ✅ | | ✅ | ✅ | ❌ |
| **Cutlass MLA** | 128 | ✅ | ✅ | ✅ | ✅ | ❌ |
| **TRTLLM MLA (Blackwell)** | 32 or 64 | ✅ | ✅ | ✅ | ✅ | ❌ |
| **FA3 (FlashAttention 3)** | n/a | ❌ | ❌ | ✅ | ✅ | ⚠️ (page_size=1 only) |
+1
View File
@@ -1607,6 +1607,7 @@ class ServerArgs:
"cutlass_mla",
"flashinfer",
"trtllm_mla",
"flashmla",
]
assert (
self.attention_backend in KV4_ATTENTION_MLA_BACKEND_CHOICES