Add KV4-capable backend flashmla and update server args (#14989)
Signed-off-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
This commit is contained in:
@@ -36,7 +36,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
|
||||
| **Backend** | **Native Page Sizes** | **FP8 KV Cache** | **FP4 KV Cache** | **Chunked Prefix Cache** | **Spec topk=1** | **Spec topk>1** |
|
||||
|----------------------------|---------------------------|------------------|------------------|--------------------------|-----------------|-----------------|
|
||||
| **FlashInfer MLA** | 1 | ❌ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **FlashMLA** | 64 | ✅ | ❌ | ✅ | ✅ | ❌ |
|
||||
| **FlashMLA** | 64 | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Cutlass MLA** | 128 | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **TRTLLM MLA (Blackwell)** | 32 or 64 | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **FA3 (FlashAttention 3)** | n/a | ❌ | ❌ | ✅ | ✅ | ⚠️ (page_size=1 only) |
|
||||
|
||||
@@ -1607,6 +1607,7 @@ class ServerArgs:
|
||||
"cutlass_mla",
|
||||
"flashinfer",
|
||||
"trtllm_mla",
|
||||
"flashmla",
|
||||
]
|
||||
assert (
|
||||
self.attention_backend in KV4_ATTENTION_MLA_BACKEND_CHOICES
|
||||
|
||||
Reference in New Issue
Block a user