chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

[NPU] update model and features supported (#16733)

Browse Source
This commit is contained in:
Hexq0210
2026-01-08 21:50:42 +08:00
committed by GitHub
parent 83abecd0c2
commit 20ca2c6e1e
2 changed files with 57 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/platforms/ascend_npu_support_features.md
View File

@@ -276,7 +276,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
| `--hicache-size` | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-write-policy` | `write_through` | `write_back`,<br/> `write_through`,<br/> `write_through_selective` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--radix-eviction-policy` | `lru` | `lru`, `lfu` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-io-backend` | `kernel` | `kernel_ascend` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-io-backend` | `kernel` | `kernel_ascend`,<br/>`direct` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-mem-layout` | `layer_first` | `page_first_direct`,<br/> `page_first_kv_split` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-storage-`<br/>`backend` | `None` | `file` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--hicache-storage-`<br/>`prefetch-policy` | `best_effort` | `best_effort`,<br/> `wait_complete`,<br/> `timeout` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -432,15 +432,15 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
## Custom weight loader
| Argument | Defaults | Options | A2 | A3 |
|-------------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------|
| `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-backend` | `nccl` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-start-seed-via-transfer-engine` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Argument | Defaults | Options | A2 | A3 |
|-------------------------------------------------------------------------|----------|---------------------------------|:----------------------------------------:|------------------------------------------|
| `--custom-weight-loader` | `None` | List[str] | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-backend` | `nccl` | `transfer_engine`, <br/> `nccl` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--remote-instance-weight-`<br/>`loader-start-seed-via-transfer-engine` | `False` | bool flag<br/> (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## For PD-Multiplexing

91
docs/platforms/ascend_npu_support_models.md
View File

@@ -6,55 +6,58 @@ You are welcome to enable various models based on your business requirements.
## Large Language Models
| Models | Model Family | A2 Supported | A3 Supported |
|-------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
| DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DeepSeek V3.2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DeepSeek R1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| DeepSeek V2 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-MoE | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-Next | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-Coder | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen2.5 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| QwQ-32B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LLM-Research/Llama-3.2-1B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| google/gemma-3-4b-it | Gemma | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| huihui-ai/grok-2 | Grok | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| HuggingFaceTB/SmolLM-1.7B | SmolLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ZhipuAI/glm-4-9b-chat | GLM-4 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| XiaomiMiMo/MiMo-7B-RL | MiMo | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| arcee-ai/AFM-4.5B-Base | Arcee AFM-4.5B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Howeee/persimmon-8b-chat | Persimmon | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| inclusionAI/Ling-lite | Ling | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ibm-granite/granite-3.1-8b-instruct | Granite | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| databricks/dbrx-instruct | DBRX (Databricks) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| openbmb/MiniCPM3-4B | MiniCPM (v3, 4B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| openai/gpt-oss-120b | GPTOSS | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Models | Model Family | A2 Supported | A3 Supported |
|--------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
| DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| vllm-ascend/DeepSeek-V3.2-Exp-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| vllm-ascend/DeepSeek-R1-0528-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| vllm-ascend/DeepSeek-V2-Lite-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-30B-A3B-Instruct-2507 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-32B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-0.6B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| vllm-ascend/Qwen3-235B-A22B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen2.5-7B-Instruct | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| vllm-ascend/QWQ-32B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LLM-Research/Llama-3.2-1B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| google/gemma-3-4b-it | Gemma | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| huihui-ai/grok-2 | Grok | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| HuggingFaceTB/SmolLM-1.7B | SmolLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ZhipuAI/glm-4-9b-chat | GLM-4 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| XiaomiMiMo/MiMo-7B-RL | MiMo | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| arcee-ai/AFM-4.5B-Base | Arcee AFM-4.5B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Howeee/persimmon-8b-chat | Persimmon | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| inclusionAI/Ling-lite | Ling | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ibm-granite/granite-3.1-8b-instruct | Granite | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| databricks/dbrx-instruct | DBRX (Databricks) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| openbmb/MiniCPM3-4B | MiniCPM (v3, 4B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| openai/gpt-oss-120b | GPTOSS | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Multimodal Language Models
| Models | Model Family (Variants) | A2 Supported | A3 Supported |
|-----------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
| Qwen2.5-VL-72B-Instruct-w8a8 | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen2.5-VL-3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen2.5-VL-72B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Qwen/Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |