diff --git a/docs/platforms/ascend_npu_support_features.md b/docs/platforms/ascend_npu_support_features.md index 55b7360d0..0a0060d8b 100644 --- a/docs/platforms/ascend_npu_support_features.md +++ b/docs/platforms/ascend_npu_support_features.md @@ -276,7 +276,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--hicache-size` | `0` | Type: int | **** | **** | | `--hicache-write-policy` | `write_through` | `write_back`,
`write_through`,
`write_through_selective` | **** | **** | | `--radix-eviction-policy` | `lru` | `lru`, `lfu` | **** | **** | -| `--hicache-io-backend` | `kernel` | `kernel_ascend` | **** | **** | +| `--hicache-io-backend` | `kernel` | `kernel_ascend`,
`direct` | **** | **** | | `--hicache-mem-layout` | `layer_first` | `page_first_direct`,
`page_first_kv_split` | **** | **** | | `--hicache-storage-`
`backend` | `None` | `file` | **** | **** | | `--hicache-storage-`
`prefetch-policy` | `best_effort` | `best_effort`,
`wait_complete`,
`timeout` | **×** | **×** | @@ -432,15 +432,15 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen ## Custom weight loader -| Argument | Defaults | Options | A2 | A3 | -|-------------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------| -| `--custom-weight-loader` | `None` | List[str] | **×** | **×** | -| `--weight-loader-disable-`
`mmap` | `False` | bool flag
(set to enable) | **** | **** | -| `--remote-instance-weight-`
`loader-seed-instance-ip` | `None` | Type: str | **×** | **×** | -| `--remote-instance-weight-`
`loader-seed-instance-service-port` | `None` | Type: int | **×** | **×** | -| `--remote-instance-weight-`
`loader-send-weights-group-ports` | `None` | Type: JSON
list | **×** | **×** | -| `--remote-instance-weight-`
`loader-backend` | `nccl` | Type: str | **×** | **×** | -| `--remote-instance-weight-`
`loader-start-seed-via-transfer-engine` | `False` | bool flag
(set to enable) | **×** | **×** | +| Argument | Defaults | Options | A2 | A3 | +|-------------------------------------------------------------------------|----------|---------------------------------|:----------------------------------------:|------------------------------------------| +| `--custom-weight-loader` | `None` | List[str] | **×** | **×** | +| `--weight-loader-disable-`
`mmap` | `False` | bool flag
(set to enable) | **** | **** | +| `--remote-instance-weight-`
`loader-seed-instance-ip` | `None` | Type: str | **×** | **×** | +| `--remote-instance-weight-`
`loader-seed-instance-service-port` | `None` | Type: int | **×** | **×** | +| `--remote-instance-weight-`
`loader-send-weights-group-ports` | `None` | Type: JSON
list | **×** | **×** | +| `--remote-instance-weight-`
`loader-backend` | `nccl` | `transfer_engine`,
`nccl` | **×** | **×** | +| `--remote-instance-weight-`
`loader-start-seed-via-transfer-engine` | `False` | bool flag
(set to enable) | **×** | **×** | ## For PD-Multiplexing diff --git a/docs/platforms/ascend_npu_support_models.md b/docs/platforms/ascend_npu_support_models.md index 3a7a3af96..701aa1ac7 100644 --- a/docs/platforms/ascend_npu_support_models.md +++ b/docs/platforms/ascend_npu_support_models.md @@ -6,55 +6,58 @@ You are welcome to enable various models based on your business requirements. ## Large Language Models -| Models | Model Family | A2 Supported | A3 Supported | -|-------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:| -| DeepSeek V3/V3.1 | DeepSeek | **** | **** | -| DeepSeek V3.2 | DeepSeek | **** | **** | -| DeepSeek R1 | DeepSeek | **** | **** | -| DeepSeek V2 | DeepSeek | **** | **** | -| Qwen3 | Qwen | **** | **** | -| Qwen3-MoE | Qwen | **** | **** | -| Qwen3-Next | Qwen | **** | **** | -| Qwen3-Coder | Qwen | **** | **** | -| Qwen2.5 | Qwen | **** | **** | -| QwQ-32B | Qwen | **** | **** | -| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **** | **** | -| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **** | **** | -| LLM-Research/Llama-3.2-1B-Instruct | Llama | **** | **** | -| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **** | **** | -| google/gemma-3-4b-it | Gemma | **** | **** | -| microsoft/Phi-4-multimodal-instruct | Phi | **** | **** | -| allenai/OLMoE-1B-7B-0924 | OLMoE | **** | **** | -| stabilityai/stablelm-2-1_6b | StableLM | **** | **** | -| CohereForAI/c4ai-command-r-v01 | Command-R | **** | **** | -| huihui-ai/grok-2 | Grok | **** | **** | -| ZhipuAI/chatglm2-6b | ChatGLM | **** | **** | -| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **** | **** | -| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **** | **** | -| xverse/XVERSE-MoE-A36B | XVERSE | **** | **** | -| HuggingFaceTB/SmolLM-1.7B | SmolLM | **** | **** | -| ZhipuAI/glm-4-9b-chat | GLM-4 | **** | **** | -| XiaomiMiMo/MiMo-7B-RL | MiMo | **** | **** | -| arcee-ai/AFM-4.5B-Base | Arcee AFM-4.5B | **** | **** | -| Howeee/persimmon-8b-chat | Persimmon | **** | **** | -| inclusionAI/Ling-lite | Ling | **** | **** | -| ibm-granite/granite-3.1-8b-instruct | Granite | **** | **** | -| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **** | **** | -| databricks/dbrx-instruct | DBRX (Databricks) | **** | **** | -| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **** | **** | -| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **** | **** | -| openbmb/MiniCPM3-4B | MiniCPM (v3, 4B) | **** | **** | -| openai/gpt-oss-120b | GPTOSS | **×** | **×** | +| Models | Model Family | A2 Supported | A3 Supported | +|--------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:| +| DeepSeek V3/V3.1 | DeepSeek | **** | **** | +| vllm-ascend/DeepSeek-V3.2-Exp-W8A8 | DeepSeek | **** | **** | +| vllm-ascend/DeepSeek-R1-0528-W8A8 | DeepSeek | **** | **** | +| vllm-ascend/DeepSeek-V2-Lite-W8A8 | DeepSeek | **** | **** | +| Qwen/Qwen3-30B-A3B-Instruct-2507 | Qwen | **** | **** | +| Qwen/Qwen3-32B | Qwen | **** | **** | +| Qwen/Qwen3-0.6B | Qwen | **** | **** | +| vllm-ascend/Qwen3-235B-A22B-W8A8 | Qwen | **** | **** | +| Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen | **** | **** | +| Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot | Qwen | **** | **** | +| Qwen/Qwen2.5-7B-Instruct | Qwen | **** | **** | +| vllm-ascend/QWQ-32B-W8A8 | Qwen | **** | **** | +| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **** | **** | +| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **** | **** | +| LLM-Research/Llama-3.2-1B-Instruct | Llama | **** | **** | +| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **** | **** | +| google/gemma-3-4b-it | Gemma | **** | **** | +| microsoft/Phi-4-multimodal-instruct | Phi | **** | **** | +| allenai/OLMoE-1B-7B-0924 | OLMoE | **** | **** | +| stabilityai/stablelm-2-1_6b | StableLM | **** | **** | +| CohereForAI/c4ai-command-r-v01 | Command-R | **** | **** | +| huihui-ai/grok-2 | Grok | **** | **** | +| ZhipuAI/chatglm2-6b | ChatGLM | **** | **** | +| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **** | **** | +| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **** | **** | +| xverse/XVERSE-MoE-A36B | XVERSE | **** | **** | +| HuggingFaceTB/SmolLM-1.7B | SmolLM | **** | **** | +| ZhipuAI/glm-4-9b-chat | GLM-4 | **** | **** | +| XiaomiMiMo/MiMo-7B-RL | MiMo | **** | **** | +| arcee-ai/AFM-4.5B-Base | Arcee AFM-4.5B | **** | **** | +| Howeee/persimmon-8b-chat | Persimmon | **** | **** | +| inclusionAI/Ling-lite | Ling | **** | **** | +| ibm-granite/granite-3.1-8b-instruct | Granite | **** | **** | +| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **** | **** | +| databricks/dbrx-instruct | DBRX (Databricks) | **** | **** | +| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **** | **** | +| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **** | **** | +| openbmb/MiniCPM3-4B | MiniCPM (v3, 4B) | **** | **** | +| openai/gpt-oss-120b | GPTOSS | **×** | **×** | ## Multimodal Language Models | Models | Model Family (Variants) | A2 Supported | A3 Supported | |-----------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:| -| Qwen2.5-VL-72B-Instruct-w8a8 | Qwen-VL | **** | **** | -| Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **** | **** | -| Qwen3-VL-8B-Instruct | Qwen-VL | **** | **** | -| Qwen3-VL-4B-Instruct | Qwen-VL | **** | **** | -| Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **** | **** | +| Qwen/Qwen2.5-VL-3B-Instruct | Qwen-VL | **** | **** | +| Qwen/Qwen2.5-VL-72B-Instruct | Qwen-VL | **** | **** | +| Qwen/Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **** | **** | +| Qwen/Qwen3-VL-8B-Instruct | Qwen-VL | **** | **** | +| Qwen/Qwen3-VL-4B-Instruct | Qwen-VL | **** | **** | +| Qwen/Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **** | **** | | deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **** | **** | | deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **** | **** | | openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **** | **** |