chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

[NPU][Doc] updated installation guide for Ascend NPU (#13585)

Browse Source 
Co-authored-by: Howeee <15935120809@163.com>
Co-authored-by: ronnie_zheng <zl19940307@163.com>
This commit is contained in:
VDV1985
2025-12-03 16:58:49 +03:00
committed by GitHub
co-authored by Howeee ronnie_zheng
parent 24903b88ba
commit dc1635023f
6 changed files with 509 additions and 104 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/basic_usage/deepseek_v3.md
+2 -2
View File
@@ -22,7 +22,7 @@ To run DeepSeek V3.1/V3/R1 models, the recommended settings are as follows:
| **Quantized weights ([INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8))** | 16 x A100/800 |
| | 32 x L40S |
| | Xeon 6980P CPU |
| | 2 x Atlas 800I A3 |
| | 4 x Atlas 800I A3 |
| **Quantized weights ([W4A8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8))** | 8 x H20/100, 4 x H200 |
| **Quantized weights ([AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ))** | 8 x H100/800/20 |
| | 8 x A100/A800 |
@@ -72,7 +72,7 @@ Detailed commands for reference:
- [16 x A100 (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
- [32 x L40S (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
- [Xeon 6980P CPU](../platforms/cpu_server.md#example-running-deepseek-r1)
- [2 x Atlas 800I A3 (INT8)](../platforms/ascend_npu.md#running-deepseek-v3)
- [4 x Atlas 800I A3 (int8)](../platforms/ascend_npu_deepseek_example.md#running-deepseek-with-pd-disaggregation-on-4-x-atlas-800i-a3)
### Download Weights
If you encounter errors when starting the server, ensure the weights have finished downloading. It's recommended to download them beforehand or restart multiple times until all weights are downloaded. Please refer to [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base#61-inference-with-deepseek-infer-demo-example-only) official guide to download the weights.
docs/index.rst
+1 -1
View File
@@ -75,7 +75,7 @@ Its core features include:
platforms/cpu_server.md
platforms/tpu.md
platforms/nvidia_jetson.md
platforms/ascend_npu.md
platforms/ascend_npu_support.rst
platforms/xpu.md
.. toctree::
docs/platforms/ascend_npu.md
+70 -101
View File
@@ -1,38 +1,8 @@
# Ascend NPUs
# SGLang installation with NPUs support
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
## System Settings
### CPU performance power scheme
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
```shell
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Make sure changes are applied successfully
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
```
### Disable NUMA balancing
```shell
sudo sysctl -w kernel.numa_balancing=0
# Check
cat /proc/sys/kernel/numa_balancing # shows 0
```
### Prevent swapping out system memory
```shell
sudo sysctl -w vm.swappiness=10
# Check
cat /proc/sys/vm/swappiness # shows 10
```
## Installing SGLang
### Method 1: Installing from source with prerequisites
@@ -46,11 +16,13 @@ conda create --name sglang_npu python=3.11
conda activate sglang_npu
```
#### CANN
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC1 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit)
#### MemFabric Adaptor
_TODO: MemFabric is still a working project yet open sourced til end of year 2025. We will release it as prebuilt wheel package for now._
MemFabric Adaptor is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
If you want to use PD disaggregation mode, you need to install MemFabric Adaptor. MemFabric Adaptor is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
```shell
pip install mf-adapter==1.0.0
@@ -58,24 +30,62 @@ pip install mf-adapter==1.0.0
#### Pytorch and Pytorch Framework Adaptor on Ascend
```shell
PYTORCH_VERSION="2.8.0"
TORCHVISION_VERSION="0.23.0"
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
At the moment NPUGraph optimizations are supported only in `torch_npu==2.6.0.post3` that requires 'torch==2.6.0'.
_TODO: NPUGraph optimizations will be supported in future releases of 'torch_npu' 2.7.1, 2.8.0 and 2.9.0_
PTA_VERSION="2.8.0"
pip install torch-npu==$PTA_VERSION
```shell
PYTORCH_VERSION=2.6.0
TORCHVISION_VERSION=0.21.0
TORCH_NPU_VERSION=2.6.0.post3
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION
```
While there is no resleased versions of 'torch_npu' for 'torch==2.7.1' and 'torch==2.8.0' we provide custom builds of 'torch_npu'. PLATFORM can be 'aarch64' or 'x86_64'
```shell
PLATFORM="aarch64"
PYTORCH_VERSION=2.8.0
TORCHVISION_VERSION=0.23.0
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/torch_npu/torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl
pip install torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl
```
If you are using other versions of 'torch' install 'torch_npu' from sources, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)
#### Triton on Ascend
_Notice:_ We recommend installing triton-ascend from source due to its rapid development, the version on PYPI can't keep up for know. This problem will be solved on Sep. 2025, afterwards `pip install` would be the one and only installing method.
We provide our own implementation of Triton for Ascend.
Please follow Triton-on-Ascend's [installation guide from source](https://gitee.com/ascend/triton-ascend#2%E6%BA%90%E4%BB%A3%E7%A0%81%E5%AE%89%E8%A3%85-triton-ascend) to install the latest `triton-ascend` package.
```shell
BISHENG_NAME="Ascend-BiSheng-toolkit_aarch64_20251121.run"
BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/${BISHENG_NAME}"
wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}"
```
```shell
pip install triton-ascend==3.2.0rc4
```
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
#### SGLang Kernels NPU
We provide our own set of SGL kernels, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md).
#### DeepEP-compatible Library
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).
We are also providing a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).
#### CustomOps
_TODO: to be removed once merged into sgl-kernel-npu._
Additional package with custom operations. DEVICE_TYPE can be "a3" for Atlas A3 server or "910b" for Atlas A2 server.
```shell
DEVICE_TYPE="a3"
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run
chmod a+x ./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run
./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run --quiet --install-path=/usr/local/Ascend/ascend-toolkit/latest/opp
wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl
pip install ./custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl
```
#### Installing SGLang from source
@@ -83,9 +93,7 @@ We are also providing a DeepEP-compatible Library as a drop-in replacement of de
# Use the last release branch
git clone -b v0.5.6 https://github.com/sgl-project/sglang.git
cd sglang
pip install --upgrade pip
rm -vf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
mv python/pyproject_other.toml python/pyproject.toml
pip install -e python[srt_npu]
```
@@ -119,72 +127,33 @@ drun --env "HF_TOKEN=<secret>" \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000
```
## Examples
## System Settings
### Running DeepSeek-V3
### CPU performance power scheme
Running DeepSeek with PD disaggregation on 2 x Atlas 800I A3.
Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
Prefill:
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
drun <image_name> \
python3 -m sglang.launch_server --model-path State_Cloud/DeepSeek-R1-bf16-hfd-w8a8 \
--trust-remote-code \
--attention-backend ascend \
--mem-fraction-static 0.8 \
--quantization w8a8_int8 \
--tp-size 16 \
--dp-size 1 \
--nnodes 1 \
--node-rank 0 \
--disaggregation-mode prefill \
--disaggregation-bootstrap-port 6657 \
--disaggregation-transfer-backend ascend \
--dist-init-addr <PREFILL_HOST_IP>:6688 \
--host <PREFILL_HOST_IP> \
--port 8000
# Make sure changes are applied successfully
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
```
Decode:
### Disable NUMA balancing
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
export HCCL_BUFFSIZE=200
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24
export SGLANG_NPU_USE_MLAPO=1
sudo sysctl -w kernel.numa_balancing=0
drun <image_name> \
python3 -m sglang.launch_server --model-path State_Cloud/DeepSeek-R1-bf16-hfd-w8a8 \
--trust-remote-code \
--attention-backend ascend \
--mem-fraction-static 0.8 \
--quantization w8a8_int8 \
--enable-deepep-moe \
--deepep-mode low_latency \
--tp-size 16 \
--dp-size 1 \
--ep-size 16 \
--nnodes 1 \
--node-rank 0 \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--dist-init-addr <DECODE_HOST_IP>:6688 \
--host <DECODE_HOST_IP> \
--port 8001
# Check
cat /proc/sys/kernel/numa_balancing # shows 0
```
Mini_LB:
### Prevent swapping out system memory
```shell
drun <image_name> \
python -m sglang.srt.disaggregation.launch_lb \
--prefill http://<PREFILL_HOST_IP>:8000 \
--decode http://<DECODE_HOST_IP>:8001 \
--host 127.0.0.1 --port 5000
sudo sysctl -w vm.swappiness=10
# Check
cat /proc/sys/vm/swappiness # shows 10
```
docs/platforms/ascend_npu_deepseek_example.md
+332
View File
@@ -0,0 +1,332 @@
## DeepSeek examples
### Running DeepSeek-V3
#### Running DeepSeek on 1 x Atlas 800I A3.
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export HCCL_BUFFSIZE=1600
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
python3 -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--tp 16 \
--trust-remote-code \
--attention-backend ascend \
--device npu \
--quantization w8a8_int8 \
--watchdog-timeout 9000 \
--host 127.0.0.1 \
--port 6688 \
--cuda-graph-bs 8 16 24 28 32 \
--mem-fraction-static 0.68 \
--max-running-requests 128 \
--context-length 8188 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 6000 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--enable-dp-attention \
--dp-size 4 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--dtype bfloat16
```
#### Running DeepSeek with PD disaggregation on 2 x Atlas 800I A3.
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
Prefill:
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#PD
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1536
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
export TASK_QUEUE_ENABLE=2
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode prefill \
--host $PREFILL_HOST_IP \
--port 8000 \
--disaggregation-bootstrap-port 8996 \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--mem-fraction-static 0.6 \
--attention-backend ascend \
--device npu \
--quantization w8a8_int8 \
--disaggregation-transfer-backend ascend \
--max-running-requests 8 \
--context-length 8192 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 28680 \
--moe-a2a-backend deepep \
--deepep-mode normal \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--dp-size 2 \
--enable-dp-attention \
--disable-shared-experts-fusion \
--dtype bfloat16
```
Decode:
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#PD
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
#Deepep communication settings
export HCCL_BUFFSIZE=720
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
unset TASK_QUEUE_ENABLE
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode decode \
--host $DECODE_HOST_IP \
--port 8001 \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--dp-size 16 \
--mem-fraction-static 0.8 \
--max-running-requests 352 \
--attention-backend ascend \
--device npu \
--quantization w8a8_int8 \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--tokenizer-worker-num 4
```
sglang router:
```shell
python -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
--decode http://<DECODE_HOST_IP>:8001 \
--host 127.0.0.1 \
--port 6688
```
#### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3.
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
Prefill:
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#PD
P_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx')
export ASCEND_MF_STORE_URL="tcp://<P_HOST_IP[0]>:<PORT>"
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1536
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
export TASK_QUEUE_ENABLE=2
for i in "${!P_HOST_IP[@]}";
do
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode prefill \
--host ${P_HOST_IP[$i]} \
--port 8000 \
--disaggregation-bootstrap-port $((8996+$i)) \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--mem-fraction-static 0.81 \
--attention-backend ascend \
--device npu \
--quantization w8a8_int8 \
--disaggregation-transfer-backend ascend \
--max-running-requests 8 \
--context-length 8192 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 28680 \
--moe-a2a-backend deepep \
--deepep-mode normal \
--speculative-algorithm NEXTN \
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 2 \
--enable-dp-attention \
--disable-shared-experts-fusion \
--dtype bfloat16
done
```
Decode:
```shell
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#PD
export ASCEND_MF_STORE_URL="tcp://<P_HOST_IP[0]>:<PORT>"
#Deepep communication settings
export HCCL_BUFFSIZE=600
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
unset TASK_QUEUE_ENABLE
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
D_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx')
for i in "${!D_HOST_IP[@]}";
do
python -m sglang.launch_server
--model-path ${MODEL_PATH} \
--disaggregation-mode decode \
--host ${D_HOST_IP[$i]} \
--port 8001 \
--trust-remote-code \
--dist-init-addr ${D_HOST_IP[0]}:5000 \
--nnodes 2 \
--node-rank $i \
--tp-size 32 \
--dp-size 32 \
--mem-fraction-static 0.8 \
--max-running-requests 832 \
--attention-backend ascend \
--device npu \
--quantization w8a8_int8 \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--cuda-graph-bs 8 10 12 14 16 18 20 22 24 26 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--tokenizer-worker-num 4 \
--prefill-round-robin-balance \
--disable-shared-experts-fusion \
--dtype bfloat16
done
```
sglang router:
```shell
python -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://<P_HOST_IP[0]>:8000 8996 \
--prefill http://<P_HOST_IP[1]>:8000 8997 \
--decode http://<D_HOST_IP[0]>:8001 \
--host 127.0.0.1 \
--port 6688
```
#### test gsm8k
```python
from types import SimpleNamespace
from sglang.test.few_shot_gsm8k import run_eval
def gsm8k():
args = SimpleNamespace(
num_shots=5,
data_path=None,
num_questions=200,
max_new_tokens=512,
parallel=32,
host=f"http://127.0.0.1",
port=6688,
)
metrics = run_eval(args)
print(f"{metrics=}")
print(f"{metrics['accuracy']=}")
if __name__ == "__main__":
gsm8k()
```
docs/platforms/ascend_npu_qwen3_examples.md
+95
View File
@@ -0,0 +1,95 @@
## Qwen3 examples
### Running Qwen3
#### Running Qwen3-32B on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
```shell
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1536
export HCCL_OP_EXPANSION_MODE=AIV
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \
--device npu \
--attention-backend ascend \
--trust-remote-code \
--tp-size 4 \
--model-path Qwen/Qwen3-32B \
--port 30111 \
--mem-fraction-static 0.8
```
#### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B)
```shell
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1536
export HCCL_OP_EXPANSION_MODE=AIV
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export SGLANG_DEEPEP_BF16_DISPATCH=1
export ENABLE_ASCEND_MOE_NZ=1
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \
--device npu \
--attention-backend ascend \
--trust-remote-code \
--tp-size 4 \
--model-path Qwen/Qwen3-30B-A3B \
--port 30111 \
--mem-fraction-static 0.8
```
#### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
```shell
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1536
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export SGLANG_DEEPEP_BF16_DISPATCH=1
export ENABLE_ASCEND_MOE_NZ=1
python -m sglang.launch_server \
--model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
--tp-size 16 \
--trust-remote-code \
--attention-backend ascend \
--device npu \
--watchdog-timeout 9000 \
--port 30111 \
--mem-fraction-static 0.8
```
#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
```shell
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1536
export HCCL_OP_EXPANSION_MODE=AIV
ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \
--device npu \
--enable-multimodal \
--attention-backend ascend \
--mm-attention-backend ascend_attn \
--trust-remote-code \
--tp-size 4 \
--model-path Qwen/Qwen3-VL-8B-Instruct \
--port 30111 \
--mem-fraction-static 0.8
```
docs/platforms/ascend_npu_support.rst
+9
View File
@@ -0,0 +1,9 @@
Ascend NPUs
===============================================================
.. toctree::
:maxdepth: 1
ascend_npu.md
ascend_npu_deepseek_example.md
ascend_npu_qwen3_examples.md