diff --git a/docs/basic_usage/deepseek_v3.md b/docs/basic_usage/deepseek_v3.md index d19934aed..8396b2f50 100644 --- a/docs/basic_usage/deepseek_v3.md +++ b/docs/basic_usage/deepseek_v3.md @@ -22,7 +22,7 @@ To run DeepSeek V3.1/V3/R1 models, the recommended settings are as follows: | **Quantized weights ([INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8))** | 16 x A100/800 | | | 32 x L40S | | | Xeon 6980P CPU | -| | 2 x Atlas 800I A3 | +| | 4 x Atlas 800I A3 | | **Quantized weights ([W4A8](https://huggingface.co/novita/Deepseek-R1-0528-W4AFP8))** | 8 x H20/100, 4 x H200 | | **Quantized weights ([AWQ](https://huggingface.co/QuixiAI/DeepSeek-R1-0528-AWQ))** | 8 x H100/800/20 | | | 8 x A100/A800 | @@ -72,7 +72,7 @@ Detailed commands for reference: - [16 x A100 (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization) - [32 x L40S (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization) - [Xeon 6980P CPU](../platforms/cpu_server.md#example-running-deepseek-r1) -- [2 x Atlas 800I A3 (INT8)](../platforms/ascend_npu.md#running-deepseek-v3) +- [4 x Atlas 800I A3 (int8)](../platforms/ascend_npu_deepseek_example.md#running-deepseek-with-pd-disaggregation-on-4-x-atlas-800i-a3) ### Download Weights If you encounter errors when starting the server, ensure the weights have finished downloading. It's recommended to download them beforehand or restart multiple times until all weights are downloaded. Please refer to [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base#61-inference-with-deepseek-infer-demo-example-only) official guide to download the weights. diff --git a/docs/index.rst b/docs/index.rst index 1eeb41e17..cdd1aa6f7 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -75,7 +75,7 @@ Its core features include: platforms/cpu_server.md platforms/tpu.md platforms/nvidia_jetson.md - platforms/ascend_npu.md + platforms/ascend_npu_support.rst platforms/xpu.md .. toctree:: diff --git a/docs/platforms/ascend_npu.md b/docs/platforms/ascend_npu.md index cd251ba30..a1357a623 100644 --- a/docs/platforms/ascend_npu.md +++ b/docs/platforms/ascend_npu.md @@ -1,38 +1,8 @@ -# Ascend NPUs + +# SGLang installation with NPUs support You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. -## System Settings - -### CPU performance power scheme - -The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended. - -```shell -echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor - -# Make sure changes are applied successfully -cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance -``` - -### Disable NUMA balancing - -```shell -sudo sysctl -w kernel.numa_balancing=0 - -# Check -cat /proc/sys/kernel/numa_balancing # shows 0 -``` - -### Prevent swapping out system memory - -```shell -sudo sysctl -w vm.swappiness=10 - -# Check -cat /proc/sys/vm/swappiness # shows 10 -``` - ## Installing SGLang ### Method 1: Installing from source with prerequisites @@ -46,11 +16,13 @@ conda create --name sglang_npu python=3.11 conda activate sglang_npu ``` +#### CANN + +Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC1 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit) + #### MemFabric Adaptor -_TODO: MemFabric is still a working project yet open sourced til end of year 2025. We will release it as prebuilt wheel package for now._ - -MemFabric Adaptor is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters. +If you want to use PD disaggregation mode, you need to install MemFabric Adaptor. MemFabric Adaptor is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters. ```shell pip install mf-adapter==1.0.0 @@ -58,24 +30,62 @@ pip install mf-adapter==1.0.0 #### Pytorch and Pytorch Framework Adaptor on Ascend -```shell -PYTORCH_VERSION="2.8.0" -TORCHVISION_VERSION="0.23.0" -pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu +At the moment NPUGraph optimizations are supported only in `torch_npu==2.6.0.post3` that requires 'torch==2.6.0'. +_TODO: NPUGraph optimizations will be supported in future releases of 'torch_npu' 2.7.1, 2.8.0 and 2.9.0_ -PTA_VERSION="2.8.0" -pip install torch-npu==$PTA_VERSION +```shell +PYTORCH_VERSION=2.6.0 +TORCHVISION_VERSION=0.21.0 +TORCH_NPU_VERSION=2.6.0.post3 +pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu +pip install torch_npu==$TORCH_NPU_VERSION ``` +While there is no resleased versions of 'torch_npu' for 'torch==2.7.1' and 'torch==2.8.0' we provide custom builds of 'torch_npu'. PLATFORM can be 'aarch64' or 'x86_64' + +```shell +PLATFORM="aarch64" +PYTORCH_VERSION=2.8.0 +TORCHVISION_VERSION=0.23.0 +pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu +wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/torch_npu/torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl +pip install torch_npu-${PYTORCH_VERSION}.post2.dev20251120-cp311-cp311-manylinux_2_28_${PLATFORM}.whl +``` + +If you are using other versions of 'torch' install 'torch_npu' from sources, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md) + #### Triton on Ascend -_Notice:_ We recommend installing triton-ascend from source due to its rapid development, the version on PYPI can't keep up for know. This problem will be solved on Sep. 2025, afterwards `pip install` would be the one and only installing method. +We provide our own implementation of Triton for Ascend. -Please follow Triton-on-Ascend's [installation guide from source](https://gitee.com/ascend/triton-ascend#2%E6%BA%90%E4%BB%A3%E7%A0%81%E5%AE%89%E8%A3%85-triton-ascend) to install the latest `triton-ascend` package. +```shell +BISHENG_NAME="Ascend-BiSheng-toolkit_aarch64_20251121.run" +BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/${BISHENG_NAME}" +wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}" +``` +```shell +pip install triton-ascend==3.2.0rc4 +``` +For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md) + +#### SGLang Kernels NPU +We provide our own set of SGL kernels, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md). #### DeepEP-compatible Library +We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md). -We are also providing a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md). +#### CustomOps +_TODO: to be removed once merged into sgl-kernel-npu._ +Additional package with custom operations. DEVICE_TYPE can be "a3" for Atlas A3 server or "910b" for Atlas A2 server. + +```shell +DEVICE_TYPE="a3" +wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run +chmod a+x ./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run +./CANN-custom_ops-8.2.0.0-$DEVICE_TYPE-linux.aarch64.run --quiet --install-path=/usr/local/Ascend/ascend-toolkit/latest/opp +wget https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/ops/custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl +pip install ./custom_ops-1.0.$DEVICE_TYPE-cp311-cp311-linux_aarch64.whl +``` #### Installing SGLang from source @@ -83,9 +93,7 @@ We are also providing a DeepEP-compatible Library as a drop-in replacement of de # Use the last release branch git clone -b v0.5.6 https://github.com/sgl-project/sglang.git cd sglang - -pip install --upgrade pip -rm -vf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml +mv python/pyproject_other.toml python/pyproject.toml pip install -e python[srt_npu] ``` @@ -119,72 +127,33 @@ drun --env "HF_TOKEN=" \ python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000 ``` -## Examples +## System Settings -### Running DeepSeek-V3 +### CPU performance power scheme -Running DeepSeek with PD disaggregation on 2 x Atlas 800I A3. -Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8). - -Prefill: +The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended. ```shell -export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True -export ASCEND_MF_STORE_URL="tcp://:" +echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor -drun \ - python3 -m sglang.launch_server --model-path State_Cloud/DeepSeek-R1-bf16-hfd-w8a8 \ - --trust-remote-code \ - --attention-backend ascend \ - --mem-fraction-static 0.8 \ - --quantization w8a8_int8 \ - --tp-size 16 \ - --dp-size 1 \ - --nnodes 1 \ - --node-rank 0 \ - --disaggregation-mode prefill \ - --disaggregation-bootstrap-port 6657 \ - --disaggregation-transfer-backend ascend \ - --dist-init-addr :6688 \ - --host \ - --port 8000 +# Make sure changes are applied successfully +cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance ``` -Decode: +### Disable NUMA balancing ```shell -export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True -export ASCEND_MF_STORE_URL="tcp://:" -export HCCL_BUFFSIZE=200 -export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24 -export SGLANG_NPU_USE_MLAPO=1 +sudo sysctl -w kernel.numa_balancing=0 -drun \ - python3 -m sglang.launch_server --model-path State_Cloud/DeepSeek-R1-bf16-hfd-w8a8 \ - --trust-remote-code \ - --attention-backend ascend \ - --mem-fraction-static 0.8 \ - --quantization w8a8_int8 \ - --enable-deepep-moe \ - --deepep-mode low_latency \ - --tp-size 16 \ - --dp-size 1 \ - --ep-size 16 \ - --nnodes 1 \ - --node-rank 0 \ - --disaggregation-mode decode \ - --disaggregation-transfer-backend ascend \ - --dist-init-addr :6688 \ - --host \ - --port 8001 +# Check +cat /proc/sys/kernel/numa_balancing # shows 0 ``` -Mini_LB: +### Prevent swapping out system memory ```shell -drun \ - python -m sglang.srt.disaggregation.launch_lb \ - --prefill http://:8000 \ - --decode http://:8001 \ - --host 127.0.0.1 --port 5000 +sudo sysctl -w vm.swappiness=10 + +# Check +cat /proc/sys/vm/swappiness # shows 10 ``` diff --git a/docs/platforms/ascend_npu_deepseek_example.md b/docs/platforms/ascend_npu_deepseek_example.md new file mode 100644 index 000000000..90e2b9738 --- /dev/null +++ b/docs/platforms/ascend_npu_deepseek_example.md @@ -0,0 +1,332 @@ +## DeepSeek examples + +### Running DeepSeek-V3 + +#### Running DeepSeek on 1 x Atlas 800I A3. + +W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8). + +```shell +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +#Deepep communication settings +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export HCCL_BUFFSIZE=1600 + +#spec overlap +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +#npu acceleration operator +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 + +python3 -m sglang.launch_server \ + --model-path ${MODEL_PATH} \ + --tp 16 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --quantization w8a8_int8 \ + --watchdog-timeout 9000 \ + --host 127.0.0.1 \ + --port 6688 \ + --cuda-graph-bs 8 16 24 28 32 \ + --mem-fraction-static 0.68 \ + --max-running-requests 128 \ + --context-length 8188 \ + --disable-radix-cache \ + --chunked-prefill-size -1 \ + --max-prefill-tokens 6000 \ + --moe-a2a-backend deepep \ + --deepep-mode auto \ + --enable-dp-attention \ + --dp-size 4 \ + --enable-dp-lm-head \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --dtype bfloat16 +``` + +#### Running DeepSeek with PD disaggregation on 2 x Atlas 800I A3. + +W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8). + + +Prefill: + +```shell +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +#PD +export ASCEND_MF_STORE_URL="tcp://:" + +#Deepep communication settings +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export HCCL_BUFFSIZE=1536 + +#npu acceleration operator +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 +export TASK_QUEUE_ENABLE=2 + +python -m sglang.launch_server \ + --model-path ${MODEL_PATH} \ + --disaggregation-mode prefill \ + --host $PREFILL_HOST_IP \ + --port 8000 \ + --disaggregation-bootstrap-port 8996 \ + --trust-remote-code \ + --nnodes 1 \ + --node-rank 0 \ + --tp-size 16 \ + --mem-fraction-static 0.6 \ + --attention-backend ascend \ + --device npu \ + --quantization w8a8_int8 \ + --disaggregation-transfer-backend ascend \ + --max-running-requests 8 \ + --context-length 8192 \ + --disable-radix-cache \ + --chunked-prefill-size -1 \ + --max-prefill-tokens 28680 \ + --moe-a2a-backend deepep \ + --deepep-mode normal \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --dp-size 2 \ + --enable-dp-attention \ + --disable-shared-experts-fusion \ + --dtype bfloat16 +``` + +Decode: + +```shell +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +#PD +export ASCEND_MF_STORE_URL="tcp://:" + +#Deepep communication settings +export HCCL_BUFFSIZE=720 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88 + +#spec overlap +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +#npu acceleration operator +unset TASK_QUEUE_ENABLE +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 + +# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade. +python -m sglang.launch_server \ + --model-path ${MODEL_PATH} \ + --disaggregation-mode decode \ + --host $DECODE_HOST_IP \ + --port 8001 \ + --trust-remote-code \ + --nnodes 1 \ + --node-rank 0 \ + --tp-size 16 \ + --dp-size 16 \ + --mem-fraction-static 0.8 \ + --max-running-requests 352 \ + --attention-backend ascend \ + --device npu \ + --quantization w8a8_int8 \ + --moe-a2a-backend deepep \ + --enable-dp-attention \ + --deepep-mode low_latency \ + --enable-dp-lm-head \ + --cuda-graph-bs 8 10 12 14 16 18 20 22 \ + --disaggregation-transfer-backend ascend \ + --watchdog-timeout 9000 \ + --context-length 8192 \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --prefill-round-robin-balance \ + --disable-shared-experts-fusion \ + --dtype bfloat16 \ + --tokenizer-worker-num 4 +``` + +sglang router: + +```shell +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://:8000 8996 \ + --decode http://:8001 \ + --host 127.0.0.1 \ + --port 6688 +``` + +#### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3. + +W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8). + +Prefill: + +```shell +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +#PD +P_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx') +export ASCEND_MF_STORE_URL="tcp://:" + +#Deepep communication settings +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export HCCL_BUFFSIZE=1536 + +#npu acceleration operator +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 +export TASK_QUEUE_ENABLE=2 + +for i in "${!P_HOST_IP[@]}"; +do + python -m sglang.launch_server \ + --model-path ${MODEL_PATH} \ + --disaggregation-mode prefill \ + --host ${P_HOST_IP[$i]} \ + --port 8000 \ + --disaggregation-bootstrap-port $((8996+$i)) \ + --trust-remote-code \ + --nnodes 1 \ + --node-rank 0 \ + --tp-size 16 \ + --mem-fraction-static 0.81 \ + --attention-backend ascend \ + --device npu \ + --quantization w8a8_int8 \ + --disaggregation-transfer-backend ascend \ + --max-running-requests 8 \ + --context-length 8192 \ + --disable-radix-cache \ + --chunked-prefill-size -1 \ + --max-prefill-tokens 28680 \ + --moe-a2a-backend deepep \ + --deepep-mode normal \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 1 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 2 \ + --dp-size 2 \ + --enable-dp-attention \ + --disable-shared-experts-fusion \ + --dtype bfloat16 +done +``` + +Decode: + +```shell +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +#PD +export ASCEND_MF_STORE_URL="tcp://:" + +#Deepep communication settings +export HCCL_BUFFSIZE=600 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78 + +#spec overlap +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +#npu acceleration operator +unset TASK_QUEUE_ENABLE +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 + +D_HOST_IP=('xx,xx,xx,xx' 'xx,xx,xx,xx') + +for i in "${!D_HOST_IP[@]}"; +do + python -m sglang.launch_server + --model-path ${MODEL_PATH} \ + --disaggregation-mode decode \ + --host ${D_HOST_IP[$i]} \ + --port 8001 \ + --trust-remote-code \ + --dist-init-addr ${D_HOST_IP[0]}:5000 \ + --nnodes 2 \ + --node-rank $i \ + --tp-size 32 \ + --dp-size 32 \ + --mem-fraction-static 0.8 \ + --max-running-requests 832 \ + --attention-backend ascend \ + --device npu \ + --quantization w8a8_int8 \ + --moe-a2a-backend deepep \ + --enable-dp-attention \ + --deepep-mode low_latency \ + --enable-dp-lm-head \ + --cuda-graph-bs 8 10 12 14 16 18 20 22 24 26 \ + --disaggregation-transfer-backend ascend \ + --watchdog-timeout 9000 \ + --context-length 8192 \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 2 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 3 \ + --tokenizer-worker-num 4 \ + --prefill-round-robin-balance \ + --disable-shared-experts-fusion \ + --dtype bfloat16 +done +``` + +sglang router: + +```shell +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://:8000 8996 \ + --prefill http://:8000 8997 \ + --decode http://:8001 \ + --host 127.0.0.1 \ + --port 6688 +``` + +#### test gsm8k + +```python +from types import SimpleNamespace +from sglang.test.few_shot_gsm8k import run_eval + +def gsm8k(): + args = SimpleNamespace( + num_shots=5, + data_path=None, + num_questions=200, + max_new_tokens=512, + parallel=32, + host=f"http://127.0.0.1", + port=6688, + ) + metrics = run_eval(args) + print(f"{metrics=}") + print(f"{metrics['accuracy']=}") +if __name__ == "__main__": + gsm8k() +``` diff --git a/docs/platforms/ascend_npu_qwen3_examples.md b/docs/platforms/ascend_npu_qwen3_examples.md new file mode 100644 index 000000000..3c35b1ba9 --- /dev/null +++ b/docs/platforms/ascend_npu_qwen3_examples.md @@ -0,0 +1,95 @@ +## Qwen3 examples + +### Running Qwen3 + +#### Running Qwen3-32B on 1 x Atlas 800I A3. + +Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B) + +```shell +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=1536 +export HCCL_OP_EXPANSION_MODE=AIV + +ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \ + --device npu \ + --attention-backend ascend \ + --trust-remote-code \ + --tp-size 4 \ + --model-path Qwen/Qwen3-32B \ + --port 30111 \ + --mem-fraction-static 0.8 +``` + +#### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3. + +Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B) + +```shell +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=1536 +export HCCL_OP_EXPANSION_MODE=AIV +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export SGLANG_DEEPEP_BF16_DISPATCH=1 +export ENABLE_ASCEND_MOE_NZ=1 + +ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \ + --device npu \ + --attention-backend ascend \ + --trust-remote-code \ + --tp-size 4 \ + --model-path Qwen/Qwen3-30B-A3B \ + --port 30111 \ + --mem-fraction-static 0.8 +``` + +#### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3. + +Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) + +```shell +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=1536 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export SGLANG_DEEPEP_BF16_DISPATCH=1 +export ENABLE_ASCEND_MOE_NZ=1 + +python -m sglang.launch_server \ + --model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \ + --tp-size 16 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --port 30111 \ + --mem-fraction-static 0.8 +``` + +#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3. + +Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct) + +```shell +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=1536 +export HCCL_OP_EXPANSION_MODE=AIV + +ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server \ + --device npu \ + --enable-multimodal \ + --attention-backend ascend \ + --mm-attention-backend ascend_attn \ + --trust-remote-code \ + --tp-size 4 \ + --model-path Qwen/Qwen3-VL-8B-Instruct \ + --port 30111 \ + --mem-fraction-static 0.8 +``` diff --git a/docs/platforms/ascend_npu_support.rst b/docs/platforms/ascend_npu_support.rst new file mode 100644 index 000000000..1437515f8 --- /dev/null +++ b/docs/platforms/ascend_npu_support.rst @@ -0,0 +1,9 @@ +Ascend NPUs +=============================================================== + +.. toctree:: + :maxdepth: 1 + + ascend_npu.md + ascend_npu_deepseek_example.md + ascend_npu_qwen3_examples.md