diff --git a/docs/platforms/ascend_npu.md b/docs/platforms/ascend_npu.md
index f1820a8e1..369933333 100644
--- a/docs/platforms/ascend_npu.md
+++ b/docs/platforms/ascend_npu.md
@@ -3,6 +3,27 @@
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
+## Component Version Mapping For SGLang
+| Component | Version | Obtain Way |
+|-------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| HDK | 25.2.1 | [link](https://support.huawei.com/carrier/productNewOffering?col=product&path=PBI1-262732867/PBI1-262735886/PBI1-262735910/PBI1-261410188/PBI1-252764743&pVR=PBI1-263550357&pC=PBI1-264360782&pSPC=PBI1-266220744&resTab=SW) |
+| CANN | 8.3.rc2 | [Obtain Images](#obtain-cann-image) |
+| Pytorch Adapter | 7.3.0 | [link](https://gitcode.com/Ascend/pytorch/releases) |
+| MemFabric | 0.1.0 | [link](https://gitcode.com/Ascend/memfabric_hybrid/releases) |
+| Triton | 3.2.0.dev2025112116 | [link](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/triton_ascend-3.2.0.dev2025112116-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl) |
+| Bisheng | 20251121 | [link](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/Ascend-BiSheng-toolkit_aarch64_20251121.run) |
+| SGLang NPU Kernel | NA | [link](https://github.com/sgl-project/sgl-kernel-npu/releases) |
+
+
+### Obtain CANN Image
+You can obtain the dependency of a specified version of CANN through an image.
+```shell
+# for Atlas 800I A3 and Ubuntu OS
+docker pull quay.io/ascend/cann:8.3.rc2-a3-ubuntu22.04-py3.11
+# for Atlas 800I A2 and Ubuntu OS
+docker pull quay.io/ascend/cann:8.3.rc2-910b-ubuntu22.04-py3.11
+```
+
## Preparing the Running Environment
### Method 1: Installing from source with prerequisites
diff --git a/docs/platforms/ascend_npu_best_practice.md b/docs/platforms/ascend_npu_best_practice.md
new file mode 100644
index 000000000..d78547d86
--- /dev/null
+++ b/docs/platforms/ascend_npu_best_practice.md
@@ -0,0 +1,3386 @@
+# Best Practice on Ascend NPU
+
+This section describes the best practice data of mainstream LLM models such as DeepSeek and Qwen on the Ascend Npu.If
+you encounter issues or have any questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
+
+## DeepSeek Series Models
+
+### Low Latency
+
+| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | TPOT(ms) | Output TPS(per card) | Configuration |
+|---------------|----------|---------|---------------|-----------|--------------|----------|----------------------|-----------------------------------------------------------------|
+| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K-1.6K | W8A8 | 19.81 | 36.906 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p32-6k1.6k-20ms) |
+| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K-1K | W8A8 | 19.77 | 35.625 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p32-3.9k1k-20ms) |
+| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | W8A8 | 19.92 | 36.980 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p32-3.5k1.5k-20ms) |
+| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1K | W8A8 | 19.52 | 36.344 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p32-3.5k1k-20ms) |
+| Deepseek-V3.2 | Atlas 800I A3 | 32 | PD Separation | 64K-1K | W8A8 | 25.36 | 14.74 | [Optimal Configuration](#ds-v32-atlas-800i-a3-p32-64k1k-30ms) |
+
+### High Throughput
+
+| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | TPOT(ms) | Output TPS(per card) | Configuration |
+|-------------|----------|---------|---------------|-----------|--------------|----------|----------------------|-----------------------------------------------------------|
+| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | W8A8 | 48.10 | 396.796 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p32-3.5k1.5k-50ms) |
+| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | W4A8 | 49.67 | 528.375 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p8-2k2k-50ms) |
+| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K-2K | W4A8 | 47.76 | 452.227 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p16-2k2k-50ms) |
+| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1K | W4A8 | 49.77 | 312.077 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p8-3.5k1.5k-50ms) |
+| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K-1K | W4A8 | 49.43 | 361.500 | [Optimal Configuration](#ds-r1-atlas-800i-a3-p16-3.5k1.5k-50ms) |
+
+## Qwen Series Models
+
+### Low Latency
+
+| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | TPOT(ms) | Output TPS(per card) | Configuration |
+|------------|---------------|---------|-------------|---------|--------------|----------|----------------------|------------------------------------------------------------------|
+| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 11K-1K | BF16 | 9.70 | 11.690 | [Optimal Configuration](#qwen3-235b-atlas-800i-a3-p8-11k1k-10ms) |
+| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K-1.5K | W8A8 | 16.87 | 311.750 | [Optimal Configuration](#qwen3-32b-atlas-800i-a3-p4-6k1.5k-18ms) |
+| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K-1.5K | BF16 | 9.46 | 25.850 | [Optimal Configuration](#qwen3-32b-atlas-800i-a3-p4-4k1.5k-11ms) |
+| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K-4K | BF16 | 12.27 | 9.955 | [Optimal Configuration](#qwen3-32b-atlas-800i-a3-p8-18k4k-12ms) |
+| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K-1.5K | W8A8 | 16.46 | 296 | [Optimal Configuration](#qwen3-32b-atlas-800i-a2-p8-6k1.5k-18ms) |
+| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K-1.5K | BF16 | 10.18 | 12 | [Optimal Configuration](#qwen3-32b-atlas-800i-a2-p8-4k1.5k-11ms) |
+
+### High Throughput
+
+| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | TPOT(ms) | Output TPS(per card) | Configuration |
+|------------|---------------|---------|---------------|-----------|--------------|----------|----------------------|----------------------------------------------------------------------|
+| Qwen3-235B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | W8A8 | 40.75 | 467.416 | [Optimal Configuration](#qwen3-235b-atlas-800i-a3-p24-3.5k1.5k-50ms) |
+| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | 51.51 | 477.625 | [Optimal Configuration](#qwen3-235b-atlas-800i-a3-p8-3.5k1.5k-50ms) |
+| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | W8A8 | 54.78 | 790.071 | [Optimal Configuration](#qwen3-235b-atlas-800i-a3-p8-2k2k-50ms) |
+| Qwen3-235B | Atlas 800I A3 | 16 | PD Mixed | 2K-2K | W8A8 | 50.12 | 519.625 | [Optimal Configuration](#qwen3-235b-atlas-800i-a3-p16-2k2k-50ms) |
+| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | W8A8 | 49.20 | 707.500 | [Optimal Configuration](#qwen3-32b-atlas-800i-a3-p2-3.5k1.5k-50ms) |
+| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K-2K | W8A8 | 48.30 | 986.150 | [Optimal Configuration](#qwen3-32b-atlas-800i-a3-p2-2k2k-50ms) |
+| Qwen3-30B | Atlas 800I A3 | 1 | PD Mixed | 3.5K-1.5K | W8A8 | 44.35 | 3166.030 | [Optimal Configuration](#qwen3-30b-atlas-800i-a3-p1-3.5k1.5k-50ms) |
+| Qwen3-480B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | W8A8 | 48.27 | 266.250 | [Optimal Configuration](#qwen3-480b-atlas-800i-a3-p24-3.5k1.5k-50ms) |
+| Qwen3-480B | Atlas 800I A3 | 16 | PD Mixed | 3.5K-1.5K | W8A8 | 50.34 | 289.813 | [Optimal Configuration](#qwen3-480b-atlas-800i-a3-p16-3.5k1.5k-50ms) |
+| Qwen3-480B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | 48.20 | 187.500 | [Optimal Configuration](#qwen3-480b-atlas-800i-a3-p8-3.5k1.5k-50ms) |
+| Qwen3-Next | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | W8A8 | 49.91 | 702.83 | [Optimal Configuration](#qwen3-next-atlas-800i-a3-p2-3.5k1.5k-50ms) | |
+| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | 48.97 | 348.75 | [Optimal Configuration](#qwen3-32b-atlas-800i-a2-p8-3.5k1.5k-50ms) |
+| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K-2K | W8A8 | 45.88 | 512 | [Optimal Configuration](#qwen3-32b-atlas-800i-a2-p8-2k2k-50ms) |
+
+## Optimal Configuration
+
+
+
+### Deepseek-R1 Atlas 800I A3-32Card PD Separation 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+export SGLANG_SET_CPU_AFFINITY=1
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+
+export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
+
+P_IP=('your prefill ip1' 'your prefill ip2')
+
+D_IP=('your prefill ip1' 'your prefill ip2')
+
+MODEL_PATH=xxx
+
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_USE_FIA_NZ=1
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+# prefill
+for i in "${!P_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+ then
+ echo "${P_IP[$i]}"
+ export HCCL_BUFFSIZE=1536
+ export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+ export TASK_QUEUE_ENABLE=2
+
+ export HCCL_SOCKET_IFNAME=lo
+ export GLOO_SOCKET_IFNAME=lo
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
+ --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
+ --tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
+ --disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
+ --chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
+ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
+ --dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
+ NODE_RANK=$i
+ break
+ fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+ then
+ echo "${D_IP[$i]}"
+ export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+ export SGLANG_ENABLE_SPEC_V2=1
+ export HCCL_BUFFSIZE=650
+ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
+ export TASK_QUEUE_ENABLE=0
+ export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
+ export HCCL_SOCKET_IFNAME=xxx
+ export GLOO_SOCKET_IFNAME=xxx
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
+ --port 8001 --trust-remote-code --dist-init-addr DIP1:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
+ --mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
+ --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
+ --cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
+ --tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 \
+ --load-balance-method decode_round_robin
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://P_IP:8000 8998 \
+ --prefill http://P_IP:8000 8999 \
+ --decode http://D_IP:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 832 --random-input-len 3500 --random-output-len 1500 --num-prompts 3328 --random-range-ratio 1
+```
+
+```angular2html
+============ Serving Benchmark Result ============
+Backend: sqlang
+Traffic request rate: inf
+Max request concurrency: 832
+Successful requests: 3328
+Benchmark duration (s): 393.15
+Total input tokens: 11202837
+Total input text tokens: 11202837
+Total input vision tokens: 0
+Total generated tokens (retokenized): 4992000
+Request throughput (req/s): 8.465
+Output token throughput (tok/s): 12697.4868
+Total token throughput (tok/s): 41192.654
+Concurrency: 696.859
+-----------------End-to-End Latency----------------
+Mean E2E Latency (ms): 82322.96
+Median E2E Latency (ms): 82395.75
+----------------Time to First Token----------------
+Mean TTFT (ms): 10170.34
+Median TTFT (ms): 8273.99
+P99 TTFT (ms): 29787.93
+----------------Output Token (excl. 1st token)----------------
+Mean PTOT (ms): 48.10
+Median PTOT (ms): 49.12
+P99 PTOT (ms): 55.02
+----------------Inter-Token Latency----------------
+Mean ITL (ms): 122.53
+Median ITL (ms): 120.08
+P99 ITL (ms): 278.32
+Max ITL (ms): 838.11
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-32Card PD Separation 6K-1.6K 20ms
+
+
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+export SGLANG_SET_CPU_AFFINITY=1
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
+
+P_IP=('your prefill ip1' 'your prefill ip2')
+
+D_IP=('your decode ip1' 'your decode ip2')
+
+MODEL_PATH=xxx
+
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_USE_FIA_NZ=1
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+# prefill
+for i in "${!P_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+ then
+ echo "${P_IP[$i]}"
+ export HCCL_BUFFSIZE=1536
+ export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+ export TASK_QUEUE_ENABLE=2
+
+ export HCCL_SOCKET_IFNAME=lo
+ export GLOO_SOCKET_IFNAME=lo
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
+ --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
+ --tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
+ --disaggregation-transfer-backend ascend --max-running-requests 4 --context-length 8192 --disable-radix-cache \
+ --chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
+ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
+ --dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
+ NODE_RANK=$i
+ break
+ fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+ then
+ echo "${D_IP[$i]}"
+ export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+ export SGLANG_ENABLE_SPEC_V2=1
+ export HCCL_BUFFSIZE=650
+ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12
+ export TASK_QUEUE_ENABLE=0
+ export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
+ export HCCL_SOCKET_IFNAME=xxx
+ export GLOO_SOCKET_IFNAME=xxx
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
+ --port 8001 --trust-remote-code --dist-init-addr DIP1:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 16 \
+ --mem-fraction-static 0.75 --max-running-requests 32 --attention-backend ascend --device npu --quantization modelslim \
+ --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
+ --cuda-graph-bs 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 \
+ --load-balance-method decode_round_robin
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://P_IP:8000 8998 \
+ --prefill http://P_IP:8000 8999 \
+ --decode http://D_IP:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 6000 --random-output-len 1600 --num-prompts 32 --random-range-ratio 1
+```
+
+```angular2html
+========== Serving Benchmark Result ==========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 32
+Successful requests: 32
+Benchmark duration (s): 43.34
+Total input tokens: 192000
+Total input text tokens: 192000
+Total input vision tokens: 0
+Total generated tokens: 51200
+Total generated tokens (retokenized): 50985
+Request throughput (req/s): 0.74
+Input token throughput (tok/s): 4429.93
+Output token throughput (tok/s): 1181.31
+Peak output token throughput (tok/s): 1667.00
+Peak concurrent requests: 32
+Total token throughput (tok/s): 5611.24
+Concurrency: 27.19
+Accept length: 2.59
+---------- End-To-End Latency ----------
+Mean E2E Latency (ms): 36921.96
+Median E2E Latency (ms): 36785.75
+---------- Time to First Token ----------
+Mean TTFT (ms): 5149.34
+Median TTFT (ms): 5146.99
+P99 TTFT (ms): 9015.93
+---------- Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 19.81
+Median TPOT (ms): 20.22
+P99 TPOT (ms): 23.74
+---------- Inter-Token Latency ----------
+Mean ITL (ms): 19.81
+Median ITL (ms): 19.08
+P95 ITL (ms): 29.95
+P99 ITL (ms): 55.85
+Max ITL (ms): 123.03
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-32Card PD Separation 3.9K-1K 20ms
+
+#### Model Deployment
+
+Please Turn to [Model Deployment](#ds-r1-low-latency-deploy)
+
+#### Benchmark
+
+```bash
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 3900 --random-output-len 1000 --num-prompts 32 --random-range-ratio 1
+```
+
+```angular2html
+========== Serving Benchmark Result ==========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 32
+Successful requests: 32
+Benchmark duration (s): 28.07
+Total input tokens: 124800
+Total input text tokens: 0
+Total generated vision tokens: 32000
+Total generated tokens (retokenized): 31809
+Request throughput (req/s): 1.14
+Input token throughput (tok/s): 4446.13
+Output token throughput (tok/s): 1148.03
+Peak output token throughput (tok/s): 1702.00
+Peak concurrent requests: 32
+Output token throughput (tok/s): 5586.16
+Concurrency: 25.82
+Accept length: 2.88
+---------- End-to-End Latency ----------
+Mean E2E Latency (ms): 22650.75
+Median E2E Latency (ms): 22649.04
+---------- Time to First Token ----------
+Mean TTFT (ms): 2901.77
+Median TTFT (ms): 2357.79
+P99 TTFT (ms): 4240.04
+---------- Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 19.77
+Median TPOT (ms): 19.68
+P99 TPOT (ms): 23.90
+---------- Inter-Token Latency ----------
+Mean ITL (ms): 19.77
+Median ITL (ms): 19.93
+P95 ITL (ms): 29.72
+P99 ITL (ms): 57.10
+Max ITL (ms): 122.71
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-32Card PD Separation 3.5K-1.5K 20ms
+
+#### Model Deployment
+
+Please turn to [Model Deployment](#ds-r1-low-latency-deploy)
+
+#### Benchmark
+
+```bash
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 3500 --random-output-len 1500 --num-prompts 32 --random-range-ratio 1
+```
+
+```angular2html
+========== Serving Benchmark Result ==========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 32
+Successful requests: 32
+Benchmark duration (s): 40.56
+Total input tokens: 112000
+Total input text tokens: 112000
+Total input vision tokens: 0
+Total generated tokens: 48000
+Total generated tokens (retokenized): 47787
+Request throughput (req/s): 0.79
+Input token throughput (tok/s): 2761.16
+Output token throughput (tok/s): 1183.35
+Peak output token throughput (tok/s): 1665.00
+Peak concurrent requests: 32
+Total token throughput (tok/s): 3944.51
+Concurrency: 25.66
+Accept length: 2.84
+---------- End-to-End Latency ----------
+Mean E2E Latency (ms): 32526.35
+Median E2E Latency (ms): 32200.67
+---------- Time to First Token ----------
+Mean TTFT (ms): 2671.91
+Median TTFT (ms): 2208.48
+P99 TTFT (ms): 3960.99
+---------- Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 19.92
+Median TPOT (ms): 19.91
+P99 TPOT (ms): 24.64
+---------- Inter-Token Latency ----------
+Mean ITL (ms): 19.92
+Median ITL (ms): 18.91
+P95 ITL (ms): 30.06
+P99 ITL (ms): 57.76
+Max ITL (ms): 189.36
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-32Card PD Separation 3.5K-1K 20ms
+
+#### Model Deployment
+
+Please turn to [Model Deployment](#ds-r1-low-latency-deploy)
+
+#### Benchmark
+
+```bash
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 3500 --random-output-len 1000 --num-prompts 32 --random-range-ratio 1
+```
+
+```angular2html
+Backend:
+================ Serving Benchmark Result ===================
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 32
+Successful requests: 32
+Benchmark duration (s): 27.50
+Total input tokens: 112000
+Total input text tokens: 0
+Total input vision tokens: 0
+Total generated tokens: 32000
+Total generated tokens (retokenized): 31837
+Request throughput (req/s): 1.16
+Input token throughput (tok/s): 4072.55
+Output token throughput (tok/s): 1163.59
+Peak output token throughput (tok/s): 1692.00
+Peak concurrent requests: 32
+Total token throughput (tok/s): 5236.14
+Concurrency: 25.78
+Accept length: 2.84
+------------------ End-To-End Latency ------------------
+Mean E2E Latency (ms): 22154.99
+Median E2E Latency (ms): 22262.70
+------------------ Time to First Token ------------------
+Mean TTFT (ms): 2655.44
+Median TTFT (ms): 2205.10
+P99 TTFT (ms): 4446.79
+------------------ Time per Output Token (excl. 1st token) ------------------
+Mean TPOT (ms): 19.52
+Median TPOT (ms): 19.82
+P99 TPOT (ms): 25.19
+------------------ Inter-Token Latency ------------------
+Mean ITL (ms): 19.52
+Median ITL (ms): 18.81
+P99 ITL (ms): 29.56
+Max ITL (ms): 65.46
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-8Card PD Mixed 2K-2K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
+
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64
+export HCCL_BUFFSIZE=1600
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=10
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512
+
+MODEL_PATH=xxx
+
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_USE_FIA_NZ=1
+export ENABLE_MOE_NZ=1
+
+python3 -m sglang.launch_server --model-path ${MODEL_PATH} \
+--tp 16 \
+--trust-remote-code \
+--attention-backend ascend \
+--device npu \
+--quantization modelslim \
+--watchdog-timeout 9000 \
+--host 127.0.0.1 --port 6699 \
+--cuda-graph-bs 4 8 16 \
+--mem-fraction-static 0.74 \
+--max-running-requests 256 \
+--disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 1500 \
+--moe-a2a-backend deepep --deepep-mode auto \
+--enable-dp-attention --dp-size 16 --enable-dp-lm-head \
+--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+--dtype bfloat16
+
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 256 --random-input-len 2048 --random-output-len 2048 --num-prompts 1024 --random-range-ratio 1
+```
+
+```angular2html
+============= Serving Benchmark Result ================
+Backend sglang
+Traffic request rate: inf
+Max request concurrency: 255
+Successful requests: 1024
+Benchmark duration (s): 496.11
+Total input tokens: 2097152
+Total input text tokens: 2097152
+Total input vision tokens: 0
+Total generated tokens: 2097152
+Total generated tokens (retokenized): 2091432
+Request throughput (req/s): 2.06
+Input token throughput (tok/s): 4227.20
+Output token throughput (tok/s): 4227.20
+Peak output token throughput (tok/s): 7098.00
+Peak concurrent requests: 275
+Total token throughput (tok/s): 8454.39
+Concurrency: 231.08
+Accept length: 2.82
+====================End-To-End Latency============
+Mean E2E Latency (ms): 111956.34
+Median E2E Latency (ms): 113251.95
+====================Time to First Token===========
+Mean TTFT (ms): 10273.92
+Median TTFT (ms): 8592.03
+P99 TTFT (ms): 28535.07
+=== Time per Output Token (excl. 1st tok) ===
+Mean TPOT (ms): 49.67
+Median TPOT (ms): 50.30
+P99 TPOT (ms): 66.12
+====================Inter-Token Latency===========
+Mean ITL (ms): 49.67
+Median ITL (ms): 34.83
+P99 ITL (ms): 75.91
+Max ITL (ms): 18621.17
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-16Card PD Separation 2K-2K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+
+export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24667"
+
+P_IP=('your prefill ip1')
+
+D_IP=('your decode ip1')
+
+MODEL_PATH=xxx
+
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_USE_FIA_NZ=1
+export ENABLE_MOE_NZ=1
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+# prefill
+for i in "${!P_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+ then
+ echo "${P_IP[$i]}"
+ export HCCL_BUFFSIZE=1536
+ export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+ export TASK_QUEUE_ENABLE=2
+
+ export HCCL_SOCKET_IFNAME=lo
+ export GLOO_SOCKET_IFNAME=lo
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
+ --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
+ --tp-size 16 --mem-fraction-static 0.6 --attention-backend ascend --device npu --quantization modelslim \
+ --disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
+ --chunked-prefill-size 32768 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
+ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
+ --dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16
+ NODE_RANK=$i
+ break
+ fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+ then
+ echo "${D_IP[$i]}"
+ export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+ export SGLANG_ENABLE_SPEC_V2=1
+ export HCCL_BUFFSIZE=720
+ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
+ export TASK_QUEUE_ENABLE=0
+ export HCCL_SOCKET_IFNAME=xxx
+ export GLOO_SOCKET_IFNAME=xxx
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
+ --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \
+ --mem-fraction-static 0.8 --max-running-requests 384 --attention-backend ascend --device npu --quantization modelslim \
+ --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \
+ --cuda-graph-bs 8 10 12 14 16 18 20 22 24 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
+ --load-balance-method decode_round_robin
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://P_IP:8000 8998 \
+ --decode http://D_IP:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 400 --random-input-len 2048 --random-output-len 2048 --num-prompts 3200 --random-range-ratio 1 --request-rate 8
+```
+
+```angular2html
+==================== Serving Benchmark Result ====================
+Backend: sglang
+Traffic request rate: 8.0
+Max request concurrency: 400
+Successful requests: 3200
+Benchmark duration (s): 905.74
+Total input tokens: 6553600
+Total input text tokens: 6553600
+Total input vision tokens: 0
+Total generated tokens (retokenized): 6534368
+Request throughput (req/s): 3.53
+Input token throughput (tok/s): 7235.64
+Output token throughput (tok/s): 7235.64
+Peak output token throughput (tok/s): 9112.00
+Peak concurrent requests: 411
+Total token throughput (tok/s): 14471.28
+Concurrency: 363.54
+Accept length: 2.86
+------------------------ End-to-End Latency ------------------------
+Mean E2E Latency (ms): 102896.35
+Median E2E Latency (ms): 104894.80
+------------------------ Time to First Token -----------------------
+Mean TTFT (ms): 5138.54
+Median TTFT (ms): 3356.93
+P99 TTFT (ms): 20223.25
+------------------ Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 47.76
+Median TPOT (ms): 48.84
+P99 TPOT (ms): 62.18
+------------------------ Inter-Token Latency -----------------------
+Mean ITL (ms): 47.76
+Median ITL (ms): 40.66
+P95 ITL (ms): 83.83
+P99 ITL (ms): 147.10
+Max ITL (ms): 674.12
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-8Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+export STREAMS_PER_DEVICE=32
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=36
+export HCCL_BUFFSIZE=1600
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_USE_FIA_NZ=1
+export ENABLE_MOE_NZ=1
+
+MODEL_PATH=xxx
+
+python3 -m sglang.launch_server --model-path ${MODEL_PATH} \
+--tp 16 \
+--trust-remote-code \
+--attention-backend ascend \
+--device npu \
+--quantization modelslim \
+--watchdog-timeout 9000 \
+--host 127.0.0.1 --port 6699 \
+--cuda-graph-bs 8 16 24 28 32 36 \
+--mem-fraction-static 0.71 \
+--max-running-requests 144 \
+--context-length 8188 --disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 9000 \
+--moe-a2a-backend deepep --deepep-mode auto \
+--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
+--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+--dtype bfloat16
+
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 144 --random-input-len 3500 --random-output-len 1500 --num-prompts 576 --random-range-ratio 1
+```
+
+```angular2html
+--------------- Serving Benchmark Result -----------------
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 144
+Successful requests: 576
+Benchmark duration (s): 346.07
+Total input tokens: 2016000
+Total input text tokens: 2016000
+Total input vision tokens: 0
+Total generated tokens: 864000
+Total generated tokens (retokenized): 861521
+Request throughput (req/s): 1.66
+Input token throughput (tok/s): 5825.46
+Output token throughput (tok/s): 2496.62
+Peak output token throughput (tok/s): 4724.00
+Peak concurrent requests: 157
+Total token throughput (tok/s): 8322.08
+Concurrency: 135.48
+Accept length: 2.83
+-----------------------------------------------------------
+-------- End-to-End Latency --------
+Mean E2E Latency (ms): 81395.45
+Median E2E Latency (ms): 82243.44
+-------- Time to First Token --------
+Mean TTFT (ms): 6788.93
+Median TTFT (ms): 3911.21
+P99 TTFT (ms): 26898.02
+-------- Time per Output Token (excl. 1st token) --------
+Mean TPOT (ms): 49.77
+Median TPOT (ms): 50.36
+P99 TPOT (ms): 68.57
+-------- Inter-Token Latency --------
+Mean ITL (ms): 49.77
+Median ITL (ms): 29.30
+P95 ITL (ms): 92.28
+P99 ITL (ms): 509.53
+Max ITL (ms): 19821.92
+```
+
+
+
+### Deepseek-R1 Atlas 800I A3-16Card PD Separation 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+
+export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24667"
+
+P_IP=('your prefill ip1')
+
+D_IP=('your decode ip1')
+
+MODEL_PATH=xxx
+
+export SGLANG_NPU_USE_MLAPO=1
+export SGLANG_USE_FIA_NZ=1
+export ENABLE_MOE_NZ=1
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+# prefill
+for i in "${!P_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+ then
+ echo "${P_IP[$i]}"
+ export HCCL_BUFFSIZE=1536
+ export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+ export TASK_QUEUE_ENABLE=2
+
+ export HCCL_SOCKET_IFNAME=lo
+ export GLOO_SOCKET_IFNAME=lo
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
+ --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
+ --tp-size 16 --mem-fraction-static 0.6 --attention-backend ascend --device npu --quantization modelslim \
+ --disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
+ --chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
+ --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
+ --dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16
+ NODE_RANK=$i
+ break
+ fi
+done
+
+# decode
+for i in "${!D_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+ then
+ echo "${D_IP[$i]}"
+ export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+ export SGLANG_ENABLE_SPEC_V2=1
+ export HCCL_BUFFSIZE=720
+ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96
+ export TASK_QUEUE_ENABLE=0
+ export HCCL_SOCKET_IFNAME=xxx
+ export GLOO_SOCKET_IFNAME=xxx
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
+ --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \
+ --mem-fraction-static 0.8 --max-running-requests 384 --attention-backend ascend --device npu --quantization modelslim \
+ --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \
+ --cuda-graph-bs 8 10 12 14 16 18 20 22 24 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
+ --load-balance-method decode_round_robin
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://P_IP:8000 8998 \
+ --decode http://D_IP:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 384 --random-input-len 3500 --random-output-len 1500 --num-prompts 1536 --random-range-ratio 1
+```
+
+```angular2html
+=======================================
+Serving Benchmark Result
+=======================================
+Backend: sglang
+Traffic request rate: 8.0
+Max request concurrency: 384
+Successful requests: 1536
+Benchmark duration (s): 398.28
+Total input tokens: 5376000
+Total input text tokens: 5376000
+Total input vision tokens: 0
+Total generated tokens (retokenized): 2297661
+Request throughput (req/s): 3.86
+Input token throughput (tok/s): 13498.06
+Output token throughput (tok/s): 5784.88
+Peak output token throughput (tok/s): 8359.00
+Peak concurrent requests: 395
+Total token throughput (tok/s): 19282.95
+Concurrency: 323.46
+Accept length: 2.87
+---------------------------------------
+End-to-End Latency
+---------------------------------------
+Mean E2E Latency (ms): 83871.76
+Median E2E Latency (ms): 84454.65
+---------------------------------------
+Time to First Token
+---------------------------------------
+Mean TTFT (ms): 9784.06
+Median TTFT (ms): 6542.10
+P99 TTFT (ms): 31487.26
+---------------------------------------
+Time per Output Token (excl. 1st token)
+---------------------------------------
+Mean TPOT (ms): 49.42
+Median TPOT (ms): 50.13
+P99 TPOT (ms): 66.06
+---------------------------------------
+Inter-Token Latency
+---------------------------------------
+Mean ITL (ms): 49.43
+Median ITL (ms): 41.87
+P99 ITL (ms): 85.42
+Max ITL (ms): 672.12
+```
+
+
+
+### Deepseek-V3.2 Atlas 800I A3-32Card PD Separation 64K-1K 30ms
+
+#### Model Deployment
+
+Deploy Prefill Instance
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH}
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+
+export HCCL_BUFFSIZE=1024
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=5
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512
+
+MODEL_PATH=xxx
+
+export SGLANG_NPU_USE_MLAPO=1
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_NPU_USE_MULTI_STREAM=1
+export HCCL_OP_EXPANSION_MODE=AIV
+
+IPs=('your prefill ip1' 'your prefill ip2')
+
+# get IP in current node
+LOCAL_HOST=`hostname -I|awk -F " " '{print$1}'`
+echo "LOCAL_HOST = " ${LOCAL_HOST}
+# get node index
+for i in "${!IPs[@]}";
+do
+ echo "LOCAL_HOST=${LOCAL_HOST}, IPs[${i}]=${IPs[$i]}"
+ if [ "$LOCAL_HOST" == "${IPs[$i]}" ]; then
+ echo "Node Rank : ${i}"
+ VC_TASK_INDEX=$i
+ break
+ fi
+done
+
+IFNAMES=('xxx' 'xxx')
+
+export HCCL_SOCKET_IFNAME=${IFNAMES[$VC_TASK_INDEX]}
+export GLOO_SOCKET_IFNAME=${HCCL_SOCKET_IFNAME}
+echo "HCCL_SOCKET_IFNAME : ${HCCL_SOCKET_IFNAME}"
+nnodes=${#IPs[@]}
+tp_size=`expr 16 \* ${nnodes}`
+export ASCEND_MF_STORE_URL=tcp://${IPs[0]}:24667
+
+python3 -m sglang.launch_server --model-path ${MODEL_PATH} \
+--tp $tp_size \
+--trust-remote-code \
+--attention-backend ascend \
+--device npu \
+--watchdog-timeout 9000 \
+--host ${IPs[$VC_TASK_INDEX]} --port 8000 \
+--mem-fraction-static 0.73 \
+--disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 68000 \
+--max-running-requests 1 \
+--moe-a2a-backend deepep --deepep-mode normal \
+--quantization modelslim \
+--disaggregation-transfer-backend ascend \
+--disaggregation-mode prefill \
+--disable-cuda-graph \
+--nnodes $nnodes --node-rank $VC_TASK_INDEX \
+--disaggregation-bootstrap-port 8995 \
+--enable-nsa-prefill-context-parallel --moe-dense-tp-size 1 \
+--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
+--dist-init-addr ${IPs[0]}:10000
+```
+
+Deploy Decode Instance
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH}
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+export ASCEND_HOME_PATH=/usr/local/Ascend/ascend-toolkit/latest
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+
+MODEL_PATH=xxx
+
+export SGLANG_NPU_USE_MULTI_STREAM=1
+export SGLANG_NPU_USE_MLAPO=1
+export HCCL_OP_EXPANSION_MODE=AIV
+export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
+export TASK_QUEUE_ENABLE=0
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+IPs=('your decode ip1' 'your decode ip2')
+
+export prefill_ip=your prefill ip1
+# get IP in current node
+LOCAL_HOST=`hostname -I|awk -F " " '{print$1}'`
+echo "LOCAL_HOST = " ${LOCAL_HOST}
+# get node index
+for i in "${!IPs[@]}";
+do
+ echo "LOCAL_HOST=${LOCAL_HOST}, IPs[${i}]=${IPs[$i]}"
+ if [ "$LOCAL_HOST" == "${IPs[$i]}" ]; then
+ echo "Node Rank : ${i}"
+ VC_TASK_INDEX=$i
+ break
+ fi
+done
+
+IFNAMES=('xxx' 'xxx')
+
+export HCCL_SOCKET_IFNAME=${IFNAMES[$VC_TASK_INDEX]}
+export GLOO_SOCKET_IFNAME=${HCCL_SOCKET_IFNAME}
+nnodes=${#IPs[@]}
+tp_size=`expr 16 \* ${nnodes}`
+export ASCEND_MF_STORE_URL=tcp://${prefill_ip}:24667
+
+CHUNKED_SIZE=65536
+DP=8
+export HCCL_BUFFSIZE=400
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8
+
+python3 -m sglang.launch_server --model-path ${MODEL_PATH} \
+--tp $tp_size \
+--dp ${DP} \
+--ep $tp_size \
+--moe-dense-tp-size 1 \
+--enable-dp-attention \
+--enable-dp-lm-head \
+--trust-remote-code \
+--attention-backend ascend \
+--device npu \
+--watchdog-timeout 9000 \
+--host ${IPs[$VC_TASK_INDEX]} --port 8001 \
+--mem-fraction-static 0.79 \
+--disable-radix-cache \
+--chunked-prefill-size -1 --max-prefill-tokens 68000 \
+--max-running-requests 32 \
+--cuda-graph-max-bs 4 \
+--moe-a2a-backend deepep \
+--deepep-mode low_latency \
+--quantization modelslim \
+--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+--disaggregation-transfer-backend ascend \
+--disaggregation-mode decode \
+--prefill-round-robin-balance \
+--load-balance-method round_robin \
+--nnodes $nnodes --node-rank $VC_TASK_INDEX \
+--dist-init-addr ${IPs[0]}:10000 --load-balance-method decode_round_robin
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://PIP1:8000 8998 \
+ --prefill http://PIP2:8000 8999 \
+ --decode http://DIP1:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 64000 --random-output-len 1000 --num-prompts 64 --random-range-ratio 1 --request-rate 0.25
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: 0.25
+Max request concurrency: 32
+Successful requests: 64
+Benchmark duration (s): 416.83
+Total input tokens: 4096000
+Total input text tokens: 4096000
+Total generated tokens (retokenized): 196608
+Total generated tokens (retokenized): 196530
+Request throughput (req/s): 0.15
+Input token throughput (tok/s): 98866.58
+Output token throughput (tok/s): 471.68
+Peak output token throughput (tok/s): 821.00
+Peak concurrent requests: 33
+Total token throughput (tok/s): 10298.25
+Concurrency: 17.99
+Accept length: 3.18
+----------------- End-to-End Latency -----------------
+Mean E2E Latency (ms): 117181.11
+Median E2E Latency (ms): 100840.21
+Mean TTFT (ms): 39313.10
+Median TTFT (ms): 19198.87
+P99 TTFT (ms): 153478.05
+Mean Time per Output Token (excl. 1st token) (ms): 25.25
+Mean TPOT (ms): 25.52
+P99 TPOT (ms): 29.63
+----------------- Inter-Token Latency -----------------
+Mean ITL (ms): 25.36
+Median ITL (ms): 20.47
+P95 ITL (ms): 40.62
+P99 ITL (ms): 54.63
+Max ITL (ms): 248.81
+```
+
+
+
+### Qwen3-235B Atlas 800I A3-24Card PD Separation 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
+
+MODEL_PATH=xxx
+export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24667"
+P_IP=('your prefill ip1')
+D_IP=('your decode ip1' 'your decode ip2')
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_DP_ROUND_ROBIN=1
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+
+for i in "${!P_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+ then
+ echo "${P_IP[$i]}"
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+ source /usr/local/Ascend/nnal/atb/set_env.sh
+ export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
+ export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
+ export HCCL_BUFFSIZE=4300
+ export TASK_QUEUE_ENABLE=2
+ export HCCL_SOCKET_IFNAME=lo
+ export GLOO_SOCKET_IFNAME=lo
+ export STREAMS_PER_DEVICE=32
+ export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+
+ # P节点
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill \
+ --host ${P_IP[$i]} --port 8000 --disaggregation-bootstrap-port 8995 --trust-remote-code \
+ --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --mem-fraction-static 0.6 \
+ --disable-radix-cache \
+ --ep-dispatch-algorithm static --init-expert-location /mnt/share/chenxu/hot_map/expert_distribution_recorder_1765615213.9892833.pt \
+ --attention-backend ascend --device npu --quantization modelslim --disaggregation-transfer-backend ascend \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --speculative-draft-model-quantization unquant \
+ --max-running-requests 128 --chunked-prefill-size 262144 --max-prefill-tokens 262144 \
+ --enable-dp-attention \
+ --moe-a2a-backend deepep --deepep-mode normal --dtype bfloat16
+ NODE_RANK=$i
+ break
+ fi
+done
+
+
+for i in "${!D_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+ then
+ echo "${D_IP[$i]}"
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+ source /usr/local/Ascend/nnal/atb/set_env.sh
+ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24
+ export HCCL_BUFFSIZE=512
+ export HCCL_SOCKET_IFNAME=data0.3001
+ export GLOO_SOCKET_IFNAME=data0.3001
+ export STREAMS_PER_DEVICE=32
+
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode \
+ --host ${D_IP[$i]} --port 8001 --trust-remote-code \
+ --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --mem-fraction-static 0.83 --max-running-requests 768 \
+ --attention-backend ascend --device npu --quantization modelslim --enable-dp-attention \
+ --moe-a2a-backend ascend_fuseep --cuda-graph-bs 6 8 12 15 18 20 22 24 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-draft-model-quantization unquant \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --dist-init-addr xxx:5000 \
+ --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --prefill-round-robin-balance --enable-dp-lm-head --dtype bfloat16 --tokenizer-worker-num 4 \
+ --load-balance-method decode_round_robin
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://PIP:8000 8995 \
+ --decode http://DIP:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 3072 --random-range-ratio 1
+```
+
+```angular2htm
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 768
+Successful requests: 3072
+Benchmark duration (s): 410.76
+Total input tokens: 10752000
+Total input text tokens: 10752000
+Total generated tokens (retokenized): 46080000
+Request throughput (req/s): 7.48
+Input token throughput (tok/s): 26176.11
+Output token throughput (tok/s): 11218.33
+Peak output token throughput (tok/s): 15636.00
+Peak concurrent requests: 787
+Total token throughput (tok/s): 37394.45
+Concurrency: 658.11
+Accept length: 2.87
+----------------- End-to-End Latency -----------------
+Mean E2E Latency (ms): 87995.10
+Median E2E Latency (ms): 80821.00
+Time to First Token
+Mean TTFT (ms): 26924.13
+Median TTFT (ms): 19415.39
+P99 TTFT (ms): 84458.36
+--- Mean Time per Output Token (excl. 1st token) -----
+Mean TPOT (ms): 40.74
+Median TPOT (ms): 38.67
+P99 TPOT (ms): 76.83
+----------------- Inter-Token Latency -----------------
+Mean ITL (ms): 40.75
+Median ITL (ms): 33.99
+P99 ITL (ms): 87.67
+Max ITL (ms): 788.71
+```
+
+
+
+### Qwen3-235B Atlas 800I A3-8Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=1600
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 272 --context-length 8192 --dtype bfloat16 \
+ --chunked-prefill-size 32768 --max-prefill-tokens 32768 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto --speculative-draft-model-quantization unquant \
+ --tp 16 --dp-size 16 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.8 --cuda-graph-bs 3 4 6 8 10 12 13 14 15 16 17
+
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 288 --random-input-len 3500 --random-output-len 1500 --num-prompts 1088 --random-range-ratio 1
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 272
+Successful requests: 1088
+Benchmark duration (s): 427.04
+Total input tokens: 3808000
+Total input text tokens: 0
+Total input vision tokens: 0
+Total generated tokens: 1632000
+Total generated tokens (retokenized): 1631408
+Request throughput (req/s): 2.55
+Input token throughput (tok/s): 8917.16
+Output token throughput (tok/s): 3821.64
+Peak output token throughput (tok/s): 9304.00
+Peak concurrent requests: 313
+Total token throughput (tok/s): 12738.79
+Concurrency: 239.32
+Accept length: 2.83
+=== End-to-End Latency ===
+Mean E2E Latency (ms): 93931.66
+Median E2E Latency (ms): 88251.02
+=== Time to First Token ===
+Mean TTFT (ms): 16721.52
+Median TTFT (ms): 15137.02
+P99 TTFT (ms): 42314.45
+=== Time per Output Token (excl. 1st token) ===
+Mean TPOT (ms): 51.51
+Median TPOT (ms): 48.14
+P99 TPOT (ms): 109.71
+=== Inter-Token Latency ===
+Mean ITL (ms): 51.51
+Median ITL (ms): 25.60
+P95 ITL (ms): 103.61
+P99 ITL (ms): 461.44
+Max ITL (ms): 35459.74
+```
+
+
+
+### Qwen3-235B Atlas 800I A3-8Card PD Mixed 2K-2K 100ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=1200
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 576 --context-length 8192 --dtype bfloat16 \
+ --chunked-prefill-size 32768 --max-prefill-tokens 458880 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto --speculative-draft-model-quantization unquant \
+ --tp 16 --dp-size 16 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.81 --cuda-graph-bs 8 16 20 24 32 36
+
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 576 --random-input-len 2000 --random-output-len 2000 --num-prompts 576 --random-range-ratio 1
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 576
+Successful requests: 576
+Benchmark duration (s): 182.26
+Total input tokens: 1152000
+Total input text tokens: 1152000
+Total input vision tokens: 0
+Total generated tokens: 1152000
+Total generated tokens (retokenized): 1151728
+Request throughput (req/s): 3.16
+Input token throughput (tok/s): 6320.57
+Output token throughput (tok/s): 6320.57
+Peak output token throughput (tok/s): 14513.00
+Peak concurrent requests: 576
+Total token throughput (tok/s): 12641.14
+Concurrency: 358.87
+Accept length: 2.94
+
+--------End-to-End Latency--------
+Mean E2E Latency (ms): 113557.20
+Median E2E Latency (ms): 113056.54
+
+--------Time to First Token--------
+Mean TTFT (ms): 4049.94
+Median TTFT (ms): 4004.61
+P99 TTFT (ms): 5926.98
+
+--------Time per Output Token (excl. 1st token)--------
+Mean TPOT (ms): 54.78
+Median TPOT (ms): 54.67
+P99 TPOT (ms): 79.92
+
+--------Inter-Token Latency--------
+Mean ITL (ms): 54.78
+Median ITL (ms): 38.01
+P95 ITL (ms): 82.41
+P99 ITL (ms): 152.56
+Max ITL (ms): 40909.44
+```
+
+
+
+### Qwen3-235B Atlas 800I A3-8Card PD Mixed 2K-2K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=1600
+export HCCL_SOCKET_IFNAME=xxx
+export GLOO_SOCKET_IFNAME=xxx
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 480 --context-length 8192 --dtype bfloat16 \
+ --chunked-prefill-size -1 --max-prefill-tokens 4096 --speculative-draft-model-quantization unquant \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto \
+ --tp 16 --dp-size 16 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.75 --cuda-graph-bs 6 8 10 12 15 18 28 30
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 480 --random-input-len 2048 --random-output-len 2048 --num-prompts 480 --random-range-ratio 1
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 480
+Successful requests: 480
+Benchmark duration (s): 166.51
+Total input tokens: 983040
+Total input text tokens: 983040
+Total input vision tokens: 0
+Total generated tokens: 982844
+Total generated tokens (retokenized): 982844
+Request throughput (req/s): 2.88
+Input token throughput (tok/s): 5903.96
+Output token throughput (tok/s): 5903.96
+Peak output token throughput (tok/s): 13839.00
+Peak concurrent requests: 480
+Total token throughput (tok/s): 11807.92
+Concurrency: 297.23
+Accept length: 2.95
+
+---End-to-End Latency---
+Mean E2E Latency (ms): 103105.60
+Median E2E Latency (ms): 101298.94
+Mean TTFT (ms): 4457.97
+Median TTFT (ms): 4381.36
+P99 TTFT (ms): 6589.31
+Mean Time per Output Token (excl. 1st token): 48.19
+Median TPOt (ms): 47.40
+P99 TPOt (ms): 70.18
+
+---Inter-Token Latency---
+Mean ITL (ms): 48.19
+Median ITL (ms): 33.67
+P99 ITL (ms): 76.04
+Max ITL (ms): 33993.29
+```
+
+
+
+### Qwen3-235B Atlas 800I A3-16Card PD Mixed 2K-2K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
+
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=1600
+export HCCL_SOCKET_IFNAME=xxx
+export GLOO_SOCKET_IFNAME=xxx
+export HCCL_OP_EXPANSION_MODE="AIV"
+
+MIX_IP=('IP1' 'IP2')
+
+for i in "${!MIX_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${MIX_IP[$i]}" || "$LOCAL_HOST2" == "${MIX_IP[$i]}" ]];
+ then
+ echo "${MIX_IP[$i]}"+
+ export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+ export SGLANG_ENABLE_SPEC_V2=1
+ export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1
+
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode \
+ --host 127.0.0.1 --port 7439 --trust-remote-code \
+ --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --mem-fraction-static 0.8 --max-running-requests 768 \
+ --attention-backend ascend --device npu --quantization modelslim --enable-dp-attention \
+ --moe-a2a-backend ascend_fuseep --cuda-graph-bs 6 8 10 12 18 24 \
+ --dist-init-addr 141.61.105.131:5000 --chunked-prefill-size 32768 --max-prefill-tokens 458880 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --prefill-round-robin-balance --enable-dp-lm-head --dtype bfloat16 --tokenizer-worker-num 4
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 768 --random-input-len 2000 --random-output-len 2000 --num-prompts 768 --random-range-ratio 1
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 768
+Successful requests: 768
+Benchmark duration (s): 199.18
+Total input tokens: 1572864
+Total input text tokens: 0
+Total input vision tokens: 0
+Total generated tokens (retokenized): 1572864
+Request throughput (req/s): 4.06
+Input token throughput (tok/s): 8314.05
+Output token throughput (tok/s): 8314.05
+Peak output token throughput (tok/s): 20655.00
+Peak concurrent requests: 768
+Total token throughput (tok/s): 16628.10
+Concurrency: 529.54
+Accept length: 3.20
+
+=== End-to-End Latency ===
+Mean E2E Latency (ms): 130442.47
+Median E2E Latency (ms): 127654.62
+
+=== Time to First Token ===
+Mean TTFT (ms): 27838.48
+Median TTFT (ms): 29338.37
+P99 TTFT (ms): 48169.79
+
+=== Time per Output Token (excl. 1st token) ===
+Mean TPOT (ms): 50.12
+Median TPOT (ms): 49.54
+P99 TPOT (ms): 72.80
+
+=== Inter-Token Latency ===
+Mean ITL (ms): 50.12
+Median ITL (ms): 30.66
+P99 ITL (ms): 133.63
+Max ITL (ms): 36393.04
+```
+
+
+
+### Qwen3-235B Atlas 800I A3-8Card PD Mixed 11K-1K 10ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=1600
+export HCCL_SOCKET_IFNAME=xxx
+export GLOO_SOCKET_IFNAME=xxx
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 1 --dtype bfloat16 \
+ --chunked-prefill-size -1 --max-prefill-tokens 16384 --speculative-draft-model-quantization unquant \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
+ --disable-radix-cache --enable-dp-lm-head \
+ --tp 16 --mem-fraction-static 0.78 --cuda-graph-bs 1
+
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 1 --random-input-len 11000 --random-output-len 1000 --num-prompts 1 --random-range-ratio 1
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 1
+Successful requests: 10690
+Benchmark duration (s): 10.69
+Total input tokens: 11000
+Total input text tokens: 0
+Total generated tokens: 1000
+Total generated tokens (retokenized): 1000
+Request throughput (req/s): 0.09
+Input token throughput (tok/s): 1028.75
+Output token throughput (tok/s): 93.52
+Peak output token throughput (tok/s): 110.00
+Peak concurrent requests: 1
+Total token throughput (tok/s): 1122.27
+Concurrency: 1.60
+Accept length: 4.03
+--- End-to-End Latency ---
+Mean E2E Latency (ms): 10661.60
+Median E2E Latency (ms): 10661.60
+--- Time to First Token ---
+Mean TTFT (ms): 973.98
+Median TTFT (ms): 973.98
+P99 TTFT (ms): 973.98
+--- Time per Output Token (excl. 1st token) ---
+Mean TPOT (ms): 9.70
+Median TPOT (ms): 9.70
+P99 TPOT (ms): 9.70
+--- Inter-Token Latency ---
+Mean ITL (ms): 9.70
+Median ITL (ms): 9.28
+P95 ITL (ms): 13.75
+P99 ITL (ms): 14.99
+Max ITL (ms): 19.83
+```
+
+
+
+### Qwen3-32B Atlas 800I A3-4Card PD Mixed 6K-1.5K 18ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=xxx
+export GLOO_SOCKET_IFNAME=xxx
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 32 \
+ --disable-radix-cache \
+ --chunked-prefill-size 32768 --max-prefill-tokens 65536 --speculative-draft-model-quantization unquant \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
+ --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 8 16 24 32 --dtype bfloat16
+
+```
+
+#### Benchmark
+
+We tested it based on the GSM8K dataset.
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 32
+Successful requests: 32
+Benchmark duration (s): 38.48
+Total input tokens: 199000
+Total input text tokens: 192000
+Total input vision tokens: 0
+Total generated tokens (retokenized): 48000
+Request throughput (req/s): 0.83
+Input token throughput (tok/s): 4988.98
+Output token throughput (tok/s): 1247.24
+Peak output token throughput (tok/s): 2245.00
+Peak concurrent requests: 32
+Total token throughput (tok/s): 6236.22
+Concurrency: 28.44
+Accept length: 2.15
+
+=== End-to-End Latency ===
+Mean E2E Latency (ms): 34202.32
+Median E2E Latency (ms): 33902.93
+
+=== Time to First Token ===
+Mean TTFT (ms): 8908.53
+Median TTFT (ms): 8557.18
+P99 TTFT (ms): 12373.42
+
+=== Time per Output Token (excl. 1st token) ===
+Mean TPOT (ms): 16.87
+Median TPOT (ms): 16.75
+P99 TPOT (ms): 22.71
+
+=== Inter-Token Latency ===
+Mean ITL (ms): 16.87
+Median ITL (ms): 10.77
+P95 ITL (ms): 32.01
+P99 ITL (ms): 32.62
+Max ITL (ms): 7912.53
+```
+
+
+
+### Qwen3-32B Atlas 800I A3-4Card PD Mixed 4K-1.5K 11ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=xxx
+export GLOO_SOCKET_IFNAME=xxx
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export DISABLE_EAGLE3_QUANT=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu \
+ --max-running-requests 32 \
+ --disable-radix-cache \
+ --base-gpu-id 4 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
+ --chunked-prefill-size -1 --max-prefill-tokens 65536 \
+ --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 1 4 6 12 18 24 30 32 --dtype bfloat1
+
+```
+
+#### Benchmark
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --random-range-ratio 1 -max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 1
+Successful requests: 58.03
+Benchmark duration (s): 16000
+Total input tokens: 16000
+Total input text tokens: 0
+Total input vision tokens: 0
+Total generated tokens (retokenized): 6000
+Request throughput (req/s): 0.07
+Input token throughput (tok/s): 275.74
+Output token throughput (tok/s): 103.40
+Peak output token throughput (tok/s): 132.00
+Peak concurrent requests: 2
+Total token throughput (tok/s): 379.14
+Concurrency: 1.00
+Accept length: 2.03
+
+---End-to-End Latency---
+Mean E2E Latency (ms): 14502.58
+Median E2E Latency (ms): 14752.39
+
+---Time to First Token---
+Mean TTFT (ms): 317.68
+Median TTFT (ms): 315.90
+P99 TTFT (ms): 324.07
+
+---Time per Output Token (excl. 1st token)---
+Mean TPOT (ms): 9.46
+Median TPOT (ms): 9.63
+P99 TPOT (ms): 10.87
+
+---Inter-Token Latency---
+Mean ITL (ms): 9.46
+Median ITL (ms): 6.64
+P95 ITL (ms): 19.78
+P99 ITL (ms): 20.73
+Max ITL (ms): 33.84
+```
+
+
+
+### Qwen3-32B Atlas 800I A3-8Card PD Mixed 18K-4K 12ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export DISABLE_EAGLE3_QUANT=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu \
+ --max-running-requests 1 \
+ --disable-radix-cache \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
+ --chunked-prefill-size -1 --max-prefill-tokens 65536 \
+ --tp-size 16 --mem-fraction-static 0.72 --cuda-graph-bs 1 --dtype bfloat16
+```
+
+#### Benchmark
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 18000 --random-input-len 4000 --num-prompts 1
+```
+
+```angular2html
+========= Serving Benchmark Result =========
+Max request concurrency: 1
+Successful requests: 1
+Benchmark duration (s): 50.22
+Total input tokens: 18000
+Total input text tokens: 18000
+Total input vision tokens: 0
+Total generated tokens: 4000
+Total generated tokens (retokenized): 4000
+Request throughput (req/s): 0.02
+Input token throughput (tok/s): 358.40
+Output token throughput (tok/s): 79.64
+Peak output token throughput (tok/s): 100.00
+Peak concurrent requests: 1
+Total token throughput (tok/s): 438.05
+Concurrency: 1.00
+Accept length: 2.15
+----------------------------------------
+---End-to-End Latency---
+Mean E2E Latency (ms): 50204.39
+Median E2E Latency (ms): 50204.39
+---Time to First Token---
+Mean TTFT (ms): 1132.62
+Median TTFT (ms): 1132.62
+P99 TTFT (ms): 1132.62
+---Time per Output Token (excl. 1st token)---
+Mean TPOT (ms): 12.27
+Median TPOT (ms): 12.27
+P99 TPOT (ms): 12.27
+---Inter-Token Latency---
+Mean ITL (ms): 12.27
+Median ITL (ms): 7.52
+P99 ITL (ms): 27.23
+Max ITL (ms): 32.56
+```
+
+
+
+### Qwen3-32B Atlas 800I A3-2Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 78 \
+ --disable-radix-cache --speculative-draft-model-quantization unquant \
+ --chunked-prefill-size -1 --max-prefill-tokens 65536 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --tp-size 4 --mem-fraction-static 0.72 --cuda-graph-bs 16 32 64 68 72 78 --dtype bfloat16
+```
+
+#### Benchmark
+
+We tested it based on the GSM8K dataset.
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 78
+Successful requests: 312
+Benchmark duration (s): 330.60
+Total input tokens: 1092000
+Total input text tokens: 1092000
+Total input vision tokens: 0
+Total generated tokens: 468000
+Total generated tokens (retokenized): 467994
+Request throughput (req/s): 0.94
+Input token throughput (tok/s): 3303.13
+Output token throughput (tok/s): 1415.63
+Peak output token throughput (tok/s): 2344.00
+Peak concurrent requests: 83
+Peak token throughput (tok/s): 4718.76
+Concurrency: 74.73
+Accept length: 2.05
+-------------------------------------------
+Mean E2E Latency (ms): 79188.24
+Median E2E Latency (ms): 78524.54
+-------------------------------------------
+Time to First Token
+Mean TTFT (ms): 5371.45
+Median TTFT (ms): 485.89
+P99 TTFT (ms): 24619.48
+-------------------------------------------
+Time per Output Token (excl. 1st token)
+Mean TPOT (ms): 49.24
+Median TPOT (ms): 49.50
+P99 TPOT (ms): 66.33
+-------------------------------------------
+Inter-Token Latency
+Mean ITL (ms): 49.24
+Median ITL (ms): 32.50
+P95 ITL (ms): 127.73
+P99 ITL (ms): 380.22
+Max ITL (ms): 12362.57
+```
+
+
+
+### Qwen3-32B Atlas 800I A3-2Card PD Mixed 2K-2K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export DISABLE_EAGLE3_QUANT=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 120 \
+ --disable-radix-cache \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --chunked-prefill-size -1 --max-prefill-tokens 49152 \
+ --tp-size 4 --mem-fraction-static 0.7 --cuda-graph-bs 54 60 66 72 78 84 90 108 114 120 --dtype bfloat16
+
+```
+
+#### Benchmark
+
+We tested it based on the GSM8K dataset.
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 120
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 120
+Successful requests: 120
+Benchmark duration (s): 121.69
+Total input tokens: 240000
+Total input text tokens: 240000
+Total input vision tokens: 0
+Total generated tokens: 240000
+Total generated tokens (retokenized): 240000
+Request throughput (req/s): 0.99
+Input token throughput (tok/s): 1972.30
+Output token throughput (tok/s): 1972.30
+Peak output token throughput (tok/s): 2983.00
+Peak concurrent requests: 120
+Total token throughput (tok/s): 3944.60
+Concurrency: 111.22
+Accept length: 1.69
+
+---End-to-End Latency---
+Mean E2E Latency (ms): 112785.77
+Median E2E Latency (ms): 112758.61
+
+---Time to First Token---
+Mean TTFT (ms): 16144.60
+Median TTFT (ms): 17229.95
+P99 TTFT (ms): 21366.64
+
+---Time per Output Token (excl. 1st token)---
+Mean TPOT (ms): 48.34
+Median TPOT (ms): 48.06
+P99 TPOT (ms): 55.74
+
+---Inter-Token Latency---
+Mean ITL (ms): 48.34
+Median ITL (ms): 39.36
+P99 ITL (ms): 81.71
+Max ITL (ms): 12772.77
+```
+
+
+
+### Qwen3-30B Atlas 800I A3-1Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export DISABLE_EAGLE3_QUANT=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 192 \
+ --disable-radix-cache \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --chunked-prefill-size -1 --max-prefill-tokens 32768 \
+ --tp-size 2 --mem-fraction-static 0.86 --cuda-graph-bs 42 88 96 132 144 156 172 178 192 --dtype bfloat16
+```
+
+#### Benchmark
+
+```angular2html
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 156 --random-input-len 3500 --random-output-len 1500 --num-prompts 624 --random-range-ratio 1
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: 1nf
+Max request concurrency: 156
+Successful requests: 624
+Benchmark duration (s): 295.64
+Total input tokens: 2184000
+Total input text tokens: 0
+Total input vision tokens: 0
+Total generated tokens: 936000
+Total generated tokens (retokenized): 935999
+Request throughput (req/s): 2.11
+Input token throughput (tok/s): 7387.40
+Output token throughput (tok/s): 3166.03
+Peak output token throughput (tok/s): 5631.00
+Peak concurrent requests: 166
+Total token throughput (tok/s): 10553.42
+Concurrency: 148.81
+Accept length: 3.17
+
+---End-to-End Latency---
+Mean E2E Latency (ms): 70504.60
+Median E2E Latency (ms): 70144.81
+
+---Time to First Token---
+Mean TTFT (ms): 4020.19
+Median TTFT (ms): 734.31
+P99 TTFT (ms): 22554.77
+
+---Time per Output Token (excl. 1st token)---
+Mean TPOT (ms): 44.35
+Median TPOT (ms): 44.46
+P99 TPOT (ms): 62.78
+
+---Inter-Token Latency---
+Mean ITL (ms): 44.35
+Median ITL (ms): 26.29
+P99 ITL (ms): 118.72
+P95 ITL (ms): 240.57
+Max ITL (ms): 18382.51
+```
+
+
+
+### Qwen3-480B Atlas 800I A3-24Card PD Separation 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
+
+MODEL_PATH=xxx
+export ASCEND_MF_STORE_URL="tcp://PIP:24667"
+P_IP=('PIP')
+D_IP=('DIP1' 'DIP2')
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+
+for i in "${!P_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
+ then
+ echo "${P_IP[$i]}"
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+ source /usr/local/Ascend/nnal/atb/set_env.sh
+ export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
+ export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
+ export HCCL_BUFFSIZE=4300
+ export TASK_QUEUE_ENABLE=2
+ export HCCL_SOCKET_IFNAME=lo
+ export GLOO_SOCKET_IFNAME=lo
+ export STREAMS_PER_DEVICE=32
+ export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill \
+ --host ${P_IP[$i]} --port 8000 --disaggregation-bootstrap-port 8995 --trust-remote-code \
+ --nnodes 1 --node-rank $i --tp-size 16 --dp-size 2 --mem-fraction-static 0.6 \
+ --disable-radix-cache \
+ --attention-backend ascend --device npu --quantization modelslim --disaggregation-transfer-backend ascend \
+ --max-running-requests 128 --chunked-prefill-size 65536 --max-prefill-tokens 262144 \
+ --enable-dp-attention \
+ --moe-a2a-backend deepep --deepep-mode normal --dtype bfloat16
+ NODE_RANK=$i
+ break
+ fi
+done
+
+for i in "${!D_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
+ then
+ echo "${D_IP[$i]}"
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+ source /usr/local/Ascend/nnal/atb/set_env.sh
+ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=72
+ export HCCL_BUFFSIZE=512
+ export HCCL_SOCKET_IFNAME=xxx
+ export GLOO_SOCKET_IFNAME=xxx
+ export STREAMS_PER_DEVICE=32
+
+ python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode \
+ --host ${D_IP[$i]} --port 8001 --trust-remote-code \
+ --nnodes 2 --node-rank $i --tp-size 32 --dp-size 4 --mem-fraction-static 0.73 --max-running-requests 384 \
+ --attention-backend ascend --device npu --quantization modelslim --enable-dp-attention \
+ --moe-a2a-backend ascend_fuseep --cuda-graph-bs 16 32 48 56 64 72 80 88 96 \
+ --dist-init-addr DIP1:5000 \
+ --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
+ --prefill-round-robin-balance --enable-dp-lm-head --dtype bfloat16 --tokenizer-worker-num 4 --load-balance-method decode_round_robin
+ NODE_RANK=$i
+ break
+ fi
+done
+
+```
+
+```shell
+export SGLANG_DP_ROUND_ROBIN=1
+python -m sglang_router.launch_router \
+ --pd-disaggregation \
+ --policy cache_aware \
+ --prefill http://PIP:8000 8995 \
+ --decode http://DIP:8001 \
+ --host 127.0.0.1 \
+ --port 6688 \
+ --mini-lb
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 410 --random-input-len 3500 --random-output-len 1500 --num-prompts 1640 --random-range-ratio 1 --request-rate 8
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: 8.0
+Max request concurrency: 410
+Successful requests: 1640
+Benchmark duration (s): 384.95
+Total input tokens: 5740000
+Total input text tokens: 5740000
+Total input vision tokens: 0
+Total generated tokens: 2460000
+Total generated tokens (retokenized): 2449627
+Request throughput (req/s): 4.26
+Output token throughput (tok/s): 14911.21
+Peak output token throughput (tok/s): 6390.52
+Peak concurrent requests: 429
+Total token throughput (tok/s): 21301.73
+Concurrency: 327.33
+
+---End-to-End Latency---
+Mean E2E Latency (ms): 76831.52
+Median E2E Latency (ms): 77111.46
+
+---Time to First Token---
+Mean TTFT (ms): 4470.95
+Median TTFT (ms): 3432.63
+P99 TTFT (ms): 17805.87
+
+---Time per Output Token (excl. 1st token)---
+Mean TPOT (ms): 48.27
+Median TPOT (ms): 49.12
+P99 TPOT (ms): 50.17
+
+---Inter-Token Latency---
+Mean ITL (ms): 48.27
+Median ITL (ms): 45.43
+P99 ITL (ms): 128.57
+Max ITL (ms): 728.88
+```
+
+
+
+### Qwen3-480B Atlas 800I A3-16Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16
+
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=1800
+export HCCL_SOCKET_IFNAME=xxx
+export GLOO_SOCKET_IFNAME=xxx
+export HCCL_OP_EXPANSION_MODE="AIV"
+
+MIX_IP=('IP1' 'IP2')
+
+for i in "${!MIX_IP[@]}";
+do
+ if [[ "$LOCAL_HOST1" == "${MIX_IP[$i]}" || "$LOCAL_HOST2" == "${MIX_IP[$i]}" ]];
+ then
+ echo "${MIX_IP[$i]}"
+
+ python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 2 --node-rank $i \
+ --dist-init-addr 141.61.133.128:5000 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 288 --context-length 8192 --dtype bfloat16 \
+ --chunked-prefill-size 114688 --max-prefill-tokens 458880 \
+ --disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto \
+ --tp 32 --dp-size 4 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.7 --cuda-graph-bs 56 64 72
+ NODE_RANK=$i
+ break
+ fi
+done
+```
+
+#### Benchmark
+
+```shell
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 288 --random-input-len 3500 --random-output-len 1500 --num-prompts 1152 --random-range-ratio 1 --request-rate 20
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: 20.0
+Max request concurrency: 288
+Successful requests: 1152
+Benchmark duration (s): 372.58
+Total input tokens: 4032000
+Total input text tokens: 4932000
+Total input vision tokens: 0
+Total generated tokens: 1728000
+Total generated tokens (retokenized): 1723296
+Request throughput (req/s): 3.09
+Input token throughput (tok/s): 10821.87
+Output token throughput (tok/s): 4637.95
+Peak output token throughput (tok/s): 6912.00
+Peak concurrent requests: 408
+Total token throughput (tok/s): 15459.82
+Concurrency: 281.13
+End-to-End Latency
+Mean E2E Latency (ms): 90923.48
+Median E2E Latency (ms): 92344.71
+Time to First Token
+Mean TTFT (ms): 15466.24
+Median TTFT (ms): 14426.93
+P99 TTFT (ms): 26498.92
+Time per Output Token (excl. 1st token)
+Mean TPOT (ms): 50.34
+Median TPOT (ms): 49.91
+P99 TPOT (ms): 59.31
+Inter-Token Latency
+Mean ITL (ms): 50.34
+Median ITL (ms): 42.35
+P95 ITL (ms): 90.51
+P99 ITL (ms): 389.41
+Max ITL (ms): 24730.36
+```
+
+
+
+### Qwen3-480B Atlas 800I A3-8Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```angular2html
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=2100
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+--host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+--attention-backend ascend --device npu --quantization modelslim \
+--max-running-requests 80 --context-length 8192 --dtype bfloat16 \
+--chunked-prefill-size 28672 --max-prefill-tokens 458880 \
+--disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto --enable-dp-attention --enable-dp-lm-head \
+--tp 16 --dp-size 4 --mem-fraction-static 0.7 --cuda-graph-bs 16 20
+```
+
+#### Benchmark
+
+```angular2html
+python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 80 --random-input-len 3500 --random-output-len 1500 --num-prompts 320 --random-range-ratio 1 --request-rate 8
+```
+
+```angular2html
+Backend: sglang
+Traffic request rate: 8.0
+Max request concurrency: 80
+Successful requests: 320
+Benchmark duration (s): 319.89
+Total input tokens: 1120000
+Total input text tokens: 1120000
+Total input vision tokens: 0
+Total generated tokens (retokenized): 477727
+Request throughput (req/s): 1.00
+Output token throughput (tok/s): 3501.16
+Input token throughput (tok/s): 1500.50
+Peak output token throughput (tok/s): 1840.00
+Peak concurrent requests: 160
+Total token throughput (tok/s): 5001.66
+Concurrency: 78.91
+----------------- End-to-End Latency -----------------
+Mean E2E Latency (ms): 78883.93
+Median E2E Latency (ms): 79526.23
+----------------- Time to First Token -----------------
+Mean TTFT (ms): 6627.14
+Median TTFT (ms): 6103.91
+P99 TTFT (ms): 11953.96
+Mean TPTOT (ms): 48.20
+Median TPTOT (ms): 48.10
+P99 TPTOT (ms): 52.33
+----------------- Inter-Token Latency -----------------
+Mean ITL (ms): 48.20
+Median ITL (ms): 45.16
+P99 ITL (ms): 51.04
+P99 ITL (ms): 55.76
+Max ITL (ms): 16869.36
+```
+
+
+
+### Qwen3-Next Atlas 800I A3-2Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+export cann_path=/usr/local/Ascend/ascend-toolkit/latest
+source /usr/local/Ascend/driver/bin/setenv.bash
+source ${cann_path}/../set_env.sh
+source ${cann_path}/../../nnal/atb/set_env.sh
+source ${cann_path}/opp/vendors/customize/bin/set_env.bash
+export ASCEND_HOME_PATH=${cann_path}
+source /usr/local/Ascend/8.5.0/bisheng_toolkit/set_env.sh
+
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export STREAMS_PER_DEVICE=32
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+
+export HCCL_OP_EXPANSION_MODE=AIV
+export HCCL_ALGO="level0:NA;level1:ring"
+
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=20
+export HCCL_BUFFSIZE=2000
+
+python -m sglang.launch_server \
+ --model-path /mnt/share/weight/Qwen3-Next-80B-A3B-Instruct-W8A8-3 \
+ --host 127.0.0.1 \
+ --port 6699 \
+ --tp-size 4 \
+ --attention-backend ascend \
+ --mem-fraction-static 0.685 \
+ --max-running-requests 80 \
+ --watchdog-timeout 3600 \
+ --disable-radix-cache \
+ --cuda-graph-bs 80 \
+ --max-prefill-tokens 28672 --max-total-tokens 450560 \
+ --moe-a2a-backend deepep --deepep-mode auto \
+ --quantization modelslim \
+ --chunked-prefill-size -1
+```
+
+#### Benchmark
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 80 --random-output-len 1536 --random-input-len 3584 --num-prompts 160
+```
+
+```angular2html
+==================== Serving Benchmark Result ====================
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 80
+Successful requests: 160
+Benchmark duration (s): 174.90
+Total input tokens: 573440
+Total input text tokens: 573440
+Total input vision tokens: 0
+Total generated tokens: 245760
+Total generated tokens (retokenized): 245760
+Request throughput (req/s): 0.91
+Input token throughput (tok/s): 3278.74
+Output token throughput (tok/s): 1405.17
+Peak output token throughput (tok/s): 1840.00
+Peak concurrent requests: 160
+Total token throughput (tok/s): 4683.91
+Concurrency: 79.87
+------------------- End-to-End Latency -------------------
+Mean E2E Latency (ms): 87303.49
+Median E2E Latency (ms): 87283.35
+------------------- Time to First Token -------------------
+Mean TTFT (ms): 10688.91
+Median TTFT (ms): 10365.16
+P99 TTFT (ms): 19285.56
+------------------- Time per Output Token (excl. 1st token) -------------------
+Mean TPOT (ms): 49.91
+Median TPOT (ms): 49.93
+P99 TPOT (ms): 55.63
+------------------- Inter-Token Latency -------------------
+Mean ITL (ms): 49.91
+Median ITL (ms): 44.75
+P95 ITL (ms): 45.54
+P99 ITL (ms): 46.08
+Max ITL (ms): 17803.50
+```
+
+
+
+### Qwen3-32B Atlas 800I A2-8Card PD Mixed 6K-1.5K 18ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 32 \
+ --disable-radix-cache \
+ --chunked-prefill-size -1 --max-prefill-tokens 135168 --speculative-draft-model-quantization unquant \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
+ --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 1 4 8 16 24 28 32 --dtype bfloat16
+```
+
+#### Benchmark
+
+We tested it based on the GSM8K dataset.
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32
+```
+
+```angular2html
+-------- Serving Benchmark Result --------
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 32
+Successful requests: 32
+Benchmark duration (s): 40.52
+Total input tokens: 192000
+Total input text tokens: 192000
+Total input vision tokens: 0
+Total generated tokens: 48000
+Total generated tokens (retokenized): 47984
+Request throughput (req/s): 0.79
+Input token throughput (tok/s): 4738.46
+Output token throughput (tok/s): 1184.62
+Peak output token throughput (tok/s): 2045.00
+Peak concurrent requests: 32
+Total token throughput (tok/s): 5923.08
+Concurrency: 28.60
+Accept length: 2.11
+
+-------- End-to-End Latency --------
+Mean E2E Latency (ms): 36214.23
+Median E2E Latency (ms): 36358.20
+
+-------- Time to First Token --------
+Mean TTFT (ms): 11544.97
+Median TTFT (ms): 11587.54
+P99 TTFT (ms): 11879.45
+
+-------- Time per Output Token (excl. 1st token) --------
+Mean TPOT (ms): 16.46
+Median TPOT (ms): 16.50
+P99 TPOT (ms): 21.34
+
+-------- Inter-Token Latency --------
+Mean ITL (ms): 16.46
+Median ITL (ms): 11.75
+P95 ITL (ms): 34.80
+P99 ITL (ms): 35.88
+Max ITL (ms): 1991.03
+```
+
+
+
+### Qwen3-32B Atlas 800I A2-8Card PD Mixed 4K-1.5K 11ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+#export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export DISABLE_EAGLE3_QUANT=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu \
+ --max-running-requests 32 \
+ --disable-radix-cache \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \
+ --chunked-prefill-size -1 --max-prefill-tokens 65536 \
+ --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 1 4 6 12 18 24 30 32 --dtype bfloat16
+```
+
+#### Benchmark
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4
+```
+
+```angular2html
+========== Serving Benchmark Result ==========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 1
+Successful requests: 4
+Benchmark duration (s): 62.33
+Total input tokens: 16000
+Total input text tokens: 16000
+Total input vision tokens: 0
+Total generated tokens: 6000
+Total generated tokens (retokenized): 6000
+Request throughput (req/s): 0.06
+Input token throughput (tok/s): 256.71
+Output token throughput (tok/s): 96.26
+Peak output token throughput (tok/s): 124.00
+Peak concurrent requests: 2
+Total token throughput (tok/s): 352.97
+Concurrency: 1.00
+Accept length: 2.04
+---------- End-to-End Latency ----------
+Mean E2E Latency (ms): 15577.88
+Median E2E Latency (ms): 15965.08
+---------- Time to First Token ----------
+Mean TTFT (ms): 312.47
+Median TTFT (ms): 312.07
+P99 TTFT (ms): 317.13
+---------- Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 10.18
+Median TPOT (ms): 10.44
+P99 TPOT (ms): 11.51
+---------- Inter-Token Latency ----------
+Mean ITL (ms): 10.18
+Median ITL (ms): 7.07
+P95 ITL (ms): 21.00
+Max ITL (ms): 27.61
+```
+
+
+
+### Qwen3-32B Atlas 800I A2-8Card PD Mixed 3.5K-1.5K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 78 \
+ --disable-radix-cache --speculative-draft-model-quantization unquant \
+ --chunked-prefill-size -1 --max-prefill-tokens 65536 \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+ --tp-size 4 --mem-fraction-static 0.72 --cuda-graph-bs 1 4 8 16 32 64 68 72 78 --dtype bfloat16 --base-gpu-id 4
+```
+
+#### Benchmark
+
+We tested it based on the GSM8K dataset.
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312
+```
+
+```angular2html
+========== Serving Benchmark Result ==========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 78
+Successful requests: 312
+Benchmark duration (s): 335.40
+Total input tokens: 1092000
+Total input text tokens: 1092000
+Total input vision tokens: 0
+Total generated tokens: 468000
+Total generated tokens (retokenized): 467916
+Request throughput (req/s): 0.93
+Input token throughput (tok/s): 3255.77
+Output token throughput (tok/s): 1395.33
+Peak output token throughput (tok/s): 2417.00
+Peak concurrent requests: 84
+Total token throughput (tok/s): 4651.10
+Concurrency: 73.66
+Accept length: 2.05
+---------- End-to-End Latency ----------
+Mean E2E Latency (ms): 79186.66
+Median E2E Latency (ms): 78608.81
+---------- Time to First Token ----------
+Mean TTFT (ms): 5786.98
+Median TTFT (ms): 645.64
+P99 TTFT (ms): 26569.53
+---------- Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 48.97
+Median TPOT (ms): 49.06
+P99 TPOT (ms): 67.17
+---------- Inter-Token Latency ----------
+Mean ITL (ms): 48.97
+Median ITL (ms): 33.00
+P95 ITL (ms): 131.60
+P99 ITL (ms): 391.27
+Max ITL (ms): 13391.14
+```
+
+
+
+### Qwen3-32B Atlas 800I A2-8Card PD Mixed 2K-2K 50ms
+
+#### Model Deployment
+
+```shell
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+export SGLANG_SET_CPU_AFFINITY=1
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+source /usr/local/Ascend/nnal/atb/set_env.sh
+source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash
+export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
+
+MODEL_PATH=xxx
+
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
+
+LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
+LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
+
+echo "${LOCAL_HOST1}"
+echo "${LOCAL_HOST2}"
+
+export HCCL_BUFFSIZE=400
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export HCCL_OP_EXPANSION_MODE="AIV"
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
+export SGLANG_ENABLE_SPEC_V2=1
+export DISABLE_EAGLE3_QUANT=1
+
+python -m sglang.launch_server --model-path $MODEL_PATH \
+ --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \
+ --attention-backend ascend --device npu --quantization modelslim \
+ --max-running-requests 120 \
+ --disable-radix-cache \
+ --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \
+ --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant \
+ --chunked-prefill-size -1 --max-prefill-tokens 49152 --base-gpu-id 4 \
+ --tp-size 4 --mem-fraction-static 0.7 --cuda-graph-bs 54 60 66 72 78 84 90 108 114 120 --dtype bfloat16
+```
+
+#### Benchmark
+
+We tested it based on the GSM8K dataset.
+
+```shell
+python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 120
+```
+
+```angular2html
+========== Serving Benchmark Result ==========
+Backend: sglang
+Traffic request rate: inf
+Max request concurrency: 120
+Successful requests: 120
+Benchmark duration (s): 117.18
+Total input tokens: 2400000
+Total input text tokens: 240000
+Total input vision tokens: 0
+Total generated tokens: 2400000
+Total generated tokens (retokenized): 239960
+Request throughput (req/s): 1.02
+Input token throughput (tok/s): 2048.10
+Output token throughput (tok/s): 2048.10
+Peak output token throughput (tok/s): 3133.00
+Peak concurrent requests: 120
+Total token throughput (tok/s): 4096.20
+Concurrency: 111.43
+Accept length: 1.69
+---------- End-to-End Latency ----------
+Mean E2E Latency (ms): 108815.31
+Median E2E Latency (ms): 108737.30
+---------- Time to First Token ----------
+Mean TTFT (ms): 17096.53
+Median TTFT (ms): 18159.44
+P99 TTFT (ms): 22681.91
+---------- Time per Output Token (excl. 1st token) ----------
+Mean TPOT (ms): 45.88
+Median TPOT (ms): 45.71
+P99 TPOT (ms): 53.49
+---------- Inter-Token Latency ----------
+Mean ITL (ms): 45.88
+Median ITL (ms): 37.46
+P95 ITL (ms): 77.55
+P99 ITL (ms): 81.31
+Max ITL (ms): 13530.47
+```
diff --git a/docs/platforms/ascend_npu_support.rst b/docs/platforms/ascend_npu_support.rst
index 494ace44d..e786a1ef7 100644
--- a/docs/platforms/ascend_npu_support.rst
+++ b/docs/platforms/ascend_npu_support.rst
@@ -9,3 +9,4 @@ Ascend NPUs
ascend_npu_support_features.md
ascend_npu_deepseek_example.md
ascend_npu_qwen3_examples.md
+ ascend_npu_best_practice.md
diff --git a/docs/platforms/ascend_npu_support_features.md b/docs/platforms/ascend_npu_support_features.md
index df1f8eae7..d8a3abd1e 100644
--- a/docs/platforms/ascend_npu_support_features.md
+++ b/docs/platforms/ascend_npu_support_features.md
@@ -3,398 +3,400 @@
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
+If you want to know the meaning and usage of each parameter, click [Service Arguments](https://docs.sglang.io/advanced_features/server_arguments.html).
+
## Model and tokenizer
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--model-path`
`--model` | The path of the model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **√** | **√** |
-| `--tokenizer-path` | The path of the tokenizer. | `None` | Type: str | **√** | **√** |
-| `--tokenizer-mode` | Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. | `auto` | `auto`, `slow` | **√** | **√** |
-| `--tokenizer-worker-num` | The worker num of the tokenizer manager. | `1` | Type: int | **√** | **√** |
-| `--skip-tokenizer-init` | If set, skip init tokenizer and pass input_ids in generate request. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--load-format` | The format of the model weights to load.
`auto` will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available.
`pt` will load the weights in the pytorch bin format.
`safetensors` will load the weights in the safetensors format.
`npcache` will load the weights in pytorch format and store a numpy cache to speed up the loading.
`dummy` will initialize the weights with random values, which is mainly for profiling.
`gguf` will load the weights in the gguf format.
`bitsandbytes` will load the weights using bitsandbytes quantization.
`layered` loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. | `auto` | `auto`, `safetensors` | **√** | **√** |
-| `--model-loader-extra-config` | Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. | {} | Type: str
for example: {"enable_multithread_load": true,"num_threads": 64} | **√** | **√** |
-| `--trust-remote-code` | Whether or not to allow for custom models defined on the Hub in their own modeling files. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--context-length` | The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). | `None` | Type: int | **√** | **√** |
-| `--is-embedding` | Whether to use a CausalLM as an embedding model. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-multimodal` | Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen | `None` | bool flag (set to enable) | **√** | **√** |
-| `--revision` | The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **×** | **×** |
-| `--model-impl` | Which implementation of the model to use.
`auto` will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available.
`sglang` will use the sglang model implementation.
`transformers` will use the transformers model implementation.
`mindspore` will use the MindSpore model implementation. | `None` | `auto`, `sglang`, `transformers` | **×** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------------|----------|---------------------------------------|------------------------------------------|------------------------------------------|
+| `--model-path`
`--model` | `None` | Type: str | **√** | **√** |
+| `--tokenizer-path` | `None` | Type: str | **√** | **√** |
+| `--tokenizer-mode` | `auto` | `auto`, `slow` | **√** | **√** |
+| `--tokenizer-worker-num` | `1` | Type: int | **√** | **√** |
+| `--skip-tokenizer-init` | `False` | bool flag (set to enable) | **√** | **√** |
+| `--load-format` | `auto` | `auto`, `safetensors` | **√** | **√** |
+| `--model-loader-`
`extra-config` | {} | Type: str | **√** | **√** |
+| `--trust-remote-code` | `False` | bool flag (set to enable) | **√** | **√** |
+| `--context-length` | `None` | Type: int | **√** | **√** |
+| `--is-embedding` | `False` | bool flag (set to enable) | **√** | **√** |
+| `--enable-multimodal` | `None` | bool flag (set to enable) | **√** | **√** |
+| `--revision` | `None` | Type: str | **×** | **×** |
+| `--model-impl` | `None` | `auto`, `sglang`,
`transformers` | **×** | **√** |
## HTTP server
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--host` | The host of the HTTP server. | `127.0.0.1` | Type: str | **√** | **√** |
-| `--port` | The port of the HTTP server. | `30000` | Type: int | **√** | **√** |
-| `--skip-server-warmup` | If set, skip warmup. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--warmups` | Specify custom warmup functions (csv) to run before server starts eg. `--warmups=warmup_name1,warmup_name2` will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests | `None` | Type: str | **√** | **√** |
-| `--nccl-port` | The port for NCCL distributed environment setup. Defaults to a random port. | `None` | Type: int | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|------------------------|-------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--host` | `127.0.0.1` | Type: str | **√** | **√** |
+| `--port` | `30000` | Type: int | **√** | **√** |
+| `--skip-server-warmup` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--warmups` | `None` | Type: str | **√** | **√** |
+| `--nccl-port` | `None` | Type: int | **√** | **√** |
## Quantization and data type
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-----------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--dtype` | Data type for model weights and activations. * "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * "half" for FP16. Recommended for AWQ quantization. * "float16" is the same as "half". * "bfloat16" for a balance between precision and range. | `auto` | `auto`, `float16`, `bfloat16` | **√** | **√** |
-| `--quantization` | The quantization method. | `None` | `modelslim` | **√** | **√** |
-| `--quantization-param-path` | Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. | `None` | Type: Optional[str] | **×** | **×** |
-| `--kv-cache-dtype` | Data type for kv cache storage. "auto" will use model data type. "fp8_e5m2" and "fp8_e4m3" is supported for CUDA 11.8+. | `auto` | `auto` | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|-----------------------------|----------|-----------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--dtype` | `auto` | `auto`,
`float16`,
`bfloat16` | **√** | **√** |
+| `--quantization` | `None` | `modelslim` | **√** | **√** |
+| `--quantization-param-path` | `None` | Type: str | **×** | **×** |
+| `--kv-cache-dtype` | `auto` | `auto` | **√** | **√** |
## Memory and scheduling
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|----------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--mem-fraction-static` | The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. | `None` | Type: float | **√** | **√** |
-| `--max-running-requests` | The maximum number of running requests. | `None` | Type: int | **√** | **√** |
-| `--prefill-max-requests` | The maximum number of requests in a prefill batch. If not specified, there is no limit.. | `None` | Type: int | **√** | **√** |
-| `--max-queued-requests` | The maximum number of queued requests. This option is ignored when using disaggregation-mode. | `None` | Type: int | **√** | **√** |
-| `--max-total-tokens` | The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes. | `None` | Type: int | **√** | **√** |
-| `--chunked-prefill-size` | The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill. | `None` | Type: int | **√** | **√** |
-| `--max-prefill-tokens` | The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length. | `16384` | Type: int | **√** | **√** |
-| `--schedule-policy` | The scheduling policy of the requests. | `fcfs` | `lpm`, `fcfs` | **√** | **√** |
-| `--enable-priority-scheduling` | Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--schedule-low-priority-values-first` | If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--priority-scheduling-preemption-threshold` | Minimum difference in priorities for an incoming request to have to preempt running request(s). | `10` | Type: int | **√** | **√** |
-| `--schedule-conservativeness` | How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. | `1.0` | Type: float | **√** | **√** |
-| `--page-size` | The number of tokens in a page, auto set 128 to Ascend NPU. | `128` | Type: int | **√** | **√** |
-| `--hybrid-kvcache-ratio` | Mix ratio in [0,1] between uniform and hybrid kv buffers (0.0 = pure uniform: swa_size / full_size = 1)(1.0 = pure hybrid: swa_size / full_size = local_attention_size / context_length) | `None` | Optional[float] | **×** | **×** |
-| `--swa-full-tokens-ratio` | The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens. | `0.8` | Type: float | **×** | **×** |
-| `--disable-hybrid-swa-memory` | Disable the hybrid SWA memory. | `False` | bool flag (set to enable) | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-----------------------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--mem-fraction-static` | `None` | Type: float | **√** | **√** |
+| `--max-running-requests` | `None` | Type: int | **√** | **√** |
+| `--prefill-max-requests` | `None` | Type: int | **√** | **√** |
+| `--max-queued-requests` | `None` | Type: int | **√** | **√** |
+| `--max-total-tokens` | `None` | Type: int | **√** | **√** |
+| `--chunked-prefill-size` | `None` | Type: int | **√** | **√** |
+| `--max-prefill-tokens` | `16384` | Type: int | **√** | **√** |
+| `--schedule-policy` | `fcfs` | `lpm`, `fcfs` | **√** | **√** |
+| `--enable-priority-`
`scheduling` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--schedule-low-priority-`
`values-first` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--priority-scheduling-`
`preemption-threshold` | `10` | Type: int | **√** | **√** |
+| `--schedule-conservativeness` | `1.0` | Type: float | **√** | **√** |
+| `--page-size` | `128` | Type: int | **√** | **√** |
+| `--hybrid-kvcache-ratio` | `None` | Optional[float] | **×** | **×** |
+| `--swa-full-tokens-ratio` | `0.8` | Type: float | **×** | **×** |
+| `--disable-hybrid-swa-memory` | `False` | bool flag
(set to enable) | **×** | **×** |
## Runtime options
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--device` | The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified. | `None` | Type: str | **√** | **√** |
-| `--tensor-parallel-size`
`--tp-size` | The tensor parallelism size. | `1` | Type: int | **√** | **√** |
-| `--pipeline-parallel-size`
`--pp-size` | The pipeline parallelism size. | `1` | Type: int | **×** | **×** |
-| `--pp-max-micro-batch-size` | The maximum micro batch size in pipeline parallelism. | `None` | Type: int | **×** | **×** |
-| `--stream-interval` | The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher | `1` | Type: int | **√** | **√** |
-| `--stream-output` | Whether to output as a sequence of disjoint segments. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--random-seed` | The random seed. | `None` | Type: int | **√** | **√** |
-| `--constrained-json-whitespace-pattern` | (outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to [\n\t ]* | `None` | Type: str | **×** | **×** |
-| `--constrained-json-disable-any-whitespace` | (xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--watchdog-timeout` | Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging. | `300` | Type: float | **√** | **√** |
-| `--soft-watchdog-timeout` | Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging. | `300` | Type: float | **√** | **√** |
-| `--dist-timeout` | Set timeout for torch.distributed initialization. | `None` | Type: int | **×** | **×** |
-| `--base-gpu-id` | The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine. | `0` | Type: int | **√** | **√** |
-| `--gpu-id-step` | The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,... | `1` | Type: int | **√** | **√** |
-| `--sleep-on-idle` | Reduce CPU usage when sglang is idle. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--mm-process-config` | A JSON string for multimodal preprocessing configuration. It can contain keys: `image`, `video`, `audio`. | `{}` | | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--device` | `None` | Type: str | **√** | **√** |
+| `--tensor-parallel-size`
`--tp-size` | `1` | Type: int | **√** | **√** |
+| `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int | **×** | **×** |
+| `--pp-max-micro-batch-size` | `None` | Type: int | **×** | **×** |
+| `--stream-interval` | `1` | Type: int | **√** | **√** |
+| `--stream-output` | `False` | bool flag (set to enable) | **√** | **√** |
+| `--random-seed` | `None` | Type: int | **√** | **√** |
+| `--constrained-json-`
`whitespace-pattern` | `None` | Type: str | **×** | **×** |
+| `--constrained-json-`
`disable-any-whitespace` | `False` | bool flag (set to enable) | **×** | **×** |
+| `--watchdog-timeout` | `300` | Type: float | **√** | **√** |
+| `--soft-watchdog-timeout` | `300` | Type: float | **√** | **√** |
+| `--dist-timeout` | `None` | Type: int | **×** | **×** |
+| `--base-gpu-id` | `0` | Type: int | **√** | **√** |
+| `--gpu-id-step` | `1` | Type: int | **√** | **√** |
+| `--sleep-on-idle` | `False` | bool flag (set to enable) | **√** | **√** |
+| `--mm-process-config` | `{}` | | **×** | **×** |
## Logging
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--log-level` | The logging level of all loggers. | `info` | Type: str | **√** | **√** |
-| `--log-level-http` | The logging level of HTTP server. If not set, reuse --log-level by default. | `None` | Type: str | **√** | **√** |
-| `--log-requests` | Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level | `False` | bool flag (set to enable) | **√** | **√** |
-| `--log-requests-level` | 0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output. | `2` | `0`, `1`, `2`, `3` | **√** | **√** |
-| `--crash-dump-folder` | Folder path to dump requests from the last 5 min before a crash (if any). If not specified, crash dumping is disabled. | `None` | Type: str | **×** | **×** |
-| `--crash-on-nan` | Crash the server on nan logprobs. | `False` | Type: str | **×** | **×** |
-| `--enable-metrics` | Enable log prometheus metrics. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-metrics-for-all-schedulers` | Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--tokenizer-metrics-custom-labels-header` | Specify the HTTP header for passing custom labels for tokenizer metrics. | `x-custom-labels` | Type: str | **×** | **×** |
-| `--tokenizer-metrics-allowed-custom-labels` | The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., {'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set. | `None` | List[str] | **×** | **×** |
-| `--bucket-time-to-first-token` | The buckets of time to first token, specified as a list of floats. | `None` | List[float] | **×** | **×** |
-| `--bucket-inter-token-latency` | The buckets of inter-token latency, specified as a list of floats. | `None` | List[float] | **×** | **×** |
-| `--bucket-e2e-request-latency` | The buckets of end-to-end request latency, specified as a list of floats. | `None` | List[float] | **×** | **×** |
-| `--collect-tokens-histogram` | Collect prompt/generation tokens histogram. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--prompt-tokens-buckets` | The buckets rule of prompt tokens. Supports 3 rule types: 'default' uses predefined buckets; 'tse ' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). | `None` | List[str] | **×** | **×** |
-| `--generation-tokens-buckets` | The buckets rule for generation tokens histogram. Supports 3 rule types: 'default' uses predefined buckets; 'tse ' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). | `None` | List[str] | **×** | **×** |
-| `--gc-warning-threshold-secs` | The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable. | `0.0` | Type: float | **×** | **×** |
-| `--decode-log-interval` | The log interval of decode batch. | `40` | Type: int | **√** | **√** |
-| `--enable-request-time-stats-logging` | Enable per request time stats logging | `False` | bool flag (set to enable) | **×** | **×** |
-| `--kv-events-config` | Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. | `None` | Type: str | **×** | **×** |
-| `--enable-trace` | Enable opentelemetry trace | `False` | bool flag (set to enable) | **×** | **×** |
-| `--oltp-traces-endpoint` | Config opentelemetry collector endpoint if --enable-trace is set. format: : | `localhost:4317` | Type: str | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------------------------|-------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--log-level` | `info` | Type: str | **√** | **√** |
+| `--log-level-http` | `None` | Type: str | **√** | **√** |
+| `--log-requests` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | **√** | **√** |
+| `--crash-dump-folder` | `None` | Type: str | **×** | **×** |
+| `--crash-on-nan` | `False` | Type: str | **×** | **×** |
+| `--enable-metrics` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-metrics-for-`
`all-schedulers` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--tokenizer-metrics-`
`custom-labels-header` | `x-custom-labels` | Type: str | **×** | **×** |
+| `--tokenizer-metrics-`
`allowed-custom-labels` | `None` | List[str] | **×** | **×** |
+| `--bucket-time-to-`
`first-token` | `None` | List[float] | **×** | **×** |
+| `--bucket-inter-token-`
`latency` | `None` | List[float] | **×** | **×** |
+| `--bucket-e2e-request-`
`latency` | `None` | List[float] | **×** | **×** |
+| `--collect-tokens-`
`histogram` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--prompt-tokens-buckets` | `None` | List[str] | **×** | **×** |
+| `--generation-tokens-buckets` | `None` | List[str] | **×** | **×** |
+| `--gc-warning-threshold-secs` | `0.0` | Type: float | **×** | **×** |
+| `--decode-log-interval` | `40` | Type: int | **√** | **√** |
+| `--enable-request-time-`
`stats-logging` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--kv-events-config` | `None` | Type: str | **×** | **×** |
+| `--enable-trace` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--oltp-traces-endpoint` | `localhost:4317` | Type: str | **×** | **×** |
## API related
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|----------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--api-key` | Set API key of the server. It is also used in the OpenAI API compatible server. | `None` | Type: str | **√** | **√** |
-| `--served-model-name` | Override the model name returned by the v1/models endpoint in OpenAI API server. | `None` | Type: str | **√** | **√** |
-| `--weight-version` | Version identifier for the model weights. Defaults to 'default' if not specified. | `default` | Type: str | **√** | **√** |
-| `--chat-template` | The buliltin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. | `None` | Type: str | **√** | **√** |
-| `--completion-template` | The buliltin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. | `None` | Type: str | **√** | **√** |
-| `--enable-cache-report` | Return number of cached tokens in usage.prompt_tokens_details for each openai request. | `True` | bool flag (set to enable) | **√** | **√** |
-| `--reasoning-parser` | Specify the parser for reasoning models. Supported parsers: [deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3]. | `None` | `deepseek-r1`, `deepseek-v3`, `glm45`, `gpt-oss`, `kimi`, `qwen3`, `qwen3-thinking`, `step3` | **√** | **√** |
-| `--tool-call-parser` | Specify the parser for handling tool-call interactions. Supported parsers: [ llama3, qwen]. | `None` | `llama3`,`qwen` | **√** | **√** |
-| `--sampling-defaults` | Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). 'model' uses the model's generation_config.json to get the recommended sampling parameters if available. Default is 'model'. | `model` | `openai`, `model` | **×** | **×** |
-| `--tool-server` | Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no tool server will be used. | `None` | Type: str | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-------------------------|-----------|---------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--api-key` | `None` | Type: str | **√** | **√** |
+| `--served-model-name` | `None` | Type: str | **√** | **√** |
+| `--weight-version` | `default` | Type: str | **√** | **√** |
+| `--chat-template` | `None` | Type: str | **√** | **√** |
+| `--completion-template` | `None` | Type: str | **√** | **√** |
+| `--enable-cache-report` | `True` | bool flag
(set to enable) | **√** | **√** |
+| `--reasoning-parser` | `None` | `deepseek-r1`,
`deepseek-v3`,
`glm45`,
`gpt-oss`,
`kimi`,
`qwen3`,
`qwen3-thinking`,
`step3` | **√** | **√** |
+| `--tool-call-parser` | `None` | `llama3`,`qwen` | **√** | **√** |
+| `--sampling-defaults` | `model` | `openai`, `model` | **×** | **×** |
+| `--tool-server` | `None` | Type: str | **×** | **×** |
## Data parallelism
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|---------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--data-parallel-size`
`--dp-size` | The data parallelism size. | `1` | Type: int | **√** | **√** |
-| `--load-balance-method` | The load balancing strategy for data parallelism. The Minimum Token algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. | `round_robin` | `round_robin`, `shortest_queue`, `minimum_tokens` | **√** | **√** |
-| `--prefill-round-robin-balance` | Prefill is round robin balanced. This is used to promise decode server can get the correct dp rank. | `False` | bool flag (set to enable) | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------------|---------------|-------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--data-parallel-size`
`--dp-size` | `1` | Type: int | **√** | **√** |
+| `--load-balance-method` | `round_robin` | `round_robin`,
`shortest_queue`,
`minimum_tokens` | **√** | **√** |
+| `--prefill-round-robin-balance` | `False` | bool flag
(set to enable) | **√** | **√** |
## Multi-node distributed serving
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|------------------------------------------|------------------------------------------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
-| `--dist-init-addr`
`--nccl-init-addr` | The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). | `None` | Type: str | **√** | **√** |
-| `--nnodes` | The number of nodes. | `1` | Type: int | **√** | **√** |
-| `--node-rank` | The node rank. | `0` | Type: int | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|-------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
+| `--dist-init-addr`
`--nccl-init-addr` | `None` | Type: str | **√** | **√** |
+| `--nnodes` | `1` | Type: int | **√** | **√** |
+| `--node-rank` | `0` | Type: int | **√** | **√** |
## Model override args
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-------------------------------|-----------------------------------------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
-| `--json-model-override-args` | A dictionary in JSON string format used to override default model configurations. | `{}` | Type: str | **√** | **√** |
-| `--preferred-sampling-params` | json-formatted sampling settings that will be returned in /get_model_info | `None` | Type: str | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|--------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
+| `--json-model-override-`
`args` | `{}` | Type: str | **√** | **√** |
+| `--preferred-sampling-`
`params` | `None` | Type: str | **√** | **√** |
## LoRA
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--enable-lora` | Enable LoRA support for the model. This argument is automatically set to `True` if `--lora-paths` is provided for backward compatibility. | `False` | Bool flag (set to enable) | **√** | **√** |
-| `--max-lora-rank` | The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. | `None` | Type: int | **√** | **√** |
-| `--lora-target-modules` | The union set of all target modules where LoRA should be applied (e.g., `q_proj`, `k_proj`, `gate_proj`). If not specified, it will be automatically inferred from the adapters provided in `--lora-paths`. You can also set it to `all` to enable LoRA for all supported modules; note this may introduce minor performance overhead. | `None` | `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, `qkv_proj`, `gate_up_proj`, `all` | **×** | **×** |
-| `--lora-paths` | The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: `` \| `=` \| JSON with schema `{"lora_name": str, "lora_path": str, "pinned": bool}`. | `None` | Type: List[str] / JSON objects | **√** | **√** |
-| `--max-loras-per-batch` | Maximum number of adapters for a running batch, including base-only requests. | `8` | Type: int | **×** | **×** |
-| `--max-loaded-loras` | If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ `--max-loras-per-batch`. | `None` | Type: int | **×** | **×** |
-| `--lora-eviction-policy` | LoRA adapter eviction policy when the GPU memory pool is full. | `lru` | `lru`, `fifo` | **×** | **×** |
-| `--lora-backend` | Choose the kernel backend for multi-LoRA serving. | `triton` | `triton`, `csgmv` | **×** | **×** |
-| `--max-lora-chunk-size` | Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when `--lora-backend` is `csgmv`. Larger values may improve performance. | `16` | `16`, `32`, `64`, `128` | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|--------------------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--enable-lora` | `False` | Bool flag
(set to enable) | **√** | **√** |
+| `--max-lora-rank` | `None` | Type: int | **√** | **√** |
+| `--lora-target-modules` | `None` | `q_proj`,
`k_proj`,
`v_proj`,
`o_proj`,
`gate_proj`,
`up_proj`,
`down_proj`,
`qkv_proj`,
`gate_up_proj`,
`all` | **×** | **×** |
+| `--lora-paths` | `None` | Type: List[str] /
JSON objects | **√** | **√** |
+| `--max-loras-per-batch` | `8` | Type: int | **×** | **×** |
+| `--max-loaded-loras` | `None` | Type: int | **×** | **×** |
+| `--lora-eviction-policy` | `lru` | `lru`,
`fifo` | **×** | **×** |
+| `--lora-backend` | `triton` | `triton`,
`csgmv` | **×** | **×** |
+| `--max-lora-chunk-size` | `16` | `16`, `32`,
`64`, `128` | **×** | **×** |
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--attention-backend` | Choose the kernels for attention layers. | `None` | `ascend` | **√** | **√** |
-| `--prefill-attention-backend` | Choose the kernels for prefill attention layers (have priority over --attention-backend). | `None` | `ascend` | **√** | **√** |
-| `--decode-attention-backend` | Choose the kernels for decode attention layers (have priority over --attention-backend). | `None` | `ascend` | **√** | **√** |
-| `--sampling-backend` | Choose the kernels for sampling layers. | `None` | `pytorch`,`ascend` | **√** | **√** |
-| `--grammar-backend` | Choose the backend for grammar-guided decoding. | `None` | `xgrammar` | **√** | **√** |
-| `--mm-attention-backend` | Set multimodal attention backend. | `None` | `ascend_attn` | **√** | **√** |
-| `--nsa-prefill-backend` | Choose the NSA backend for the prefill stage (overrides `--attention-backend` when running DeepSeek NSA-style attention). | `flashmla_sparse` | `flashmla_sparse`, `flashmla_decode`, `fa3`, `tilelang`, `aiter` | **×** | **×** |
-| `--nsa-decode-backend` | Choose the NSA backend for the decode stage when running DeepSeek NSA-style attention. Overrides `--attention-backend` for decoding. | `flashmla_kv` | `flashmla_prefill`, `flashmla_kv`, `fa3`, `tilelang`, `aiter` | **×** | **×** |
-| `--fp8-gemm-backend` | Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (optimal for Blackwell and low-latency), 'cutlass' (optimal for Hopper/Blackwell GPUs and high-throughput), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). **NOTE**: This replaces the deprecated environment variables SGLANG_ENABLE_FLASHINFER_FP8_GEMM and SGLANG_SUPPORT_CUTLASS_BLOCK_FP8. | `auto` | `auto`, `deep_gemm`, `flashinfer_trtllm`, `cutlass`, `triton`, `aiter` | **×** | **×** |
-| `--disable-flashinfer-autotune` | Flashinfer autotune is enabled by default. Set this flag to disable the autotune. | `False` | bool flag (set to enable) | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------------|-------------------|-------------------------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--attention-backend` | `None` | `ascend` | **√** | **√** |
+| `--prefill-attention-backend` | `None` | `ascend` | **√** | **√** |
+| `--decode-attention-backend` | `None` | `ascend` | **√** | **√** |
+| `--sampling-backend` | `None` | `pytorch`,
`ascend` | **√** | **√** |
+| `--grammar-backend` | `None` | `xgrammar` | **√** | **√** |
+| `--mm-attention-backend` | `None` | `ascend_attn` | **√** | **√** |
+| `--nsa-prefill-backend` | `flashmla_sparse` | `flashmla_sparse`,
`flashmla_decode`,
`fa3`,
`tilelang`,
`aiter` | **×** | **×** |
+| `--nsa-decode-backend` | `flashmla_kv` | `flashmla_prefill`,
`flashmla_kv`,
`fa3`,
`tilelang`,
`aiter` | **×** | **×** |
+| `--fp8-gemm-backend` | `auto` | `auto`,
`deep_gemm`,
`flashinfer_trtllm`,
`cutlass`,
`triton`,
`aiter` | **×** | **×** |
+| `--disable-flashinfer-`
`autotune` | `False` | bool flag
(set to enable) | **×** | **×** |
## Speculative decoding
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|-----------|---------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--speculative-algorithm` | Speculative algorithm. | `None` | `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM` | **√** | **√** |
-| `--speculative-draft-model-path`
`--speculative-draft-model` | The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **√** | **√** |
-| `--speculative-draft-model-revision` | The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **×** | **×** |
-| `--speculative-num-steps` | The number of steps sampled from draft model in Speculative Decoding. | `None` | Type: int | **√** | **√** |
-| `--speculative-eagle-topk` | The number of tokens sampled from the draft model in eagle2 each step. | `None` | Type: int | **√** | **√** |
-| `--speculative-num-draft-tokens` | The number of tokens sampled from the draft model in Speculative Decoding. | `None` | Type: int | **√** | **√** |
-| `--speculative-accept-threshold-single` | Accept a draft token if its probability in the target model is greater than this threshold. | `1.0` | Type: float | **√** | **√** |
-| `--speculative-accept-threshold-acc` | The accept probability of a draft token is raised from its target probability p to min(1, p / threshold_acc). | `1.0` | Type: float | **√** | **√** |
-| `--speculative-token-map` | The path of the draft model's small vocab table. | `None` | Type: str | **×** | **×** |
-| `--speculative-attention-mode` | Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'. | `prefill` | `prefill`, `decode` | **√** | **√** |
-| `--speculative-moe-runner-backend` | MOE backend for EAGLE speculative decoding, see --moe-runner-backend for options. Same as moe runner backend if unset. | `None` | | **√** | **√** |
-| `--speculative-moe-a2a-backend` | MOE A2A backend for EAGLE speculative decoding, see --moe-a2a-backend for options. Same as moe a2a backend if unset. | `None` | | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|------------------------------------------------------------------|-----------|-----------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--speculative-algorithm` | `None` | `EAGLE`,
`EAGLE3`,
`NEXTN`,
`STANDALONE`,
`NGRAM` | **√** | **√** |
+| `--speculative-draft-model-path`
`--speculative-draft-model` | `None` | Type: str | **√** | **√** |
+| `--speculative-draft-model-`
`revision` | `None` | Type: str | **×** | **×** |
+| `--speculative-num-steps` | `None` | Type: int | **√** | **√** |
+| `--speculative-eagle-topk` | `None` | Type: int | **√** | **√** |
+| `--speculative-num-draft-tokens` | `None` | Type: int | **√** | **√** |
+| `--speculative-accept-`
`threshold-single` | `1.0` | Type: float | **√** | **√** |
+| `--speculative-accept-`
`threshold-acc` | `1.0` | Type: float | **√** | **√** |
+| `--speculative-token-map` | `None` | Type: str | **×** | **×** |
+| `--speculative-attention-`
`mode` | `prefill` | `prefill`,
`decode` | **√** | **√** |
+| `--speculative-moe-runner-`
`backend` | `None` | | **√** | **√** |
+| `--speculative-moe-a2a-`
`backend` | `None` | | **√** | **√** |
## Ngram speculative decoding
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------------------------|-----------------------------------------------------------------------------------|------------|---------------|:--------------------------------------:|:--------------------------------------:|
-| `--speculative-ngram-min-match-window-size` | The minimum window size for pattern matching in ngram speculative decoding. | `1` | Type: int | **×** | **×** |
-| `--speculative-ngram-max-match-window-size` | The maximum window size for pattern matching in ngram speculative decoding. | `12` | Type: int | **×** | **×** |
-| `--speculative-ngram-min-bfs-breadth` | The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | `1` | Type: int | **×** | **×** |
-| `--speculative-ngram-max-bfs-breadth` | The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. | `10` | Type: int | **×** | **×** |
-| `--speculative-ngram-match-type` | The match type for cache tree. | `BFS` | `BFS`, `PROB` | **×** | **×** |
-| `--speculative-ngram-branch-length` | The branch length for ngram speculative decoding. | `18` | Type: int | **×** | **×** |
-| `--speculative-ngram-capacity` | The cache capacity for ngram speculative decoding. | `10000000` | Type: int | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------------------------|------------|--------------------|:--------------------------------------:|:--------------------------------------:|
+| `--speculative-ngram-`
`min-match-window-size` | `1` | Type: int | **×** | **×** |
+| `--speculative-ngram-`
`max-match-window-size` | `12` | Type: int | **×** | **×** |
+| `--speculative-ngram-`
`min-bfs-breadth` | `1` | Type: int | **×** | **×** |
+| `--speculative-ngram-`
`max-bfs-breadth` | `10` | Type: int | **×** | **×** |
+| `--speculative-ngram-`
`match-type` | `BFS` | `BFS`,
`PROB` | **×** | **×** |
+| `--speculative-ngram-`
`branch-length` | `18` | Type: int | **×** | **×** |
+| `--speculative-ngram-`
`capacity` | `10000000` | Type: int | **×** | **×** |
## Expert parallelism
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|-----------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--expert-parallel-size`
`--ep-size`
`--ep` | The expert parallelism size. Default equal to tp size. | `tp-size` | Type: int | **×** | **√** |
-| `--moe-a2a-backend` | Select the backend for all-to-all communication for expert parallelism. | `none` | `none`, `deepep`, `ascend_fuseep` | **×** | **√** |
-| `--moe-runner-backend` | Choose the runner backend for MoE. | `auto` | `auto` | **×** | **√** |
-| `--flashinfer-mxfp4-moe-precision` | Choose the computation precision of flashinfer mxfp4 moe | `default` | `default`, `bf16` | **×** | **×** |
-| `--enable-flashinfer-allreduce-fusion` | Enable FlashInfer allreduce fusion with Residual RMSNorm. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--deepep-mode` | Select the mode when enable DeepEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch. | `auto` | `normal`, `low_latency`, `auto` | **×** | **√** |
-| `--deepep-config` | Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. | `None` | Type: str | **×** | **×** |
-| `--ep-num-redundant-experts` | Allocate this number of redundant experts in expert parallel. | `0` | Type: int | **×** | **×** |
-| `--ep-dispatch-algorithm` | The algorithm to choose ranks for redundant experts in expert parallel. | `None` | Type: str | **×** | **×** |
-| `--init-expert-location` | Initial location of EP experts. | `trivial` | Type: str | **×** | **×** |
-| `--enable-eplb` | Enable EPLB algorithm | `False` | bool flag (set to enable) | **×** | **×** |
-| `--eplb-algorithm` | Chosen EPLB algorithm | `auto` | Type: str | **×** | **×** |
-| `--eplb-rebalance-layers-per-chunk` | Number of layers to rebalance per forward pass. | `None` | Type: int | **×** | **×** |
-| `--eplb-min-rebalancing-utilization-threshold` | Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range [0.0, 1.0]. | `1.0` | Type: float | **×** | **×** |
-| `--expert-distribution-recorder-mode` | Mode of expert distribution recorder. | `None` | Type: str | **×** | **×** |
-| `--expert-distribution-recorder-buffer-size` | Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. | `None` | Type: int | **×** | **×** |
-| `--enable-expert-distribution-metrics` | Enable logging metrics for expert balancedness | `False` | bool flag (set to enable) | **×** | **×** |
-| `--moe-dense-tp-size` | TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. | `None` | Type: int | **√** | **√** |
-| `--elastic-ep-backend` | Select the collective communication backend for elastic EP. Currently supports 'mooncake'. | None | N/A | **×** | **×** |
-| `--mooncake-ib-device` | The InfiniBand devices for Mooncake Backend, accepts multiple comma-separated devices. Default is None, which triggers automatic device detection when Mooncake Backend is enabled. | None | N/A | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-------------------------------------------------------|-----------|---------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--expert-parallel-size`
`--ep-size`
`--ep` | `tp-size` | Type: int | **×** | **√** |
+| `--moe-a2a-backend` | `none` | `none`,
`deepep`,
`ascend_fuseep` | **×** | **√** |
+| `--moe-runner-backend` | `auto` | `auto` | **×** | **√** |
+| `--flashinfer-mxfp4-`
`moe-precision` | `default` | `default`,
`bf16` | **×** | **×** |
+| `--enable-flashinfer-`
`allreduce-fusion` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--deepep-mode` | `auto` | `normal`,
`low_latency`,
`auto` | **×** | **√** |
+| `--deepep-config` | `None` | Type: str | **×** | **×** |
+| `--ep-num-redundant-experts` | `0` | Type: int | **×** | **×** |
+| `--ep-dispatch-algorithm` | `None` | Type: str | **×** | **×** |
+| `--init-expert-location` | `trivial` | Type: str | **×** | **×** |
+| `--enable-eplb` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--eplb-algorithm` | `auto` | Type: str | **×** | **×** |
+| `--eplb-rebalance-layers-`
`per-chunk` | `None` | Type: int | **×** | **×** |
+| `--eplb-min-rebalancing-`
`utilization-threshold` | `1.0` | Type: float | **×** | **×** |
+| `--expert-distribution-`
`recorder-mode` | `None` | Type: str | **×** | **×** |
+| `--expert-distribution-`
`recorder-buffer-size` | `None` | Type: int | **×** | **×** |
+| `--enable-expert-distribution-`
`metrics` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--moe-dense-tp-size` | `None` | Type: int | **√** | **√** |
+| `--elastic-ep-backend` | None | N/A | **×** | **×** |
+| `--mooncake-ib-device` | None | N/A | **×** | **×** |
## Mamba Cache
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|-------------------------------------|:--------------------------------------:|:--------------------------------------:|
-| `--max-mamba-cache-size` | The maximum size of the mamba cache. | `None` | Type: int | **×** | **×** |
-| `--mamba-ssm-dtype` | The data type of the SSM states in mamba cache. | `float32` | `float32`, `bfloat16` | **×** | **×** |
-| `--mamba-full-memory-ratio` | The ratio of mamba state memory to full kv cache memory. | `0.2` | Type: float | **×** | **×** |
-| `--mamba-scheduler-strategy` | The strategy to use for mamba scheduler.
`auto` currently defaults to `no_buffer`.
1. `no_buffer` does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented.
2. `extra_buffer` supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes `2x` for non-spec; `1+(1/(2+speculative_num_draft_tokens))x` for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)).
2a. `extra_buffer` is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests.
2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support (currently only FLA backend), currently only extra_buffer supports branching. | `auto` | `auto`, `no_buffer`, `extra_buffer` | **×** | **×** |
-| `--mamba-track-interval` | The interval (in tokens) to track the mamba state during decode. Only used when `--mamba-scheduler-strategy` is `extra_buffer`. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. | `256` | Type: int | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|------------------------------|-----------|-----------------------------------------------|:--------------------------------------:|:--------------------------------------:|
+| `--max-mamba-cache-size` | `None` | Type: int | **×** | **×** |
+| `--mamba-ssm-dtype` | `float32` | `float32`,
`bfloat16` | **×** | **×** |
+| `--mamba-full-memory-ratio` | `0.2` | Type: float | **×** | **×** |
+| `--mamba-scheduler-strategy` | `auto` | `auto`,
`no_buffer`,
`extra_buffer` | **×** | **×** |
+| `--mamba-track-interval` | `256` | Type: int | **×** | **×** |
## Args for multi-item scoring
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|----------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
-| `--multi-item-scoring-delimiter` | Delimiter token ID for multi-item scoring. Used to combine Query and Items into a single sequence: QueryItem1Item2... This enables efficient batch processing of multiple items against a single query. | `None` | Type: int | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|----------------------------------|----------|-----------|:--------------------------------------:|:--------------------------------------:|
+| `--multi-item-scoring-delimiter` | `None` | Type: int | **×** | **×** |
## Hierarchical cache
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--enable-hierarchical-cache` | Enable hierarchical cache | `False` | bool flag (set to enable) | **√** | **√** |
-| `--hicache-ratio` | The ratio of the size of host KV cache memory pool to the size of device pool. | `2.0` | Type: float | **√** | **√** |
-| `--hicache-size` | The size of host KV cache memory pool in gigabytes, which will override the hicache_ratio if set. | `0` | Type: int | **√** | **√** |
-| `--hicache-write-policy` | The write policy of hierarchical cache. | `write_through` | `write_back`, `write_through`, `write_through_selective` | **√** | **√** |
-| `--radix-eviction-policy` | The eviction policy of radix trees. 'lru' stands for Least Recently Used, 'lfu' stands for Least Frequently Used. | `lru` | `lru`, `lfu` | **√** | **√** |
-| `--hicache-io-backend` | The IO backend for KV cache transfer between CPU and GPU | `kernel` | `kernel_ascend` | **√** | **√** |
-| `--hicache-mem-layout` | The layout of host memory pool for hierarchical cache. | `layer_first` | `page_first_direct`, `page_first_kv_split` | **√** | **√** |
-| `--hicache-storage-backend` | The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend_name (custom name), module_path (Python module path), class_name (backend class name). | `None` | `file` | **√** | **√** |
-| `--hicache-storage-prefetch-policy` | Control when prefetching from the storage backend should stop. | `best_effort` | `best_effort`, `wait_complete`, `timeout` | **×** | **×** |
-| `--hicache-storage-backend-extra-config` | A dictionary in JSON string format containing extra configuration for the storage backend. | `None` | Type: str | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-------------------------------------------------|-----------------|--------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--enable-hierarchical-`
`cache` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--hicache-ratio` | `2.0` | Type: float | **√** | **√** |
+| `--hicache-size` | `0` | Type: int | **√** | **√** |
+| `--hicache-write-policy` | `write_through` | `write_back`,
`write_through`,
`write_through_selective` | **√** | **√** |
+| `--radix-eviction-policy` | `lru` | `lru`, `lfu` | **√** | **√** |
+| `--hicache-io-backend` | `kernel` | `kernel_ascend` | **√** | **√** |
+| `--hicache-mem-layout` | `layer_first` | `page_first_direct`,
`page_first_kv_split` | **√** | **√** |
+| `--hicache-storage-`
`backend` | `None` | `file` | **√** | **√** |
+| `--hicache-storage-`
`prefetch-policy` | `best_effort` | `best_effort`,
`wait_complete`,
`timeout` | **×** | **×** |
+| `--hicache-storage-`
`backend-extra-config` | `None` | Type: str | **×** | **×** |
## LMCache
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|--------------------|-------------------------------------------------------------|----------|---------------------------|:--------------------------------------:|:--------------------------------------:|
-| `--enable-lmcache` | Using LMCache as an alternative hierarchical cache solution | `False` | bool flag (set to enable) | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|--------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:|
+| `--enable-lmcache` | `False` | bool flag
(set to enable) | **×** | **×** |
## Double Sparsity
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:--------------------------------------:|:--------------------------------------:|
-| `--enable-double-sparsity` | Enable double sparsity attention | `False` | bool flag (set to enable) | **×** | **×** |
-| `--ds-channel-config-path` | The path of the double sparsity channel config | `None` | Type: str | **×** | **×** |
-| `--ds-heavy-channel-num` | The number of heavy channels in double sparsity attention | `32` | Type: int | **×** | **×** |
-| `--ds-heavy-token-num` | The number of heavy tokens in double sparsity attention | `256` | Type: int | **×** | **×** |
-| `--ds-heavy-channel-type` | The type of heavy channels in double sparsity attention | `qk` | Type: str | **×** | **×** |
-| `--ds-sparse-decode-threshold` | The minimum decode sequence length required before the double-sparsity backend switches from the dense fallback to the sparse decode kernel. | `4096` | Type: int | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|---------------------------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:|
+| `--enable-double-sparsity` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--ds-channel-config-path` | `None` | Type: str | **×** | **×** |
+| `--ds-heavy-channel-num` | `32` | Type: int | **×** | **×** |
+| `--ds-heavy-token-num` | `256` | Type: int | **×** | **×** |
+| `--ds-heavy-channel-type` | `qk` | Type: str | **×** | **×** |
+| `--ds-sparse-decode-`
`threshold` | `4096` | Type: int | **×** | **×** |
## Offloading
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|---------------------------|----------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
-| `--cpu-offload-gb` | How many GBs of RAM to reserve for CPU offloading. | `0` | Type: int | **√** | **√** |
-| `--offload-group-size` | Number of layers per group in offloading. | `-1` | Type: int | **×** | **×** |
-| `--offload-num-in-group` | Number of layers to be offloaded within a group. | `1` | Type: int | **×** | **×** |
-| `--offload-prefetch-step` | Steps to prefetch in offloading. | `1` | Type: int | **×** | **×** |
-| `--offload-mode` | Mode of offloading. | `cpu` | Type: str | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|---------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
+| `--cpu-offload-gb` | `0` | Type: int | **√** | **√** |
+| `--offload-group-size` | `-1` | Type: int | **×** | **×** |
+| `--offload-num-in-group` | `1` | Type: int | **×** | **×** |
+| `--offload-prefetch-step` | `1` | Type: int | **×** | **×** |
+| `--offload-mode` | `cpu` | Type: str | **×** | **×** |
## Optimization/debug options
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|--------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--disable-radix-cache` | Disable RadixAttention for prefix caching. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--cuda-graph-max-bs` | Set the maximum batch size for cuda graph. It will extend the cuda graph capture batch size to this value. | `None` | Type: int | **×** | **√** |
-| `--cuda-graph-bs` | Set the list of batch sizes for cuda graph. | `None` | List[int] | **×** | **√** |
-| `--disable-cuda-graph` | Disable cuda graph. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--disable-cuda-graph-padding` | Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed. | `False` | bool flag (set to enable) | **×** | **√** |
-| `--enable-profile-cuda-graph` | Enable profiling of cuda graph capture. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-cudagraph-gc` | Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-nccl-nvls` | Enable NCCL NVLS for prefill heavy requests when available. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-symm-mem` | Enable NCCL symmetric memory for fast collectives. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--disable-flashinfer-cutlass-moe-fp4-allgather` | Disables quantize before all-gather for flashinfer cutlass moe. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-tokenizer-batch-encode` | Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input_ids, or input_embeds. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--disable-outlines-disk-cache` | Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--disable-custom-all-reduce` | Disable the custom all-reduce kernel and fall back to NCCL. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-mscclpp` | Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-torch-symm-mem` | Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--disable-overlap-schedule` | Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-mixed-chunk` | Enabling mixing prefill and decode in a batch when using chunked prefill. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-dp-attention` | Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-dp-lm-head` | Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-two-batch-overlap` | Enabling two micro batches to overlap. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-single-batch-overlap` | Let computation and communication overlap within one micro batch. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--tbo-token-distribution-threshold` | The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap. | `0.48` | Type: float | **×** | **×** |
-| `--enable-torch-compile` | Optimize the model with torch.compile. Experimental feature. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-torch-compile-debug-mode` | Enable debug mode for torch compile. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--enable-piecewise-cuda-graph` | Optimize the model with piecewise cuda graph for extend/prefill only. Experimental feature. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--piecewise-cuda-graph-tokens` | Set the list of tokens when using piecewise cuda graph. | `None` | Type: JSON list | **×** | **×** |
-| `--torch-compile-max-bs` | Set the maximum batch size when using torch compile. | `32` | Type: int | **×** | **×** |
-| `--piecewise-cuda-graph-max-tokens` | Set the maximum tokens when using piecewise cuda graph. | `4096` | Type: int | **×** | **√** |
-| `--torchao-config` | Optimize the model with torchao. Experimental feature. Current choices are: int8dq, int8wo, int4wo-, fp8wo, fp8dq-per_tensor, fp8dq-per_row | `` | Type: str | **×** | **×** |
-| `--enable-nan-detection` | Enable the NaN detection for debugging purposes. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-p2p-check` | Enable P2P check for GPU access, otherwise the p2p access is allowed by default. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--triton-attention-reduce-in-fp32` | Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16. This only affects Triton attention kernels. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--triton-attention-num-kv-splits` | The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8. | `8` | Type: int | **×** | **×** |
-| `--triton-attention-split-tile-size` | The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference. | `None` | Type: int | **×** | **×** |
-| `--num-continuous-decode-steps` | Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time. | `1` | Type: int | **×** | **×** |
-| `--delete-ckpt-after-loading` | Delete the model checkpoint after loading the model. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-memory-saver` | Allow saving memory using release_memory_occupation and resume_memory_occupation | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-weights-cpu-backup` | Save model weights to CPU memory during release_weights_occupation and resume_weights_occupation | `False` | bool flag (set to enable) | **×** | **×** |
-| `--allow-auto-truncate` | Allow automatically truncating requests that exceed the maximum input length instead of returning an error. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-custom-logit-processor` | Enable users to pass custom logit processors to the server (disabled by default for security) | `False` | bool flag (set to enable) | **×** | **×** |
-| `--flashinfer-mla-disable-ragged` | Not using ragged prefill wrapper when running flashinfer mla | `False` | bool flag (set to enable) | **×** | **×** |
-| `--disable-shared-experts-fusion` | Disable shared experts fusion optimization for deepseek v3/r1. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--disable-chunked-prefix-cache` | Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--disable-fast-image-processor` | Adopt base image processor instead of fast image processor. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--keep-mm-feature-on-device` | Keep multimodal feature tensors on device after processing to save D2H copy. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-return-hidden-states` | Enable returning hidden states with responses. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--scheduler-recv-interval` | The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this. | `1` | Type: int | **×** | **×** |
-| `--numa-node` | Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. | `None` | List[int] | **×** | **×** |
-| `--enable-layerwise-nvtx-marker` | Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-attn-tp-input-scattered` | Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--enable-nsa-prefill-context-parallel` | Context parallelism used in the long sequence prefill phase of DeepSeek v3.2 | `False` | bool flag (set to enable) | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|---------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--disable-radix-cache` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--cuda-graph-max-bs` | `None` | Type: int | **×** | **√** |
+| `--cuda-graph-bs` | `None` | List[int] | **×** | **√** |
+| `--disable-cuda-graph` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--disable-cuda-graph-`
`padding` | `False` | bool flag
(set to enable) | **×** | **√** |
+| `--enable-profile-`
`cuda-graph` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-cudagraph-gc` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-nccl-nvls` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-symm-mem` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-tokenizer-`
`batch-encode` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--disable-outlines-`
`disk-cache` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--disable-custom-`
`all-reduce` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-mscclpp` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-torch-`
`symm-mem` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--disable-overlap`
`-schedule` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--enable-mixed-`
`chunk` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--enable-dp-attention` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--enable-dp-lm-head` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--enable-two-`
`batch-overlap` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-single-`
`batch-overlap` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--tbo-token-`
`distribution-threshold` | `0.48` | Type: float | **×** | **×** |
+| `--enable-torch-`
`compile` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--enable-torch-`
`compile-debug-mode` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--enable-piecewise-`
`cuda-graph` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--piecewise-cuda-`
`graph-tokens` | `None` | Type: JSON
list | **×** | **×** |
+| `--torch-compile-max-bs` | `32` | Type: int | **×** | **×** |
+| `--piecewise-cuda-`
`graph-max-tokens` | `4096` | Type: int | **×** | **√** |
+| `--torchao-config` | `` | Type: str | **×** | **×** |
+| `--enable-nan-detection` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-p2p-check` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--triton-attention-`
`reduce-in-fp32` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--triton-attention-`
`num-kv-splits` | `8` | Type: int | **×** | **×** |
+| `--triton-attention-`
`split-tile-size` | `None` | Type: int | **×** | **×** |
+| `--num-continuous-`
`decode-steps` | `1` | Type: int | **×** | **×** |
+| `--delete-ckpt-`
`after-loading` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-memory-saver` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-weights-`
`cpu-backup` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--allow-auto-truncate` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-custom-`
`logit-processor` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--flashinfer-mla-`
`disable-ragged` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--disable-shared-`
`experts-fusion` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--disable-chunked-`
`prefix-cache` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--disable-fast-`
`image-processor` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--keep-mm-feature-`
`on-device` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-return-`
`hidden-states` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--scheduler-recv-`
`interval` | `1` | Type: int | **×** | **×** |
+| `--numa-node` | `None` | List[int] | **×** | **×** |
+| `--enable-layerwise-`
`nvtx-marker` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-attn-tp-`
`input-scattered` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--enable-nsa-prefill-`
`context-parallel` | `False` | bool flag
(set to enable) | **×** | **×** |
## Forward hooks
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-----------------|:--------------------------------------:|:--------------------------------------:|
-| `--forward-hooks` | JSON-formatted list of forward hook specifications. Each element must include `target_modules` (list of glob patterns matched against `model.named_modules()` names) and `hook_factory` (Python import path to a factory, e.g. `my_package.hooks:make_hook`). An optional `name` field is used for logging, and an optional `config` object is passed as a `dict` to the factory. | `None` | Type: JSON list | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-------------------|----------|-----------------|:--------------------------------------:|:--------------------------------------:|
+| `--forward-hooks` | `None` | Type: JSON list | **×** | **×** |
## Debug tensor dumps
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-------------------------------------------|---------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--debug-tensor-dump-input-file` | The input filename for dumping tensors | `None` | Type: str | **×** | **×** |
-| `--debug-tensor-dump-inject` | Inject the outputs from jax as the input of every layer. | `False` | Type: str | **×** | **×** |
-| `--enable-dynamic-batch-tokenizer` | Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--dynamic-batch-tokenizer-batch-size` | [Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer. | `32` | Type: int | **√** | **√** |
-| `--dynamic-batch-tokenizer-batch-timeout` | [Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests. | `0.002` | Type: float | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|--------------------------------------------------|----------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--debug-tensor-dump-`
`input-file` | `None` | Type: str | **×** | **×** |
+| `--debug-tensor-dump-`
`inject` | `False` | Type: str | **×** | **×** |
+| `--enable-dynamic-`
`batch-tokenizer` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--dynamic-batch-`
`tokenizer-batch-size` | `32` | Type: int | **√** | **√** |
+| `--dynamic-batch-`
`tokenizer-batch-timeout` | `0.002` | Type: float | **√** | **√** |
## PD disaggregation
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|--------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------|--------------------------------------|:----------------------------------------:|:----------------------------------------:|
-| `--disaggregation-mode` | Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated | `null` | `null`, `prefill`, `decode` | **√** | **√** |
-| `--disaggregation-transfer-backend` | The backend for disaggregation transfer. Default is mooncake. | `mooncake` | `mooncake`, `nixl`, `ascend`, `fake` | **√** | **√** |
-| `--disaggregation-bootstrap-port` | Bootstrap server port on the prefill server. Default is 8998. | `8998` | Type: int | **√** | **√** |
-| `--disaggregation-decode-tp` | Decode tp size. If not set, it matches the tp size of the current engine. This is only set on the prefill server. | `None` | Type: int | **×** | **×** |
-| `--disaggregation-decode-dp` | Decode dp size. If not set, it matches the dp size of the current engine. This is only set on the prefill server. | `None` | Type: int | **×** | **×** |
-| `--disaggregation-prefill-pp` | Prefill pp size. If not set, it is default to 1. This is only set on the decode server. | `1` | Type: int | **×** | **×** |
-| `--disaggregation-ib-device` | The InfiniBand devices for disaggregation transfer, accepts single device (e.g., --disaggregation-ib-device mlx5_0) or multiple comma-separated devices (e.g., --disaggregation-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when mooncake backend is enabled. | `None` | Type: str | **×** | **×** |
-| `--disaggregation-decode-enable-offload-kvcache` | Enable async KV cache offloading on decode server (PD mode). | `False` | bool flag (set to enable) | **×** | **×** |
-| `--num-reserved-decode-tokens` | Number of decode tokens that will have memory reserved when adding new request to the running batch. | `512` | Type: int | **√** | **√** |
-| `--disaggregation-decode-polling-interval` | The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this. | `1` | Type: int | **√** | **√** |
+| Argument | Defaults | Options | A2 | A3 |
+|---------------------------------------------------------|------------|-----------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
+| `--disaggregation-mode` | `null` | `null`,
`prefill`,
`decode` | **√** | **√** |
+| `--disaggregation-transfer-backend` | `mooncake` | `mooncake`,
`nixl`,
`ascend`,
`fake` | **√** | **√** |
+| `--disaggregation-bootstrap-port` | `8998` | Type: int | **√** | **√** |
+| `--disaggregation-decode-tp` | `None` | Type: int | **×** | **×** |
+| `--disaggregation-decode-dp` | `None` | Type: int | **×** | **×** |
+| `--disaggregation-prefill-pp` | `1` | Type: int | **×** | **×** |
+| `--disaggregation-ib-device` | `None` | Type: str | **×** | **×** |
+| `--disaggregation-decode-`
`enable-offload-kvcache` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--num-reserved-decode-tokens` | `512` | Type: int | **√** | **√** |
+| `--disaggregation-decode-`
`polling-interval` | `1` | Type: int | **√** | **√** |
## Custom weight loader
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|--------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|------------------------------------------|
-| `--custom-weight-loader` | The custom dataloader which used to update the model. Should be set with a valid import path, such as my_package.weight_load_func | `None` | List[str] | **×** | **×** |
-| `--weight-loader-disable-mmap` | Disable mmap while loading weight using safetensors. | `False` | bool flag (set to enable) | **√** | **√** |
-| `--remote-instance-weight-loader-seed-instance-ip` | The ip of the seed instance for loading weights from remote instance. | `None` | Type: str | **×** | **×** |
-| `--remote-instance-weight-loader-seed-instance-service-port` | The service port of the seed instance for loading weights from remote instance. | `None` | Type: int | **×** | **×** |
-| `--remote-instance-weight-loader-send-weights-group-ports` | The communication group ports for loading weights from remote instance. | `None` | Type: JSON list | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|---------------------------------------------------------------------|----------|--------------------------------|:----------------------------------------:|------------------------------------------|
+| `--custom-weight-loader` | `None` | List[str] | **×** | **×** |
+| `--weight-loader-disable-`
`mmap` | `False` | bool flag
(set to enable) | **√** | **√** |
+| `--remote-instance-weight-`
`loader-seed-instance-ip` | `None` | Type: str | **×** | **×** |
+| `--remote-instance-weight-`
`loader-seed-instance-service-port` | `None` | Type: int | **×** | **×** |
+| `--remote-instance-weight-`
`loader-send-weights-group-ports` | `None` | Type: JSON
list | **×** | **×** |
## For PD-Multiplexing
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|-----------------------|--------------------------------------------------------|----------|---------------------------|:--------------------------------------:|:--------------------------------------:|
-| `--enable-pdmux` | Enable PD-Multiplexing, PD running on greenctx stream. | `False` | bool flag (set to enable) | **×** | **×** |
-| `--pdmux-config-path` | The path of the PD-Multiplexing config file. | `None` | Type: str | **×** | **×** |
-| `--sm-group-num` | Number of sm partition groups. | `8` | Type: int | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-----------------------|----------|--------------------------------|:--------------------------------------:|:--------------------------------------:|
+| `--enable-pdmux` | `False` | bool flag
(set to enable) | **×** | **×** |
+| `--pdmux-config-path` | `None` | Type: str | **×** | **×** |
+| `--sm-group-num` | `8` | Type: int | **×** | **×** |
## For deterministic inference
-| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
-|------------------------------------|---------------------------------------------------------------|----------|---------------------------|----------------------------------------|----------------------------------------|
-| `--enable-deterministic-inference` | Enable deterministic inference mode with batch invariant ops. | `False` | bool flag (set to enable) | **×** | **×** |
+| Argument | Defaults | Options | A2 | A3 |
+|-------------------------------------------|----------|--------------------------------|----------------------------------------|----------------------------------------|
+| `--enable-deterministic-`
`inference` | `False` | bool flag
(set to enable) | **×** | **×** |