From 99101ce30b2de4ca25366d91f3dee8457f954670 Mon Sep 17 00:00:00 2001 From: husf Date: Tue, 10 Feb 2026 21:52:27 +0800 Subject: [PATCH] [NPU][docs] improve docs for Best Practice on Ascend NPU (#18360) --- docs/platforms/ascend_npu_best_practice.md | 371 +++++++++++++-------- 1 file changed, 241 insertions(+), 130 deletions(-) diff --git a/docs/platforms/ascend_npu_best_practice.md b/docs/platforms/ascend_npu_best_practice.md index f3e02e043..dfea11e2e 100644 --- a/docs/platforms/ascend_npu_best_practice.md +++ b/docs/platforms/ascend_npu_best_practice.md @@ -7,58 +7,59 @@ you encounter issues or have any questions, please [open an issue](https://githu ### Low Latency -| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration | -|---------------|---------------|---------|---------------|-----------|--------------|----------------------------------------------------------| -| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K-1.6K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-1) | -| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K-1K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-2) | -| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-3) | -| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-4) | -| Deepseek-V3.2 | Atlas 800I A3 | 32 | PD Separation | 64K-3K | W8A8 | [Optimal Configuration](#deepseek-v32-low-latency-30ms) | +| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | +|-------------------|---------------|-------|---------------|-----------|------|--------------|--------------------------------------------------------------------------------------| +| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K-1.6K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode) | +| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K-1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-39k1k-20ms-on-a3-32-cards-separation-mode) | +| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-20ms-on-a3-32-cards-separation-mode) | +| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k1k-20ms-on-a3-32-cards-separation-mode) | +| DeepSeek-V3.2-Exp | Atlas 800I A3 | 32 | PD Separation | 64K-3K | 30ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v32-exp-64k3k-30ms-on-a3-32-cards-separation-mode) | ### High Throughput -| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration | -|-------------|---------------|---------|---------------|-----------|--------------|---------------------------------------------------------------| -| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-1) | -| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-2) | -| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K-2K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-3) | -| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-4) | -| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K-1.5K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-5) | +| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | +|-------------|---------------|-------|---------------|-----------|------|--------------|----------------------------------------------------------------------------------| +| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-32-cards-separation-mode) | +| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k2k-50ms-on-a3-8-cards-mixed-mode) | +| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K-2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k2k-50ms-on-a3-16-cards-separation-mode) | +| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-8-cards-mixed-mode) | +| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K-1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-16-cards-separation-mode) | ## Qwen Series Models ### Low Latency -| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration | -|------------|---------------|---------|-------------|---------|--------------|---------------------------------------------------------| -| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 11K-1K | BF16 | [Optimal Configuration](#qwen3-235b-low-latency-10ms) | -| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K-1.5K | BF16 | [Optimal Configuration](#qwen3-32b-low-latency-18ms) | -| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K-1.5K | BF16 | [Optimal Configuration](#qwen3-32b-low-latency-11ms) | -| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K-4K | BF16 | [Optimal Configuration](#qwen3-32b-low-latency-12ms) | -| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K-1.5K | W8A8 | [Optimal Configuration](#qwen3-32b-a2-low-latency-18ms) | -| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K-1.5K | BF16 | [Optimal Configuration](#qwen3-32b-a2-low-latency-11ms) | +| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | +|-----------------|---------------|-------|-------------|---------|------|--------------|-------------------------------------------------------------------------------| +| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11K-1K | 10ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-11k1k-10ms-on-a3-8-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K-1.5K | 18ms | BF16 | [Optimal Configuration](#qwen3-32b-6k15k-18ms-on-a3-4-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K-1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k15k-11ms-on-a3-4-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K-4K | 12ms | BF16 | [Optimal Configuration](#qwen3-32b-18k4k-12ms-on-a3-8-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K-1.5K | 18ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-6k15k-18ms-on-a2-8-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K-1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k15k-11ms-on-a2-8-cards-mixed-mode) | ### High Throughput -| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration | -|------------|---------------|---------|---------------|-----------|--------------|---------------------------------------------------------------| -| Qwen3-235B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-1) | -| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-2) | -| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-3) | -| Qwen3-235B | Atlas 800I A3 | 16 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-4) | -| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-32b-high-throughput-50ms-1) | -| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-32b-high-throughput-50ms-2) | -| Qwen3-30B | Atlas 800I A3 | 1 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-30b-high-throughput-50ms-3) | -| Qwen3-480B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-480b-high-throughput-50ms-1) | -| Qwen3-480B | Atlas 800I A3 | 16 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-480b-high-throughput-50ms-2) | -| Qwen3-480B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-480b-high-throughput-50ms-3) | -| Qwen3-Next | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-next-high-throughput-50ms) | -| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-32b-a2-high-throughput-50ms-1) | -| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-32b-a2-high-throughput-50ms-2) | +| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | +|--------------------------------|---------------|-------|---------------|-----------|-------|--------------|-----------------------------------------------------------------------------------------------------| +| Qwen3-235B-A22B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-35k15k-50ms-on-a3-24-cards-separation-mode) | +| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-35k15k-50ms-on-a3-8-cards-mixed-mode) | +| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 100ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-100ms-on-a3-8-cards-mixed-mode) | +| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-50ms-on-a3-8-cards-mixed-mode) | +| Qwen3-235B-A22B | Atlas 800I A3 | 16 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-50ms-on-a3-16-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-35k15k-50ms-on-a3-2-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k2k-50ms-on-a3-2-cards-mixed-mode) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-35k15k-50ms-on-a3-1-card-mixed-mode) | +| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-24-cards-separation-mode) | +| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 16 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-16-cards-mixed-mode) | +| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-8-cards-mixed-mode) | +| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80B-a3b-instruct-35k15k-50ms-on-a3-2-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-35k15k-50ms-on-a2-8-cards-mixed-mode) | +| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k2k-50ms-on-a2-8-cards-mixed-mode) | ## Optimal Configuration -### DeepSeek R1 High Performance 50ms 1 +### DeepSeek-R1 3.5K+1.5K 50ms on A3 32 Cards Separation Mode Model: Deepseek R1 @@ -66,7 +67,9 @@ Hardware: Atlas 800I A3 32Card DeployMode: PD Separation -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -169,11 +172,13 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 3072 --random-range-ratio 1 --request-rate 16 ``` -### DeepSeek R1 Low Latency 20ms 1 +### DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode Model: Deepseek R1 @@ -181,7 +186,9 @@ Hardware: Atlas 800I A3 32Card DeployMode: PD Separation -DataSets: 6K1.6K +Dataset: random + +Input Output Length: 6K+1.6K TPOT: 20ms @@ -282,11 +289,13 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 6000 --random-output-len 1600 --num-prompts 32 --random-range-ratio 1 ``` -### DeepSeek R1 Low Latency 20ms 2 +### DeepSeek-R1 3.9K+1K 20ms on A3 32 Cards Separation Mode Model: Deepseek R1 @@ -294,21 +303,26 @@ Hardware: Atlas 800I A3 32Card DeployMode: PD Separation -DataSets: 3.9K1K +Dataset: random + +Input Output Length: 3.9K+1K TPOT: 20ms #### Model Deployment -Please Turn to [DeepSeek R1 Low Latency 20ms](#deepSeek-r1-low-latency-20ms-1) +Please Turn +to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode) #### Benchmark -```bash +We tested it based on the `RANDOM` dataset. + +```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3900 --random-output-len 1000 --num-prompts 768 --random-range-ratio 1 --request-rate 16 ``` -### DeepSeek R1 Low Latency 20ms 3 +### DeepSeek-R1 3.5K+1.5K 20ms on A3 32 Cards Separation Mode Model: Deepseek R1 @@ -316,21 +330,25 @@ Hardware: Atlas 800I A3 32Card DeployMode: PD Separation -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 20ms #### Model Deployment -Please Turn to [DeepSeek R1 Low Latency 20ms](#deepSeek-r1-low-latency-20ms-1) +Please Turn to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode) #### Benchmark +We tested it based on the `RANDOM` dataset. + ```bash python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 768 --random-range-ratio 1 --request-rate 16 ``` -### DeepSeek R1 Low Latency 20ms 4 +### DeepSeek-R1 3.5K+1K 20ms on A3 32 Cards Separation Mode Model: Deepseek R1 @@ -338,21 +356,26 @@ Hardware: Atlas 800I A3 32Card DeployMode: PD Separation -DataSets: 3.5K1K +Dataset: random + +Input Output Length: 3.5K+1K TPOT: 20ms #### Model Deployment -Please Turn to [DeepSeek R1 Low Latency 20ms](#deepSeek-r1-low-latency-20ms-1) +Please Turn +to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode) #### Benchmark -```bash +We tested it based on the `RANDOM` dataset. + +```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1000 --num-prompts 768 --random-range-ratio 1 --request-rate 16 ``` -### DeepSeek R1 High Performance 50ms 2 +### DeepSeek-R1 2K+2K 50ms on A3 8 Cards Mixed Mode Model: Deepseek R1 @@ -360,7 +383,9 @@ Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 50ms @@ -424,11 +449,13 @@ python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 256 --random-input-len 2048 --random-output-len 2048 --num-prompts 1024 --random-range-ratio 1 ``` -### DeepSeek R1 High Performance 50ms 3 +### DeepSeek-R1 2K+2K 50ms on A3 16 Cards Separation Mode Model: Deepseek R1 @@ -436,7 +463,9 @@ Hardware: Atlas 800I A3 16Card DeployMode: PD Separation -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 50ms @@ -545,11 +574,13 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 400 --random-input-len 2048 --random-output-len 2048 --num-prompts 3200 --random-range-ratio 1 --request-rate 8 ``` -### DeepSeek R1 High Performance 50ms 4 +### DeepSeek-R1 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode Model: Deepseek R1 @@ -557,7 +588,9 @@ Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -618,11 +651,13 @@ python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 144 --random-input-len 3500 --random-output-len 1500 --num-prompts 576 --random-range-ratio 1 ``` -### DeepSeek R1 High Performance 50ms 5 +### DeepSeek-R1 3.5K+1.5K 50ms on A3 16 Cards Separation Mode Model: Deepseek R1 @@ -630,7 +665,9 @@ Hardware: Atlas 800I A3 16Card DeployMode: PD Separation -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -738,19 +775,23 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 384 --random-input-len 3500 --random-output-len 1500 --num-prompts 1536 --random-range-ratio 1 ``` -### Deepseek V32 Low Latency 30ms +### DeepSeek-V3.2-Exp 64K+3K 30ms on A3 32 Cards Separation Mode -Model: Deepseek V3.2 +Model: DeepSeek-V3.2-Exp-W8A8 Hardware: Atlas 800I A3 32Card DeployMode: PD Separation -DataSets: 64K3K +Dataset: random + +Input Output Length: 64K+3K TPOT: 30ms @@ -945,19 +986,23 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 64000 --random-output-len 3000 --num-prompts 64 --random-range-ratio 1 ``` -### Qwen3 235B High Throughput 50ms 1 +### Qwen3-235B-A22B 3.5K+1.5K 50ms on A3 24 Cards Separation Mode -Model: Qwen3 235B +Model: Qwen3-235B-A22B-W8A8 Hardware: Atlas 800I A3 24Card DeployMode: PD Separation -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -1077,19 +1122,23 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang-oai --host 127.0.0.1 --port 7239 --max-concurrency 860 --random-input-len 3500 --random-output-len 1500 --num-prompts 3440 --random-range-ratio 1 ``` -### Qwen3 235B High Throughput 50ms 2 +### Qwen3-235B-A22B 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode -Model: Qwen3 235B +Model: Qwen3-235B-A22B-W8A8 Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -1146,19 +1195,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 272 --random-input-len 3500 --random-output-len 1500 --num-prompts 1088 --random-range-ratio 1 ``` -### Qwen3-235B Atlas 800I A3-8Card PD Mixed 2K-2K 100ms +### Qwen3-235B-A22B 2K+2K 100ms on A3 8 Cards Mixed Mode -Model: Qwen3 235B +Model: Qwen3-235B-A22B-W8A8 Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 100ms @@ -1214,19 +1267,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 576 --random-input-len 2000 --random-output-len 2000 --num-prompts 576 --random-range-ratio 1 ``` -### Qwen3 235B High Throughput 50ms 3 +### Qwen3-235B-A22B 2K+2K 50ms on A3 8 Cards Mixed Mode -Model: Qwen3 235B +Model: Qwen3-235B-A22B-W8A8 Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 50ms @@ -1282,19 +1339,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 480 --random-input-len 2048 --random-output-len 2048 --num-prompts 480 --random-range-ratio 1 ``` -### Qwen3 235B High Throughput 50ms 4 +### Qwen3-235B-A22B 2K+2K 50ms on A3 16 Cards Mixed Mode -Model: Qwen3 235B +Model: Qwen3-235B-A22B-W8A8 Hardware: Atlas 800I A3 16Card DeployMode: PD Mixed -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 50ms @@ -1363,19 +1424,23 @@ done #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 768 --random-input-len 2000 --random-output-len 2000 --num-prompts 768 --random-range-ratio 1 ``` -### Qwen3 235B Low Latency 10ms +### Qwen3-235B-A22B 11K+1K 10ms on A3 8 Cards Mixed Mode -Model: Qwen3 235B +Model: Qwen3-235B-A22B-W8A8 Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 11K1K +Dataset: random + +Input Output Length: 11K+1K TPOT: 10ms @@ -1432,19 +1497,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 1 --random-input-len 11000 --random-output-len 1000 --num-prompts 1 --random-range-ratio 1 ``` -### Qwen3 32B Low Latency 18ms +### Qwen3-32B 6K+1.5K 18ms on A3 4 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A3 4Card DeployMode: PD Mixed -DataSets: 6K1.5K +Dataset: random + +Input Output Length: 6K+1.5K TPOT: 18ms @@ -1499,21 +1568,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark -We tested it based on the GSM8K dataset. +We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1 ``` -### Qwen3 32B Low Latency 11ms +### Qwen3-32B 4K+1.5K 11ms on A3 4 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A3 4Card DeployMode: PD Mixed -DataSets: 4K1.5K +Dataset: random + +Input Output Length: 4K+1.5K TPOT: 11ms @@ -1567,19 +1638,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4 ``` -### Qwen3 32B Low Latency 12ms +### Qwen3-32B 18K+4K 12ms on A3 8 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 18K4K +Dataset: random + +Input Output Length: 18K+4K TPOT: 12ms @@ -1632,19 +1707,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 18000 --random-input-len 4000 --num-prompts 1 ``` -### Qwen3 32B High Throughput 50ms 1 +### Qwen3-32B 3.5K+1.5K 50ms on A3 2 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A3 2Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -1698,21 +1777,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark -We tested it based on the GSM8K dataset. +We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1 ``` -### Qwen3 32B High Throughput 50ms 2 +### Qwen3-32B 2K+2K 50ms on A3 2 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A3 2Card DeployMode: PD Mixed -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 50ms @@ -1766,21 +1847,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark -We tested it based on the GSM8K dataset. +We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 480 --random-range-ratio 1 ``` -### Qwen3 30B High Throughput 50ms 3 +### Qwen3-30B-A3B 3.5K+1.5K 50ms on A3 1 Card Mixed Mode -Model: Qwen3 30B +Model: Qwen3-30B-A3B-Instruct-2507 Hardware: Atlas 800I A3 1Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -1834,19 +1917,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 156 --random-input-len 3500 --random-output-len 1500 --num-prompts 624 --random-range-ratio 1 ``` -### Qwen3 480B High Throughput 50ms 1 +### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 24 Cards Separation Mode -Model: Qwen3 480B +Model: Qwen3-Coder-480B-A35B-Instruct Hardware: Atlas 800I A3 24Card DeployMode: PD Separation -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -1954,19 +2041,23 @@ python -m sglang_router.launch_router \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 410 --random-input-len 3500 --random-output-len 1500 --num-prompts 1640 --random-range-ratio 1 --request-rate 8 ``` -### Qwen3 480B High Throughput 50ms 2 +### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 16 Cards Mixed Mode -Model: Qwen3 480B +Model: Qwen3-Coder-480B-A35B-Instruct Hardware: Atlas 800I A3 16Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -2033,19 +2124,23 @@ done #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 288 --random-input-len 3500 --random-output-len 1500 --num-prompts 1152 --random-range-ratio 1 --request-rate 20 ``` -### Qwen3 480B High Throughput 50ms 3 +### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode -Model: Qwen3 480B +Model: Qwen3-Coder-480B-A35B-Instruct Hardware: Atlas 800I A3 8Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -2097,19 +2192,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 80 --random-input-len 3500 --random-output-len 1500 --num-prompts 320 --random-range-ratio 1 ``` -### Qwen3 Next High Throughput 50ms +### Qwen3-Next-80B-A3B-Instruct 3.5K+1.5K 50ms on A3 2 Cards Mixed Mode -Model: Qwen3 Next +Model: Qwen3-Next-80B-A3B-Instruct Hardware: Atlas 800I A3 2Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -2143,7 +2242,7 @@ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=20 export HCCL_BUFFSIZE=2000 python -m sglang.launch_server \ - --model-path /mnt/share/weight/Qwen3-Next-80B-A3B-Instruct-W8A8-3 \ + --model-path /path/to/Qwen3-Next-80B-A3B-Instruct-W8A8-3 \ --host 127.0.0.1 \ --port 6699 \ --tp-size 4 \ @@ -2162,19 +2261,23 @@ python -m sglang.launch_server \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 80 --random-output-len 1536 --random-input-len 3584 --num-prompts 160 --random-range-ratio 1 ``` -### Qwen3 32B A2 Low Latency 18ms +### Qwen3-32B 6K+1.5K 18ms on A2 8 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A2 8Card DeployMode: PD Mixed -DataSets: 6K1.5K +Dataset: random + +Input Output Length: 6K+1.5K TPOT: 18ms @@ -2227,21 +2330,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark -We tested it based on the GSM8K dataset. +We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1 ``` -### Qwen3 32B A2 Low Latency 11ms +### Qwen3-32B 4K+1.5K 11ms on A2 8 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A2 8Card DeployMode: PD Mixed -DataSets: 4K1.5K +Dataset: random + +Input Output Length: 4K+1.5K TPOT: 11ms @@ -2298,19 +2403,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark +We tested it based on the `RANDOM` dataset. + ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4 ``` -### Qwen3 32B A2 High Throughput 50ms 1 +### Qwen3-32B 3.5K+1.5K 50ms on A2 8 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A2 8Card DeployMode: PD Mixed -DataSets: 3.5K1.5K +Dataset: random + +Input Output Length: 3.5K+1.5K TPOT: 50ms @@ -2364,21 +2473,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark -We tested it based on the GSM8K dataset. +We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1 ``` -### Qwen3 32B A2 High Throughput 50ms 2 +### Qwen3-32B 2K+2K 50ms on A2 8 Cards Mixed Mode -Model: Qwen3 32B +Model: Qwen3-32B Hardware: Atlas 800I A2 8Card DeployMode: PD Mixed -DataSets: 2K2K +Dataset: random + +Input Output Length: 2K+2K TPOT: 50ms @@ -2431,7 +2542,7 @@ python -m sglang.launch_server --model-path $MODEL_PATH \ #### Benchmark -We tested it based on the GSM8K dataset. +We tested it based on the `RANDOM` dataset. ```shell python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 120 --random-range-ratio 1