chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU][docs] improve docs for Best Practice on Ascend NPU (#18360)

Browse Source
This commit is contained in:
husf
2026-02-10 21:52:27 +08:00
committed by GitHub
parent bec7fe9e65
commit 99101ce30b
1 changed files with 241 additions and 130 deletions
Download Patch File Download Diff File Expand all files Collapse all files

371
docs/platforms/ascend_npu_best_practice.md
View File

@@ -7,58 +7,59 @@ you encounter issues or have any questions, please [open an issue](https://githu
### Low Latency
| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration |
|---------------|---------------|---------|---------------|-----------|--------------|----------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K-1.6K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-1) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K-1K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-2) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-3) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1K | W8A8 | [Optimal Configuration](#deepseek-r1-low-latency-20ms-4) |
| Deepseek-V3.2 | Atlas 800I A3 | 32 | PD Separation | 64K-3K | W8A8 | [Optimal Configuration](#deepseek-v32-low-latency-30ms) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-------------------|---------------|-------|---------------|-----------|------|--------------|--------------------------------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K-1.6K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K-1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-39k1k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k1k-20ms-on-a3-32-cards-separation-mode) |
| DeepSeek-V3.2-Exp | Atlas 800I A3 | 32 | PD Separation | 64K-3K | 30ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v32-exp-64k3k-30ms-on-a3-32-cards-separation-mode) |
### High Throughput
| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration |
|-------------|---------------|---------|---------------|-----------|--------------|---------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-1) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-2) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K-2K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-3) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-4) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K-1.5K | W4A8 | [Optimal Configuration](#deepseek-r1-high-performance-50ms-5) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-------------|---------------|-------|---------------|-----------|------|--------------|----------------------------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k2k-50ms-on-a3-8-cards-mixed-mode) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K-2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k2k-50ms-on-a3-16-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-8-cards-mixed-mode) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K-1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-16-cards-separation-mode) |
## Qwen Series Models
### Low Latency
| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration |
|------------|---------------|---------|-------------|---------|--------------|---------------------------------------------------------|
| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 11K-1K | BF16 | [Optimal Configuration](#qwen3-235b-low-latency-10ms) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K-1.5K | BF16 | [Optimal Configuration](#qwen3-32b-low-latency-18ms) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K-1.5K | BF16 | [Optimal Configuration](#qwen3-32b-low-latency-11ms) |
| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K-4K | BF16 | [Optimal Configuration](#qwen3-32b-low-latency-12ms) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K-1.5K | W8A8 | [Optimal Configuration](#qwen3-32b-a2-low-latency-18ms) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K-1.5K | BF16 | [Optimal Configuration](#qwen3-32b-a2-low-latency-11ms) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-----------------|---------------|-------|-------------|---------|------|--------------|-------------------------------------------------------------------------------|
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11K-1K | 10ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-11k1k-10ms-on-a3-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K-1.5K | 18ms | BF16 | [Optimal Configuration](#qwen3-32b-6k15k-18ms-on-a3-4-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K-1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k15k-11ms-on-a3-4-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K-4K | 12ms | BF16 | [Optimal Configuration](#qwen3-32b-18k4k-12ms-on-a3-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K-1.5K | 18ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-6k15k-18ms-on-a2-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K-1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k15k-11ms-on-a2-8-cards-mixed-mode) |
### High Throughput
| Model | Hardware | CardNum | Deploy Mode | Dataset | Quantization | Configuration |
|------------|---------------|---------|---------------|-----------|--------------|---------------------------------------------------------------|
| Qwen3-235B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-1) |
| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-2) |
| Qwen3-235B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-3) |
| Qwen3-235B | Atlas 800I A3 | 16 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-235b-high-throughput-50ms-4) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-32b-high-throughput-50ms-1) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-32b-high-throughput-50ms-2) |
| Qwen3-30B | Atlas 800I A3 | 1 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-30b-high-throughput-50ms-3) |
| Qwen3-480B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-480b-high-throughput-50ms-1) |
| Qwen3-480B | Atlas 800I A3 | 16 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-480b-high-throughput-50ms-2) |
| Qwen3-480B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-480b-high-throughput-50ms-3) |
| Qwen3-Next | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-next-high-throughput-50ms) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K-1.5K | W8A8 | [Optimal Configuration](#qwen3-32b-a2-high-throughput-50ms-1) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K-2K | W8A8 | [Optimal Configuration](#qwen3-32b-a2-high-throughput-50ms-2) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|--------------------------------|---------------|-------|---------------|-----------|-------|--------------|-----------------------------------------------------------------------------------------------------|
| Qwen3-235B-A22B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-35k15k-50ms-on-a3-24-cards-separation-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-35k15k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 100ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-100ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 16 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-50ms-on-a3-16-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-35k15k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k2k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-35k15k-50ms-on-a3-1-card-mixed-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-24-cards-separation-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 16 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-16-cards-mixed-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80B-a3b-instruct-35k15k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-35k15k-50ms-on-a2-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k2k-50ms-on-a2-8-cards-mixed-mode) |
## Optimal Configuration
### DeepSeek R1 High Performance 50ms 1
### DeepSeek-R1 3.5K+1.5K 50ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -66,7 +67,9 @@ Hardware: Atlas 800I A3 32Card
DeployMode: PD Separation
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -169,11 +172,13 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 3072 --random-range-ratio 1 --request-rate 16
```
### DeepSeek R1 Low Latency 20ms 1
### DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -181,7 +186,9 @@ Hardware: Atlas 800I A3 32Card
DeployMode: PD Separation
DataSets: 6K1.6K
Dataset: random
Input Output Length: 6K+1.6K
TPOT: 20ms
@@ -282,11 +289,13 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 6000 --random-output-len 1600 --num-prompts 32 --random-range-ratio 1
```
### DeepSeek R1 Low Latency 20ms 2
### DeepSeek-R1 3.9K+1K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -294,21 +303,26 @@ Hardware: Atlas 800I A3 32Card
DeployMode: PD Separation
DataSets: 3.9K1K
Dataset: random
Input Output Length: 3.9K+1K
TPOT: 20ms
#### Model Deployment
Please Turn to [DeepSeek R1 Low Latency 20ms](#deepSeek-r1-low-latency-20ms-1)
Please Turn
to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode)
#### Benchmark
```bash
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3900 --random-output-len 1000 --num-prompts 768 --random-range-ratio 1 --request-rate 16
```
### DeepSeek R1 Low Latency 20ms 3
### DeepSeek-R1 3.5K+1.5K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -316,21 +330,25 @@ Hardware: Atlas 800I A3 32Card
DeployMode: PD Separation
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 20ms
#### Model Deployment
Please Turn to [DeepSeek R1 Low Latency 20ms](#deepSeek-r1-low-latency-20ms-1)
Please Turn to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode)
#### Benchmark
We tested it based on the `RANDOM` dataset.
```bash
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 768 --random-range-ratio 1 --request-rate 16
```
### DeepSeek R1 Low Latency 20ms 4
### DeepSeek-R1 3.5K+1K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -338,21 +356,26 @@ Hardware: Atlas 800I A3 32Card
DeployMode: PD Separation
DataSets: 3.5K1K
Dataset: random
Input Output Length: 3.5K+1K
TPOT: 20ms
#### Model Deployment
Please Turn to [DeepSeek R1 Low Latency 20ms](#deepSeek-r1-low-latency-20ms-1)
Please Turn
to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode)
#### Benchmark
```bash
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1000 --num-prompts 768 --random-range-ratio 1 --request-rate 16
```
### DeepSeek R1 High Performance 50ms 2
### DeepSeek-R1 2K+2K 50ms on A3 8 Cards Mixed Mode
Model: Deepseek R1
@@ -360,7 +383,9 @@ Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 50ms
@@ -424,11 +449,13 @@ python3 -m sglang.launch_server --model-path ${MODEL_PATH} \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 256 --random-input-len 2048 --random-output-len 2048 --num-prompts 1024 --random-range-ratio 1
```
### DeepSeek R1 High Performance 50ms 3
### DeepSeek-R1 2K+2K 50ms on A3 16 Cards Separation Mode
Model: Deepseek R1
@@ -436,7 +463,9 @@ Hardware: Atlas 800I A3 16Card
DeployMode: PD Separation
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 50ms
@@ -545,11 +574,13 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 400 --random-input-len 2048 --random-output-len 2048 --num-prompts 3200 --random-range-ratio 1 --request-rate 8
```
### DeepSeek R1 High Performance 50ms 4
### DeepSeek-R1 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode
Model: Deepseek R1
@@ -557,7 +588,9 @@ Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -618,11 +651,13 @@ python3 -m sglang.launch_server --model-path ${MODEL_PATH} \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 144 --random-input-len 3500 --random-output-len 1500 --num-prompts 576 --random-range-ratio 1
```
### DeepSeek R1 High Performance 50ms 5
### DeepSeek-R1 3.5K+1.5K 50ms on A3 16 Cards Separation Mode
Model: Deepseek R1
@@ -630,7 +665,9 @@ Hardware: Atlas 800I A3 16Card
DeployMode: PD Separation
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -738,19 +775,23 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 384 --random-input-len 3500 --random-output-len 1500 --num-prompts 1536 --random-range-ratio 1
```
### Deepseek V32 Low Latency 30ms
### DeepSeek-V3.2-Exp 64K+3K 30ms on A3 32 Cards Separation Mode
Model: Deepseek V3.2
Model: DeepSeek-V3.2-Exp-W8A8
Hardware: Atlas 800I A3 32Card
DeployMode: PD Separation
DataSets: 64K3K
Dataset: random
Input Output Length: 64K+3K
TPOT: 30ms
@@ -945,19 +986,23 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 64000 --random-output-len 3000 --num-prompts 64 --random-range-ratio 1
```
### Qwen3 235B High Throughput 50ms 1
### Qwen3-235B-A22B 3.5K+1.5K 50ms on A3 24 Cards Separation Mode
Model: Qwen3 235B
Model: Qwen3-235B-A22B-W8A8
Hardware: Atlas 800I A3 24Card
DeployMode: PD Separation
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -1077,19 +1122,23 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang-oai --host 127.0.0.1 --port 7239 --max-concurrency 860 --random-input-len 3500 --random-output-len 1500 --num-prompts 3440 --random-range-ratio 1
```
### Qwen3 235B High Throughput 50ms 2
### Qwen3-235B-A22B 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode
Model: Qwen3 235B
Model: Qwen3-235B-A22B-W8A8
Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -1146,19 +1195,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 272 --random-input-len 3500 --random-output-len 1500 --num-prompts 1088 --random-range-ratio 1
```
### Qwen3-235B Atlas 800I A3-8Card PD Mixed 2K-2K 100ms
### Qwen3-235B-A22B 2K+2K 100ms on A3 8 Cards Mixed Mode
Model: Qwen3 235B
Model: Qwen3-235B-A22B-W8A8
Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 100ms
@@ -1214,19 +1267,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 576 --random-input-len 2000 --random-output-len 2000 --num-prompts 576 --random-range-ratio 1
```
### Qwen3 235B High Throughput 50ms 3
### Qwen3-235B-A22B 2K+2K 50ms on A3 8 Cards Mixed Mode
Model: Qwen3 235B
Model: Qwen3-235B-A22B-W8A8
Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 50ms
@@ -1282,19 +1339,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 480 --random-input-len 2048 --random-output-len 2048 --num-prompts 480 --random-range-ratio 1
```
### Qwen3 235B High Throughput 50ms 4
### Qwen3-235B-A22B 2K+2K 50ms on A3 16 Cards Mixed Mode
Model: Qwen3 235B
Model: Qwen3-235B-A22B-W8A8
Hardware: Atlas 800I A3 16Card
DeployMode: PD Mixed
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 50ms
@@ -1363,19 +1424,23 @@ done
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 768 --random-input-len 2000 --random-output-len 2000 --num-prompts 768 --random-range-ratio 1
```
### Qwen3 235B Low Latency 10ms
### Qwen3-235B-A22B 11K+1K 10ms on A3 8 Cards Mixed Mode
Model: Qwen3 235B
Model: Qwen3-235B-A22B-W8A8
Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 11K1K
Dataset: random
Input Output Length: 11K+1K
TPOT: 10ms
@@ -1432,19 +1497,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 1 --random-input-len 11000 --random-output-len 1000 --num-prompts 1 --random-range-ratio 1
```
### Qwen3 32B Low Latency 18ms
### Qwen3-32B 6K+1.5K 18ms on A3 4 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A3 4Card
DeployMode: PD Mixed
DataSets: 6K1.5K
Dataset: random
Input Output Length: 6K+1.5K
TPOT: 18ms
@@ -1499,21 +1568,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the GSM8K dataset.
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1
```
### Qwen3 32B Low Latency 11ms
### Qwen3-32B 4K+1.5K 11ms on A3 4 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A3 4Card
DeployMode: PD Mixed
DataSets: 4K1.5K
Dataset: random
Input Output Length: 4K+1.5K
TPOT: 11ms
@@ -1567,19 +1638,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4
```
### Qwen3 32B Low Latency 12ms
### Qwen3-32B 18K+4K 12ms on A3 8 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 18K4K
Dataset: random
Input Output Length: 18K+4K
TPOT: 12ms
@@ -1632,19 +1707,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 18000 --random-input-len 4000 --num-prompts 1
```
### Qwen3 32B High Throughput 50ms 1
### Qwen3-32B 3.5K+1.5K 50ms on A3 2 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A3 2Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -1698,21 +1777,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the GSM8K dataset.
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1
```
### Qwen3 32B High Throughput 50ms 2
### Qwen3-32B 2K+2K 50ms on A3 2 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A3 2Card
DeployMode: PD Mixed
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 50ms
@@ -1766,21 +1847,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the GSM8K dataset.
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 480 --random-range-ratio 1
```
### Qwen3 30B High Throughput 50ms 3
### Qwen3-30B-A3B 3.5K+1.5K 50ms on A3 1 Card Mixed Mode
Model: Qwen3 30B
Model: Qwen3-30B-A3B-Instruct-2507
Hardware: Atlas 800I A3 1Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -1834,19 +1917,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 156 --random-input-len 3500 --random-output-len 1500 --num-prompts 624 --random-range-ratio 1
```
### Qwen3 480B High Throughput 50ms 1
### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 24 Cards Separation Mode
Model: Qwen3 480B
Model: Qwen3-Coder-480B-A35B-Instruct
Hardware: Atlas 800I A3 24Card
DeployMode: PD Separation
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -1954,19 +2041,23 @@ python -m sglang_router.launch_router \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 410 --random-input-len 3500 --random-output-len 1500 --num-prompts 1640 --random-range-ratio 1 --request-rate 8
```
### Qwen3 480B High Throughput 50ms 2
### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 16 Cards Mixed Mode
Model: Qwen3 480B
Model: Qwen3-Coder-480B-A35B-Instruct
Hardware: Atlas 800I A3 16Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -2033,19 +2124,23 @@ done
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 288 --random-input-len 3500 --random-output-len 1500 --num-prompts 1152 --random-range-ratio 1 --request-rate 20
```
### Qwen3 480B High Throughput 50ms 3
### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode
Model: Qwen3 480B
Model: Qwen3-Coder-480B-A35B-Instruct
Hardware: Atlas 800I A3 8Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -2097,19 +2192,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 80 --random-input-len 3500 --random-output-len 1500 --num-prompts 320 --random-range-ratio 1
```
### Qwen3 Next High Throughput 50ms
### Qwen3-Next-80B-A3B-Instruct 3.5K+1.5K 50ms on A3 2 Cards Mixed Mode
Model: Qwen3 Next
Model: Qwen3-Next-80B-A3B-Instruct
Hardware: Atlas 800I A3 2Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -2143,7 +2242,7 @@ export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=20
export HCCL_BUFFSIZE=2000
python -m sglang.launch_server \
--model-path /mnt/share/weight/Qwen3-Next-80B-A3B-Instruct-W8A8-3 \
--model-path /path/to/Qwen3-Next-80B-A3B-Instruct-W8A8-3 \
--host 127.0.0.1 \
--port 6699 \
--tp-size 4 \
@@ -2162,19 +2261,23 @@ python -m sglang.launch_server \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 80 --random-output-len 1536 --random-input-len 3584 --num-prompts 160 --random-range-ratio 1
```
### Qwen3 32B A2 Low Latency 18ms
### Qwen3-32B 6K+1.5K 18ms on A2 8 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A2 8Card
DeployMode: PD Mixed
DataSets: 6K1.5K
Dataset: random
Input Output Length: 6K+1.5K
TPOT: 18ms
@@ -2227,21 +2330,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the GSM8K dataset.
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1
```
### Qwen3 32B A2 Low Latency 11ms
### Qwen3-32B 4K+1.5K 11ms on A2 8 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A2 8Card
DeployMode: PD Mixed
DataSets: 4K1.5K
Dataset: random
Input Output Length: 4K+1.5K
TPOT: 11ms
@@ -2298,19 +2403,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4
```
### Qwen3 32B A2 High Throughput 50ms 1
### Qwen3-32B 3.5K+1.5K 50ms on A2 8 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A2 8Card
DeployMode: PD Mixed
DataSets: 3.5K1.5K
Dataset: random
Input Output Length: 3.5K+1.5K
TPOT: 50ms
@@ -2364,21 +2473,23 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the GSM8K dataset.
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1
```
### Qwen3 32B A2 High Throughput 50ms 2
### Qwen3-32B 2K+2K 50ms on A2 8 Cards Mixed Mode
Model: Qwen3 32B
Model: Qwen3-32B
Hardware: Atlas 800I A2 8Card
DeployMode: PD Mixed
DataSets: 2K2K
Dataset: random
Input Output Length: 2K+2K
TPOT: 50ms
@@ -2431,7 +2542,7 @@ python -m sglang.launch_server --model-path $MODEL_PATH \
#### Benchmark
We tested it based on the GSM8K dataset.
We tested it based on the `RANDOM` dataset.
```shell
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 120 --random-range-ratio 1