chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU][docs]fix bug about hyperlink for best practice for ascend npu (#18561)

Browse Source
This commit is contained in:
husf
2026-02-11 01:03:28 +08:00
committed by GitHub
parent 44603764d6
commit 573ff55814
1 changed files with 71 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files

144
docs/platforms/ascend_npu_best_practice.md
View File

@@ -7,59 +7,59 @@ you encounter issues or have any questions, please [open an issue](https://githu
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-------------------|---------------|-------|---------------|-----------|------|--------------|--------------------------------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K-1.6K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K-1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-39k1k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k1k-20ms-on-a3-32-cards-separation-mode) |
| DeepSeek-V3.2-Exp | Atlas 800I A3 | 32 | PD Separation | 64K-3K | 30ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v32-exp-64k3k-30ms-on-a3-32-cards-separation-mode) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-------------------|---------------|-------|---------------|-----------|------|--------------|---------------------------------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 6K+1.6K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-6k-1_6k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.9K+1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-3_9k-1k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K+1.5K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-3_5k-1_5k-20ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K+1K | 20ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-3_5k-1k-20ms-on-a3-32-cards-separation-mode) |
| DeepSeek-V3.2-Exp | Atlas 800I A3 | 32 | PD Separation | 64K+3K | 30ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v32-exp-64k-3k-30ms-on-a3-32-cards-separation-mode) |
### High Throughput
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-------------|---------------|-------|---------------|-----------|------|--------------|----------------------------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k2k-50ms-on-a3-8-cards-mixed-mode) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K-2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k2k-50ms-on-a3-16-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-8-cards-mixed-mode) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K-1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-35k15k-50ms-on-a3-16-cards-separation-mode) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-------------|---------------|-------|---------------|-----------|------|--------------|-------------------------------------------------------------------------------------|
| Deepseek-R1 | Atlas 800I A3 | 32 | PD Separation | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-3_5k-1_5k-50ms-on-a3-32-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 2K+2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k-2k-50ms-on-a3-8-cards-mixed-mode) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 2K+2K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-2k-2k-50ms-on-a3-16-cards-separation-mode) |
| Deepseek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-3_5k-1_5k-50ms-on-a3-8-cards-mixed-mode) |
| Deepseek-R1 | Atlas 800I A3 | 16 | PD Separation | 3.5K+1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-3_5k-1_5k-50ms-on-a3-16-cards-separation-mode) |
## Qwen Series Models
### Low Latency
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-----------------|---------------|-------|-------------|---------|------|--------------|-------------------------------------------------------------------------------|
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11K-1K | 10ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-11k1k-10ms-on-a3-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K-1.5K | 18ms | BF16 | [Optimal Configuration](#qwen3-32b-6k15k-18ms-on-a3-4-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K-1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k15k-11ms-on-a3-4-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K-4K | 12ms | BF16 | [Optimal Configuration](#qwen3-32b-18k4k-12ms-on-a3-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K-1.5K | 18ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-6k15k-18ms-on-a2-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K-1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k15k-11ms-on-a2-8-cards-mixed-mode) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|-----------------|---------------|-------|-------------|---------|------|--------------|--------------------------------------------------------------------------------|
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11K+1K | 10ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-11k-1k-10ms-on-a3-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 6K+1.5K | 18ms | BF16 | [Optimal Configuration](#qwen3-32b-6k-1_5k-18ms-on-a3-4-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 4 | PD Mixed | 4K+1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k-1_5k-11ms-on-a3-4-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K+4K | 12ms | BF16 | [Optimal Configuration](#qwen3-32b-18k-4k-12ms-on-a3-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 6K+1.5K | 18ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-6k-1_5k-18ms-on-a2-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 4K+1.5K | 11ms | BF16 | [Optimal Configuration](#qwen3-32b-4k-1_5k-11ms-on-a2-8-cards-mixed-mode) |
### High Throughput
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|--------------------------------|---------------|-------|---------------|-----------|-------|--------------|-----------------------------------------------------------------------------------------------------|
| Qwen3-235B-A22B | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-35k15k-50ms-on-a3-24-cards-separation-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-35k15k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 100ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-100ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 16 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k2k-50ms-on-a3-16-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-35k15k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k2k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-35k15k-50ms-on-a3-1-card-mixed-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 24 | PD Separation | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-24-cards-separation-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 16 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-16-cards-mixed-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-35k15k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80B-a3b-instruct-35k15k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K-1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-35k15k-50ms-on-a2-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K-2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k2k-50ms-on-a2-8-cards-mixed-mode) |
| Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration |
|--------------------------------|---------------|-------|---------------|-----------|-------|--------------|--------------------------------------------------------------------------------------------------------|
| Qwen3-235B-A22B | Atlas 800I A3 | 24 | PD Separation | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-3_5k-1_5k-50ms-on-a3-24-cards-separation-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-3_5k-1_5k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K+2K | 100ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k-2k-100ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 2K+2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k-2k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-235B-A22B | Atlas 800I A3 | 16 | PD Mixed | 2K+2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-2k-2k-50ms-on-a3-16-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-3_5k-1_5k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 2K+2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k-2k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-3_5k-1_5k-50ms-on-a3-1-card-mixed-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 24 | PD Separation | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-3_5k-1_5k-50ms-on-a3-24-cards-separation-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 16 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-3_5k-1_5k-50ms-on-a3-16-cards-mixed-mode) |
| Qwen3-Coder-480B-A35B-Instruct | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-coder-480b-a35b-instruct-3_5k-1_5k-50ms-on-a3-8-cards-mixed-mode) |
| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80B-a3b-instruct-3_5k-1_5k-50ms-on-a3-2-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-3_5k-1_5k-50ms-on-a2-8-cards-mixed-mode) |
| Qwen3-32B | Atlas 800I A2 | 8 | PD Mixed | 2K+2K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-2k-2k-50ms-on-a2-8-cards-mixed-mode) |
## Optimal Configuration
### DeepSeek-R1 3.5K+1.5K 50ms on A3 32 Cards Separation Mode
### DeepSeek-R1 3_5K-1_5K 50ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -178,7 +178,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 3072 --random-range-ratio 1 --request-rate 16
```
### DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode
### DeepSeek-R1 6K-1_6K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -295,7 +295,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 6000 --random-output-len 1600 --num-prompts 32 --random-range-ratio 1
```
### DeepSeek-R1 3.9K+1K 20ms on A3 32 Cards Separation Mode
### DeepSeek-R1 3_9K-1K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -311,8 +311,7 @@ TPOT: 20ms
#### Model Deployment
Please Turn
to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode)
Please Turn to [DeepSeek-R1 6K-1_6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k-1_6k-20ms-on-a3-32-cards-separation-mode)
#### Benchmark
@@ -322,7 +321,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3900 --random-output-len 1000 --num-prompts 768 --random-range-ratio 1 --request-rate 16
```
### DeepSeek-R1 3.5K+1.5K 20ms on A3 32 Cards Separation Mode
### DeepSeek-R1 3_5K-1_5K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -338,7 +337,7 @@ TPOT: 20ms
#### Model Deployment
Please Turn to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode)
Please Turn to [DeepSeek-R1 6K-1_6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k-1_6k-20ms-on-a3-32-cards-separation-mode)
#### Benchmark
@@ -348,7 +347,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1500 --num-prompts 768 --random-range-ratio 1 --request-rate 16
```
### DeepSeek-R1 3.5K+1K 20ms on A3 32 Cards Separation Mode
### DeepSeek-R1 3_5K-1K 20ms on A3 32 Cards Separation Mode
Model: Deepseek R1
@@ -364,8 +363,7 @@ TPOT: 20ms
#### Model Deployment
Please Turn
to [DeepSeek-R1 6K+1.6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k16k-20ms-on-a3-32-cards-separation-mode)
Please Turn to [DeepSeek-R1 6K-1_6K 20ms on A3 32 Cards Separation Mode](#deepseek-r1-6k-1_6k-20ms-on-a3-32-cards-separation-mode)
#### Benchmark
@@ -375,7 +373,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 768 --random-input-len 3500 --random-output-len 1000 --num-prompts 768 --random-range-ratio 1 --request-rate 16
```
### DeepSeek-R1 2K+2K 50ms on A3 8 Cards Mixed Mode
### DeepSeek-R1 2K-2K 50ms on A3 8 Cards Mixed Mode
Model: Deepseek R1
@@ -455,7 +453,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 256 --random-input-len 2048 --random-output-len 2048 --num-prompts 1024 --random-range-ratio 1
```
### DeepSeek-R1 2K+2K 50ms on A3 16 Cards Separation Mode
### DeepSeek-R1 2K-2K 50ms on A3 16 Cards Separation Mode
Model: Deepseek R1
@@ -580,7 +578,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 400 --random-input-len 2048 --random-output-len 2048 --num-prompts 3200 --random-range-ratio 1 --request-rate 8
```
### DeepSeek-R1 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode
### DeepSeek-R1 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode
Model: Deepseek R1
@@ -657,7 +655,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 144 --random-input-len 3500 --random-output-len 1500 --num-prompts 576 --random-range-ratio 1
```
### DeepSeek-R1 3.5K+1.5K 50ms on A3 16 Cards Separation Mode
### DeepSeek-R1 3_5K-1_5K 50ms on A3 16 Cards Separation Mode
Model: Deepseek R1
@@ -781,7 +779,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 384 --random-input-len 3500 --random-output-len 1500 --num-prompts 1536 --random-range-ratio 1
```
### DeepSeek-V3.2-Exp 64K+3K 30ms on A3 32 Cards Separation Mode
### DeepSeek-V3.2-Exp 64K-3K 30ms on A3 32 Cards Separation Mode
Model: DeepSeek-V3.2-Exp-W8A8
@@ -992,7 +990,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 32 --random-input-len 64000 --random-output-len 3000 --num-prompts 64 --random-range-ratio 1
```
### Qwen3-235B-A22B 3.5K+1.5K 50ms on A3 24 Cards Separation Mode
### Qwen3-235B-A22B 3_5K-1_5K 50ms on A3 24 Cards Separation Mode
Model: Qwen3-235B-A22B-W8A8
@@ -1128,7 +1126,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang-oai --host 127.0.0.1 --port 7239 --max-concurrency 860 --random-input-len 3500 --random-output-len 1500 --num-prompts 3440 --random-range-ratio 1
```
### Qwen3-235B-A22B 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode
### Qwen3-235B-A22B 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode
Model: Qwen3-235B-A22B-W8A8
@@ -1201,7 +1199,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 272 --random-input-len 3500 --random-output-len 1500 --num-prompts 1088 --random-range-ratio 1
```
### Qwen3-235B-A22B 2K+2K 100ms on A3 8 Cards Mixed Mode
### Qwen3-235B-A22B 2K-2K 100ms on A3 8 Cards Mixed Mode
Model: Qwen3-235B-A22B-W8A8
@@ -1273,7 +1271,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 576 --random-input-len 2000 --random-output-len 2000 --num-prompts 576 --random-range-ratio 1
```
### Qwen3-235B-A22B 2K+2K 50ms on A3 8 Cards Mixed Mode
### Qwen3-235B-A22B 2K-2K 50ms on A3 8 Cards Mixed Mode
Model: Qwen3-235B-A22B-W8A8
@@ -1345,7 +1343,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 480 --random-input-len 2048 --random-output-len 2048 --num-prompts 480 --random-range-ratio 1
```
### Qwen3-235B-A22B 2K+2K 50ms on A3 16 Cards Mixed Mode
### Qwen3-235B-A22B 2K-2K 50ms on A3 16 Cards Mixed Mode
Model: Qwen3-235B-A22B-W8A8
@@ -1430,7 +1428,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 768 --random-input-len 2000 --random-output-len 2000 --num-prompts 768 --random-range-ratio 1
```
### Qwen3-235B-A22B 11K+1K 10ms on A3 8 Cards Mixed Mode
### Qwen3-235B-A22B 11K-1K 10ms on A3 8 Cards Mixed Mode
Model: Qwen3-235B-A22B-W8A8
@@ -1503,7 +1501,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 1 --random-input-len 11000 --random-output-len 1000 --num-prompts 1 --random-range-ratio 1
```
### Qwen3-32B 6K+1.5K 18ms on A3 4 Cards Mixed Mode
### Qwen3-32B 6K-1_5K 18ms on A3 4 Cards Mixed Mode
Model: Qwen3-32B
@@ -1574,7 +1572,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1
```
### Qwen3-32B 4K+1.5K 11ms on A3 4 Cards Mixed Mode
### Qwen3-32B 4K-1_5K 11ms on A3 4 Cards Mixed Mode
Model: Qwen3-32B
@@ -1644,7 +1642,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4
```
### Qwen3-32B 18K+4K 12ms on A3 8 Cards Mixed Mode
### Qwen3-32B 18K-4K 12ms on A3 8 Cards Mixed Mode
Model: Qwen3-32B
@@ -1713,7 +1711,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 18000 --random-input-len 4000 --num-prompts 1
```
### Qwen3-32B 3.5K+1.5K 50ms on A3 2 Cards Mixed Mode
### Qwen3-32B 3_5K-1_5K 50ms on A3 2 Cards Mixed Mode
Model: Qwen3-32B
@@ -1783,7 +1781,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1
```
### Qwen3-32B 2K+2K 50ms on A3 2 Cards Mixed Mode
### Qwen3-32B 2K-2K 50ms on A3 2 Cards Mixed Mode
Model: Qwen3-32B
@@ -1853,7 +1851,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 480 --random-range-ratio 1
```
### Qwen3-30B-A3B 3.5K+1.5K 50ms on A3 1 Card Mixed Mode
### Qwen3-30B-A3B 3_5K-1_5K 50ms on A3 1 Card Mixed Mode
Model: Qwen3-30B-A3B-Instruct-2507
@@ -1923,7 +1921,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 156 --random-input-len 3500 --random-output-len 1500 --num-prompts 624 --random-range-ratio 1
```
### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 24 Cards Separation Mode
### Qwen3-Coder-480B-A35B-Instruct 3_5K-1_5K 50ms on A3 24 Cards Separation Mode
Model: Qwen3-Coder-480B-A35B-Instruct
@@ -2047,7 +2045,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 410 --random-input-len 3500 --random-output-len 1500 --num-prompts 1640 --random-range-ratio 1 --request-rate 8
```
### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 16 Cards Mixed Mode
### Qwen3-Coder-480B-A35B-Instruct 3_5K-1_5K 50ms on A3 16 Cards Mixed Mode
Model: Qwen3-Coder-480B-A35B-Instruct
@@ -2130,7 +2128,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 288 --random-input-len 3500 --random-output-len 1500 --num-prompts 1152 --random-range-ratio 1 --request-rate 20
```
### Qwen3-Coder-480B-A35B-Instruct 3.5K+1.5K 50ms on A3 8 Cards Mixed Mode
### Qwen3-Coder-480B-A35B-Instruct 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode
Model: Qwen3-Coder-480B-A35B-Instruct
@@ -2198,7 +2196,7 @@ We tested it based on the `RANDOM` dataset.
python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 80 --random-input-len 3500 --random-output-len 1500 --num-prompts 320 --random-range-ratio 1
```
### Qwen3-Next-80B-A3B-Instruct 3.5K+1.5K 50ms on A3 2 Cards Mixed Mode
### Qwen3-Next-80B-A3B-Instruct 3_5K-1_5K 50ms on A3 2 Cards Mixed Mode
Model: Qwen3-Next-80B-A3B-Instruct
@@ -2267,7 +2265,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 80 --random-output-len 1536 --random-input-len 3584 --num-prompts 160 --random-range-ratio 1
```
### Qwen3-32B 6K+1.5K 18ms on A2 8 Cards Mixed Mode
### Qwen3-32B 6K-1_5K 18ms on A2 8 Cards Mixed Mode
Model: Qwen3-32B
@@ -2336,7 +2334,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1
```
### Qwen3-32B 4K+1.5K 11ms on A2 8 Cards Mixed Mode
### Qwen3-32B 4K-1_5K 11ms on A2 8 Cards Mixed Mode
Model: Qwen3-32B
@@ -2409,7 +2407,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4
```
### Qwen3-32B 3.5K+1.5K 50ms on A2 8 Cards Mixed Mode
### Qwen3-32B 3_5K-1_5K 50ms on A2 8 Cards Mixed Mode
Model: Qwen3-32B
@@ -2479,7 +2477,7 @@ We tested it based on the `RANDOM` dataset.
python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1
```
### Qwen3-32B 2K+2K 50ms on A2 8 Cards Mixed Mode
### Qwen3-32B 2K-2K 50ms on A2 8 Cards Mixed Mode
Model: Qwen3-32B