add doc for dsv32 cp+pp (#16916)

This commit is contained in:
ybyang
2026-01-12 19:14:07 +08:00
committed by GitHub
parent 2b3791ed37
commit aab640c99f

View File

@@ -186,6 +186,7 @@ Latency: 29.545 s
Output throughput: 4418.617 token/s
```
### Accuracy Test with `gpqa-diamond`
Accuracy benchmark on long context can be tested on GPQA-diamond dataset with long output tokens and thinking enabled:
@@ -321,3 +322,116 @@ Example usage:
# Launch with FusedMoe + CP8
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode round-robin-split --max-running-requests 32
```
### Pipeline Parallel + Context Parallel (PP + CP)
This mode combines Pipeline Parallelism (PP) and Context Parallelism (CP) to scale across multiple nodes, which can achieve better throughput and Time To First Token (TTFT). Note that this method has only been tested on H20 96G.
#### Standard Usage
To launch with PP=2 and CP (via `round-robin-split` mode) on 2 nodes. This configuration uses the fused MoE kernel by default, which generally provides better performance.
For related development details, please refer to:
- Fused MoE + CP support: [PR #13959](https://github.com/sgl-project/sglang/pull/13959)
- PP + CP support: [Issue #15358](https://github.com/sgl-project/sglang/issues/15358) and [PR #16380](https://github.com/sgl-project/sglang/pull/16380)
Node 0:
```bash
export SGLANG_PP_LAYER_PARTITION=30,31
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--nnodes 2 --node-rank 0 \
--dist-init-addr <HEAD_NODE_IP>:62001 \
--tp 8 --pp-size 2 \
--dp-size 1 --moe-dense-tp-size 1 \
--enable-nsa-prefill-context-parallel \
--nsa-prefill-cp-mode round-robin-split \
--trust-remote-code \
--disable-radix-cache \
--mem-fraction-static 0.8 \
--max-running-requests 128 \
--chunked-prefill-size 16384 \
--cuda-graph-max-bs 8 \
--page-size 64 \
--watchdog-timeout 3600 \
--host 0.0.0.0 --port 8000 \
--tool-call-parser deepseekv32
```
Node 1:
```bash
export SGLANG_PP_LAYER_PARTITION=30,31
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--nnodes 2 --node-rank 1 \
--dist-init-addr <HEAD_NODE_IP>:62001 \
--tp 8 --pp-size 2 \
--dp-size 1 --moe-dense-tp-size 1 \
--enable-nsa-prefill-context-parallel \
--nsa-prefill-cp-mode round-robin-split \
--trust-remote-code \
--disable-radix-cache \
--mem-fraction-static 0.8 \
--max-running-requests 128 \
--chunked-prefill-size 16384 \
--cuda-graph-max-bs 8 \
--page-size 64 \
--watchdog-timeout 3600 \
--host 0.0.0.0 --port 8000 \
--tool-call-parser deepseekv32
```
#### PD Disaggregation with PP + CP
If using PD (Prefill-Decode) Disaggregation, the Prefill nodes can be configured with PP + CP as follows.
Prefill Node 0:
```bash
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--served-model-name deepseek-v32 \
--nnodes 2 --node-rank 0 \
--dist-init-addr <PREFILL_HEAD_IP>:20102 \
--tp 8 --pp-size 2 \
--dp-size 1 --moe-dense-tp-size 1 \
--enable-nsa-prefill-context-parallel \
--nsa-prefill-cp-mode continuous-split \
--disaggregation-ib-device mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 \
--trust-remote-code \
--disable-radix-cache \
--max-running-requests 512 \
--chunked-prefill-size 4096 \
--context-length 131072 \
--mem-fraction-static 0.9 \
--page-size 64 \
--enable-metrics \
--collect-tokens-histogram \
--tokenizer-worker-num 8 \
--host 0.0.0.0 --port 30000
```
Prefill Node 1:
```bash
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--served-model-name deepseek-v32-prefill \
--nnodes 2 --node-rank 1 \
--dist-init-addr <PREFILL_HEAD_IP>:20102 \
--tp 8 --pp-size 2 \
--dp-size 1 --moe-dense-tp-size 1 \
--enable-nsa-prefill-context-parallel \
--nsa-prefill-cp-mode continuous-split \
--disaggregation-ib-device mlx5_bond_0,mlx5_bond_1,mlx5_bond_2,mlx5_bond_3 \
--trust-remote-code \
--disable-radix-cache \
--max-running-requests 512 \
--chunked-prefill-size 4096 \
--context-length 131072 \
--mem-fraction-static 0.9 \
--page-size 64 \
--enable-metrics \
--collect-tokens-histogram \
--tokenizer-worker-num 8 \
--host 0.0.0.0 --port 30000
```
For the Decode nodes, it is recommended to use the **EP mode**.