chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Multiple refactors of DeepSeek V32 and context parallel (#16305)

Browse Source
This commit is contained in:
Baizhou Zhang
2026-01-03 02:21:22 +08:00
committed by GitHub
parent 1cfd2b2ded
commit f07e76b229
7 changed files with 193 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

57
docs/basic_usage/deepseek_v32.md
View File

@@ -274,31 +274,40 @@ DeepSeek-V3.2-Speciale:
## DSA long sequence context parallel optimization(experimental)
Accuracy benchmark on long context can be tested on GPQA-diamond dataset with long output tokens and thinking enabled:
**Note: This feature is only verified on Hopper machines**
For context parallel in DeepSeek V3.2 model, we provide two different modes of splitting tokens, which can be controlled with argument `--nsa-prefill-cp-mode`.
### In sequence splitting (default setting)
The first mode can be enabled by `--nsa-prefill-cp-mode in-seq-split`. This mode implements context parallel for DSA by splitting the sequence uniformly between context parallel ranks. At attention stage, each cp rank computes the indexer results of sharded sequence, and collects the whole kv cache through all gather operator.
The communication group for context parallel reuses the one for attention tp, thus `cp_size` equals `atten_tp_size = tp_size / dp_size`.
Note that in sequence splitting mode has the following restrictions:
- The batch size is restricted to 1 for prefill batches
- Multi-node/PD disaggregation is still not supported
- `moe_dense_tp_size=1`, `kv_cache_dtype = "bf16"`, `moe_a2a_backend = "deepep"`
- To ensure `cp_size > 1`, the passed in `tp_size` must be larger than `dp_size`
For more details, please refer to PR https://github.com/sgl-project/sglang/pull/12065.
Example:
```bash
# In-seq splitting mode launched with EP + DP
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --ep 8 --dp 2 --enable-dp-attention --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode in-seq-split --max-running-requests 32
```
### Round robin splitting
This mode can be enabled by specifying the parameter `--nsa-prefill-cp-mode round-robin-split`, which distributes tokens across ranks based on `token_idx % cp_size`.
In this scenario, compared with the aforementioned method, it additionally supports the fused MoE backend (the fused MoE backend may deliver better performance than DeepEP in single-machine scenarios), FP8 KV-cache, and multi-batch prefill inference. But it cannot be enabled with dp attention together.
For more details, please refer to PR https://github.com/sgl-project/sglang/pull/13959.
Example usage:
```bash
# Launch with EP + DP
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --ep 8 --dp 2 --enable-dp-attention --enable-nsa-prefill-context-parallel --max-running-requests 32
```
### Context-parallel Tips
`CP_size` reuses `atten_tp_size`, which is equal to `TP_size` / `DP_size`.
Some features are still not supported at present.
- **Multi-batch prefill**: Currently, only single-request processing is supported during the prefill process.
- **disaggregation**: P/D disaggregation.
- **Cross-machine support**: - Currently only tested on a single machine (TP=8,EP=8).
- **Other Args**: Currently only supports moe_dense_tp_size=1, kv_cache_dtype = "bf16", moe_a2a_backend = "deepep",
- **DP_size**: `CP_size` reuses `atten_tp_size`, which is equal to `TP_size` / `DP_size`. For the cp function to work correctly, `TP_size` must be divisible by `DP_size`, and TP_size / DP_size > 1 (to ensure CP_size > 1).
- **Detailed design reference**: https://github.com/sgl-project/sglang/pull/12065
### Alternative context parallel mode
You can switch the CP token splitting mode for prefill by specifying the parameter `--nsa-prefill-cp-mode round-robin-split`. It distributes tokens across ranks based on `token_idx % cp_size`.
In this scenario, compared with the aforementioned method, it additionally supports the fused MoE backend (the fused MoE backend may deliver better performance than DeepEP in single-machine scenarios),
FP8 KV-cache, and multi-batch prefill inference. For more details, please refer to PR https://github.com/sgl-project/sglang/pull/13959.
Example usage:
```bash
# Launch with FusedMoe + CP8 + DP1
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 1 --enable-dp-attention --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode round-robin-split --max-running-requests 32
# Launch with FusedMoe + CP8
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --enable-nsa-prefill-context-parallel --nsa-prefill-cp-mode round-robin-split --max-running-requests 32
```