From c52578c7fd19f64353513006b604b5050132901e Mon Sep 17 00:00:00 2001 From: husf Date: Sat, 31 Jan 2026 12:42:11 +0800 Subject: [PATCH] =?UTF-8?q?=E3=80=90docs=E3=80=91=E3=80=90NPU=E3=80=91Upda?= =?UTF-8?q?te=20Expert=20Parallelism=20docs=20for=20Ascend=20NPU=20(#17940?= =?UTF-8?q?)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/advanced_features/attention_backend.md | 2 +- docs/advanced_features/expert_parallelism.md | 42 +++++++++++++++++++- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/docs/advanced_features/attention_backend.md b/docs/advanced_features/attention_backend.md index 12649c305..046c125d3 100644 --- a/docs/advanced_features/attention_backend.md +++ b/docs/advanced_features/attention_backend.md @@ -27,7 +27,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu | **Dual Chunk FlashAttention** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **AITER (ROCm)** | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | | **Wave (ROCm)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | -| **Ascend (NPU)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | +| **Ascend (NPU)** | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | | **Intel XPU** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | | **Intel AMX (CPU)** | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | diff --git a/docs/advanced_features/expert_parallelism.md b/docs/advanced_features/expert_parallelism.md index bb4396726..9abe69b5a 100644 --- a/docs/advanced_features/expert_parallelism.md +++ b/docs/advanced_features/expert_parallelism.md @@ -21,7 +21,7 @@ SGLang's EP integrates diverse, highly efficient backends for different use case DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes. -Currently, DeepEP, Mooncake and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. +Currently, DeepEP, Mooncake, `ascend_fuseep` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. ### Backends for MoE Computation @@ -156,3 +156,43 @@ For model like `nvidia/DeepSeek-R1-0528-NVFP4-v2`, the target model uses NVFP4 p --speculative-moe-runner-backend triton \ ... ``` + + +## Ascend NPU Guidance + + +### Guidance on SGLang configuration in Ascend NPU +- `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends, + - `deepep`: The mechanism is consistent with the above description. + - `ascend_fuseep`: Offer a large fused operator which integrates all operations between dispatch and combine to boost MoE computation. Only used for decode stage in PD Disaggregation Mode. +- `--moe-runner-backend` parameter does not need to be configured. +- `--deepep-mode`: + - In PD mixed mode, please set `--deepep-mode auto`. + - In PD Disaggregation Mode, prefill instance sets `--deepep-mode normal`, and decode instance sets `--deepep-mode low_latency`. + + +### DeepEP Ascend Introduction + +DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP). +It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences. + +Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables: +- `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS`: Enable ant-moving function in dispatch stage. Indicates the number of tokens transmitted per round on each rank, default 8192. +- `DEEPEP_NORMAL_LONG_SEQ_ROUND`: Enable ant-moving function in dispatch stage. Indicates the number of rounds transmitted on each rank, default 1. +- `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ`: Enable ant-moving function in combine stage, default 0 (means disabled). + +`DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * DEEPEP_NORMAL_LONG_SEQ_ROUND` means input sequence length. When the input sequence length exceeds 8192, it is recommended to enable the ant-moving function in both dispatch and combine phase. + +The environment variable `HCCL_BUFFSIZE` is used to configure the buffer size (MB) actually allocated. Its calculation formula is as follows: +```angular2html +# Enable Ant-moving Function +HCCL_BUFFSIZE >= 2 * (102MB + 4MB + DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS * (hidden_size + hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE + +# Disable Ant-moving Function +HCCL_BUFFSIZE >= 2 * (102MB + 4MB + TOTAL_SEQ_LEN * (hidden_size + hidden_size) * topk) + PADDING_BUFFSIZE +``` +Wherein the parameters are described as follows: +- `hidden_size`: hidden size in model config. +- `topk`: The number of selected routing experts. +- `TOTAL_SEQ_LEN`: input sequence length. +- `PADDING_BUFFSIZE`: A value of 20 or greater is recommended.