[Doc]Update MTP moe backends for EP document (#16013)
This commit is contained in:
@@ -139,3 +139,17 @@ SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.c
|
||||
To enable EPLB, use the flags `--enable-eplb true --load-balance-method eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains.
|
||||
|
||||
For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb).
|
||||
|
||||
|
||||
## EP with Spectulative Decoding
|
||||
|
||||
|
||||
When utilizing speculative decoding with MTP on MoE architectures, use the `--speculative-moe-runner-backend` and `--speculative-moe-a2a-backend` arguments to customize the MoE layer behavior for the draft model. While they default to the target model’s settings, users can differentiate them for varying precisions between target and draft models.
|
||||
|
||||
For model like `nvidia/DeepSeek-R1-0528-NVFP4-v2`, the target model uses NVFP4 precision while the draft model uses BF16. To apply `flashinfer_trtllm` kernel for target MoE layer while falling back to triton fused MoE kernel for draft MoE layer, users can set the arguments as follows:
|
||||
```
|
||||
...
|
||||
--moe-runner-backend flashinfer_trtllm \
|
||||
--speculative-moe-runner-backend triton \
|
||||
...
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user