chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

feat(eplb): add eplb warmup-only mode

Browse Source 
Freeze expert layout after the initial EPLB warmup rebalances so heavy workloads avoid recurring rebalance memory spikes and OOMs.
This commit is contained in:
laoyao0822
2026-04-08 06:46:26 +08:00
committed by wxiwnd
parent d5ec233834
commit 101100e25b
4 changed files with 33 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/expert_parallelism.md
+1 -1
View File
@@ -141,7 +141,7 @@ SGLang introduces a dispatcher-hook system for Single-Batch Overlap (SBO), enabl
SGLang integrates the [Expert Parallelism Load Balancer (EPLB)](https://github.com/deepseek-ai/EPLB) from DeepSeek to address routing imbalances in MoE models. By analyzing expert activation statistics, EPLB computes an optimal expert arrangement, strategically placing or replicating experts to minimize GPU utilization variance, reduce idle cycles, and enhance scalability.
To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains.
To enable EPLB, use `--enable-eplb`. For optimal performance, increase batch sizes to stabilize activation statistics and configure periodic rebalancing (e.g., every 1000 requests) to adapt to evolving workloads. If you only want EPLB to act during its initial warmup rebalances and then freeze the resulting expert placement, add `--eplb-warmup-only`. Simulations demonstrate significant improvements in load balancedness (ratio of mean to max computation time), correlating strongly with throughput gains.
For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb).
docs/advanced_features/server_arguments.md
+1
View File
@@ -321,6 +321,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
| `--ep-dispatch-algorithm` | The algorithm to choose ranks for redundant experts in expert parallel. | `None` | Type: str |
| `--init-expert-location` | Initial location of EP experts. | `trivial` | Type: str |
| `--enable-eplb` | Enable EPLB algorithm | `False` | bool flag (set to enable) |
| `--eplb-warmup-only` | Run EPLB only during its warmup rebalances, then keep expert layout static. | `False` | bool flag (set to enable) |
| `--eplb-algorithm` | Chosen EPLB algorithm | `auto` | Type: str |
| `--eplb-rebalance-num-iterations` | Number of iterations to automatically trigger a EPLB re-balance. | `1000` | Type: int |
| `--eplb-rebalance-layers-per-chunk` | Number of layers to rebalance per forward pass. | `None` | Type: int |