chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

[Doc] Refine fused_moe_triton configs doc (#13820)

Browse Source
This commit is contained in:
Xiaoyu Zhang
2025-11-24 11:09:41 +08:00
committed by GitHub
parent dbf22152d6
commit 9ea1953331
1 changed files with 36 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
python/sglang/srt/layers/moe/fused_moe_triton/configs/README.md
View File

@@ -1,15 +1,40 @@
# Fused MoE Triton Kernel Configurations
This directory contains tuned configurations for different settings of the fused_moe kernel.
For different settings of
- E (number of experts)
- N (intermediate size)
- device_name (torch.cuda.get_device_name())
- dtype: The data type used by the fused MoE kernel for computation. Supported types include fp8_w8a8, int8_w8a8, int8_w8a16, int4_w4a16, etc. This determines the precision and quantization scheme for both weights and activations.
- block_shape: The block quantization shape introduced starting from DeepSeek V3/R1 models. This parameter defines the granularity for block-wise quantization, typically specified as `[block_n, block_k]` where `block_n` and `block_k` represent the block dimensions. For example, DeepSeek V3 commonly uses `[128, 128]` block shapes for efficient block-wise FP8 quantization.
the JSON file contains a mapping from M (batch size) to the chosen configuration.
## Configuration Parameters
The example configurations provided are for the Mixtral model for TP2 on H100
and TP4 on A100. Mixtral has intermediate size N = 14336, i.e. for TP2 we have
N = 7168 and for TP4 we have N = 3584.
Each configuration file is generated based on the following parameters:
See `benchmark/kernels/fused_moe_triton/README.md` on how to generate these config files.
- **E** (number of experts): Total number of experts in the MoE layer
- **N** (intermediate size): The intermediate/hidden dimension size
- For Tensor Parallelism (TP): `N = original_intermediate_size / tp_size`
- Example: Mixtral has N = 14336. For TP=2, N = 7168; for TP=4, N = 3584
- **device_name**: GPU device name from `torch.cuda.get_device_name()`
- Examples: `NVIDIA_H100_80GB_HBM3`, `NVIDIA_A100-SXM4-80GB`, `NVIDIA_GeForce_RTX_4090`
- **dtype**: Data type for computation
- Supported types: `fp8_w8a8`, `int8_w8a8`, `int8_w8a16`, `int4_w4a16`, etc.
- Determines precision and quantization scheme for weights and activations
- **block_shape**: Block quantization shape (for DeepSeek V3/R1 models)
- Defines granularity for block-wise quantization, specified as `[block_n, block_k]`
- Example: DeepSeek V3 commonly uses `[128, 128]` for efficient block-wise FP8 quantization
- **tp_size**: Tensor Parallelism size (affects N parameter)
- **ep_size**: Expert Parallelism size (affects E parameter when EP is enabled)
- **per_channel_quant**: Whether per-channel quantization is used
## Configuration File Format
Each JSON file contains a mapping from **M** (batch size) to the optimal kernel configuration for that batch size. The configuration includes parameters like `BLOCK_M`, `BLOCK_N`, `BLOCK_K`, `GROUP_M`, number of warps, and pipeline stages.
**Filename Format**:
```
E={E},N={N},device_name={device_name},dtype={dtype}[,block_shape={block_shape}][,per_channel_quant={bool}].json
```
## Generating Configuration Files
To generate new configuration files for your specific hardware and model settings, use the tuning tools:
**📖 Full Documentation**: [Tuning Triton MoE Kernels](https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton)
After tuning, move the generated JSON files to this directory to use them in SGLang.