chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

B300 NVFP4 MoE: slice global input scales to EP-local on the non-flashinfer process_weights path

Browse Source 
The NextN/EAGLE draft runs under speculative_moe_backend_context -> MOE_RUNNER_BACKEND=AUTO (trtllm is
forbidden as a speculative backend), so its NVFP4 MoE takes the plain-FusedMoE 'else' branch of
process_weights_after_loading. That branch left w13_input_scale/w2_input_scale GLOBAL (num_experts=256,
they are tagged _sglang_require_global_experts at load) while w13_weight_scale_2 is EP-local (64) ->
_compute_gemm1_alphas did 256*64 -> RuntimeError. trtllm/cutlass collapse to a scalar and cutedsl already
slices; the else/triton path was a latent EP>1 NVFP4 bug. Slice the global input scales to this rank's
local experts (mirrors the cutedsl _slice_scale). No-op for moe_ep_size==1; main model unaffected
(never takes this branch). opus-diagnosed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
孙远航
2026-06-21 17:02:43 +00:00
parent 1d96197d3d
commit de726b4e7e
1 changed files with 11 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
python/sglang/srt/layers/quantization/modelopt_quant.py
View File

@@ -2027,6 +2027,17 @@ class ModelOptNvFp4FusedMoEMethod(FusedMoEMethodBase):
else: else:
w13_input_scale = layer.w13_input_scale.max(dim=-1).values.to(torch.float32) w13_input_scale = layer.w13_input_scale.max(dim=-1).values.to(torch.float32)
w2_input_scale = layer.w2_input_scale w2_input_scale = layer.w2_input_scale
# w13_input_scale/w2_input_scale are loaded GLOBAL (num_experts) because
# create_weights tags them _sglang_require_global_experts, while
# w13_weight_scale_2 is EP-local (num_local_experts). Under EP>1 this
# non-flashinfer path must slice the global input scales down to this
# rank's local experts so the gemm1/gemm2 alpha products line up
# (mirrors the cutedsl branch above). No-op for moe_ep_size == 1.
if layer.moe_ep_size > 1 and w13_input_scale.shape[0] == layer.num_experts:
start = layer.moe_ep_rank * layer.num_local_experts
end = start + layer.num_local_experts
w13_input_scale = w13_input_scale[start:end]
w2_input_scale = w2_input_scale[start:end]
if self.quant_config.use_per_token_activation: if self.quant_config.use_per_token_activation:
# FlashInfer computes activation scales dynamically per token, so # FlashInfer computes activation scales dynamically per token, so