Files
sglang/python/sglang
leavelet de726b4e7e B300 NVFP4 MoE: slice global input scales to EP-local on the non-flashinfer process_weights path
The NextN/EAGLE draft runs under speculative_moe_backend_context -> MOE_RUNNER_BACKEND=AUTO (trtllm is
forbidden as a speculative backend), so its NVFP4 MoE takes the plain-FusedMoE 'else' branch of
process_weights_after_loading. That branch left w13_input_scale/w2_input_scale GLOBAL (num_experts=256,
they are tagged _sglang_require_global_experts at load) while w13_weight_scale_2 is EP-local (64) ->
_compute_gemm1_alphas did 256*64 -> RuntimeError. trtllm/cutlass collapse to a scalar and cutedsl already
slices; the else/triton path was a latent EP>1 NVFP4 bug. Slice the global input scales to this rank's
local experts (mirrors the cutedsl _slice_scale). No-op for moe_ep_size==1; main model unaffected
(never takes this branch). opus-diagnosed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-21 17:02:43 +00:00
..

Code Structure

  • eval: The evaluation utilities.
  • lang: The frontend language.
  • multimodal_gen: Inference framework for accelerated image/video generation.
  • srt: The backend engine for running local models. (SRT = SGLang Runtime).
  • test: The test utilities.
  • api.py: The public APIs.
  • bench_offline_throughput.py: Benchmark the performance in the offline mode.
  • bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
  • bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
  • bench_serving.py: Benchmark online serving with dynamic requests.
  • check_env.py: Check the environment variables and dependencies.
  • global_config.py: The global configs and constants.
  • launch_server.py: The entry point for launching a local server.
  • profiler.py: The profiling entry point to send profile requests.
  • utils.py: Common utilities.
  • version.py: Version info.