Files
sglang/python/sglang
laoyao0822 187b700406 Keep speculative grammar traversal scalar-safe
EAGLE verification can pass torch scalar tensors through the speculative tree traversal before calling grammar backends. xgrammar/tvm_ffi requires Python int token ids, so normalize traversal indices and draft token ids at the boundary while leaving tensor storage unchanged.

Constraint: xgrammar/tvm_ffi rejects torch scalar tensors for GrammarMatcher.accept_token
Rejected: Coerce tokens inside every grammar backend | the invalid value originates in speculative traversal and should be fixed before backend dispatch
Confidence: high
Scope-risk: narrow
Directive: Keep grammar backend calls scalar-Python typed; do not pass torch scalar tensors through accept_token
Tested: remote g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/speculative/test_spec_utils.py test/registered/unit/configs/test_nsa_index_layers.py test/registered/unit/models/test_deepseek_index_skip_weight_loading.py -> 19 passed
Tested: remote g0034 cjy-glm5-new py_compile for modified runtime files
Not-tested: live decode replay after this commit
2026-06-21 05:24:05 +08:00
..

Code Structure

  • eval: The evaluation utilities.
  • lang: The frontend language.
  • multimodal_gen: Inference framework for accelerated image/video generation.
  • srt: The backend engine for running local models. (SRT = SGLang Runtime).
  • test: The test utilities.
  • api.py: The public APIs.
  • bench_offline_throughput.py: Benchmark the performance in the offline mode.
  • bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
  • bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
  • bench_serving.py: Benchmark online serving with dynamic requests.
  • check_env.py: Check the environment variables and dependencies.
  • global_config.py: The global configs and constants.
  • launch_server.py: The entry point for launching a local server.
  • profiler.py: The profiling entry point to send profile requests.
  • utils.py: Common utilities.
  • version.py: Version info.