Files
sglang/python/sglang
laoyao0822 07c9544737 Preserve transferred EAGLE state past metadata-slot reuse
Decode committed EAGLE top-k and hidden-state tensors as views into reusable metadata-buffer rows. The metadata index is freed immediately after transfer commit, while the request may wait before process_prebuilt consumes the draft state. Under concurrent cache-hit traffic a later transfer can overwrite the same row, leaving output_id copied correctly but EAGLE draft state corrupted, which matches low accept length despite successful KV/state registration.

Constraint: Metadata slots are intentionally recycled right after transfer commit for throughput.

Rejected: Hold metadata slots until process_prebuilt | larger lifetime change and reduces transfer capacity; cloning the small prebuilt EAGLE state is narrower.

Confidence: high

Scope-risk: narrow

Directive: Do not store reusable metadata-buffer views on Req unless the slot lifetime is extended through all consumers.

Tested: Local py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 container py_compile for decode.py and test_decode_queue_compaction.py.

Tested: Remote g0034 focused clone-lifetime test: 1 passed.

Tested: Remote g0034 test_decode_queue_compaction.py: 10 passed, 5 warnings.

Not-tested: ETE cache-hit accept-length validation after restarting prefill/decode with this synced code.
2026-05-30 03:36:42 +08:00
..

Code Structure

  • eval: The evaluation utilities.
  • lang: The frontend language.
  • multimodal_gen: Inference framework for accelerated image/video generation.
  • srt: The backend engine for running local models. (SRT = SGLang Runtime).
  • test: The test utilities.
  • api.py: The public APIs.
  • bench_offline_throughput.py: Benchmark the performance in the offline mode.
  • bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
  • bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
  • bench_serving.py: Benchmark online serving with dynamic requests.
  • check_env.py: Check the environment variables and dependencies.
  • global_config.py: The global configs and constants.
  • launch_server.py: The entry point for launching a local server.
  • profiler.py: The profiling entry point to send profile requests.
  • utils.py: Common utilities.
  • version.py: Version info.