Files
sglang/python/sglang
laoyao0822 adf357b02c Default CP bs>1 extend admission to chunk budget
When chunked prefill is active, CP shared-KV bs>1 cannot consume more extend
tokens than the current chunk budget. If the CP-specific extend-token limit is
omitted, default it to rem_chunk_tokens so scheduler admission reflects the
reachable chunk capacity. The request-count and cached-token knobs keep their
None-as-unlimited behavior.

Constraint: CP bs>1 batching must not advertise a larger extend batch than chunked prefill can execute.
Rejected: Require users to always set --cp-shared-kv-prefill-max-total-extend-tokens | the safe default is already available from chunked prefill state.
Rejected: Default batch request or cached-token limits | those are policy knobs and None should remain unlimited.
Confidence: high
Scope-risk: narrow
Directive: Keep --cp-shared-kv-prefill-max-total-extend-tokens as min(user_limit, chunk_budget) when both exist.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote g0034 cjy-glm5-new targeted prefill_adder tests: 2 passed.
Not-tested: Full ETE scheduler batching distribution after defaulting the extend limit.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-11 03:51:41 +08:00
..

Code Structure

  • eval: The evaluation utilities.
  • lang: The frontend language.
  • multimodal_gen: Inference framework for accelerated image/video generation.
  • srt: The backend engine for running local models. (SRT = SGLang Runtime).
  • test: The test utilities.
  • api.py: The public APIs.
  • bench_offline_throughput.py: Benchmark the performance in the offline mode.
  • bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
  • bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
  • bench_serving.py: Benchmark online serving with dynamic requests.
  • check_env.py: Check the environment variables and dependencies.
  • global_config.py: The global configs and constants.
  • launch_server.py: The entry point for launching a local server.
  • profiler.py: The profiling entry point to send profile requests.
  • utils.py: Common utilities.
  • version.py: Version info.