Files
sglang/python/sglang
laoyao0822 a7472c415f Preserve request boundaries in CP shared-KV index top-k
W4-1 needs target index/top-k sync correctness before current/partial-current reuse can be restored. Batch-size>1 in-seq CP produces local q/weights in request-segment order, so top-k must consume req0_prev, req0_next, req1_prev, req1_next rather than treating the flattened batch as one scalar prev/next pair.

The implementation adds a batch dispatch for _get_topk_in_seq_cp_pair, reuses one synchronous shared-index materialization per layer, and calls _get_topk_ragged_with_cp per request segment with an explicit batch_idx. The scalar bs=1 path remains unchanged.

Constraint: This is W4-1 target index/top-k sync correctness; original W4 current/partial-current reuse remains a separate follow-up.

Constraint: Phase W4-1 must not enable bs>1 index prefetch, current reuse, partial-current reuse, or the cp_index multi-batch branch.

Rejected: Use cp_index branch for multi-batch | source marks that path as having accuracy issues.

Rejected: Pad batch requests to max length | wastes compute and violates packed/ragged batch contract.

Confidence: high

Scope-risk: moderate

Directive: Keep bs>1 target top-k ordered by request segment unless a later fused descriptor proves identical ordering and correctness.

Tested: Remote g0034 py_compile for nsa_indexer.py

Tested: Remote g0034 PYTHONPATH=python pytest test/registered/unit/layers/test_nsa_cp_utils.py -> 45 passed

Tested: Remote g0034 PYTHONPATH=python pytest test_nsa_cp_utils.py test_cp_shared_kv_layout.py test_cp_shared_kv_runtime.py -> 172 passed, 2 subtests passed

Not-tested: Full ETE bs>1 serving run with live traffic
2026-06-03 02:36:31 +08:00
..

Code Structure

  • eval: The evaluation utilities.
  • lang: The frontend language.
  • multimodal_gen: Inference framework for accelerated image/video generation.
  • srt: The backend engine for running local models. (SRT = SGLang Runtime).
  • test: The test utilities.
  • api.py: The public APIs.
  • bench_offline_throughput.py: Benchmark the performance in the offline mode.
  • bench_one_batch.py: Benchmark the latency of running a single static batch without a server.
  • bench_one_batch_server.py: Benchmark the latency of running a single batch with a server.
  • bench_serving.py: Benchmark online serving with dynamic requests.
  • check_env.py: Check the environment variables and dependencies.
  • global_config.py: The global configs and constants.
  • launch_server.py: The entry point for launching a local server.
  • profiler.py: The profiling entry point to send profile requests.
  • utils.py: Common utilities.
  • version.py: Version info.