Files
sglang/docs/advanced_features
laoyao0822 0158e28689 Plan batch-aware CP shared-KV execution
The existing CP shared-KV notes still reflected the earlier multi-slot-first conclusion, while current short cache-hit extends require true ForwardBatch>1 planning. This commit records the batch-aware contract, sequential phase plan, and parallel workstream split so implementation can proceed without rediscovering single-request blockers or accidentally removing guards into unsafe fallback paths.

Constraint: Current NSA in-seq CP shared-KV paths assume batch_size == 1 across metadata, owner-lane allocation, direct write, current reuse, prefetch, top-k, and EAGLE/draft collection.

Rejected: Remove batch_size guards directly | would treat multiple requests as one long sequence and corrupt request boundaries and page-owner state.

Rejected: Keep only the old multi-slot recommendation | online extend sizes around 200-2000 tokens make true batching a required throughput direction.

Confidence: high

Scope-risk: narrow

Directive: Implement bs>1 by per-request page-aligned planning before flattening; do not add planner collectives or silent legacy fallbacks.

Tested: Placeholder scan and referenced-path existence check for all four docs; git diff --cached --check.

Not-tested: Runtime behavior; documentation-only change.

Co-authored-by: OmX <omx@oh-my-codex.dev>
2026-06-03 00:37:54 +08:00
..