Files
sglang/docs/advanced_features
laoyao0822 ffff715f00 Plan bs>1 CP shared-KV L1 prefetch safely
Record the staged design for batch-aware L1 shared-KV prefetch under CP + HiCache. The plan separates L2->L1 load two-layer lookahead from L1 prefetch one-layer lookahead, keeps target first and draft/EAGLE out of the first phase, and calls out that current suffix reuse remains synchronous until the request-aware span contract is extended.

Constraint: Existing CP shared-KV correctness failures came from batch-global remap/req-id ambiguity, so bs>1 prefetch must carry explicit per-request spans and ids.

Rejected: Reuse the bs=1 scalar prefix_pages prefetcher for bs>1 | it aliases request rows and cannot represent mixed prefix/current spans.

Rejected: Make draft/EAGLE prefetch part of the first phase | target path must be validated first before coupling draft lifecycle.

Confidence: medium

Scope-risk: narrow

Directive: Do not open bs>1 L1 prefetch until req-id fail-fast and per-request slot span tests cover both MLA and index paths.

Tested: git diff --cached --check

Not-tested: Runtime implementation; this is a design document only.
2026-06-10 07:04:39 +08:00
..
2025-11-16 19:36:33 -08:00