Files
sglang/python
laoyao0822 4f65d7a176 Bound CP prefill batches by cached-token pressure
High cache-hit CP shared-KV batches can have small extend tokens while still carrying substantial prefix/load-back work. Add a CP-specific cached-token admission limit so operators can bound that pressure independently from extend-token batching.

Constraint: The limit must not deadlock a single high-cache-hit request; it only stops adding additional requests to a non-empty batch.

Constraint: Cached tokens are counted after L2 load-back planning via prefix_len, so L1 hits and successful L2 hits share one scheduler budget.

Rejected: Reuse max_prefill_tokens | it limits generic input budget and does not represent cached-token work.

Rejected: Count only L1 prefix before load-back | would miss L2 hit pressure, which is one of the target cases.

Confidence: high

Scope-risk: moderate

Directive: Keep cached-token and extend-token limits separate; they bound different scheduler costs.

Tested: Remote pytest targeted cached-token PrefillAdder cases: 2 passed.

Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 18 passed.

Tested: Remote ServerArgs CP validation smoke: SERVER_ARGS_CACHED_LIMIT_OK.

Not-tested: Full ETE replay with a production cached-token limit value.
2026-06-10 22:21:46 +08:00
..
2026-06-10 05:54:43 +08:00