When chunked prefill is active, CP shared-KV bs>1 cannot consume more extend
tokens than the current chunk budget. If the CP-specific extend-token limit is
omitted, default it to rem_chunk_tokens so scheduler admission reflects the
reachable chunk capacity. The request-count and cached-token knobs keep their
None-as-unlimited behavior.
Constraint: CP bs>1 batching must not advertise a larger extend batch than chunked prefill can execute.
Rejected: Require users to always set --cp-shared-kv-prefill-max-total-extend-tokens | the safe default is already available from chunked prefill state.
Rejected: Default batch request or cached-token limits | those are policy knobs and None should remain unlimited.
Confidence: high
Scope-risk: narrow
Directive: Keep --cp-shared-kv-prefill-max-total-extend-tokens as min(user_limit, chunk_budget) when both exist.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote g0034 cjy-glm5-new targeted prefill_adder tests: 2 passed.
Not-tested: Full ETE scheduler batching distribution after defaulting the extend limit.
Co-authored-by: OmX <omx@oh-my-codex.dev>
CP shared-KV bs>1 batching was only bounded by request count, extend tokens, and cached tokens. That left temporary GPU buffers such as MLA/index materialization, remap metadata, logits windows, and transfer descriptors implicit, and raw extend-token limits could exceed the active chunked-prefill budget.\n\nThis adds an explicit max-buffer-size admission gate with a CPU-only stream-aware estimator, wires it through PrefillAdder/Scheduler, performs a startup CUDA smoke allocation when configured, and reports the estimate in the scheduler admission benchmark. When chunked prefill is active, the effective CP extend-token limit is capped by the current chunk budget so the CP path does not advertise unreachable batch capacity or lift max-prefill-tokens too far.\n\nConstraint: Admission estimation must stay CPU-only on the scheduler hot path; CUDA allocation is limited to startup smoke checking.\nConstraint: Single oversized requests must still be allowed to run alone to avoid scheduler deadlock.\nRejected: Rely only on --max-prefill-tokens | it does not reliably bound the first oversized request and does not model cache-hit/load-back pressure.\nRejected: Let CP extend limit exceed chunked-prefill size | it creates an unreachable effective capacity and misleading budget lift.\nConfidence: medium\nScope-risk: moderate\nDirective: If bs>1 L1 prefetch is enabled later, update CPSharedKVPrefillBufferEstimatorContext.bs_gt1_l1_prefetch_enabled and include the live prefetch dense buffers in overlap windows.\nTested: local py_compile for touched files\nTested: local PYTHONPATH=python pytest -q test/registered/unit/managers/test_cp_shared_kv_prefill_buffer_estimator.py (4 passed)\nTested: remote cjy-glm5-new targeted pytest for new server_args, PrefillAdder, estimator, and benchmark cases (10 passed)\nTested: remote cjy-glm5-new PYTHONPATH=python pytest -q test/registered/unit/managers/test_cp_shared_kv_prefill_buffer_estimator.py test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_prefill_scheduler_admission_bench.py (29 passed before chunk cap, then test_prefill_adder.py 21 passed after chunk cap)\nNot-tested: full server_args suite because existing TestPrepareServerArgs tries to reach HuggingFace and fails under container DNS/network\nNot-tested: GLM5 ETE smoke with --cp-shared-kv-prefill-max-buffer-size
High cache-hit CP shared-KV batches can have small extend tokens while still carrying substantial prefix/load-back work. Add a CP-specific cached-token admission limit so operators can bound that pressure independently from extend-token batching.
Constraint: The limit must not deadlock a single high-cache-hit request; it only stops adding additional requests to a non-empty batch.
Constraint: Cached tokens are counted after L2 load-back planning via prefix_len, so L1 hits and successful L2 hits share one scheduler budget.
Rejected: Reuse max_prefill_tokens | it limits generic input budget and does not represent cached-token work.
Rejected: Count only L1 prefix before load-back | would miss L2 hit pressure, which is one of the target cases.
Confidence: high
Scope-risk: moderate
Directive: Keep cached-token and extend-token limits separate; they bound different scheduler costs.
Tested: Remote pytest targeted cached-token PrefillAdder cases: 2 passed.
Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 18 passed.
Tested: Remote ServerArgs CP validation smoke: SERVER_ARGS_CACHED_LIMIT_OK.
Not-tested: Full ETE replay with a production cached-token limit value.
CP shared-KV bs>1 uses cp_shared_kv_prefill_max_total_extend_tokens as its grouping admission limit, but the generic max_prefill_tokens budget could still stop batching earlier. Raise only the legacy input-token admission budget for this CP path while keeping allocator-owned capacity checks unchanged.
Constraint: CP shared-KV bs>1 needs large cache-hit batches without relying on the generic max_prefill_tokens default.
Constraint: Allocator capacity must remain enforced by rem_total_tokens, cur_rem_tokens, and prepare_for_extend().
Rejected: Increase server-wide max_prefill_tokens | would change generic scheduler behavior and non-CP paths.
Confidence: high
Scope-risk: narrow
Directive: Do not use max_prefill_tokens as the CP shared-KV bs>1 grouping limit; use the CP-specific total extend token knob.
Tested: Local py_compile for schedule_policy.py and test_prefill_adder.py.
Tested: Remote pytest targeted CP PrefillAdder cases: 5 passed.
Tested: Remote pytest test/registered/unit/managers/test_prefill_adder.py: 16 passed.
Not-tested: Full ETE replay after this scheduler-only change.
The scheduler can now admit multi-request NSA in-seq CP shared-KV prefill batches only when the shared-KV bs>1 flag is explicitly enabled. The gate is still disabled by default and is scoped to CP shared-KV so ordinary CP is not widened accidentally.
Batch admission is bounded by optional request-count and page-aligned extend-token limits while real memory capacity remains allocator-owned. This keeps bf16 and fp8 on the same scheduler path because dtype differences are already reflected in KV pool token/page capacity.
Constraint: bs>1 runtime paths remain guarded by existing CP shared-KV fail-fast checks.
Constraint: Scheduler must not duplicate bf16/fp8 byte-level capacity estimation.
Rejected: Open the old CP gate unconditionally | ordinary CP would inherit an unverified shared-KV-specific batching path.
Rejected: Treat the extend-token cap as a hard per-request limit | a single large request could deadlock the scheduler.
Confidence: medium
Scope-risk: moderate
Directive: Keep CP shared-KV batching gated until ETE validates EAGLE accept length, output length, and HiCache load/backup behavior under real traffic.
Tested: local py_compile for server_args, schedule_policy, scheduler, prefill_adder tests, and server_args tests.
Tested: remote g0034 py_compile for the same files.
Tested: remote g0034 pytest target set: 5 passed for parser, parameter validation, default single-request CP gate, enabled bs>1 gate, and page-aligned extend cap.
Tested: remote g0034 pytest test_prefill_adder.py => 13 passed.
Not-tested: full server_args test file has an unrelated HuggingFace DNS/config-download failure in TestPrepareServerArgs.test_prepare_server_args.
Not-tested: ETE production traffic with --enable-cp-shared-kv-prefill-bs-gt1.
HiCache host hits can be skipped before load-back when the quota gate only counts immediately free KV allocator space. Under CP shared-KV pressure most reusable capacity may be represented as evictable radix-cache leaves, so the gate can incorrectly reject a host hit and leave prefill with cached-token zero despite host residency. Count device evictable cache in the quota estimate while leaving actual owner-lane allocation and eviction checks in the load path.
Constraint: CP HiCache load-back still has to respect owner-lane allocation and allocator eviction semantics.
Rejected: Force load-back regardless of quota | would bypass the scheduler pressure signal and increase OOM risk.
Rejected: Treat cache-hit zero as a transfer issue | logs showed host hits were found but skipped by quota before transfer.
Confidence: medium
Scope-risk: moderate
Directive: Do not remove evictable cache from load-back capacity accounting without checking CP HiCache host-hit behavior under device pressure.
Tested: git diff --check
Tested: remote g0034 container pytest -q test/registered/unit/managers/test_prefill_adder.py test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_alloc_pages_with_owners.py (90 passed, 3 warnings)
Not-tested: Full ETE GLM5 CP+HiCache+EAGLE pressure run after this quota change
Co-authored-by: OmX <omx@oh-my-codex.dev>
