Files
sglang/python
laoyao0822 f0c6b892b6 Release CP draft-hidden buffers after EAGLE target prefill
CP draft shared-KV target prefill only needs the CP-local draft hidden side channel until draft prefill consumes it.  The reused ModelWorkerBatch was left with capture_draft_hidden_states enabled, and LogitsProcessorOutput retained the extra draft_hidden_states tensor past that point.  Restore the capture flags after target prefill, disable draft-hidden capture for draft prefill, and drop the consumed reference.

Constraint: Large CP prefill batches are memory-sensitive; an extra hidden tensor held across the speculative step materially increases peak memory.

Rejected: Leave capture_draft_hidden_states sticky across draft prefill | it can capture or retain hidden state outside the target-side CP-local side channel.

Confidence: high

Scope-risk: narrow

Directive: Keep CP-local draft hidden as a target-prefill-only side channel unless the draft worker explicitly owns a new lifetime contract.

Tested: g0034 cjy-glm5-new PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_paged_slot_remap_cache_distinguishes_same_storage_views test/registered/unit/speculative/test_eagle_worker_v2_cp_hidden.py

Not-tested: full speculative integration/E2E workload.
(cherry picked from commit f31ef2293ce0cdda2359a5f4713996d78b47f9df)
2026-06-21 18:47:09 +00:00
..
2026-06-10 05:54:43 +08:00