Preserve draft KV across CP HiCache hits

Cache-hit prefill can skip draft forward for the prefix while PD transfer still reads draft KV for that same prefix.  CP HiCache therefore needs to persist draft/MTP KV alongside target KV instead of relying on whatever remains in the draft GPU pool.

Constraint: CP HiCache is host-only here; storage backends remain unsupported for CP shared KV.

Constraint: CP shared KV must keep owner-page semantics and avoid falling back to full KV on every rank.

Rejected: Recompute cached-prefix draft KV during prefill | loses the HiCache benefit and reintroduces the large hidden/KV footprint.

Rejected: Change PD transfer to skip draft prefix KV | decode still needs draft cache continuity for MTP acceptance.

Confidence: medium

Scope-risk: moderate

Directive: Keep target and draft CP HiCache metadata/load/write/evict paths in lockstep; changing one without the other can silently reduce MTP accept length.

Tested: Remote g0034 container /sgl-workspace/sglang-tai: python3 -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 58 passed, 3 warnings

Not-tested: Full multi-node HiCache+MTP serving benchmark and accept-length recovery.
This commit is contained in:
laoyao0822
2026-05-26 23:58:55 +08:00
committed by leavelet
parent 315eaaff56
commit fa80c6278e
2 changed files with 5 additions and 2 deletions
@@ -285,7 +285,9 @@ class TestHiCacheControllerCPWrite(CustomTestCase):
controller = self.make_controller(host_pool, cp_rank=1)
logical_locs = torch.tensor([8, 9, 10], dtype=torch.int64)
with self.assertRaisesRegex(ValueError, "host_indices.*whole pages"):
with self.assertRaisesRegex(
ValueError, "(host_indices|physical_device_indices).*whole pages"
):
controller.write(logical_locs, node_id=21)
def test_cp_write_rejects_non_contiguous_owned_physical_page(self):