Preserve draft KV across CP HiCache hits
Cache-hit prefill can skip draft forward for the prefix while PD transfer still reads draft KV for that same prefix. CP HiCache therefore needs to persist draft/MTP KV alongside target KV instead of relying on whatever remains in the draft GPU pool. Constraint: CP HiCache is host-only here; storage backends remain unsupported for CP shared KV. Constraint: CP shared KV must keep owner-page semantics and avoid falling back to full KV on every rank. Rejected: Recompute cached-prefix draft KV during prefill | loses the HiCache benefit and reintroduces the large hidden/KV footprint. Rejected: Change PD transfer to skip draft prefix KV | decode still needs draft cache continuity for MTP acceptance. Confidence: medium Scope-risk: moderate Directive: Keep target and draft CP HiCache metadata/load/write/evict paths in lockstep; changing one without the other can silently reduce MTP accept length. Tested: Remote g0034 container /sgl-workspace/sglang-tai: python3 -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 58 passed, 3 warnings Not-tested: Full multi-node HiCache+MTP serving benchmark and accept-length recovery.
This commit is contained in:
@@ -285,7 +285,9 @@ class TestHiCacheControllerCPWrite(CustomTestCase):
|
||||
controller = self.make_controller(host_pool, cp_rank=1)
|
||||
logical_locs = torch.tensor([8, 9, 10], dtype=torch.int64)
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "host_indices.*whole pages"):
|
||||
with self.assertRaisesRegex(
|
||||
ValueError, "(host_indices|physical_device_indices).*whole pages"
|
||||
):
|
||||
controller.write(logical_locs, node_id=21)
|
||||
|
||||
def test_cp_write_rejects_non_contiguous_owned_physical_page(self):
|
||||
|
||||
Reference in New Issue
Block a user