Commit Graph
15 Commits
Author SHA1 Message Date
laoyao0822 315eaaff56 Preserve draft KV across CP HiCache hits
Cache-hit prefill can skip draft forward for the prefix while PD transfer still reads draft KV for that same prefix.  CP HiCache therefore needs to persist draft/MTP KV alongside target KV instead of relying on whatever remains in the draft GPU pool.

Constraint: CP HiCache is host-only here; storage backends remain unsupported for CP shared KV.

Constraint: CP shared KV must keep owner-page semantics and avoid falling back to full KV on every rank.

Rejected: Recompute cached-prefix draft KV during prefill | loses the HiCache benefit and reintroduces the large hidden/KV footprint.

Rejected: Change PD transfer to skip draft prefix KV | decode still needs draft cache continuity for MTP acceptance.

Confidence: medium

Scope-risk: moderate

Directive: Keep target and draft CP HiCache metadata/load/write/evict paths in lockstep; changing one without the other can silently reduce MTP accept length.

Tested: Remote g0034 container /sgl-workspace/sglang-tai: python3 -m pytest -q test/registered/unit/managers/test_hicache_controller_cp.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 58 passed, 3 warnings

Not-tested: Full multi-node HiCache+MTP serving benchmark and accept-length recovery.
2026-05-26 23:58:40 +08:00
ThomasX f38937723e fix: address CP HiCache review regressions 2026-05-08 04:11:25 +08:00
ThomasX 8844c303e2 fix: synchronize CP HiCache host eviction semantics 2026-05-08 03:20:47 +08:00
ThomasX 3e9152194b test: cover CP HiCache host integration 2026-05-08 02:59:27 +08:00
ThomasX 39f40c02b4 fix: handle empty HiCache match prefixes 2026-05-08 02:47:25 +08:00
ThomasX 829f3ebceb feat: restore CP HiCache host hits as logical locs 2026-05-08 02:34:16 +08:00
ThomasX 08d518c2ed fix: preserve CP host siblings during stale cleanup 2026-05-08 02:06:04 +08:00
ThomasX 2df832dec0 fix: unblock CP host eviction past stale leaves 2026-05-08 01:59:14 +08:00
ThomasX f1a7e605e8 feat: split and evict CP HiCache metadata 2026-05-08 01:50:07 +08:00
ThomasX bebbbfd0e8 fix: treat CP HiCache metadata as backup state 2026-05-08 01:41:53 +08:00
ThomasX 08a95a35a6 fix: implement CP HiCache write retry eviction 2026-05-08 01:35:55 +08:00
ThomasX 068aa2f910 feat: track CP HiCache write-back state 2026-05-08 01:29:39 +08:00
ThomasX eb0ec4299d test: cover CP HiCache metadata validation edges 2026-05-08 01:09:32 +08:00
ThomasX 1ede8bb999 test: harden CP HiCache metadata invariants 2026-05-08 01:06:58 +08:00
ThomasX 5fbe43381f feat: add CP HiCache node metadata 2026-05-08 01:00:09 +08:00