chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Prevent unattached CP HiCache write acks from crashing chunked prefill

Browse Source 
Prepared per-layer CP HiCache backups can enqueue their final write ack before the radix node is attached. Chunked prefill exposed this window when a separate catch-up backup made writing_check drain the ack queue and pop an unattached node id.

Keep ready but unattached prepared acks queued until insert either attaches the prepared backup or rollback removes the orphan ack. Also document the reactive host free-room eviction plan separately from this state-machine fix.

Constraint: CP HiCache prepared backup transfer can complete before radix insertion attaches node state

Rejected: Drop unknown ack ids | would orphan a later successful prepared attach and leak write state

Rejected: Chunked-only guard | the invalid assumption is in the generic CP write ack state machine

Confidence: high

Scope-risk: narrow

Directive: Do not drain CP write acks unless every ack id is registered in pending_host_backups or ongoing_write_through

Tested: Remote red-green test_cp_hicache_metadata.py::TestHiRadixCacheCPBackup::test_writing_check_defers_unattached_prepared_ack

Tested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py (116 passed)

Tested: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py test/registered/unit/mem_cache/test_cp_hicache_metadata.py

Not-tested: Full chunked-prefill ETE replay after this commit

Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
laoyao0822
2026-06-02 05:17:42 +08:00
parent c2d25ff591
commit 5bd68768d9
4 changed files with 313 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
python/sglang/srt/mem_cache/hiradix_cache.py
View File

@@ -2583,10 +2583,31 @@ class HiRadixCache(RadixCache):
if ack_queue_len == 0:
return
def ack_registered_for_radix_state(ack_id: int) -> bool:
return (
ack_id in self.pending_host_backups
or ack_id in self.ongoing_write_through
)
finish_count = 0
for _, finish_event, ack_list in self.cache_controller.ack_write_queue:
if not finish_event.query():
break
if not all(ack_registered_for_radix_state(ack_id) for ack_id in ack_list):
# Prepared per-layer CP backups can finish and enqueue their final
# ack before the corresponding radix node is attached. A later
# catch-up/write-back node may make ongoing_write_through non-empty
# and enter writing_check while that prepared ack is still
# unattached. Keep the ack queued until insert either attaches the
# prepared backup or rolls it back.
logger.debug(
"[HiCache-write] writing_check defers unattached ack ids: "
"ack_ids=%s ongoing=%s pending=%s",
ack_list,
list(self.ongoing_write_through.keys()),
list(self.pending_host_backups.keys()),
)
break
finish_count += 1
local_finish_count = finish_count
queue_size = torch.tensor(finish_count, dtype=torch.int, device="cpu")