chenchenghao/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 11 Packages Projects Releases Wiki Activity

Keep CP pending-split probes collective-free

Browse Source 
Radix insert can reach the pending-backup/stale-tail split probe on only a subset of CP ranks. The previous opportunistic ack drain called writing_check(), which may issue a scalar write-visibility collective while other ranks are polling inflight transfer state with a different tensor shape. That ordering mismatch matches the observed Gloo 4-vs-2 abort after CP_HICACHE_FALLBACK insert_deferred_pending_backup_split logs.

This keeps the split probe local and conservative: it leaves ready write acks queued and lets the globally ordered scheduler/cache-event path drain them. The regression test locks the contract by making any collective from the pending-split probe fail.

Constraint: CP radix insert/pending-split probing is not globally ordered across ranks.
Rejected: Drain ready write acks from the split probe | can interleave with scheduler poll collectives and corrupt distributed collective ordering.
Confidence: high
Scope-risk: narrow
Directive: Do not call writing_check() or any CP/TP collective from radix pending-split probes; drain write acks only from globally ordered paths.
Tested: Remote cjy container: PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py => 126 passed, 21 warnings.
Tested: Remote cjy container: python -m py_compile python/sglang/srt/mem_cache/hiradix_cache.py => PY_COMPILE_OK.
Not-tested: Full ETE prefill/decode replay after this change.
Co-authored-by: OmX <omx@oh-my-codex.dev>
This commit is contained in:
laoyao0822
2026-06-11 20:27:06 +08:00
committed by leavelet
parent a24111a5f4
commit f74f0a557f
2 changed files with 52 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
python/sglang/srt/mem_cache/hiradix_cache.py
View File

@@ -2440,19 +2440,21 @@ class HiRadixCache(RadixCache):
return False
def _cp_drain_write_acks_before_pending_split(self) -> None:
"""Drain already-finished CP write acks before declaring split blocked.
"""Do not drain CP write acks from the radix pending-split path.
This is intentionally only called on the rare stale-tail/pending-split
path. It may run the final write-visibility collective when a completed
ack is available, but it avoids adding a collective to the normal insert
hot path.
This helper is reached from radix insert, where CP ranks can legitimately
diverge: one rank may be checking a node-pending-backup split while
another is checking stale-tail pruning or has already returned to the
scheduler event loop. `writing_check()` may issue the final
write-visibility TP collective when a local ack is ready. Calling it
here can therefore interleave a scalar write-ack collective with the
scheduler's inflight-poll collective on other ranks and abort Gloo with
a message-size mismatch.
Keep this path local and conservative: report the split as still blocked
and let the globally ordered scheduler/cache-event path drain write acks.
"""
if not self._uses_cp_hicache or not getattr(
self, "ongoing_write_through", None
):
return
self.writing_check()
return
def _cp_node_split_still_pending(self, node: TreeNode) -> bool:
if not self._uses_cp_hicache or not self._node_host_write_pending(node):