2b.2: B1 collective-free shared-pool evict-to-fit + delete the #5 deadlock
Replaces the 2b.1b shared-L2 capacity-miss SKIP with a deterministic collective-free evict-to-fit, and deletes the #5 deadlock. This is the gate that makes --enable-cp-shared-physical-l2-hicache safe in a real run. evict-to-fit (_reserve_write_cp_shared_l2_evict_to_fit, hiradix): on a shared-L2 reserve capacity miss -- which raises identically on every CP rank over the replicated free list -- snapshot the plannable shared-L2 host leaves (excluding the node being backed up) in the replicated SLRU order (_cp_host_evict_key = Phase-1 logical clock + node.id total-order tiebreak), release the coldest one at a time (each a replicated free-list mutation via the proven _evict_cp_host_for_write_admission teardown -> evict_cp_host -> allocator.release) and retry the pooled reserve after each, stopping at the first fit. Fragmentation-correct (free coalesces in _return_range). Identical victim set + order + deterministic reserve => every rank evicts the same victims and stops at the same point (design Thm 1), so NO collective is needed. The snapshot is one-level by design (a parent promoted to a leaf mid-loop is not re-pushed; deep cascades skip-and-retry next tick -- missed identically on all ranks). Exhaustion -> loud rate-limited skip (transient: pinned/in-flight objects hold the pool), never a hang. #5 deadlock DELETED: stripped the synchronize_across_ranks all_reduce machinery (the `while len(heap) and not all_ranks_done()` ReduceOp.MIN host_evict_done_min closure + the param) from _evict_host_for_physical_slots -- never reached (no caller passed True) and the headline CP-deadlock shape. evict_host(num_tokens) still works (single-arg, equivalent non-sync loop). Opus adversarial review = SHIP (all 7 findings PASS: determinism [_cp_host_evict_key purely replicated, pin_expiry dormant under CP], teardown byte-equivalence + no double-free, no infinite recursion + clean abort-on-partial, snapshot safety [no ancestor/descendant among host leaves], bounded termination, clean #5 strip, flag-off path untouched). Tests: new test_eight_ranks_evict_to_fit_stays_identical (fill->miss->release-in-order->retry -> identical placement, exercises coalescing) + 88/88 pool suite (syh-dev-new) + import smoke. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -2779,6 +2779,51 @@ class TestCpSharedL2EightRankReserveDeterminism(unittest.TestCase):
|
||||
allocs[3].reserve("cp_hicache_node:1", PAYLOAD_TARGET_KV, 1) # rank 3 only
|
||||
self.assertGreater(len({a.placement_digest() for a in allocs}), 1)
|
||||
|
||||
def test_eight_ranks_evict_to_fit_stays_identical(self):
|
||||
# B1 (2b.2): on a shared-L2 capacity miss every rank runs the SAME
|
||||
# deterministic evict-to-fit -- release the coldest victims (here the
|
||||
# ascending-node-id analog of the replicated SLRU order) one at a time,
|
||||
# retrying the pooled reserve after each, stopping at the first success.
|
||||
# Identical free list + identical victim order + deterministic reserve =>
|
||||
# every rank evicts the same victims and stops at the same point (design
|
||||
# Thm 1), so the hiradix _reserve_write_cp_shared_l2_evict_to_fit needs NO
|
||||
# collective. Also exercises fragmentation: the win comes only after the
|
||||
# freed ranges COALESCE into a contiguous run.
|
||||
allocs = self.make_rank_allocators(pages=8)
|
||||
for nid, npages in ((0, 3), (1, 3), (2, 2)): # fill 3+3+2 = 8/8 pages
|
||||
self._reserve_on_all(allocs, nid, (PAYLOAD_TARGET_KV,), npages)
|
||||
for a in allocs:
|
||||
a.mark_object_committed(f"cp_hicache_node:{nid}")
|
||||
self.assertEqual(len({a.placement_digest() for a in allocs}), 1)
|
||||
|
||||
# New backup needs 4 contiguous pages -> capacity miss on EVERY rank.
|
||||
for a in allocs:
|
||||
with self.assertRaises(ValueError):
|
||||
a.reserve("cp_hicache_node:9", PAYLOAD_TARGET_KV, 4)
|
||||
a.abort("cp_hicache_node:9") # SF3: drop any partial reserve
|
||||
|
||||
victim_order = [f"cp_hicache_node:{nid}" for nid in (0, 1, 2)]
|
||||
evicted_counts = []
|
||||
for a in allocs:
|
||||
evicted = 0
|
||||
for victim in victim_order:
|
||||
a.release(victim)
|
||||
evicted += 1
|
||||
try:
|
||||
a.reserve("cp_hicache_node:9", PAYLOAD_TARGET_KV, 4)
|
||||
break # contiguous run reopened -> fit
|
||||
except ValueError:
|
||||
a.abort("cp_hicache_node:9")
|
||||
evicted_counts.append(evicted)
|
||||
|
||||
# Every rank evicted the SAME victims (release 0 -> free=3<4 no fit; release
|
||||
# 1 -> [0,3)+[3,6) coalesce to [0,6) -> 4 fits) and reached identical placement.
|
||||
self.assertEqual(set(evicted_counts), {2})
|
||||
self.assertEqual(len({a.placement_digest() for a in allocs}), 1)
|
||||
for a in allocs:
|
||||
self.assertTrue(a.is_committed("cp_hicache_node:2")) # untouched survivor
|
||||
self.assertIsNotNone(a.get_range("cp_hicache_node:9", PAYLOAD_TARGET_KV))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user