L2 prereqs for CP8DP2 + L3: scope HiCache consensus to the CP group + reset pooled allocator on flush
Two latent CP shared-KV L2 correctness fixes, landed BEFORE L3 (not in an L3 commit). Surfaced by a per-CP8-instance (CP8DP2EP16) scoping review. PREREQ-1 (CP-group scoping). The B1 commit/evict consensus collectives (writing_check ReduceOp.MIN, placement_digest MIN/MAX + its tp_world_size<=1 entry guard, drain_storage_control_queues, the evict/prefetch MINs, the flush barrier) AND cache_controller.prefetch_tp_group all derive from self.tp_group/self.tp_world_size, which was params.tp_cache_group. tp_cache_group equals the CP group ONLY at dp_size=1 (enable_dp_attention False -> tp_cpu_group, and attn_cp_size==tp_size -> _ATTN_CP==_TP) -- the sole reason B1 works today. Under CP8DP2 (DP attention, attn_tp_size=1) tp_cache_group is the size-1 attn-TP group, so every `tp_world_size>1` collective silently no-ops per rank and the placement assert self-disables -> divergent placement -> shared-slab corruption. Fix: for CP hicache (cp_size>1) scope self.tp_group to the CP cpu group (get_attention_cp_group().cpu_group -- already used for the slab-handle broadcast) + self.tp_world_size to its size. A no-op handle change at dp_size=1 (same group object); the intended fix at CP8DP2. The single init-point change propagates to every consensus collective + un-gates prefetch_tp_group + re-enables the placement assert. PREREQ-2 (flush reset). HiRadixCache.reset() cleared the radix tree + host pool but never reset CpSharedL2PageAllocator -> stale free list/ranges/committed after flush_cache (leak; the shared pool was never reclaimed). Added CpSharedL2PageAllocator.reset() (rebuild the per-slab free list all-free, drop ranges + committed, restore the freshly-built placement_digest) called from cache_controller.reset() after the ack queues are cleared. Safe: flush_cache is idle-gated (no in-flight backup/reserve). This is also L3's clear hookpoint. Validation: new test_reset_restores_freshly_constructed_all_free_state + 89/89 pool suite (torch-2.11 container) + import smoke. PREREQ-1 is a no-op at dp_size=1 (live no-regression confirmed on the next prefill restart); CP8DP2 correctness is by construction (CP-group membership verified) pending a 2-machine run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1019,6 +1019,28 @@ class TestCpSharedL2PageAllocator(unittest.TestCase):
|
||||
all(src == 0 and group == "fake-cpu-group" for src, group, _ in broadcasts)
|
||||
)
|
||||
|
||||
def test_reset_restores_freshly_constructed_all_free_state(self):
|
||||
# PREREQ-2: flush_cache must reclaim the pooled L2. reset() rebuilds the
|
||||
# free list all-free + drops ranges/committed, restoring the freshly-built
|
||||
# placement_digest exactly (rank-uniform at the idle flush point).
|
||||
allocator = self.make_allocator(pages=8)
|
||||
fresh_digest = allocator.placement_digest()
|
||||
allocator.reserve("n0", PAYLOAD_TARGET_KV, 3)
|
||||
allocator.reserve("n1", PAYLOAD_TARGET_KV, 2)
|
||||
allocator.mark_object_committed("n0")
|
||||
self.assertLess(allocator.free_pages(PAYLOAD_TARGET_KV), 8)
|
||||
self.assertNotEqual(allocator.placement_digest(), fresh_digest)
|
||||
|
||||
allocator.reset()
|
||||
|
||||
self.assertEqual(allocator.free_pages(PAYLOAD_TARGET_KV), 8)
|
||||
self.assertEqual(allocator.free_pages(PAYLOAD_DRAFT_KV), 8)
|
||||
self.assertFalse(allocator.is_committed("n0"))
|
||||
self.assertEqual(allocator.object_ranges("n0"), {})
|
||||
self.assertEqual(allocator.placement_digest(), fresh_digest)
|
||||
# usable after reset
|
||||
self.assertIsNotNone(allocator.reserve("n2", PAYLOAD_TARGET_KV, 4))
|
||||
|
||||
def test_release_and_abort_return_capacity_once_without_double_free(self):
|
||||
allocator = self.make_allocator(pages=5)
|
||||
allocator.reserve("release-me", PAYLOAD_TARGET_KV, 2)
|
||||
|
||||
Reference in New Issue
Block a user