Fix CP HiCache load_cp owner-pattern mismatch (cache-hit corruption)
In CP=8 + NSA-shared-KV + HiCache disagg-prefill, cache-hit prefill produced incoherent decode output. Cold prefill on CP was correct; pure CP without HiCache was correct. The bug lived at the HiCache load_cp / device-alloc interface. Root cause: cache_controller.load_cp called the plain mem_pool_device_allocator.alloc(logical_len), which returns logical pages with no CP owner-pattern preservation. Cold prefill instead uses alloc_extend_compute_owner with a zigzag owner pattern from build_in_seq_page_compute_owners. The saved CpHiCacheNodeMetadata.owned_positions records WHICH POSITIONS in the write-time alloc were owned by this rank. At load time, those same positions are applied to a new alloc whose per-position owner pattern is arbitrary -- each rank loads its host bytes into physical slots whose corresponding logical page is owned by a DIFFERENT rank. Attention's materialize_shared_token_kv_buffer reads from the owner's physical slot, which was never loaded. Result: garbage. Fix: - CpHiCacheNodeMetadata gains two required fields: page_owners (int8 per logical page, identical on all CP ranks) and page_size. __post_init__ validates; split() bisects page_owners by page index with a page-alignment check. - _write_cp derives page_owners from device_indices (page-first slot of each page -> logical page id -> layout.owner_for_logical_pages) and stores in both metadata-construction sites (zero-owned and normal). - New CPSharedPagedTokenToKVPoolAllocator.alloc_pages_with_owners() reuses _select_compute_owner_pages (with its tai-kernel fast path) and returns page-contiguous token locs whose per-page owner sequence equals the input. - load_cp now concats page_owners across nodes_to_load and calls alloc_pages_with_owners. On None (lane exhausted) the caller hits the retry-with-eviction path; further failure returns None and degrades to cache miss. No silent fallback to plain alloc -- that recreated the bug. - load_back retry path now calls _evict_for_compute_owner_lanes (module-top import) instead of plain evict(); this targets the deficit lane and gives the next alloc attempt a chance to satisfy it. - envs import moved to module top in cache_controller.py per code-review feedback. Removed an over-defensive owned_check.all().item() in load_cp that would have re-introduced the host-sync anti-pattern 97a9f850c removed -- the invariant is already guaranteed by alloc_pages_with_owners. Tests: 40 existing CpHiCacheNodeMetadata constructions migrated to pass the new required fields. 9 new metadata tests (validators + split page-alignment). 10 new allocator tests in test_alloc_pages_with_owners.py covering input-order preservation, lane exhaustion, release_pages fallback, debug-mode invariant. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -109,6 +109,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=8,
|
||||
owned_positions=torch.tensor([1, 3, 7], dtype=torch.int64),
|
||||
host_indices=torch.tensor([10, 11, 12], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(8, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
parent, child = metadata.split(0)
|
||||
@@ -125,6 +127,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=10,
|
||||
owned_positions=torch.tensor([0, 2, 5, 9], dtype=torch.int64),
|
||||
host_indices=torch.tensor([20, 21, 22, 23], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(10, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
parent, child = metadata.split(5)
|
||||
@@ -142,6 +146,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
owned_positions=torch.tensor([0, 2, 5, 9], dtype=torch.int64),
|
||||
host_indices=torch.tensor([20, 21, 22, 23], dtype=torch.int64),
|
||||
draft_host_indices=torch.tensor([120, 121, 122, 123], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(10, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
parent, child = metadata.split(5)
|
||||
@@ -157,6 +163,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=64,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int32),
|
||||
host_indices=torch.empty((0,), dtype=torch.int32),
|
||||
page_owners=torch.zeros(max(64, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
self.assertEqual(metadata.logical_len, 64)
|
||||
@@ -170,6 +178,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([1, 3], dtype=torch.int32),
|
||||
host_indices=torch.tensor([10, 11], dtype=torch.int32),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
self.assertEqual(metadata.owned_positions.dtype, torch.int64)
|
||||
@@ -181,6 +191,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=-1,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(0, dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
def test_metadata_does_not_alias_input_tensors(self):
|
||||
@@ -191,6 +203,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=owned_positions,
|
||||
host_indices=host_indices,
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
owned_positions[0] = 2
|
||||
host_indices[0] = 12
|
||||
@@ -203,6 +217,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([1], dtype=torch.int64),
|
||||
host_indices=torch.tensor([9], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "split_len"):
|
||||
@@ -214,6 +230,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([2, 1], dtype=torch.int64),
|
||||
host_indices=torch.tensor([9, 10], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
def test_duplicate_positions_raise(self):
|
||||
@@ -222,6 +240,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([1, 1], dtype=torch.int64),
|
||||
host_indices=torch.tensor([9, 10], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
def test_length_mismatch_raises(self):
|
||||
@@ -230,6 +250,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([1, 2], dtype=torch.int64),
|
||||
host_indices=torch.tensor([9], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
def test_draft_host_length_mismatch_raises(self):
|
||||
@@ -239,6 +261,8 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
owned_positions=torch.tensor([1, 2], dtype=torch.int64),
|
||||
host_indices=torch.tensor([9, 10], dtype=torch.int64),
|
||||
draft_host_indices=torch.tensor([109], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
def test_out_of_range_positions_raise(self):
|
||||
@@ -247,8 +271,129 @@ class TestCpHiCacheNodeMetadata(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([4], dtype=torch.int64),
|
||||
host_indices=torch.tensor([9], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
# ── New: validators for page_owners + page_size (the fields that carry the
|
||||
# CP owner pattern across a HiCache write→load round-trip).
|
||||
|
||||
def test_zero_page_size_raises(self):
|
||||
with self.assertRaisesRegex(ValueError, "page_size must be positive"):
|
||||
CpHiCacheNodeMetadata(
|
||||
logical_len=4,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.empty((0,), dtype=torch.int8),
|
||||
page_size=0,
|
||||
)
|
||||
|
||||
def test_logical_len_not_multiple_of_page_size_raises(self):
|
||||
with self.assertRaisesRegex(ValueError, "multiple of"):
|
||||
CpHiCacheNodeMetadata(
|
||||
logical_len=10,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(2, dtype=torch.int8),
|
||||
page_size=4, # 10 % 4 != 0
|
||||
)
|
||||
|
||||
def test_page_owners_length_mismatch_raises(self):
|
||||
with self.assertRaisesRegex(ValueError, "page_owners length"):
|
||||
CpHiCacheNodeMetadata(
|
||||
logical_len=8,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(3, dtype=torch.int8), # expected 8/4=2
|
||||
page_size=4,
|
||||
)
|
||||
|
||||
def test_negative_page_owners_raises(self):
|
||||
with self.assertRaisesRegex(ValueError, "page_owners.*non-negative"):
|
||||
CpHiCacheNodeMetadata(
|
||||
logical_len=8,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.tensor([-1, 0], dtype=torch.int8),
|
||||
page_size=4,
|
||||
)
|
||||
|
||||
def test_page_owners_normalized_to_int8_cpu(self):
|
||||
metadata = CpHiCacheNodeMetadata(
|
||||
logical_len=8,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
# Pass int64 to test normalization.
|
||||
page_owners=torch.tensor([0, 1], dtype=torch.int64),
|
||||
page_size=4,
|
||||
)
|
||||
self.assertEqual(metadata.page_owners.dtype, torch.int8)
|
||||
self.assertEqual(metadata.page_owners.device.type, "cpu")
|
||||
self.assertEqual(metadata.page_owners.tolist(), [0, 1])
|
||||
|
||||
def test_page_owners_not_aliased_to_input(self):
|
||||
page_owners = torch.tensor([0, 1, 0, 1], dtype=torch.int8)
|
||||
metadata = CpHiCacheNodeMetadata(
|
||||
logical_len=16,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=page_owners,
|
||||
page_size=4,
|
||||
)
|
||||
page_owners[0] = 1
|
||||
self.assertEqual(metadata.page_owners.tolist(), [0, 1, 0, 1])
|
||||
|
||||
def test_split_non_page_aligned_split_len_raises(self):
|
||||
metadata = CpHiCacheNodeMetadata(
|
||||
logical_len=16,
|
||||
owned_positions=torch.tensor([0, 8], dtype=torch.int64),
|
||||
host_indices=torch.tensor([20, 21], dtype=torch.int64),
|
||||
page_owners=torch.zeros(4, dtype=torch.int8),
|
||||
page_size=4,
|
||||
)
|
||||
with self.assertRaisesRegex(ValueError, "must be a multiple of page_size"):
|
||||
metadata.split(5) # 5 % 4 != 0
|
||||
|
||||
def test_split_preserves_page_owners_slice(self):
|
||||
metadata = CpHiCacheNodeMetadata(
|
||||
logical_len=32, # 8 pages of page_size=4
|
||||
owned_positions=torch.tensor([0, 4, 16, 28], dtype=torch.int64),
|
||||
host_indices=torch.tensor([100, 101, 102, 103], dtype=torch.int64),
|
||||
page_owners=torch.tensor(
|
||||
[0, 1, 0, 1, 0, 1, 0, 1], dtype=torch.int8
|
||||
),
|
||||
page_size=4,
|
||||
)
|
||||
parent, child = metadata.split(16) # split at page 4 of 8.
|
||||
self.assertEqual(parent.page_owners.tolist(), [0, 1, 0, 1])
|
||||
self.assertEqual(child.page_owners.tolist(), [0, 1, 0, 1])
|
||||
self.assertEqual(parent.page_size, 4)
|
||||
self.assertEqual(child.page_size, 4)
|
||||
|
||||
def test_split_at_zero_yields_empty_parent_page_owners(self):
|
||||
metadata = CpHiCacheNodeMetadata(
|
||||
logical_len=8,
|
||||
owned_positions=torch.tensor([0], dtype=torch.int64),
|
||||
host_indices=torch.tensor([50], dtype=torch.int64),
|
||||
page_owners=torch.tensor([0, 1], dtype=torch.int8),
|
||||
page_size=4,
|
||||
)
|
||||
parent, child = metadata.split(0)
|
||||
self.assertEqual(parent.page_owners.numel(), 0)
|
||||
self.assertEqual(child.page_owners.tolist(), [0, 1])
|
||||
|
||||
def test_split_at_logical_len_yields_empty_child_page_owners(self):
|
||||
metadata = CpHiCacheNodeMetadata(
|
||||
logical_len=8,
|
||||
owned_positions=torch.tensor([0], dtype=torch.int64),
|
||||
host_indices=torch.tensor([50], dtype=torch.int64),
|
||||
page_owners=torch.tensor([0, 1], dtype=torch.int8),
|
||||
page_size=4,
|
||||
)
|
||||
parent, child = metadata.split(8)
|
||||
self.assertEqual(parent.page_owners.tolist(), [0, 1])
|
||||
self.assertEqual(child.page_owners.numel(), 0)
|
||||
|
||||
|
||||
class FakeWriteFailure:
|
||||
metadata = None
|
||||
@@ -280,6 +425,8 @@ class FakeWriteController:
|
||||
logical_len=len(device_indices),
|
||||
owned_positions=torch.tensor([0], dtype=torch.int64),
|
||||
host_indices=torch.tensor([99], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(len(device_indices), 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -297,6 +444,8 @@ class FakeZeroOwnedWriteController:
|
||||
logical_len=len(device_indices),
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(len(device_indices), 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
)
|
||||
|
||||
@@ -328,6 +477,8 @@ class TestHiRadixCacheCPBackup(CustomTestCase):
|
||||
logical_len=8,
|
||||
owned_positions=torch.tensor([1, 2], dtype=torch.int64),
|
||||
host_indices=torch.tensor([10, 11], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(8, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
self.assertTrue(cache._node_backuped(node))
|
||||
@@ -348,6 +499,8 @@ class TestHiRadixCacheCPBackup(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([], dtype=torch.int64),
|
||||
host_indices=torch.tensor([], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
cache._inc_hit_count(node)
|
||||
@@ -378,6 +531,8 @@ class TestHiRadixCacheCPBackup(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([0], dtype=torch.int64),
|
||||
host_indices=torch.tensor([55], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
root.children[1] = evictable_node
|
||||
cache.evictable_host_leaves.add(evictable_node)
|
||||
@@ -457,6 +612,8 @@ class TestHiRadixCacheCPBackup(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([0], dtype=torch.int64),
|
||||
host_indices=torch.tensor([55], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
root.children[1] = node
|
||||
cache.evictable_leaves.add(node)
|
||||
@@ -488,6 +645,8 @@ class TestHiRadixCacheCPSplitEvict(CustomTestCase):
|
||||
logical_len=10,
|
||||
owned_positions=torch.tensor([0, 2, 5, 9], dtype=torch.int64),
|
||||
host_indices=torch.tensor([20, 21, 22, 23], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(10, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
root.children[0] = child
|
||||
|
||||
@@ -534,6 +693,8 @@ class TestHiRadixCacheCPSplitEvict(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([1], dtype=torch.int64),
|
||||
host_indices=torch.tensor([70], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
cache.root_node.children[1] = node
|
||||
cache.evictable_host_leaves.add(node)
|
||||
@@ -576,6 +737,8 @@ class TestHiRadixCacheCPSplitEvict(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([2], dtype=torch.int64),
|
||||
host_indices=torch.tensor([80], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
cache.root_node.children[1] = parent
|
||||
|
||||
@@ -630,6 +793,8 @@ class TestHiRadixCacheCPSplitEvict(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.tensor([2], dtype=torch.int64),
|
||||
host_indices=torch.tensor([80], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
cache.root_node.children[1] = parent
|
||||
|
||||
@@ -696,6 +861,8 @@ class TestHiRadixCacheCPSplitEvict(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
cache.root_node.children[1] = node
|
||||
cache.evictable_host_leaves.add(node)
|
||||
@@ -751,6 +918,8 @@ class TestHiRadixCacheCPSplitEvict(CustomTestCase):
|
||||
logical_len=4,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(4, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
cache.root_node.children[1] = node
|
||||
cache.evictable_host_leaves.add(node)
|
||||
@@ -804,6 +973,8 @@ class TestHiRadixCacheCPLoadBack(CustomTestCase):
|
||||
logical_len=8,
|
||||
owned_positions=torch.tensor([0, 1], dtype=torch.int64),
|
||||
host_indices=torch.tensor([50, 51], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(8, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
loaded = cache.load_back(node)
|
||||
@@ -839,6 +1010,8 @@ class TestHiRadixCacheCPLoadBack(CustomTestCase):
|
||||
logical_len=6,
|
||||
owned_positions=torch.empty((0,), dtype=torch.int64),
|
||||
host_indices=torch.empty((0,), dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(6, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
|
||||
loaded = cache.load_back(node)
|
||||
@@ -871,6 +1044,8 @@ class TestHiRadixCacheCPLoadBack(CustomTestCase):
|
||||
logical_len=8,
|
||||
owned_positions=torch.tensor([0, 1], dtype=torch.int64),
|
||||
host_indices=torch.tensor([50, 51], dtype=torch.int64),
|
||||
page_owners=torch.zeros(max(8, 0), dtype=torch.int8),
|
||||
page_size=1,
|
||||
)
|
||||
root.children[0] = node
|
||||
|
||||
|
||||
Reference in New Issue
Block a user