Protect CP shared-KV cache-hit correctness under batched FP8 reuse

Cache-hit GSM8K regressions only appeared after the second pass reused request-specific suffix pages, so this change adds fail-fast transfer validation, masks stale rectangular page-table tails, and extends CUDA/unit coverage across FP8 CP shared-KV write, load, top-k, and materialization paths. The temporary ledger records eliminated hypotheses to prevent re-debugging the same L2 and persistent-cache paths.\n\nConstraint: CP shared KV stores physical pages but scheduler-visible semantics must remain valid-token/page-bounded.\nConstraint: bs>1 FP8 prefill must preserve existing CP shared-KV fast paths without silent fallback.\nRejected: Blame raw HiCache L2 load without tests | L2 KV and index backup/load/materialize roundtrips pass on remote CUDA.\nRejected: Disable current/partial reuse broadly | hides the cache-hit contract regression and costs performance.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not weaken CP shared-KV fail-fast or rectangular-tail masking without rerunning second-pass cache-hit accuracy tests.\nTested: remote CUDA pytest for fused FP8 MLA store, fused persistent index store, L2-loaded FP8 KV materialize, L2-loaded index materialize, ragged top-k offset, TAI batched index MQA prepare.\nTested: local py_compile for touched test files and git diff --check.\nNot-tested: full second-pass GSM8K accuracy after these diagnostic tests; root cause remains under investigation.
This commit is contained in:
laoyao0822
2026-06-07 13:26:49 +08:00
parent b17976b60d
commit f75ffff8d9
10 changed files with 2645 additions and 1 deletions
@@ -14,6 +14,37 @@ register_cpu_ci(est_time=1, suite="stage-a-test-cpu")
class TestNSATopkTransform(unittest.TestCase):
@unittest.skipIf(not torch.cuda.is_available(), "CUDA is required")
def test_ragged_topk_transform_offsets_are_request_relative_with_row_starts(self):
from sgl_kernel import fast_topk_transform_ragged_fused
# CP bs>1 compacts each request segment into a temporary K buffer.
# The selected compact column must be translated back to the normal
# request-concatenated ragged layout as:
# request_base + (selected_compact_col - row_start)
# not request_base + selected_compact_col.
columns = 5000
logits = torch.zeros((2, columns), device="cuda", dtype=torch.float32)
logits[0, 2999] = 10.0
logits[1, 1000 + 2999] = 10.0
lengths = torch.tensor([3000, 3000], device="cuda", dtype=torch.int32)
row_starts = torch.tensor([0, 1000], device="cuda", dtype=torch.int32)
request_offsets = torch.tensor(
[0, 100000], device="cuda", dtype=torch.int32
)
out = fast_topk_transform_ragged_fused(
score=logits,
lengths=lengths,
topk_indices_offset=request_offsets,
topk=2048,
row_starts=row_starts,
)
torch.cuda.synchronize()
self.assertEqual(int(out[0, 0].item()), 2999)
self.assertEqual(int(out[1, 0].item()), 100000 + 2999)
def test_paged_topk_transform_raises_when_fused_output_is_not_from_page_table(self):
page_table = torch.tensor(
[