Protect CP shared-KV cache-hit correctness under batched FP8 reuse
Cache-hit GSM8K regressions only appeared after the second pass reused request-specific suffix pages, so this change adds fail-fast transfer validation, masks stale rectangular page-table tails, and extends CUDA/unit coverage across FP8 CP shared-KV write, load, top-k, and materialization paths. The temporary ledger records eliminated hypotheses to prevent re-debugging the same L2 and persistent-cache paths.\n\nConstraint: CP shared KV stores physical pages but scheduler-visible semantics must remain valid-token/page-bounded.\nConstraint: bs>1 FP8 prefill must preserve existing CP shared-KV fast paths without silent fallback.\nRejected: Blame raw HiCache L2 load without tests | L2 KV and index backup/load/materialize roundtrips pass on remote CUDA.\nRejected: Disable current/partial reuse broadly | hides the cache-hit contract regression and costs performance.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not weaken CP shared-KV fail-fast or rectangular-tail masking without rerunning second-pass cache-hit accuracy tests.\nTested: remote CUDA pytest for fused FP8 MLA store, fused persistent index store, L2-loaded FP8 KV materialize, L2-loaded index materialize, ragged top-k offset, TAI batched index MQA prepare.\nTested: local py_compile for touched test files and git diff --check.\nNot-tested: full second-pass GSM8K accuracy after these diagnostic tests; root cause remains under investigation.
This commit is contained in:
@@ -14,6 +14,37 @@ register_cpu_ci(est_time=1, suite="stage-a-test-cpu")
|
||||
|
||||
|
||||
class TestNSATopkTransform(unittest.TestCase):
|
||||
@unittest.skipIf(not torch.cuda.is_available(), "CUDA is required")
|
||||
def test_ragged_topk_transform_offsets_are_request_relative_with_row_starts(self):
|
||||
from sgl_kernel import fast_topk_transform_ragged_fused
|
||||
|
||||
# CP bs>1 compacts each request segment into a temporary K buffer.
|
||||
# The selected compact column must be translated back to the normal
|
||||
# request-concatenated ragged layout as:
|
||||
# request_base + (selected_compact_col - row_start)
|
||||
# not request_base + selected_compact_col.
|
||||
columns = 5000
|
||||
logits = torch.zeros((2, columns), device="cuda", dtype=torch.float32)
|
||||
logits[0, 2999] = 10.0
|
||||
logits[1, 1000 + 2999] = 10.0
|
||||
lengths = torch.tensor([3000, 3000], device="cuda", dtype=torch.int32)
|
||||
row_starts = torch.tensor([0, 1000], device="cuda", dtype=torch.int32)
|
||||
request_offsets = torch.tensor(
|
||||
[0, 100000], device="cuda", dtype=torch.int32
|
||||
)
|
||||
|
||||
out = fast_topk_transform_ragged_fused(
|
||||
score=logits,
|
||||
lengths=lengths,
|
||||
topk_indices_offset=request_offsets,
|
||||
topk=2048,
|
||||
row_starts=row_starts,
|
||||
)
|
||||
torch.cuda.synchronize()
|
||||
|
||||
self.assertEqual(int(out[0, 0].item()), 2999)
|
||||
self.assertEqual(int(out[1, 0].item()), 100000 + 2999)
|
||||
|
||||
def test_paged_topk_transform_raises_when_fused_output_is_not_from_page_table(self):
|
||||
page_table = torch.tensor(
|
||||
[
|
||||
|
||||
Reference in New Issue
Block a user