Record CP cache-hit GSM8K findings and cleanup state

The bs>1 warm-cache accuracy investigation produced several negative results, reproducer details, and the final duplicate-logical-page slot-aliasing hypothesis. Keeping the ledger prevents repeated log archaeology after context compaction and documents which probes were removed from runtime code. Constraint: The document is an investigation ledger, not active runtime instrumentation Rejected: Drop the tensor-dump history entirely | it explains why later work focused on request-slot remap instead of decode transfer Confidence: medium Scope-risk: narrow Directive: Treat historical dump sections as evidence only; do not infer that dump hooks are still present in runtime code Tested: Documentation-only; runtime verification covered by the adjacent code/test commits Not-tested: Rendered documentation formatting outside plain Markdown
2026-06-08 20:57:30 +08:00
parent a32c639147
commit 430ed85083
1 changed files with 864 additions and 0 deletions
--- a/docs/advanced_features/nsa_prefill_cp_gsm8k_cachehit_temp_findings.md
+++ b/docs/advanced_features/nsa_prefill_cp_gsm8k_cachehit_temp_findings.md
@@ -1443,3 +1443,867 @@ Next fix direction:
   - CP-local compute-padded rows (`rows == expected_compute_rows`), using `select_cp_local_valid_rows_for_cache_write()`, or
   - global valid current rows (`rows == current_extend_kv_rows_for_reuse(...)`), using `cp_split_and_rebuild_data()`.
 2. Add a unit/source-contract test covering the `rows == local_out_cache_loc.numel()` fast path before production code change.
+
+## 2026-06-07 G: retained-cache 200x2 run is not a cold/warm comparison
+
+用户提醒本轮服务未重启，cache 仍保留。因此下面结果不能作为“冷启动第一轮 vs warm-cache 第二轮”对照，只能作为已有热缓存状态下的复现：
+
+- 输出目录：`/mnt/beegfs/cjy/gsm8k_debug_20260607_195303`
+- pass1：`Accuracy=0.695`, `Invalid=0.000`, `Latency=35.049s`, `Output throughput=615.881 token/s`
+- pass2：`Accuracy=0.695`, `Invalid=0.000`, `Latency=34.227s`, `Output throughput=632.346 token/s`
+
+结论：当前服务 cache 已处于错误热缓存状态；继续重复跑 GSM8K 不会提供新的冷启动证据。该结果仍证明 warm-cache consume 路径存在稳定精度损坏。后续不要把这组数据误解为首轮冷启动掉点。
+
+## 2026-06-08 AM: retained-cache 200x2 probe is warm/warm, not cold/warm
+
+用户提醒本次服务没有重启，cache 仍然保留。复查 `/mnt/beegfs/cjy/gsm8k_debug_20260607_195303` 与日志 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_193843.log` 后确认：
+
+- pass1 Accuracy `0.695`，pass2 Accuracy `0.695`；两轮都已经是热 cache 状态。
+- 通过 `origin_sig` 关联 CP0 `scheduler_prefill_batch` 与 200 条 GSM8K prompt，pass1/pass2 的 prefix/extend 基本完全一致。例如：
+  - pid0: `(prefix=704, extend=32)` 在 19:53:08 和 19:53:50 各出现一次；
+  - pid1: `(prefix=640, extend=59)` 两轮一致；
+  - pid4: `(prefix=768, extend=13)` 两轮一致。
+- CP0 prefix 统计：`704:334`, `640:34`, `768:32`，对应 200 条 prompt 的两轮 warm/warm 调度。
+
+结论：这轮结果只能证明“保留 cache 后 warm-cache 精度稳定在约 0.69”，不能用于判断某个修复是否改善了 cold→warm 掉点，也不能作为第一轮 cold baseline。后续需要冷/warm 对比时必须重启 prefill/decode 或显式清 cache；若不重启，就只把它当热 cache 定位样本。
+
+## 2026-06-08 AM: tensor dump deferred
+
+User correction: tensor dump should be the last-resort step, only after code/log review cannot locate the divergence boundary. Current next step stays source-level/runtime-log investigation around prefill cache-hit compute and send metadata.
+
+Do not add tensor dump instrumentation by default. If later needed, keep it opt-in and scoped first to prefill handoff / `send_kv_chunk()`; decode-side dump should remain out of scope unless prefill-side evidence matches but output still diverges.
+
+## 2026-06-08 AQ: index partial-current 尚未排查干净，补充限频元数据日志
+
+结论：index 的 partial/current 方向还不能排除。
+
+依据：
+
+- 既有单测已覆盖 CP=8、bs=5、FP8、prefix+current page-slot compose、TAI batched index MQA prepare、persistent index fused-store、L2 roundtrip 等基本合同；这些降低了“单个 kernel/基础布局必错”的优先级。
+- 但最新 warm-cache 复现仍显示：没有 L2 load-back、没有 fallback/fail-fast，失败率与 cache-hit 后 current tail 大小相关。
+- 线上日志里 index 路径大量走 `has_current_index=True` + `has_shared_index_buffer=True` 的 partial-current compose；现有日志只能确认分支被使用，不能确认 request 级 current rows、slot spans、compact topk segments 与 request metadata 一致。
+
+本轮新增默认关闭的元数据日志，复用已有开关：
+
+```bash
+SGLANG_CP_SHARED_KV_BS_GT1_DEBUG=1 \
+SGLANG_CP_SHARED_KV_BS_GT1_DEBUG_LIMIT=<N>
+```
+
+新增事件：
+
+- `index_current_reuse_prepare_detail`：记录 current-index reuse 的 row source、prefix/extend、local valid loc、rank-local offsets/tokens。
+- `index_partial_current_compose_detail`：记录 prefix/current slot spans、current rows、logical page-table shape、是否命中 prefetcher。
+- `index_topk_batch_compact_detail`：记录 batch topk compact 前后的 request q/kv 长度、valid q、cp_index 与 scatter spans。
+- `index_topk_cp_segments_detail`：记录 `_get_topk_ragged_with_cp()` 内部 segment_records、k/q bases、request_kv_bases、topk offset 列表摘要。
+- `index_direct_write` 增加 rids、prefix/extend、logical/physical loc 摘要。
+
+这些日志只在 debug env 打开时执行；默认路径不新增 runtime check，也不做 tensor dump。下一轮应使用 50/200 条 warm-cache canary，把 raw prompt 的 `origin_sig/rid` 与上述 index 事件绑定，判断是否存在某个 request 的 current rows、slot span 或 topk offset 跨 request/错位。
+
+## 2026-06-08 AR: 50 条 GSM8K 三段隔离，问题收敛到 bs>1 + warm-cache
+
+远端服务健康，日志：`/mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_204531.log`。
+
+测试目录：`/mnt/beegfs/cjy/gsm8k_50_bs_probe_20260607_205120`。
+
+三段命令等价为：
+
+1. 50 条并发 `parallel=64`：`pass1_parallel64`
+2. 同一 50 条再次并发 `parallel=64`：`pass2_parallel64`
+3. 同一服务不清 cache，50 条顺序 `parallel=1`：`pass3_parallel1`
+
+结果：
+
+```text
+pass1_parallel64: Accuracy=0.980, Invalid=0.000, Latency=12.365s
+pass2_parallel64: Accuracy=0.760, Invalid=0.000, Latency=12.398s
+pass3_parallel1:  Accuracy=0.960, Invalid=0.000, Latency=94.405s
+```
+
+raw 对比：
+
+```text
+p1 bad ids: [37]
+p2 bad ids: [10,13,15,18,19,20,28,29,33,37,44,45]
+p3 bad ids: [5,12]
+p1 -> p2 regress: [10,13,15,18,19,20,28,29,33,44,45]
+p1 -> p3 regress: [5,12], recover: [37]
+```
+
+prefill/scheduler 统计：
+
+```text
+p1_parallel64:
+  scheduler bs: {5:72, 4:8, 1:8}
+  prefix top: 640 x392, 0 x8
+  extend avg: 102.62, min=59, max=736
+  CP0 prefill: newtok=7040, cached=31360
+
+p2_parallel64:
+  scheduler bs: {5:72, 4:8, 1:8}
+  prefix top: 704 x352, 640 x24, 768 x24
+  extend avg: 25.82, min=3, max=61
+  CP0 prefill: newtok=3200, cached=35200
+
+p3_parallel1:
+  scheduler bs: {1:408}
+  prefix top: 704 x352, 640 x24, 768 x24, 0 x8
+  extend avg: 25.33, min=1, max=61
+  CP0 prefill: newtok=3264, cached=35200
+```
+
+关键结论：
+
+- `pass2_parallel64` 和 `pass3_parallel1` 都是 warm-cache、短 extend、prefix 命中 704/768；但 `parallel=64/bs>1` 明显掉点，`parallel=1/bs=1` 基本恢复。
+- 因此 **cache-hit 本身不是充分条件**；问题收敛到 **bs>1 batching 与 warm-cache/current-tail 组合**。
+- `p2` 与 `p3` 均大量走 index partial-current：`has_current_index=True`、`has_shared_index_buffer=True`、`compute_padding=True`。但 bs=1 通过，说明不是“index partial-current 基础逻辑必错”，而更可能是 batch-aware compact/scatter/request-offset/slot-span 跨 request 语义问题。
+- 下一步优先审计 batch-aware index/MLA consume 路径，而不是 L2 load-back 或单 request current reuse。
+
+## 2026-06-08 AS: parallel=20 小批量 probe 强化 bs>1 触发信号
+
+用户指出 `parallel=5` 可能没有充分叠 batch，因此直接跑 20 条、`parallel=20` 的 warm-cache canary。
+
+测试目录：`g0034:/mnt/beegfs/cjy/gsm8k_tiny20_p20_probe_20260607_210046`，日志仍为：`/mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_204531.log`。
+
+结果：
+
+```text
+warm_parallel20_pass1: Accuracy=0.750, Invalid=0.000, Latency=6.370s
+warm_parallel20_pass2: Accuracy=0.750, Invalid=0.000, Latency=6.539s
+bad prompt ids: [10, 14, 15, 18, 19] in both passes
+```
+
+对照同一热 cache 状态下较低并发/顺序小样本：
+
+```text
+parallel=5, 20 questions: Accuracy=0.850, bad=[5,7,12]
+parallel=1, 20 questions: Accuracy=0.900, bad=[5,12]
+```
+
+用 `origin_sig` 绑定 prompt id 到 CP0 `scheduler_prefill_batch` 后，parallel=20 中失败请求稳定落在 bs>1 batch 内，且集中在 batch 后半段：
+
+```text
+21:00:53 bs=5 pids=[6,7,9,8,10]       bad=[10]      prefix=[704,704,704,768,704] extend=[10,34,26,6,27]
+21:00:54 bs=5 pids=[11,12,13,15,14]   bad=[15,14]   prefix=[704,704,704,704,704] extend=[30,35,26,59,18]
+21:00:55 bs=4 pids=[16,17,18,19]      bad=[18,19]   prefix=[704,704,640,704]     extend=[19,21,61,33]
+
+21:01:06 bs=5 pids=[5,7,8,9,10]       bad=[10]
+21:01:06 bs=5 pids=[12,11,13,14,15]   bad=[14,15]
+21:01:07 bs=4 pids=[17,16,18,19]      bad=[18,19]
+```
+
+结论：
+
+- `parallel=20` 比 `parallel=5` 更清楚地触发了 bs>1 错误；20 条样本即可稳定复现 `0.75` accuracy。
+- 失败不是简单 cache-hit 本身：相同热 cache 下 `parallel=1` 明显更好。
+- 失败也不是随机：同一批 prompt 在两轮 parallel=20 中坏 id 一致，并且集中在 bs=5/bs=4 batch 的后半 request。下一步优先检查 batch-aware compact/scatter/output cursor/request offset 是否在后续 request 上错位，尤其是 index topk、RAGGED MLA cache-hit compose、以及 prefill handoff 的 per-request输出对齐。
+
+## 2026-06-08 AT: parallel=20 输出显示跨 request 语义串扰
+
+继续检查 parallel=20 raw 输出后，发现失败样本不是普通推理错误，而是明显的 request 语义串扰。
+
+例子：parallel=20 中 `prompt_id=18` 的 question 是鸡蛋/dozen 问题，但输出包含：
+
+```text
+60-15-25=... cups of feed
+#### 20
+```
+
+该推理内容对应 GSM8K `prompt_id=4` 的鸡饲料题。也就是说，后半 request 的 prefill/forward 结果中混入了另一个 request 的语义，而不是单纯 cache hit 后数值漂移。
+
+对比：同一热 cache 状态下 `parallel=1/5` 中 `prompt_id=10/14/15/18/19` 正确，`parallel=20` 中这些 id 稳定错误。因此当前首要假设是 **bs>1 flatten/compact/scatter/request-offset 对齐错误**，不是 L2 load-back 基础错误，也不是 decode 单独错误。
+
+## 2026-06-08 AU: slot remap 不是 unique/dedup，排除一个旧假设
+
+检查 `build_slot_page_remap()`：
+
+- 输入是 `logical_pages` 的 flattened slot 表。
+- 输出 dense page id 是 flattened slot id + 1；每个 page-table slot 都有独立 dense slot。
+- 它不是按 logical page 去重的 unique/remap。
+
+因此先前“prefix/current spans 可能索引进 dedup 后 unique pages，导致 request slot 串位”的假设不成立。
+
+仍需注意：`page_inverse.scatter_()` 对重复 logical page 会保留最后一个 slot，但共享 prefix 的重复 slot 理论上内容相同，当前没有证据表明这是 root cause。
+
+## 2026-06-08 AV: RAGGED index batch path走 TAI packed gather，暂未证实 page-slot/packed 错配
+
+检查 `_get_topk_ragged_with_cp()` 和 TAI `cp_index_mqa_prepare.cu` 后确认：
+
+- RAGGED batch index 路径构造 `k_bases/q_bases/request_kv_bases` 等 packed descriptor。
+- `try_tai_prepare_cp_mqa_index_batch()` 的 CUDA gather 通过 `block_tables[batch_idx, logical_page]` 读取 page，再写入 `k_out[k_base + token_base]`。
+- 也就是说 index K buffer 的最终产物是按 request packed 的 dense token buffer，而不是直接把 page-slot buffer 当 packed buffer 使用。
+
+因此“index topk 直接把 page-slot dense buffer 当 request-packed buffer”的简单假设暂时排除。仍需继续验证 packed topk output 与 MLA ragged KV dequant 的 packed token layout 是否完全一致。
+
+## 2026-06-08 AW: 错误在 prefill handoff 前已经出现，不是 decode-only
+
+在 parallel=20 的 `bs=4 pids=[16,17,18,19]` batch 中，CP0 日志显示：
+
+```text
+event=prefill_result_handoff bs=4
+extend_lens=[19,21,61,33]
+prefix_lens=[704,704,640,704]
+next_token_ids=[576,5512,220,1416]
+hidden_shape=(4,6144) topk_shape=(4,1)
+```
+
+其中 `prompt_id=18` 对应 batch index 2，prefill 首 token 是 `220`（空格），后续输出转向了另一个题目的语义。由于 prefill handoff 已经携带错误 next token，问题不应优先归因到 decode-only。
+
+下一步优先检查：
+
+1. `cp_collect_last_token_hidden()` 在 compute padding + bs>1 下是否选错每个 request 的 last hidden row。
+2. RAGGED MLA cache-hit compose/dequant 的 packed KV 与 index topk 的 packed token id 是否一致。
+3. logits/topk handoff 的 per-request row 顺序是否仍等于 scheduler batch request 顺序。
+
+## 2026-06-08 AX: p20 tiny-extend last-token collect 合同通过，暂时排除 prefill handoff 行选择
+
+针对 parallel=20 中稳定失败的真实 batch 形态补了单测：
+
+```text
+extend_lens=[19,21,61,33]
+prefix_lens=[704,704,640,704]
+cp_size=8,page_size=64,cp_rank=0
+request_compute_rank_local_offsets=[0,64,128,192]
+request_last_token_local_offset=[18,20,60,32]
+request_last_token_owner=[0,0,0,0]
+```
+
+验证 `cp_collect_last_token_hidden()` 会从每个 request 的 64-row compute-padded slot 中取正确 last-token 行，并按 batch request 顺序 all-gather 回收。
+
+远端验证：
+
+```text
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_collect_last_token_hidden_matches_parallel20_tiny_extend_layout
+# 1 passed
+```
+
+结论：当前证据不支持 “prefill handoff 的 last hidden row 选择把 request 行串了” 作为 root cause。下一步继续查 RAGGED MLA/topk 的 packed token layout 对齐，特别是 cache-hit compose/dequant 后的 KV token 顺序是否和 index topk 的 request-local packed token id 一致。
+
+## 2026-06-08 AY: p20 tiny-extend batch rerange 合同通过，暂时排除 compute-padding rerange 串行
+
+补充 cp=8、parallel=20 真实形态的 `_torch_batch_in_seq_all_gather_rerange()` 单测：
+
+```text
+extend_lens=[19,21,61,33]
+prefix_lens=[704,704,640,704]
+compute rows: 每 req 64 row，rank0 有 valid current，rank1-7 只有 dummy padding
+```
+
+测试在 rank0 的每个 64-row compute slot 里填 valid rows + poison padding，在 rank1-7 填 poison，期望 rerange 后只保留每个 request 的 valid rows，且按 request 顺序拼接。
+
+远端验证：
+
+```text
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_batch_in_seq_all_gather_rerange_matches_parallel20_tiny_extend_layout \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_collect_last_token_hidden_matches_parallel20_tiny_extend_layout
+# 2 passed
+```
+
+结论：当前证据不支持 “cp_all_gather_rerange_output 在 p20 tiny compute padding 下把 dummy rows 混入 valid 输出” 作为 root cause。
+
+同时检查 `fast_topk_transform_ragged_fused` 后确认：GSM8K 700 token 级别 `kv_len < topk=2048`，ragged topk fast path 返回 `0..length-1 + topk_indices_offset`，不依赖 index logits 内容。因此 p20 掉点优先级应从 index logits/partial-current 降到 MLA KV cache-hit compose/dequant/token-content 对齐。
+
+## 2026-06-08 AZ: CP8 multi-request KV compose 的 Python/reference 合同通过
+
+新增 runtime 单测 `test_cp8_batch_kv_partial_current_keeps_request_packed_layout`：
+
+- `cp_size=8`，4 个 request；
+- prefix pages 分散在不同 CP owner；
+- current pages 全部构造为 rank0-owned；
+- 每个 request/token 写唯一值；
+- 对 8 个 rank 分别执行 `materialize_prefix_and_reuse_current_kv_page_slots()`，再把 rank 输出求和模拟 all-reduce；
+- 用返回的 `mixed_locs` 按 request-packed 顺序读取 `merged_kv`。
+
+远端验证：
+
+```text
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_cp8_batch_kv_partial_current_keeps_request_packed_layout
+# 1 passed
+```
+
+结论：Python/reference 的 page-slot compose + current fill + request-packed mixed_locs 合同目前没有暴露串 request。下一步优先检查生产 CUDA/TAI current-slot fill、FP8 paged dequant、以及 FlashMLA sparse consume 的接口形状，而不是继续怀疑 planner/rerange 的基础 CPU 逻辑。
+
+## BA. parallel=20 reproducer confirms max bs=5 stacking under current scheduler gate
+
+2026-06-08: ran GSM8K first 50 twice with `--parallel 20` against the running remote service, without restart:
+
+- Result dir: `/mnt/beegfs/cjy/gsm8k_p20_probe_20260608_055637`
+- pass1: `Accuracy: 0.740`, bad prompt ids `[10,14,15,18,19,20,28,29,33,37,39,44,45]`
+- pass2: `Accuracy: 0.760`, bad prompt ids `[10,14,15,18,20,28,29,33,37,39,44,45]`
+- Prefill log confirms `parallel=20` is sufficient to hit the configured scheduler cap: many batches are `#new-seq: 5`, `#new-token: 320`, cached-token around `3456-3584`.
+
+Conclusion: `parallel=5` can under-drive batching; use `parallel=20` for the small GSM8K reproducer. The current failure is not merely warm-cache drift: p20 immediately exposes the bs>1 tiny-extend path with poor accuracy on first pass as well.
+
+Next focus remains the production CUDA/TAI MLA current-slot compose/dequant/FlashMLA consume path; CPU/reference batch rerange and slot-compose contracts already passed.
+
+## BB. Same-cache parallel=1 recovers accuracy, isolating failure to bs>1 consume path
+
+2026-06-08: after the p20 reproducer above, ran the same first 50 GSM8K questions on the same running service/cache with `--parallel 1`:
+
+- Result dir: `/mnt/beegfs/cjy/gsm8k_p1_after_p20_probe_20260608_060435`
+- Result: `Accuracy: 0.960`, `Invalid: 0.000`
+
+This is a strong isolation result: the cache contents are not globally corrupted by prior p20 traffic. If L2/L1 cache storage or load-back had generally poisoned the cached prefix, p1 warm-cache should also stay bad. Instead p1 recovers to expected accuracy while p20 remains bad.
+
+Conclusion: continue focusing on bs>1 runtime consume/metadata semantics. L2 cache backup/load-back remains lower priority for this accuracy bug.
+
+Additional diagnostic test:
+
+- `test_tai_current_slot_fill_matches_torch_for_parallel20_tiny_fp8_rows` compares TAI current-slot fill against torch reference for the p20 tiny-extend shape, `page_size=64`, `prefix_lens=[704,704,640,704,704]`, `extend_lens=[29,9,60,9,17]`, row width 656 bytes.
+- Remote result: `1 passed`.
+
+Conclusion: current evidence does not support TAI current-slot fill byte-copy as the root cause for p20 accuracy loss.
+
+## BC. FP8 paged dequant p20 slot-loc contract passed
+
+Added CUDA diagnostic test `test_fp8_paged_dequant_matches_reference_for_parallel20_slot_locs`:
+
+- Shape mirrors p20 tiny warm-cache requests: `prefix_lens=[704,704,640,704,704]`, `extend_lens=[29,9,60,9,17]`, `page_size=64`.
+- Builds slot-dense FP8 MLA rows (`dim=656`) and request-packed `mixed_locs` through the same slot-remap helpers.
+- Compares `dequantize_k_cache_paged()` output with a direct per-loc reference dequant.
+
+Remote result: `1 passed`.
+
+Conclusion: current evidence does not support FP8 paged dequant or slot-loc remapping as the root cause. Remaining high-priority suspects are now narrower:
+
+1. FlashMLA sparse consuming bs>1 compute-padded query rows and topk rows with an ordering/shape mismatch.
+2. Index/topk batch path producing topk rows whose row order is not exactly the same as the q rows consumed by FlashMLA sparse.
+3. Later layer pipeline preserving compute-padded rows through attention/MoE but selecting valid rows only at handoff; if an intermediate op reorders or compacts rows without preserving the compute-padded contract, p1 can pass while p20 fails.
+
+## BD. FlashMLA sparse dummy-row contract passed
+
+Added CUDA diagnostic test `test_flashmla_sparse_valid_rows_match_compact_with_compute_padding`:
+
+- Builds p20-like compute-padded q rows: 5 requests, 64 rows/request, valid rows `[29,9,60,9,17]`.
+- Fills dummy rows' sparse indices with `-1`.
+- Runs `flash_mla_sparse_fwd()` once on padded rows and once on compact valid-only rows.
+- Compares only valid-row outputs.
+
+Remote result: `1 passed`.
+
+Conclusion: FlashMLA sparse itself handles compute-padded dummy rows with `-1` indices for this contract. The remaining likely bug is not the sparse kernel independently, but a production metadata/layout mismatch among:
+
+- `q_all` row order consumed by FlashMLA,
+- `topk_indices` row order/offset returned by batch indexer,
+- dequantized KV request-packed row order from `page_table_1_flattened`.
+
+## BE. parallel=20 isolates a compute-padding semantic hazard
+
+2026-06-08: `--parallel 20` is now the preferred small GSM8K reproducer because it reaches the current scheduler cap (`bs=5`). The p20 first-50 run produced `Accuracy: 0.740/0.760`, while the same cache with `--parallel 1` recovered to `Accuracy: 0.960`.
+
+Additional log evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_204531.log`:
+
+```text
+event=batch_plan cp_rank=0 bs=5 extend_lens=[29,9,60,9,17]
+valid_pages=[1,1,1,1,1] compute_pages=[8,8,8,8,8]
+valid_local_tokens=[29,9,60,9,17] compute_local_tokens=[64,64,64,64,64]
+compute_padding=True padding_tokens=[483,503,452,503,495]
+
+cp_rank=1..7 valid_local_tokens=[0,0,0,0,0]
+compute_local_tokens=[64,64,64,64,64]
+```
+
+This means tiny requests are not just padded to the physical tail page. They are inflated from one valid page to `cp_size` compute pages, and every CP rank runs 64 synthetic rows per request through the full model stack. That is semantically risky with MoE/DeepEP because dummy rows can enter routing/dispatch/capacity/accounting even if cache writes later drop them. It also explains why targeted tests for current-slot fill, FP8 dequant, and FlashMLA sparse dummy indices passed while p20 accuracy still drops: those tests cover attention/cache consume locally, not full-layer MoE interaction with synthetic rows.
+
+Current hypothesis: compute-padding-to-`cp_size` is not a safe correctness mechanism for tiny bs>1. The next fix should prefer valid page-granular splits with zero-lane support over fabricating full-model dummy compute rows. If zero-lane support is incomplete, fail-fast at the precise unsupported boundary rather than silently padding through MoE.
+
+## BF. next evidence target: layer0 tensor dump for compute-padding validation
+
+当前 p20 失败已收敛到 bs>1 full-layer consume 语义，而不是单个低层 copy/kernel 单测独立失败。下一步不再继续猜测修复，改为在 target model layer0 加窄门控 tensor dump：
+
+- dump p1 与 p20 的 layer0 valid-path tensor；
+- 对比阶段：layer input、prepare_attn 后、self_attn 后、prepare_mlp 后、mlp 后/postprocess 后；
+- 若 layer input/prepare_attn valid rows 一致而 self_attn 或 MLP 后开始偏离，则可以定位到 attention/cache consume 或 MoE/DeepEP 与 compute-padding 的交互；
+- 若 layer input 已不一致，则问题在 scheduler/split/position/input metadata 进入 layer 前。
+
+实现约束：只用 `DUMPER_ENABLE` 触发；默认生产路径零行为变化；跳过 warmup 极短请求；每个 rank/stage 只 dump 首个 bs=1 与首个 bs>1 目标层 batch，避免日志/磁盘爆炸。
+
+## BG. layer0 dump probe data file prepared
+
+为避免 layer0 dump 被前几个正常样本消耗，本轮准备了只包含已知 p20 稳定失败样本的 5 条 GSM8K 子集：
+
+```text
+g0034:/mnt/beegfs/cjy/data/gsm8k_bad_probe_10_14_15_18_19.jsonl
+orig ids: [10, 14, 15, 18, 19]
+```
+
+建议采集顺序：在启用 `DUMPER_ENABLE=1` 的新 prefill 进程中，先 `parallel=20` 打这 5 条形成一个 bs=5 batch，再 `parallel=1` 打同一文件。当前 dump 代码会跳过 warmup，并分别保存首个 `bs_gt1` 与首个 `bs1` target layer0 batch 的各阶段 tensor。
+
+## BH. first layer0 dump attempt failed due dumper ctx duplicate layer_id
+
+首次启用 `DUMPER_ENABLE=1` 后，远端日志 `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_223657.log` 出现：
+
+```text
+CP shared-KV layer0 tensor dump failed at stage=layer_input: dict() got multiple values for keyword argument 'layer_id'
+```
+
+原因：dumper 的全局 ctx 已经可能包含 `layer_id`，本次新增 dump tags 又显式传入 `layer_id`，触发 dumper 内部 `dict(..., **extra_kwargs, **global_ctx)` 重复 keyword。该问题只影响采证 dump，不影响模型推理语义。
+
+修正：layer0 dump tags 不再传 `layer_id`，仍然通过 stage/cp_rank/batch_kind/compute_padding 区分文件；必要时从 dumper 全局 ctx 或 meta 读取层号。
+
+## BI. layer0 dump window widened for p20 first-20 reproducer
+
+只打 `[10,14,15,18,19]` 五条子集时，远端返回 `Accuracy=1.000`，没有复现 p20 串扰。这说明错误可能依赖前序 batch/队列状态，而不是单独这五个 request 本身。
+
+采证策略修正：
+
+- `bs_gt1`：每个 CP rank/stage dump 前 4 个 layer0 batch；覆盖此前 p20 first-20 中第 2/3/4 个稳定失败 batch。
+- `bs1`：每个 CP rank/stage dump 前 20 个 layer0 request；用于同一 first-20 prompt 顺序对照。
+
+dump 文件增加 `dump_batch_index` tag，便于按 batch 顺序匹配。
+
+## BJ. first p20 pass only populates cache; dump quota must cover second p20 warm pass
+
+用户指出 layer0 dump 必须先跑一轮填 cache。验证结果：
+
+```text
+first p20 first-20:  Accuracy=0.950
+second p20 first-20: Accuracy=0.750
+```
+
+因此第一轮 p20 不能作为失败样本；它会消耗原先 `bs_gt1` 前 4 个 batch 的 dump quota，导致第二轮 warm-cache 失败 batch 不再 dump。
+
+修正：`bs_gt1` dump 窗口从 4 个 batch 扩大到 10 个 batch，用来覆盖“第一轮 populate-cache + 第二轮 warm-cache”。`bs1` 仍保留 20 个 request 用于顺序对照。当前进程已经消耗旧 quota，需要重启后重新采集。
+
+## BK. layer0 dump quota increased for router probes and two-pass warm-cache capture
+
+用户提醒 router 可能发送 health check / probe 请求。当前代码已经在计数前跳过 `total_extend_tokens < 16` 的短请求，但为了避免额外调度或非 GSM8K 请求消耗窗口，采证 quota 继续放大：
+
+- `bs_gt1` 与 `bs1` 均保留前 40 个有效 layer0 batch/request；
+- 短 warmup/health 请求仍不会消耗 quota。
+
+这会多占 dump 空间，但仅在 `DUMPER_ENABLE=1` 下生效，用于本轮 root-cause 取证。
+
+## BL. layer0 dump evidence: matched valid rows are identical between p20 warm-hit and p1
+
+采集流程：新进程开启 `DUMPER_ENABLE=1`，先跑 p20 first-20 填 cache，再跑 p20 first-20 复现，再跑 p1 first-20 对照。
+
+结果：
+
+```text
+p20 pass1: Accuracy=1.000
+p20 pass2: Accuracy=0.850
+p1  pass3: Accuracy=0.900
+Dump dir: /ssd/dump/cp_layer0_20260607_225959
+Log:      /mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_225959.log
+```
+
+对 p20 第二轮 warm-hit 的 `bs_gt1 dump_batch_index=4..7` 与 p1 顺序请求按 `origin_sig` 对齐，在 CP0 上比较 valid rows：
+
+- 比较阶段：`layer_input`, `after_prepare_attn`, `after_self_attn`, `after_prepare_mlp`, `after_mlp`, `layer_output`。
+- 已匹配 request 的所有阶段 `max=0, mean=0, rel=0`。
+- positions 也未发现 mismatch。
+
+样例：
+
+```text
+BATCH 6 ext [30,35,26,18,59] prefix [704,704,704,704,704]
+req0..req4: layer_input/after_prepare_attn/after_self_attn/after_prepare_mlp/after_mlp/layer_output all max=0
+
+BATCH 7 ext [21,19,61,33] prefix [704,704,640,704]
+req0..req3: all stages max=0
+```
+
+解释：这是一条反证。至少在 target model layer0、CP0 valid rows 上，compute padding 没有导致 p20 与 p1 的语义差异；dummy rows 也没有通过 layer0 attention/MLP/DeepEP 影响这些 valid rows。当前不能把 root cause 直接归因于 layer0 compute padding 污染。
+
+仍未排除：
+
+1. 后续层才出现 divergence；
+2. 其他 CP rank 的 valid rows（非 CP0 owner）或跨层 all-gather/rerange 有问题；
+3. EAGLE/draft 路径或 prefill handoff/logits/topk 在 layer0 之后出现 batch 对齐错误；
+4. 少数 p1 对照没有 dump，因为当前 dump 仍跳过 `total_extend_tokens < 16`，导致 ext=3/6/10/13 等短请求无法比较。
+
+下一步更有效的证据点应从 layer0 后移：dump final collected hidden/logits 或在 selected later layers 做同样 valid-row 对比，而不是继续只盯 layer0。
+
+## BM. Next evidence boundary: keep layer0 and add final-layer/final-output dumps
+
+User direction: focus on layer0 and the last layer, i.e. beginning and end.
+
+Current evidence from BL shows CP0 valid rows at target layer0 are bitwise identical between bs>1 warm-cache and bs=1 matched prompts. Therefore continuing to only inspect layer0 is unlikely to isolate the accuracy drop.
+
+Planned opt-in evidence points:
+
+- keep existing target layer0 stage dumps;
+- add target last-layer input/output dumps only, not every middle layer;
+- add final model hidden dumps around norm and CP last-token collect;
+- add compact logits top-k/argmax dump after `LogitsProcessor`, avoiding full-vocab tensor dumps unless later needed.
+
+All points must stay gated by existing dumper enablement. To work with the current remote `DUMPER_FILTER="name.startswith('cp_shared_kv_layer0')"`, new dump records intentionally keep the `cp_shared_kv_layer0_*` prefix while encoding `dump_scope` and `stage` in metadata/tags.
+
+## BN. Implemented final-boundary dump hooks for the next cache-hit accuracy probe
+
+Implemented opt-in dump hooks in `python/sglang/srt/models/deepseek_v2.py`:
+
+- existing layer0 dumps remain unchanged in semantics;
+- last target layer now dumps `last_layer_input` and `last_layer_output`;
+- final model path dumps `model_final_pre_norm`, `model_final_post_norm_pre_collect`, and post CP collect/rerange hidden;
+- logits path dumps only top-16 indices/values plus argmax at `logits_processor_next_token_topk`, not full-vocab logits.
+
+All dumps are still gated by existing dumper enablement and preserve the `cp_shared_kv_layer0_*` filename prefix so the current remote filter keeps capturing them. Metadata now includes `dump_scope` (`layer0`, `last_layer`, `model_final`, `logits`) and `target_layer_id` where applicable.
+
+Verification so far: local `python -m py_compile python/sglang/srt/models/deepseek_v2.py` passed. Runtime evidence requires service restart because the running process has the old module loaded.
+
+## BO. Prepared boundary comparison script for layer0/final-layer probes
+
+Remote helper script prepared at:
+
+```text
+/mnt/beegfs/cjy/compare_cp_boundaries.py
+```
+
+It parses dump tags and scheduler `origin_sig`, then compares bs>1 requests with bs=1 requests for the same prompt signature on CP rank 0. It supports the new boundary scopes:
+
+- `layer0/layer_input`, `layer0/layer_output`;
+- `last_layer/last_layer_input`, `last_layer/last_layer_output`;
+- `model_final/model_final_pre_norm`, `model_final/model_final_post_norm_pre_collect`, post collect/rerange compact hidden;
+- `logits/logits_processor_next_token_topk` top-index equality and top-value drift.
+
+Usage after a restarted service captures new dumps:
+
+```bash
+python /mnt/beegfs/cjy/compare_cp_boundaries.py \
+  --dump-dir /ssd/dump/<new_cp_layer0_dump_dir> \
+  --log /mnt/beegfs/cjy/log/<new_sglang_cp_hicache_log> \
+  --max-batches 12
+```
+
+The script can run against old layer0-only dumps, but last-layer/model-final/logits comparisons require the new `deepseek_v2.py` hooks and therefore a service restart after code sync.
+
+## BP. Mooncake version/path triage for current bs>1 cache-hit accuracy issue
+
+User note: another per-layer development environment upgraded Mooncake to `0.3.11.post1`; check whether current issue is Mooncake-related.
+
+Findings on `g0034` container:
+
+- `import mooncake` succeeds from `/usr/local/lib/python3.12/dist-packages/mooncake`.
+- `mooncake.__version__`, `mooncake.version`, and `mooncake.VERSION` are all `None`.
+- `pip show mooncake` reports `Package(s) not found: mooncake`; the installed package does not expose normal wheel metadata, so the exact pip version cannot be confirmed from Python metadata.
+- `mooncake/engine.cpython-312-x86_64-linux-gnu.so` is present and logs normal Transfer Engine startup (`Using RDMA transport`, topology discovery, etc.).
+
+Current service path:
+
+- launch args show `disaggregation_transfer_backend='mooncake'`, so ordinary prefill->decode KV transfer uses Mooncake;
+- code gate for per-layer transfer is `SGLANG_CP_SHARED_KV_PER_LAYER_TRANSFER`, default `False`;
+- current launch args/env do not include `SGLANG_CP_SHARED_KV_PER_LAYER_TRANSFER=1`;
+- latest log contains no `CP_PER_LAYER_TRANSFER` registration/finish lines, so current reproduction is **not** exercising the per-layer Mooncake transfer path.
+
+Root-cause relevance:
+
+- Existing evidence shows some bad cases already have suspect `prefill_result_handoff` next-token behavior before decode consumes transferred KV.
+- Therefore Mooncake can still affect decode continuation / KV integrity / hangs, but it is unlikely to be the primary root cause of the current bs>1 warm-cache GSM8K accuracy drop if final prefill logits/next-token are already divergent.
+- The next boundary dump should decide this cleanly: if `model_final_post_cp_collect` or `logits_processor_next_token_topk` diverges in prefill, Mooncake is outside the first failing boundary. If prefill logits match but decode output diverges, then Mooncake transfer/version becomes a high-priority suspect.
+
+Action: keep Mooncake in the risk ledger, but do not change Mooncake or per-layer transfer while investigating the prefill final-boundary divergence.
+
+## BQ. Boundary dump localizes warm-cache bs>1 divergence after layer0 and before final target layer
+
+Latest restarted service with final-boundary dumps:
+
+```text
+Log:      /mnt/beegfs/cjy/log/sglang_cp_hicache_20260607_231847.log
+Dump dir: /ssd/dump/cp_layer0_20260607_231847
+Probe:    /mnt/beegfs/cjy/gsm8k_boundary_probe_20260607_232341
+Compare:  /mnt/beegfs/cjy/compare_cp_boundaries_20260607_232341.txt
+```
+
+Probe results:
+
+```text
+p20 populate: Accuracy=1.000
+p20 warm:     Accuracy=0.800
+p1 warm:      Accuracy=0.900
+```
+
+Boundary comparison for p20 warm-cache batches (`dump_batch_index=4..7`) against p1 same-origin prompts on CP0:
+
+- `layer0/layer_input`: max diff 0 for matched valid rows.
+- `layer0/layer_output`: max diff 0 for matched valid rows.
+- `last_layer/last_layer_input`: already diverged for several requests.
+- Divergence persists through `last_layer_output`, `model_final_post_norm`, `model_final_post_cp_collect`, and logits top-k.
+
+Representative rows from the comparison:
+
+```text
+batch4 req3 ext=18 prefix=704: layer0 max=0, last_layer_input max=11.9844, post_collect max=3.71484
+batch5 req2 ext=34 prefix=704: layer0 max=0, last_layer_input max=27.625, post_collect max=6.59717, logits arg0 differs 92983 vs 5512
+batch6 req3 ext=18 prefix=704: layer0 max=0, last_layer_input max=20.1562, logits arg0 differs 62792 vs 220
+batch7 req2 ext=61 prefix=640: layer0 max=0, last_layer_input max=22.7188, logits arg0 differs 220 vs 2932
+```
+
+Conclusion:
+
+- The current accuracy regression is not caused by layer0 input/output corruption on the compared CP0 valid rows.
+- The first observed failing boundary is between target layer0 output and target last-layer input.
+- Because final prefill hidden/logits are already divergent, ordinary Mooncake prefill->decode transfer is not the first failing boundary for this specific GSM8K warm-cache bs>1 correctness issue.
+
+Next evidence step:
+
+- Add an opt-in configurable intermediate-layer dump list (for example `SGLANG_CP_SHARED_KV_DUMP_LAYERS=0,20,40,60,77`) and binary-search the first layer where bs>1 warm-cache diverges from bs=1 same-origin prompts.
+- Keep dumps gated and sparse; do not add always-on runtime checks or production overhead.
+
+## BR. Scope correction: do not investigate Mooncake for current prefill KV correctness probe
+
+User clarified that the local environment is not suitable for Mooncake investigation and that per-layer transfer is not enabled in the current run.
+
+Current debugging scope is therefore narrowed to prefill-side correctness only:
+
+- do not spend more time on Mooncake version/path checks for this issue;
+- verify whether CP shared-KV/HiCache warm-cache bs>1 produces or consumes incorrect KV inside prefill forward;
+- use opt-in dumper evidence rather than adding production-time checks;
+- focus on KV cache content at the attention consume boundary for layer0/intermediate/final layers.
+
+Rationale: prior boundary dumps show layer0 hidden rows match but final prefill hidden/logits diverge. Since per-layer transfer is disabled and final prefill output is already suspect, the next useful boundary is the paged KV cache consumed by target attention, not prefill->decode transfer.
+
+## BS. Current pass: stop remote/dump, unit-test intermediate prefill invariants first
+
+User direction for this pass:
+
+- remote service is stopped;
+- do not add another dump yet;
+- do not investigate Mooncake/per-layer transfer for this issue;
+- continue reading the prefill middle path and add unit tests for suspicious invariants.
+
+Current localized evidence remains:
+
+- warm-cache bs>1 and bs=1 matched prompts are bitwise equal on CP0 target layer0 input/output valid rows;
+- divergence is first observed later, before the final target layer input;
+- final prefill hidden/logits already diverge, so decode transfer is not the first failing boundary.
+
+Most useful next invariant to test without runtime dumps:
+
+1. RAGGED MLA cache-hit compose must produce a compact KV row space whose row IDs match the row space used by NSA index/top-k offsets for every request in a bs>1 batch.
+2. Persistent cache writes must use only valid rows and preserve request boundaries across layers; compute-padding/page-tail rows must not shift later-layer request-local coordinates.
+3. The cached remap for page-slot materialization must be batch/request-layout scoped enough that a later layer cannot reuse a stale row mapping from another request shape.
+
+This pass will prioritize small CPU/unit tests around those mapping contracts before any code fix.
+
+## BT. Suspicious invariant: compute-padding rows are not a suffix before sparse/MoE top-k
+
+New read-only evidence for the current no-dump pass:
+
+- GLM-5.1 FP8 config on `g0034:/ssd/models/GLM-5.1-FP8/config.json`:
+  - `num_hidden_layers=78`
+  - `first_k_dense_replace=3`
+  - `moe_layer_freq=1`
+  - `n_routed_experts=256`
+  - `num_experts_per_tok=8`
+- This matches the previous boundary dump: target layer0 can match exactly while later layers diverge, because layer0 is dense and sparse/MoE begins at layer3.
+
+Code path evidence:
+
+- `build_batch_page_aligned_in_seq_split_plan()` enables compute padding when a request has fewer physical pages than `cp_size`.
+- For tiny extends, each CP rank receives compute rows, but many of those rows are synthetic page-fill rows. In the common 1-page suffix case, non-owner ranks can have compute rows with zero valid rows.
+- `DeepseekV2DecoderLayer.forward()` passes the full `hidden_states` tensor through `self.mlp(...)` after `prepare_mlp()`; there is no CP shared-KV valid-row compaction before sparse/MoE.
+- `DeepseekV2MoE.forward_deepep()` calls `self.topk(..., num_token_non_padded=forward_batch.num_token_non_padded, ...)`.
+- `forward_batch.num_token_non_padded` is a scalar based on total original input tokens. It can only represent a contiguous valid prefix/suffix-padded layout. CP compute-padding invalid rows are per-request/per-rank and can be interleaved inside the local tensor, so the scalar cannot mask them correctly.
+
+Concrete suspected failure shape:
+
+- For a bs>1 warm-cache batch with tiny suffixes like `extend=[29,9,60,9,17]`, `page_size=64`, `cp_size=8`, compute padding creates `64` local rows per request on every CP rank.
+- On non-owner ranks, valid local current rows for those requests can be `0`, but the local compute tensor still has `5 * 64 = 320` rows.
+- The scalar `num_token_non_padded=sum(extend_lens)=124` or similar would leave the first 124 synthetic local rows active in MoE routing, rather than masking all zero-valid rows on that rank.
+
+Hypothesis:
+
+- The cache-hit accuracy drop is caused by compute-padding rows entering sparse/MoE routing/dispatch/capacity as real tokens in bs>1 tiny-extend batches.
+- This explains why layer0 input/output valid rows match while final prefill hidden/logits diverge: layer0 is dense; the first likely semantic contamination point is sparse layer3.
+
+Next unit-test target:
+
+- Add a CPU/unit contract test showing that CP compute-padding creates non-suffix invalid rows and that a single scalar `num_token_non_padded` is insufficient for sparse/MoE top-k masking. This should become the guardrail before choosing the final fix (compact-before-MoE, row-mask-capable MoE, or disabling compute padding for sparse layers).
+
+### BT unit tests added
+
+Added CPU/unit guardrails in `test/registered/unit/layers/test_nsa_cp_utils.py`:
+
+- `test_compute_padding_tiny_batch_valid_rows_are_not_suffix_maskable_for_moe`
+  - Builds the p20-like tiny batch `extend=[29,9,60,9,17]`, `prefix=[704,704,640,704,704]`, `page_size=64`, `cp_size=8`, `cp_rank=0`.
+  - Confirms compute rows are `320`, valid rows are `124`, and valid rows are not a contiguous tensor prefix. Therefore a scalar suffix mask (`num_token_non_padded`) cannot express the valid-row layout.
+- `test_compute_padding_non_owner_rank_scalar_non_padded_would_unmask_dummy_rows`
+  - Same batch on `cp_rank=1`.
+  - Confirms compute rows are `320`, valid rows are `0`, while the global non-padded count is positive. Reusing the global scalar would unmask dummy rows on a zero-valid rank.
+
+Verification:
+
+```bash
+ssh g0034 "docker exec sglang-glm5-dev-2 bash -lc 'cd /sgl-workspace/sglang-tai && PYTHONPATH=python python -m pytest -q test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_compute_padding_tiny_batch_valid_rows_are_not_suffix_maskable_for_moe test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_compute_padding_non_owner_rank_scalar_non_padded_would_unmask_dummy_rows'"
+```
+
+Result: `2 passed, 5 warnings`.
+
+Local verification remains blocked by missing `orjson` in `/root` Python environment; remote container has the required dependencies.
+
+Additional related verification for the same invariant group:
+
+```bash
+ssh g0034 "docker exec sglang-glm5-dev-2 bash -lc 'cd /sgl-workspace/sglang-tai && PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_batch_plan_compute_padding_only_pads_tiny_request_in_mixed_batch \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_batch_plan_compute_padding_is_per_request_not_batch_total \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_compute_padding_tiny_batch_valid_rows_are_not_suffix_maskable_for_moe \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_compute_padding_non_owner_rank_scalar_non_padded_would_unmask_dummy_rows \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_collect_last_token_hidden_matches_parallel20_tiny_extend_layout \
+  test/registered/unit/layers/test_nsa_cp_utils.py::TestNSAInSeqCPUtils::test_batch_in_seq_all_gather_rerange_matches_parallel20_tiny_extend_layout'"
+```
+
+Result: `6 passed, 5 warnings`.
+
+## 2026-06-08 BU. Agent2 page_inverse duplicate-alias hypothesis
+
+Question checked: whether bs>1 dense materialize/current-reuse can corrupt cache-hit requests because `page_inverse` is batch-global and keyed only by logical page id.
+
+Evidence:
+
+- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:2902-2931`
+  `build_slot_page_remap()` intentionally preserves every flattened page-table slot and documents that duplicate logical pages may become duplicate dense slots.
+- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:2943-2974`
+  `build_slot_page_inverse()` builds a single `logical_page -> dense_slot` map with `scatter_`. Duplicate logical pages cannot be represented; the later slot wins in the Python fallback.
+- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:2977-3012`
+  `remap_logical_locs_to_slot_dense_locs()` maps all token locs through that one inverse, so all occurrences of the same logical page id target the same dense slot.
+- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:3584-3716`
+  MLA prefix+current compose materializes prefix slots, then remaps `logical_locs` and fills current rows via the same `slot_remap.page_inverse`.
+- `python/sglang/srt/layers/attention/nsa/cp_shared_kv_runtime.py:3772-3905`
+  Index prefix+current compose has the same inverse-based slot contract.
+- `python/tai_kernel/nsa_prefill/csrc/cp_shared_kv_materialize.cu:163-176` and `202-226`
+  TAI kernels assign `page_inverse[lp] = dense_page` without atomic/duplicate handling. Duplicate logical pages are therefore last-writer/race dependent on GPU. This is not only a TAI issue: the Python fallback also collapses duplicates by construction.
+- `python/tai_kernel/nsa_prefill/csrc/cp_index_mqa_prepare.cu:79-116` and `374-444`
+  The TAI index-MQA prepare path is segment/request-aware through `batch_indices`, `kv_lens`, `k_bases`, and `q_bases`; it does not rely only on a global logical-page inverse.
+
+Inference:
+
+- The hypothesis is structurally valid: the current slot-remap contract says duplicate logical pages can have duplicate dense slots, but its inverse cannot address duplicate slots. Any consumer that needs request-concat positional identity can be redirected to another request's slot.
+- This fits the observed shape: bs=1 has no cross-request duplicate slots; warm-cache bs>1 with shared prefixes/current reuse can create repeated logical page ids; second GSM8K pass loses accuracy while first pass is mostly clean.
+- The strongest corrupting case is duplicate logical page ids that are not semantically identical, or current/suffix locs that collide. If duplicates are true shared-prefix pages with identical KV content, content aliasing alone may be safe, but positional/dequant/top-k consumers can still be wrong if they expect one dense slot per request slot.
+- Agent2's `index-GetK path has k_bases, so it is safe` is only true for the fused index-MQA prepare/GetK concat path. The separate index page-slot materialize path still uses `page_inverse` and should be checked under bs>1 cache-hit.
+
+Unknown / next discriminating checks:
+
+1. Add a unit test with two bs>1 requests whose `remap_logical_pages` contains a repeated positive logical page and whose downstream `logical_locs` require per-request slot identity. Current code should show both requests remap to one dense slot.
+2. Add a production-shape test for RAGGED cache-hit compose where prefix pages are shared but current pages differ; verify whether dequant/top-k consumers require duplicate dense slots or can safely dedupe shared-prefix content.
+3. Under existing `SGLANG_CP_SHARED_KV_BS_GT1_DEBUG`, count duplicate positive pages in `metadata.real_page_table`, classified by prefix slots vs current slots. Current/suffix duplicates should be fail-fast; prefix duplicates require a clear dedupe-or-slot contract.
+
+Status: hypothesis has medium-high confidence as a real correctness risk; root cause of the GSM8K warm-cache drop is not fully proven until the duplicate class and consumer contract are verified.
+
+### BU reproduction test added
+
+Added a focused RED unit test:
+
+- `test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_duplicate_pages_keep_request_slot_identity`
+
+Shape:
+
+- `page_size=4`, `cp_size=1`.
+- `remap_logical_pages=[[1,2],[1,3]]`: two requests share logical page 1 as their first slot.
+- Slot layout says dense pages should be `[1,2,3,4]` for the four request slots.
+- Desired request-slot-preserving dense locs: `[[4,8],[12,16]]`.
+
+Observed failure on `g0034` remote container:
+
+```text
+E AssertionError: Lists differ: [[12, 8], [12, 16]] != [[4, 8], [12, 16]]
+```
+
+Interpretation:
+
+- `slot_logical_pages=[1,2,1,3]` is preserved, but `page_inverse[1]` points to the second occurrence's dense slot.
+- Req0's logical loc on page 1 is remapped from expected row `4` to row `12`, aliasing req1's page-1 slot.
+- This reproduces the duplicate-page alias at the CPU/fallback contract level. It is therefore not only a TAI-kernel issue; TAI adds non-atomic/race behavior on top of an already lossy inverse contract.
+
+Verification command:
+
+```bash
+ssh g0034 "docker exec sglang-glm5-dev-2 bash -lc 'cd /sgl-workspace/sglang-tai && PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_duplicate_pages_keep_request_slot_identity'"
+```
+
+Result: expected RED, `1 failed` with the alias shown above.
+
+### BU fix: row-scoped slot inverse for bs>1 duplicate logical pages
+
+Root cause fixed in this pass:
+
+- The dense slot layout is request-slot based: duplicate logical pages are allowed because two batch rows can reference the same logical page id at different request-local positions.
+- The old `page_inverse` was batch-global (`logical_page -> dense_page`) and therefore lossy under duplicates. Python `scatter_` deterministically kept one occurrence; TAI kernels used non-atomic writes and could be nondeterministic.
+- Cache-hit bs>1 materialize/current-reuse paths then remapped token/page indices through that lossy inverse, so one request could alias another request's dense slot.
+
+Implemented contract:
+
+- Keep the existing global `page_inverse` only for legacy/no-row-context paths.
+- Add `page_inverse_by_row` for rectangular bs>1 page tables: `request_row, logical_page -> dense_page`.
+- Add row-aware token/page remap helpers. If row context exists, remap through `page_inverse_by_row`; otherwise fall back to the old global inverse.
+- Pass flattened request row ids into the FP8 RAGGED MLA cache-hit compose path, where `logical_locs` is flattened but the page table is request-shaped.
+- Update MLA/index prefetch consume paths to use the shared row-aware helpers when the cached remap carries row-scoped inverse metadata.
+
+Verification on `g0034` remote container:
+
+```bash
+# RED test now GREEN
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_duplicate_pages_keep_request_slot_identity
+# result: 1 passed
+
+# Related materialize/current-reuse regression set
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_build_shared_token_kv_slot_remap_reuses_slot_layout \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_duplicate_pages_keep_request_slot_identity \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_materialize_batch_prefix_span_and_reuse_current_kv_page_slots \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_cp8_batch_kv_partial_current_keeps_request_packed_layout \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_fp8_ragged_mla_uses_page_slot_current_compose_for_cache_hit \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_index_prefetch_partial_current_compose_fills_current_page_slots \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_cp8_index_partial_current_compose_matches_rank_merged_reference_for_bs5
+# result: 7 passed
+```
+
+Remaining risk:
+
+- The TAI full-materialize kernels still expose only a global inverse. The Python runtime now avoids that path when row-scoped remap is required, but the CUDA kernel contract should eventually be upgraded to row-scoped descriptors instead of relying on the Python fallback for bs>1 duplicate-page cases.
+- `page_inverse_by_row` memory is `O(batch_size * logical_page_capacity)`. Current production gates use small bs, so this is acceptable as a correctness fix; if bs grows substantially, replace it with a compact per-row descriptor/kernel.
+
+### BU follow-up: replace dense row inverse with compact row-slot descriptor
+
+Correction after review:
+
+- The first row-scoped fix used a dense CUDA tensor shaped `[batch_size, logical_page_capacity]`.
+- That was correctness-safe but not acceptable for production memory use, especially with `bs` up to 10 and long contexts.
+
+Updated implementation contract:
+
+- Removed the dense `page_inverse_by_row` allocation entirely.
+- Store only compact descriptors with the same shape as the actual batch page table:
+  - `slot_sorted_logical_pages_by_row`: `[batch_size, actual_pages_per_row]`
+  - `slot_sorted_dense_pages_by_row`: `[batch_size, actual_pages_per_row]`
+- Row-aware remap loops over request rows and uses `torch.searchsorted` on that row's compact sorted page list.
+- Space is now `O(actual_batch_pages)`, not `O(batch_size * logical_page_capacity)`.
+
+Tradeoff:
+
+- This adds a small per-row remap compute cost versus direct dense inverse indexing.
+- Current production gate keeps `bs` small enough that this is acceptable. If this path becomes a CPU/GPU launch bottleneck, the next step is a TAI row-aware remap kernel over the same compact descriptors.
+
+Verification on `g0034` remote container:
+
+```bash
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py::TestCpSharedKVRuntimeHelpers::test_token_slot_remap_duplicate_pages_keep_request_slot_identity
+# result: 1 passed
+
+PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py
+# result: 126 passed, 2 subtests passed
+
+PYTHONPATH=python python -m pytest -q \
+  test/registered/unit/layers/test_nsa_cp_utils.py \
+  test/registered/unit/mem_cache/test_cp_shared_kv_layout.py
+# result: 137 passed
+```
+
+### BV cleanup: remove opt-in tensor dump hooks and temporary index trace logs
+
+Cleanup status for the current code state:
+
+- Removed the temporary `deepseek_v2.py` tensor-dump runtime hooks used for layer0/last-layer/model-final/logits boundary evidence.
+- Removed temporary `nsa_indexer.py` request-origin/top-k/direct-write detail logs that were only intended for the GSM8K bs>1 investigation.
+- Kept semantic regression tests, fail-fast checks, and explicit fallback/warning paths.
+- Historical dump findings above remain as an investigation ledger only; they should not be read as active runtime instrumentation.
+
+Directive: do not reintroduce tensor dumps or high-cardinality per-request logs into the default runtime path. If another boundary probe is needed, make it opt-in, narrow, and remove it after collecting evidence.