Both cached KV components round-trip byte-perfect, so the bug is downstream in the
cross-rank COMPOSE/GATHER on a reload forward. Add level-3 hashes targeting the
active suspect = materialize_prefix_and_reuse_current_kv_page_slots (reloaded
prefix gathered via modulo-owner IPC descriptors + fresh current via page_inverse
staging; the abutting prefix|current boundary is the suspect):
- gather_out: the live composed dense KV the attention actually reads (mixed_locs
gather), with prefix_pages/current_pages; nz==0 = zero/uninitialized composed KV
(the observed 0|0|0), self-contained.
- span_owner: per prefix/current span -> owner ranks (modulo) + physical pages +
logical pages + per-span hash/nz, to verify the prefix span maps to modulo
owners and catch a boundary conflation/off-by-one.
- ipc_desc: owner/src/logical-page ranges in build_cp_shared_kv_ipc_page_descriptors
(the modulo-owner prefix gather chokepoint).
- MoE stages (forward_deepep, via fwd_hash, valid-row local tensors only, never the
a2a-permuted intermediate): moe_postsel, router_logits, topk_ids/topk_w,
experts_out, moe_out_compact, moe_out_restored -> brackets select/router/topk/
dispatch-combine/shared-add/row-restore.
(The symm ComposePlan is dormant/unwired, so excluded.) All level 3, eager-extend
only, try/except-guarded, helpers khash/knz/rng. Analyzer flags zero composed KV +
zero spans + MoE-stage zeros.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
leavelet
50d1c2fc5a