The index trace proved per-node addressing is self-consistent, so the
corruption must be in the KV VALUES, not the bookkeeping. Add a full-tensor
position-weighted fingerprint (khash) of the actual device KV at two points,
both keyed by (node_id, layer_id) in owned-position order so they compare
directly:
- backup_kv_hash: device KV at backup, taken on the DEFAULT stream (outside the
write_stream block) so it is the correct POST-STORE value, not the racy
write_stream copy's view.
- reload_kv_hash: device KV after the H2D load (on load_stream, in-order after
the copy).
If a reload hash matches no backup hash for that node+layer, the round-trip
delivered wrong KV -- catching BOTH a per-layer store-vs-copy ordering race
(backup-correct != reloaded-stale) AND transfer corruption, in one run. Since
the compose/attention is identical fresh-vs-reload and stateless-per-forward,
all-ranks byte-correct shards => correct output, so this per-rank value check is
complete for the round-trip. Gated at SGLANG_CP_HICACHE_KV_TRACE=2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
leavelet
5ed20f493a