Heavily-forked CP HiCache produces KV corruption only after an L1->L2->L1
round-trip (eviction + reload), scaling with cached volume. Static reading and
upstream-diff are exhausted, so trace the data flow and check four invariants
directly, gated by SGLANG_CP_HICACHE_KV_TRACE (0=off, 1=structural lifecycle,
2=+compose/free/ack), off by default and cheap when off.
Trace points (keyed by node_id; rid_map ties client rid->node_id):
- backup_reserve / backup_d2h: host<->phys slots written (H2/H4)
- split: prefix+suffix partition vs parent (H1)
- evict: write_pending at device free (H4)
- reload_node: host slots read on reload (H2)
- reload_assign: reloaded device locs -> forward
- dev_free / write_ack: device free vs backup-complete ordering (H4)
- compose / remap: per-forward row-id cache reuse + unmapped count (H3)
New mem_cache/cp_hicache_trace.py (cptrace/rng helper; values rendered
space-free for offline parsing). The companion analyzer joins these by
node_id; the decisive, request-independent check is H2: any node whose reload
host-slot fingerprint differs from its backup fingerprint is the corruption
source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
leavelet
38ef664b4c