Keep CP HiCache reuse page-safe without eviction log churn

Page-aligned CP shared KV can pad out_cache_loc beyond valid current rows, so current reuse now gates MLA composition on the valid extend rows and permits draft partial-current reuse once the TAI sparse-page capability check passes. The TAI current-slot path self-tests sparse pages before use and falls back to the torch reference when the installed kernel is stale.

Eviction success and no-op diagnostics were also moved from INFO to DEBUG so owner-lane and host-admission churn does not flood production logs; true write failures remain WARNING.

Constraint: CP shared KV uses page-aligned physical reservations where valid suffix rows can be shorter than padded out_cache_loc.

Constraint: Production failure/fallback logs must remain visible, but hot successful eviction paths should not emit INFO per victim/rank.

Rejected: Keep draft partial-current reuse disabled | would preserve avoidable full materialization on draft cache-hit suffixes.

Rejected: Trust the TAI current-slot kernel unconditionally | stale kernels can corrupt sparse current-page composition.

Confidence: medium

Scope-risk: moderate

Directive: Do not reintroduce INFO logging in eviction hot paths without rate limiting and runtime evidence.

Tested: local py_compile for touched Python files

Tested: local git diff --check

Tested: remote container py_compile for touched Python files

Tested: remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py -> 82 passed, 5 warnings, 2 subtests passed

Not-tested: full ETE traffic after this commit; draft partial-current accept length still needs user-driven runtime validation

docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md

@@ -3365,3 +3365,251 @@ Verification on g0034 container:
 
 Remaining risk:
 - This makes the all-layer direct/page-first backup API submit one TAI per-layer memcpy-batch op instead of one stale sgl-kernel all-layer op. It is correct and CUDA-13-safe, but it can increase CPU submit count on all-layer backup paths. Production forward-overlap paths already operate per-layer, so this is acceptable for the current branch goal.
+
+### C81 — 2026-05-31 correction: do not reopen draft partial-current reuse before fixing root cause
+
+Correction:
+
+- Draft partial-current reuse must not be reopened merely by deleting the
+  `draft_partial_current_reuse_disabled` guard.
+- The observed accept-length regression must be fixed first, then the default
+  can be reopened without an env gate.
+- Until that root cause is fixed and verified, the safe contract remains:
+  draft current-only reuse is allowed, draft cache-hit partial-current reuse is
+  disabled with the explicit `draft_partial_current_reuse_disabled` warning.
+
+Current known facts:
+
+- Draft HiCache L2(host)→L1(device) load-back exists via `CacheController.load_cp()`
+  and `CacheController.start_loading()` using `draft_load_queue` and
+  `draft_mem_pool_host.load_to_device_per_layer(...)` when `has_draft_hicache` is
+  true.
+- That L2→L1 load-back is separate from the CP shared-KV MLA/index next-layer
+  prefetcher; the latter is still target-only because EAGLE/NextN has one
+  executable draft layer.
+
+Next target:
+
+- Audit the draft partial-current compose path (`nsa_backend.py` plus
+  `materialize_prefix_and_reuse_current_kv_page_slots`) for target/draft layout,
+  layer-id, current-suffix placement, and page-slot mapping mismatches before
+  reopening the guard.
+
+### C82 — 2026-05-31 fixed-before-open path for draft partial-current reuse
+
+Root-cause guard added before reopening draft partial-current reuse:
+
+- The risky failure mode was not the draft predicate itself; it was accepting a
+  stale TAI current-slot fill kernel whose masking logic assumed the current
+  suffix covered a dense page range from first current page to last current page.
+- For sparse current pages, that stale kernel could mask unrelated prefix pages
+  as suffix slack, corrupting partial-current compose while tensor shapes stayed
+  valid.
+- `cp_shared_kv_runtime.py` now validates the installed TAI current-slot fill
+  kernel once per CUDA device with a sparse-page self-test before using it. If
+  the self-test fails, the TAI result is not used and the code falls back to the
+  torch reference path, which preserves correctness.
+
+After this capability fix:
+
+- Draft cache-hit partial-current reuse is reopened directly, without an env
+  gate.
+- Draft next-layer MLA/index prefetcher creation remains disabled; draft has one
+  executable layer, so a future draft prefetch optimization should use an
+  explicit same-layer contract.
+
+Verification:
+
+- Remote RED before runtime helper existed:
+  `test_tai_current_slot_fill_is_skipped_when_sparse_page_self_test_fails`
+  failed with missing `_tai_current_slot_fill_supports_sparse_pages`.
+- Remote GREEN after runtime helper:
+  `test_tai_current_slot_fill_is_skipped_when_sparse_page_self_test_fails` → pass.
+- Remote installed-kernel evidence:
+  `test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel`
+  → pass.
+- Remote tai-kernel sparse-page tests:
+  `test_fill_current_token_kv_page_slots_handles_sparse_current_pages`,
+  `test_fill_current_token_kv_page_slots_handles_nonzero_first_page_offset`, and
+  `test_fill_current_token_kv_page_slots_masks_current_page_slack` → `3 passed`.
+
+Remaining risk:
+
+- Need ETE run with draft `branch=partial_current_sync` active to confirm decode
+  aggregate accept remains healthy. The code-level stale-kernel corruption path
+  is now blocked, but runtime traffic is still required before treating this as
+  final performance/correctness validation.
+
+### C83 — 2026-05-31 Mooncake P2P handshake receives HTTP traffic on port 17000
+
+Observed remote log:
+
+- `readString: too large length from socket: 3564087971010531152`
+- `SocketHandShakePlugin: failed to receive handshake message, malformed json`
+
+Root-cause evidence:
+
+- `3564087971010531152 == 0x31762f2054534f50`, little-endian bytes are
+  `POST /v1`.
+- Tcpdump on g0034 captured the offending packet:
+  `10.20.34.3:<ephemeral> > 10.20.32.34:17000` with HTTP payload
+  `POST /v1/chat/completions HTTP/1.1` and `Host: 10.20.32.34:17000`.
+- In the same run, Mooncake P2P handshake selected port `17000` for one prefill
+  scheduler process:
+  `Transfer Engine RPC using P2P handshake, listening on 10.20.32.34:17000`.
+- The errors repeat every ~30 seconds and start before request traffic while the
+  model is still loading, so this is not caused by HiCache per-layer backup
+  payload corruption.
+
+Current conclusion:
+
+- This is a port/protocol collision: an external HTTP client/prober is sending
+  OpenAI HTTP traffic to a Mooncake metadata/P2P handshake listener.
+- It is noisy and can mask real transfer errors, but it is not evidence that
+  HiCache write/read payloads are corrupted.
+
+Decision:
+
+- Do not keep the temporary SGLang-side Mooncake port-range override.
+- The code change that set `MC_MAX_PRC_PORT=16999` by default was reverted on
+  2026-05-31 at user request.
+- If this noise matters operationally, fix the external endpoint/prober or set
+  Mooncake `MC_MIN_PRC_PORT`/`MC_MAX_PRC_PORT` explicitly in the launch
+  environment instead of silently changing SGLang defaults.
+
+### C84 — 2026-05-31 draft partial-current reuse also requires valid-row, not padded-loc, MLA gating
+
+Finding:
+
+- Reopening draft partial-current reuse still was not enough when `out_cache_loc`
+  was page padded. The MLA gate in `nsa_backend.py` required
+  `k.shape[0] == forward_batch.out_cache_loc.numel()` and the same for
+  `k_rope`.
+- Under the page-aligned cache contract, `out_cache_loc` may cover the padded
+  physical page extent while MLA projection tensors only contain the valid
+  current suffix rows from `extend_seq_lens_cpu`.
+- That strict equality silently forces cache-hit draft suffixes (for example
+  `extend_len=65`, padded locs `128`) away from partial-current reuse and back
+  toward full materialization.
+
+Fix direction:
+
+- Validate current MLA tensors against valid current rows, not padded loc count.
+- Slice `k`, `k_rope`, and `out_cache_loc` to `extend_seq_lens_cpu[0]` before
+  composing current rows. Padded tail locs are physical reservation/slack, not
+  visible KV rows.
+- Keep draft next-layer MLA/index prefetcher disabled; this is only restoring the
+  synchronous partial-current compose path for draft cache-hit suffixes.
+
+Verification added:
+
+- RED on remote before implementation:
+  `test_current_extend_kv_rows_for_reuse_accepts_padded_out_cache_loc` failed
+  because `current_extend_kv_rows_for_reuse` did not exist.
+- RED on remote before implementation:
+  `test_mla_current_reuse_gate_accepts_padded_out_cache_loc` found the stale
+  `k.shape[0] == forward_batch.out_cache_loc.numel()` gate.
+
+Remaining runtime check:
+
+- Remote GREEN after implementation:
+  `test_should_reuse_current_extend_kv_enables_draft_partial_cache_hit_suffix`,
+  `test_current_extend_kv_rows_for_reuse_accepts_padded_out_cache_loc`,
+  `test_mla_current_reuse_gate_accepts_padded_out_cache_loc`,
+  `test_tai_current_slot_fill_is_skipped_when_sparse_page_self_test_fails`, and
+  `test_tai_current_slot_fill_sparse_page_self_test_passes_on_installed_kernel`
+  → `5 passed`.
+- Remote broader runtime helper suite:
+  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_shared_kv_runtime.py`
+  → `81 passed, 5 warnings, 2 subtests passed`.
+- Still requires user-driven ETE traffic to verify that draft `partial_current_sync`
+  appears without accept-length collapse.
+
+### C85 — 2026-05-31 CP HiCache eviction path and overhead findings
+
+Code path checked:
+
+- Extend allocation under CP shared KV uses compute-owner page allocation in
+  `alloc_paged_token_slots_extend()`.
+- If owner-lane allocation fails, `_evict_for_compute_owner_lanes()` computes
+  per-owner deficits and calls `tree_cache.evict(EvictParams(...,
+  owner_lane_deficits=...))`.
+- CP HiCache routes that to `_evict_cp_owner_lane_deficit_nodes()`, which plans
+  victim nodes by owner-lane contribution and frees backed GPU nodes through
+  `_evict_backuped()` -> `CacheController.evict_device()` ->
+  `mem_pool_device_allocator.free()`.
+- Load-back can also evict device nodes through `_evict_cp_load_back_owner_lanes()`.
+- Host write admission evicts host nodes through
+  `_reserve_write_cp_indices_no_collective()` ->
+  `_evict_cp_host_for_write_admission()` -> `CacheController.evict_cp_host()`.
+
+Remote log evidence from `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260530_184620.log`:
+
+- `MemCache-evict _evict_for_compute_owner_lanes`: 3224 attempt lines across 8
+  ranks, about 403 logical eviction cycles in ~18 minutes.
+- Average owner-lane request was ~18.5k tokens; average actual evicted was ~38k
+  tokens, i.e. about ~19.5k token overshoot per rank-line.
+- Small requests were common: 1336 owner-lane evict lines requested less than
+  4096 tokens, but eviction can still remove an entire victim node.
+- Host write-admission eviction appeared too: 1088 lines, about 136 logical
+  host eviction cycles. Average local host freed was ~4.7k token slots.
+
+Current conclusion:
+
+- Eviction does not copy KV for already-backed nodes; it mainly frees allocator
+  slots and updates radix/HiCache metadata. It should not consume PCIe bandwidth.
+- It can still be CPU/GPU-scheduler expensive because it is synchronous on the
+  scheduler path, scans/heapifies evictable leaves, emits many INFO logs, and
+  `PagedTokenToKVPoolAllocator.free()` uses `torch.unique(free_index // page_size)`
+  per victim.
+- Per-owner capacity stats also do CUDA tensor reductions with `.item()` per
+  owner lane, so failed owner-lane allocations can introduce host syncs.
+- The large overshoot is structural: eviction frees whole radix nodes, not the
+  exact deficit. This can cause L1 churn even after removing the older `* cp_size`
+  multiplier.
+
+Candidate optimizations to consider:
+
+1. Rate-limit or demote hot-path INFO eviction logs; current log volume alone is
+   large enough to affect performance during heavy churn.
+2. Add a page-aligned fast path in `PagedTokenToKVPoolAllocator.free()` to use
+   `free_index[::page_size] // page_size` when inputs are page-shaped, avoiding
+   `torch.unique` for normal HiCache node eviction.
+3. Cache/maintain owner-lane free counts incrementally in the CP allocator to
+   avoid repeated `.sum().item()` reductions during owner-lane admission.
+4. Improve victim selection to reduce overshoot for small deficits, or keep a
+   small-node victim pool for small owner-lane shortages.
+
+### C86 — 2026-05-31 Eviction hot-path logs demoted to DEBUG
+
+User decision:
+
+- Keep failure/fallback logs visible.
+- Demote successful/no-op eviction hot-path diagnostics from INFO to DEBUG.
+
+Implemented scope:
+
+- `mem_cache/common.py`
+  - `evict_from_tree_cache()` capacity-trigger message is DEBUG.
+  - `_evict_for_compute_owner_lanes()` no-evictable, attempt, and result messages are DEBUG.
+- `mem_cache/hiradix_cache.py`
+  - CP load-back owner-lane eviction summary is DEBUG.
+  - CP owner-lane no-victim / stale-victim / END summaries are DEBUG.
+  - Deterministic CP host eviction success summary is DEBUG.
+  - Generic `evict START/END`, `_evict_backuped`, `_evict_regular`, and host-slot eviction start are DEBUG.
+  - `write_backup CP retry after deterministic host eviction` is DEBUG.
+  - `write_backup CP FAILED after deterministic retry` is promoted to WARNING so true failures remain visible.
+
+Not changed:
+
+- Existing `warning`/`error` fallback, capacity failure, OOM, pin-release, and plan-divergence logs remain visible.
+- Eviction semantics are unchanged; this only changes log levels.
+
+Verification:
+
+- Added `TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only`.
+- RED on remote before implementation: failed because `MemCache-evict` success-path logs still used `logger.info`.
+- GREEN on remote after implementation:
+  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py::TestHiCacheEvictLoggingLevels::test_evict_hot_path_success_logs_are_debug_only`
+  → `1 passed`.
+- Local and remote `py_compile` passed for touched Python files.