Prevent CP HiCache pending-backup splits from killing chunked prefill

Chunked prefill can revisit a sub-page CP HiCache tail while a per-layer backup is still in flight. The old insert path split the radix node first and only then tried to prune the stale tail, so an unprunable pending backup raised after tree mutation and propagated to the scheduler.

This makes split/prune atomic from the radix-tree perspective: drain completed write acks only on the conflict path, preflight pending/unprunable state before split, and return a deferred insert result when the backup is still in flight. Unfinished requests keep their KV ownership for transfer/release instead of rematching or freeing pages under a stale tree state.

Constraint: CP HiCache backup metadata is node/page owned and cannot be repartitioned while per-layer D2H is pending
Rejected: Split the pending backup node | would require repartitioning in-flight backup metadata and host reservations
Rejected: Delete the stale tail unconditionally | risks freeing device/host pages still owned by pending backup
Confidence: high
Scope-risk: moderate
Directive: Do not move stale-tail prune after split without a preflight; pending backup split conflicts must remain non-mutating
Tested: remote g0034 container py_compile for mem_cache files
Tested: remote g0034 PYTHONPATH=python pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py test/registered/unit/mem_cache/test_cp_hicache_load_back_owner_lanes.py => 115 passed
Tested: local git diff --check
Not-tested: full chunked prefill ETE after service restart
Co-authored-by: OmX <omx@oh-my-codex.dev>

docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md

@@ -5021,3 +5021,18 @@ Implication:
 - Production LPF speedup requires changing allocation/residency policy, not only
   tensor layout: allocate compact/extent-aware host pages for each backed node or
   add a transfer-time gather/staging path.
+
+### C118 — 2026-06-02 Chunked prefill can hit CP HiCache stale-tail split while backup is pending
+
+Finding:
+
+- Remote crash at `process_batch_result_disagg_prefill -> cache_unfinished_req -> insert -> _cp_prune_stale_tail_after_page_floor` raised `HiCachePendingBackupSplit` for a stale tail while CP HiCache backup was pending.
+- The current insert path performs `_split_node()` first and only then calls `_cp_prune_stale_tail_after_page_floor()`.  If the stale tail subtree is unprunable because a backup lock/ref is pending, the exception is raised after the radix tree has already been mutated.
+- `_split_node()` only checks `pending_host_backups`; the prune guard checks both `ongoing_write_through` and `pending_host_backups` via `_node_host_write_pending()`.  Therefore an ongoing-only write can pass the split precheck and fail during prune.
+- `match_prefix()` catches `HiCachePendingBackupSplit` and defers scheduling, but `insert()` / `cache_unfinished_req()` do not convert this condition into a non-fatal deferred insertion.  Under chunked prefill the conflict can surface after forward, where requeueing the request is no longer possible.
+
+Required correction:
+
+- Make stale-tail split/prune atomic from the radix tree perspective: preflight pending/unprunable state before `_split_node()` and drain completed write acks once on the conflict path.
+- If the stale tail is still pending, do not mutate the tree and do not crash the scheduler.  Defer/skip this cache insertion attempt while keeping request KV ownership intact for transfer/release.
+- Do not delete the stale tail or split an in-flight backup node.  Per-layer backup ownership remains node/page based and cannot be repartitioned mid-flight.