Prevent stale CP HiCache tails from overlapping new page owners

CP HiCache owns KV at page granularity, but exact valid-tail extension and backed partial-tail split could leave an old sub-page tail child beside a new suffix that reuses the same physical page. That makes radix residency ambiguous across device, host, and draft mirrors. The insert/match split paths now prune stale floored tails when safe, and defer/fail through the existing pending-split path when the subtree is protected or has in-flight backup state.\n\nThis also keeps a temporary scheduler boundary warning for externally observed zero-output responses so future ETE runs can classify whether zero visible output reaches SGLang's output processor.\n\nConstraint: CP shared KV and HiCache manage physical KV by page, while radix keys retain valid-token lengths.\nRejected: Keep overlapping old tail nodes after page-floor split | leaves two independent cache states for one physical tail page.\nRejected: Force-prune protected or in-flight backup tails | can mutate cache state still used by active transfer or inference.\nConfidence: medium\nScope-risk: moderate\nDirective: Do not remove the stale-tail prune without replacing it with another page-granular ownership rule for CP HiCache radix splits.\nTested: Remote py_compile for hiradix_cache.py and scheduler_output_processor_mixin.py in g0034 container.\nTested: Remote PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py -> 97 passed, 5 warnings.\nNot-tested: Full ETE recovery of EAGLE accept length; latest ETE still shows accept collapse, documented in C55.\nNot-tested: Router/client-side output_len=0 correlation when server-side OUTPUT_ZERO_DEBUG does not fire.

docs/advanced_features/nsa_prefill_cp_page_aligned_cache_contract.md

@@ -2323,3 +2323,190 @@ Interpretation:
 
 - Do not use this process to judge C48.
 - If the earlier held-slot fix was active in this process, low accept length is not fully explained by missing EAGLE metadata lifetime alone; next investigation should compare a fresh C48 runtime and then inspect EAGLE state correctness/content, not just tensor presence.
+
+### C50. 2026-05-30 output_len=0 is confirmed by router/client; server-side checked points
+
+New user-provided fact:
+
+- `output_len=0` has been confirmed from router/client logs.  Treat it as a
+  real external symptom even if the currently inspected SGLang server logs do
+  not print full response bodies.
+
+Server-side evidence checked so far:
+
+- The latest valid decode logs around the previous failure window contain a
+  burst of `Decode transfer failed ... Aborted by AbortReq` at
+  `2026-05-29 21:01:40`, and the paired prefill log contains matching
+  `Prefill bootstrap failed` / `Prefill transfer failed` aborts at
+  `2026-05-29 21:01:41`.
+- A request aborted before any decoded token can produce a zero-length final
+  output at the scheduler boundary (`stream_output_generation()` reports
+  `completion_tokens=len(req.output_ids_through_stop)`).  The OpenAI serving
+  layer should normally convert scheduler abort finish reasons to HTTP errors,
+  so a client-observed successful empty response still needs a request-id/rid
+  correlation or a server-side zero-output warning to determine whether the
+  empty result is `abort`, `length=0`, stop-trimming, or router/client handling.
+- The newer prefill process PID `2763878` started at `2026-05-30 05:01:46 CST`
+  from the synced code, but at the time of this check it was still in startup /
+  weight-load/warmup logs (`sglang_cp_hicache_20260529_210146.log`).  It cannot
+  yet validate whether C48/C49 fixed or changed the output-len symptom.
+
+Do-not-repeat:
+
+- Do not conclude “server did not output zero” only because the SGLang log does
+  not contain `completion_tokens=0`; those bodies are usually not logged.
+- Do not treat router `Prefill server failed (CRITICAL)` as an independent
+  router bug when it aligns with prefill/decode KV-transfer aborts.
+
+Next useful evidence:
+
+- Add a low-frequency/anomaly-only server warning at the scheduler output
+  boundary for finished generation responses with `completion_tokens == 0`,
+  including `rid`, `finish_reason`, `max_new_tokens`, `origin_len`, cached
+  tokens, disaggregation mode, and bootstrap room.  This should fire only on the
+  confirmed anomalous path and should be removed or gated after root cause is
+  found.
+
+### C51. 2026-05-30 split hypothesis for output_len=0
+
+New user-provided hypothesis:
+
+- `output_len=0` may be caused by a split path.
+
+Checked points:
+
+- There are two distinct split mechanisms in the current failure surface:
+  1. NSA in-seq CP split (`can_cp_split()` / `build_page_aligned_in_seq_split_list()`),
+     which affects forward compute distribution.
+  2. Radix/HiCache tree split (`HiRadixCache._split_node()`), which affects
+     device/host cache residency and prefix matching.
+- C24 already disables NSA in-seq CP split for cache-hit suffixes with fewer
+  physical suffix pages than CP lanes.  This removes the known `extend_len=65`
+  mostly-zero-lane split path, but it does not address long suffixes or radix
+  structural splits.
+- Current radix exact-valid-tail extension handling floors the matched prefix
+  to the previous page boundary.  During insertion it then splits the old
+  valid-tail node and can leave two children under the same page-boundary parent
+  whose keys start at the same logical tail page, e.g. with `page_size=4`:
+  old child `(4, 5)` and new longer child `(4, 5, 6, 7)`.
+- This overlapping-child shape is not yet proven to produce `output_len=0`, but
+  it is a real radix invariant risk: two children can represent overlapping
+  logical token ranges with independent device/host/draft HiCache state.  Under
+  eviction/load-back or EAGLE target/draft mirror pressure, that can create
+  target/draft cache-state divergence or unnecessary transfer/abort pressure.
+- The latest inspected runtime did not include the new `OUTPUT_ZERO_DEBUG`
+  scheduler boundary warning, so server-side classification of confirmed
+  client/router `output_len=0` is still pending a fresh process.  Existing logs
+  still show the direct zero-output-compatible server symptom as large bursts of
+  prefill/decode `KVTransferError(...): Aborted by AbortReq` followed by decode
+  losing the prefill connection.
+
+Next checks:
+
+- Add a focused radix regression that demonstrates the overlapping-child shape
+  for exact valid-tail extension and then replace it with a page-granular policy
+  that sacrifices the old sub-page tail instead of keeping two overlapping
+  children.
+- Keep the scheduler `OUTPUT_ZERO_DEBUG` anomaly warning until a fresh runtime
+  confirms whether client/router `output_len=0` corresponds to abort,
+  max_new_tokens=0/length finish, stop trimming, or another finish reason.
+
+### C52. 2026-05-30 exact valid-tail split currently preserves stale sub-page child
+
+Finding:
+
+- The existing focused unit test `test_cp_insert_extends_from_page_boundary_after_exact_valid_tail`
+  encodes the risky behavior from C51 as expected behavior: after an old valid-tail
+  CP node `(0..5)` is extended by a new request `(0..9)`, the radix tree keeps both
+  the stale old child `(4,5)` and the new longer child `(4,5,6,7,8,9)` under the
+  page-boundary parent `(0..3)`.
+- This violates the page-granular cache contract we now want for CP HiCache: a
+  logical sub-page tail is not an independent cache-management unit once a request
+  extends beyond it.  The old tail page should be sacrificed/freed, and the new
+  suffix should own that page from the boundary.
+
+Next implementation guardrail:
+
+- Add/adjust the regression first so the expected tree has only the new suffix
+  child after exact-valid-tail extension.  Then implement a narrow CP-only prune
+  path for the stale tail child, with pending/in-flight backup split still fail-fast
+  or deferred instead of silently mutating an in-flight node.
+
+### C53. 2026-05-30 stale-tail pruning must cover backed partial-tail splits too
+
+Finding:
+
+- The exact-valid-tail extension case is only one instance of the same bug.
+  `test_cp_insert_floors_backed_tail_split_to_page_boundary` shows a backed CP
+  HiCache node `(0..5)` split by a shorter/new partial request `(0..4)`.  The old
+  test also preserved the stale child `(4,5)` beside the new child `(4)`, creating
+  the same overlapping-page state.
+- Therefore the prune condition cannot be limited to exact valid-tail extension.
+  Whenever CP split length is floored from inside a valid/padded tail page to the
+  previous page boundary, the stale tail subtree created by `_split_node()` must
+  be pruned unless it is protected or has an in-flight backup.
+
+Guardrail:
+
+- Probe-only paths may continue to floor without mutating.  Mutating match/insert
+  paths must prune stale tails after page-floor split; protected/in-flight tails
+  must defer/fail rather than silently keeping overlapping children.
+
+### C54. 2026-05-30 stale CP tail pruning implemented and tested
+
+Implemented:
+
+- CP mutating radix match/insert paths now detect when a split length was floored
+  from inside a page to the previous page boundary.
+- After `_split_node()` creates the page-boundary parent, the stale sub-page tail
+  subtree is pruned instead of being kept as an overlapping child.  This applies
+  both to exact valid-tail extension and backed partial-tail split cases.
+- The prune path first checks the stale subtree for lock refs, host refs, pending
+  backup, or ongoing write-through state.  Such in-flight/protected tails defer
+  or fail via the existing pending-split path rather than silently mutating cache
+  state used by active work.
+- Pruning frees device indices and CP host/draft metadata when the corresponding
+  cache-controller eviction hooks are available, then unlinks the stale subtree
+  and refreshes leaf state when the cache was fully initialized.
+
+Verification:
+
+- Remote py_compile passed for `python/sglang/srt/mem_cache/hiradix_cache.py` and
+  `python/sglang/srt/managers/scheduler_output_processor_mixin.py` in container
+  `/sgl-workspace/sglang-tai`.
+- Remote targeted red tests first failed on the old overlapping-child behavior.
+- Remote full focused suite passed after the fix:
+  `PYTHONPATH=python python -m pytest -q test/registered/unit/mem_cache/test_cp_hicache_metadata.py`
+  -> `97 passed, 5 warnings`.
+
+Remaining risk:
+
+- This removes one concrete split-induced overlapping-cache invariant violation,
+  but it does not yet prove the external `output_len=0` symptom is fixed.  A fresh
+  ETE run still needs to check for `[OUTPUT_ZERO_DEBUG]`, decode accept length,
+  and router/client zero-output correlation.
+
+### C55. 2026-05-30 latest split-prune runtime: zero-output not seen in server logs, EAGLE accept still collapsed
+
+Runtime identity:
+
+- New prefill process on `g0034` PID `2796105` started at `2026-05-30 05:36:48 CST`, after the split-prune and `OUTPUT_ZERO_DEBUG` source sync (`05:33-05:34 CST`).
+- Decode processes on `g0035`/`g0036` started at `05:36:52/05:36:55 CST` and see the same synced source mtimes.
+- Latest logs inspected:
+  - prefill: `/mnt/beegfs/cjy/log/sglang_cp_hicache_20260529_213648.log`
+  - decode0: `/mnt/beegfs/cjy/log/decode0_20260529_213652.log`
+  - decode1: `/mnt/beegfs/cjy/log/decode1_20260529_213656.log`
+
+Checked result:
+
+- No `[OUTPUT_ZERO_DEBUG]` line was found in prefill/decode logs, and no server-side `completion_tokens: 0` / `output_len=0` text was found in these logs.  This does not prove the router/client symptom is gone, but the scheduler boundary did not observe a finished generation with zero visible tokens in the inspected log window.
+- No current prefill/decode `KVTransferError`, transfer abort burst, traceback, health-check failure, or scheduler exception was found in this log window.  The only `out of memory` text was warmup max_m reduction, not runtime OOM.
+- EAGLE accept remains bad in the inspected decode logs:
+  - decode0: `17649` accept-debug rows, average `avg_accept=0.073`, median `0.0`, `15650` zero rows, `17544` rows at `<=1.05`; last 200 rows average `0.135`.
+  - decode1: `17541` accept-debug rows, average `avg_accept=0.065`, median `0.0`, `15569` zero rows, `17495` rows at `<=1.05`; last 200 rows average `0.030`.
+- Metadata presence is not missing in the samples: `has_pd_hidden=True`, `pd_hidden_shape=(6144,)`, `pd_topk_shape=(16,)`, no `has_pd_hidden=False` / `pd_topk_shape=None` lines.  `spec_topk_shape=(0,1)` appears occasionally but is not the dominant count; most rows have `(1,1)` or `(2,1)`.
+
+Interpretation:
+
+- Split-prune removes a real overlapping-radix invariant violation, but it did not recover EAGLE accept length in this run.
+- The current accept collapse is more likely in the semantic correctness/content of EAGLE prebuilt state or draft/target KV alignment than in missing metadata lifetime or gross transfer failure.  Next debugging should compare the actual prebuilt top-k token/content and draft input state against a non-HiCache or non-CP baseline, not just tensor shape/presence.